Complex Usage (Script)

After the installation is outlined, the next sections describe how to get started. The sections configuration and logging describe the general methods we use, this is helpful to understand how you can change model parameters and similar.

Script Installation (Linux)

This is a quick guide on how to install AvaFrame and the required dependencies on your machine. AvaFrame is developed on Linux machines (Ubuntu/Manjaro/Arch) with recent Python versions > 3.8. These instructions assume you are familiar with working in a terminal. This guide is described for Linux. For Windows, see Script Installation (Windows).

Requirements

Install git and pixi. Some operating systems might require the python headers (e.g python-dev on ubuntu) or other supporting libraries/packages (e.g. Visual Studio on Windows needs the c++ compiler components).

Note

This documentation uses pixi environments. You can either prefix commands with pixi run <command>, which runs the command in the pixi environment without requiring you to activate it first, or you can run pixi shell once to start an interactive environment and then run the commands without the pixi run prefix (e.g. pixi run python script.py is equivalent to pixi shell then python script.py).

Setup AvaFrame

Clone the AvaFrame repository (in a directory of your choice: [YOURDIR]) and change into it:

cd [YOURDIR]
git clone https://github.com/OpenNHM/AvaFrame.git
cd AvaFrame

Compile the cython com1DFA part. You might also have to install a c-compiler (gcc or similar) through your systems package manager:

pixi run build

Warning

You will have to do this compilation every time something changes in the cython code. We also suggest to do this everytime updates from the repositories are pulled. Use pixi run rebuild to clean and rebuild in one step.

All this installs avaframe in editable mode, so every time you import avaframe the current (local) version will be used.

Test it by running:

pixi run python -c "import avaframe"

If you want to use the lastet stable release instead, run:

pixi run --environment prod python -c "import avaframe"

If no error comes up, you are good to go.

To see the current version, you can use:

pixi list avaframe

Head over to First run for the next steps.

Script Installation (Windows)

This is a quick guide on how to install AvaFrame and the required dependencies on your machine. AvaFrame is developed on Linux machines (Ubuntu/Manjaro/Arch) with recent Python versions > 3.8. These instructions assume you are familiar with working in a terminal. This guide is described for Windows. For Linux, see Script Installation (Linux).

Requirements

Install git and pixi.

Install Microsoft C++ compiler. Follow the installation steps for the version corresponding to the installed python version.

Note

This documentation uses pixi environments. You can either prefix commands with pixi run <command>, which runs the command in the pixi environment without requiring you to activate it first, or you can run pixi shell once to start an interactive environment and then run the commands without the pixi run prefix (e.g. pixi run python script.py is equivalent to pixi shell then python script.py).

Setup AvaFrame

Clone the AvaFrame repository (in a directory of your choice: [YOURDIR]) and change into it:

cd [YOURDIR]
git clone https://github.com/OpenNHM/AvaFrame.git
cd AvaFrame

Note

If you get some access denied error in Powershell, you might need to run the command

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser

Compile the cython com1DFA part:

pixi run build

Warning

You will have to do this compilation every time something changes in the cython code. We also suggest to do this everytime updates from the repositories are pulled. Use pixi run rebuild to clean and rebuild in one step.

Before compilation in Windows, make sure to delete the AvaFrame/build directory.

This installs avaframe in editable mode, so every time you import avaframe the current (local) version will be used.

Test it by running:

pixi run python -c "import avaframe"

If you want to use the lastet stable release instead, run:

pixi run --environment prod python -c "import avaframe"

If no error comes up, you are good to go.

Head over to First run for the next steps.

Using QGIS from within an script installation

To run QGIS (this will install the environment on first use):

pixi run --environment qgis qgis

Depending on your mode of QGis installation (direct installer, conda, OSGeo4W…) your script AvaFrame installation might be overruled…

Now you can install the OpenNHMConnector plugin via QGIS as per usual (Standard Usage (QGis)).

First run

Follow these steps to run your first simulation (assuming you finished the complex usage installation steps). The commands below use pixi run, so you do not need to activate the environment via pixi shell first:

  • change into your AvaFrame directory (replace [YOURDIR] with your path from the installation steps):

    cd [YOURDIR]/AvaFrame/avaframe

  • run:

    pixi run python runCom1DFA.py

  • a similar output should show up:

    logUtils - INFO -  Started logging at: 03.11.2020 22:42:04
logUtils - INFO -  Also logging to: data/avaParabola/runCom1DFA.log
runCom1DFA - INFO -  MAIN SCRIPT
runCom1DFA - INFO -  Current avalanche: data/avaParabola
...

This will perform a dense flow avalanche simulation using the com1DFA module. The results are saved to data/avaParabola/Outputs/com1DFA. For a first look at the results, got to the folder reports, there you can find a markdown report of the simulations performed including some plots of the results.

To display markdown files in a nice way use a markdown viewer of your choice. Some other options are:

  • Use the Atom editor with a markdown plugin

  • If you have pandoc installed use this to convert it to pdf/html

  • Some browsers have markdown extensions you can install easily

Workflow example

The following example should make it easier for you to find your way in AvaFrame and setup your own AvaFrame workflow after you did the full setup. There is also a directory with examples for different workflows, see more here: Example runscripts.

Make sure you change to your AvaFrame directory by:

cd [YOURDIR]/AvaFrame

Replace [YOURDIR] with the directory from your installation step.

Initialize project

To create the folder where the input data lies and where the output results will be saved, specify the full path to the folder in the local_avaframeCfg.ini (which is a copy of avaframeCfg.ini that you need to create). So:

cd avaframe
cp avaframeCfg.ini local_avaframeCfg.ini

and edit local_avaframeCfg.ini with your favorite text editor and adjust the variable avalancheDir.

Then run

pixi run python runScripts/runInitializeProject.py

This will create a new directory with the input required by AvaFrame structured as described in Initialize Project.

Input data

Check the input data required by the different modules you want to use and fill the Inputs/ inside the [avalancheDir] folder from the initialize step accordingly.

For example the com1DFA module needs input as described in Input. You can also have a look at the default setting for the module you want to use (for example com1DFACfg.ini for module com1DFA). If you want to use different settings, create a local_ copy of the .ini file and modify the desired parameters.

More information about the configuration can be found here: Configuration

Building your run script

Create your own workflow by taking the runOperational.py script as template.

We suggest you copy it and adjust it to your liking. There are annotations in the code that should help you to understand the structure.

A lot more examples can be found in the runScripts directory (see also Example runscripts).

Configuration

In order to set the configurations required by all the modules within Avaframe, the python module configparser is used.

This is done in two steps. The first step fetches the main settings:

from avaframe.in3Utils import cfgUtils
# Load avalanche directory from general configuration file
cfgMain = cfgUtils.getGeneralConfig()
avalancheDir = cfgMain['MAIN']['avalancheDir']

In the second step the specific settings to a given module are imported:

from avaframe.tmp1Ex import tmp1Ex
# Load all input Parameters from config file
# get the configuration of an already imported module
# Write config to log file
cfg = cfgUtils.getModuleConfig(tmp1Ex)

The in3Utils.cfgUtils.getModuleConfig() function reads the settings from a configuration file (tmpExCfg.ini in our example) and writes these settings to the log file. The default settings can be found in the configuration file provided within each module.

It is possible to modify these settings. The main options are:

  • provide the path to your own configuration file using the fileOverride parameter when calling in3Utils.cfgUtils.getModuleConfig()

  • create an expert configuration file at {avalancheDir}/Inputs/CFGs/{moduleName}Cfg.ini (see Advanced Usage (QGis, Script) for details)

  • create a copy of the module configuration file called local_ followed by the name of the original configuration file and set the desired values of the individual parameters. local_ needs to be in the same folder as the original.

  • see Override configuration for additional options to modify configuration

So the order is as follows:

  1. if batchCfgDir is provided, the path to the batch configuration directory is returned (this is used for batch processing and returns a pathlib.Path instead of a ConfigParser object).

  2. if onlyDefault=True is passed to in3Utils.cfgUtils.getModuleConfig(), only the default configuration is used (all overrides are skipped).

  3. if there is a path provided via the fileOverride parameter, configuration is read from this file.

  4. if the avalancheDir is provided and {avalancheDir}/Inputs/CFGs/{moduleName}Cfg.ini exists, this expert config is used (see Advanced Usage (QGis, Script) for details).

  5. if there is no expert config, the local_... configuration file is read if it exists.

  6. if there is no local_..., the getModuleConfig function reads the settings from the default configuration file with the default settings.

The following flowchart illustrates this priority order:

digraph config_priority { rankdir=TB newrank=true graph [ranksep=0.6, nodesep=0.5, splines=ortho, fontname="helvetica", fontsize="20", fontcolor="#0E4160"] node [shape=box, style="rounded,filled", fontsize=16, fontcolor="#0E4160", fontname="helvetica", fillcolor="#51ADE5cf", penwidth=1.5] edge [fontname="helvetica", fontsize="12", penwidth=1.5] // Entry and exit points start [label="getModuleConfig()", shape=ellipse, fillcolor="#8BC9F0"] done [label="Return ConfigParser", shape=ellipse, fillcolor="#8BC9F0"] // Question nodes (left column) batch [label="batchCfgDir\nprovided?", shape=box, fillcolor="#5ce0e6", margin="0.3,0.2"] only [label="onlyDefault\n= True?", shape=box, fillcolor="#5ce0e6", margin="0.3,0.2"] file [label="fileOverride\nprovided?", shape=box, fillcolor="#5ce0e6", margin="0.3,0.2"] expert [label="Expert config\nexists?", shape=box, fillcolor="#5ce0e6", margin="0.3,0.2"] local [label="local_* file\nexists?", shape=box, fillcolor="#5ce0e6", margin="0.3,0.2"] // Action nodes (right column) batch_ret [label="Return Path\n(batch mode)", shape=ellipse] only_ret [label="Return default\nconfig only"] file_load [label="Load fileOverride"] expert_load [label="Load expert config\nInputs/CFGs/"] local_load [label="Load local_* file"] merge [label="Read default config\nFill missing values\nfrom default"] // Force horizontal alignment of question-answer pairs {rank=same; batch; batch_ret} {rank=same; only; only_ret} {rank=same; file; file_load} {rank=same; expert; expert_load} {rank=same; local; local_load} // Invisible chain to enforce vertical alignment of questions (left column) batch -> only -> file -> expert -> local [style=invis, weight=100] // Invisible chain to enforce vertical alignment of actions (right column) batch_ret -> only_ret -> file_load -> expert_load -> local_load [style=invis, weight=100] // Main flow: entry start -> batch // Main flow: "no" path (down the left column) batch -> only [xlabel="no"] only -> file [xlabel="no"] file -> expert [xlabel="no"] expert -> local [xlabel="no"] // Invisible node to route "no" edge from local down then right to merge local_merge_route [shape=point, width=0, height=0] local -> local_merge_route [dir=none, xlabel="no"] local_merge_route -> merge // Main flow: "yes" path (to the right column) batch -> batch_ret [xlabel="yes"] only -> only_ret [xlabel="yes"] file -> file_load [xlabel="yes"] expert -> expert_load [xlabel="yes"] local -> local_load [xlabel="yes"] // Convergence to merge and exit file_load:e -> merge:ne expert_load:se -> merge:ne local_load -> merge only_ret -> done merge -> done }

In the configuration file itself, there are multiple options to vary a parameter:

  • replace the default parameter value with desired value

  • provide a number of parameter values separated by | (e.g. relTh=1.|2.|3.)

  • provide a number of parameter values using start:stop:numberOfSteps (e.g. relTh=1.:3.:3)) - a single value can be added by appending &4.0 for example

Override configuration

When module B calls tools from module A, you can override A’s configuration directly in B’s config file. This keeps all settings for a workflow in one place.

How it works:

  1. Add a section named [collectionName_moduleName_override] to module B’s config file

  2. List the parameters you want to override from module A

  3. When B runs, it loads A’s default config and applies your overrides (happens in cfgHandling.applyCfgOverride)

Example: ana1Tests/energyLineTestCfg.ini overrides com1DFA settings:

[energyLineTest]
# energyLineTest's own settings
runDFAModule = True
pathFromPart = True

[com1DFA_com1DFA_override]
# these override com1DFA's defaults when called from energyLineTest
defaultConfig = True
simTypeList = null
relTh = 1
frictModel = Coulomb

The defaultConfig = True parameter ensures the override starts from A’s default configuration (rather than a local_ file if one exists).

Logging

In order to generate simulation logs and to control what is prompted to the terminal, we use the python module logging.

Let’s have a look at the simple example in runScripts.runTmp1Ex and tmp1Ex.tmp1Ex() on how this is used within AvaFrame.

In your main script call:

from avaframe.in3Utils import logUtils
# log file name; leave empty to use default runLog.log
logName = 'runTmp1Ex'
# specify the working directory
avalancheDir = './'
# ---------------------------------------------
# Start logging
log = logUtils.initiateLogger(avalancheDir, logName)

This will configure the logging (it sets the console output as well as the log file). In your modules/subscripts add:

import logging
log = logging.getLogger(__name__)

So you can use:

log.debug('Should be here')
log.info('DEM : %s',variable)

To get output that looks like this in your console:

tmp1Ex:DEBUG - Should be here
tmp1Ex:INFO - DEM : /path/to/DEM

and something similar in the .log file which is saved in ./runTmp1Ex.log in this example. The logging configuration is set in AvaFrame/avaframe/in3Utils/logging.conf.

You can modify this logging.conf file to modify the levels or format of the messages to display (python doc will help you).

Example runscripts

In runScripts we provide ready-to-use scripts for different applications of the modules provided within AvaFrame.

Derive input data

  • runScripts.runComputeDist

  • runScripts.runD2Th

  • runScripts.runRenaming

Create a new project

  • runScripts.runInitializeProject

Generate idealized/generic topography data

  • runScripts.runGenerateTopo

  • runScripts.runGenProjTopoRelease

  • runScripts.runRotationTest

  • runScripts.runEnergyLineTest

Postprocessing

  • runScripts.runAna3AIMEC

  • runScripts.runAna3AIMECCompMods

  • runScripts.runAimecSummaryPlot

  • runScripts.runStatsExample

  • runScripts.runStatsExampleSimple

  • runScripts.runStatsPlots

  • runScripts.runProbAna

  • runScripts.runCreateProbCfgsFlowPy

Visualisation

  • runScripts.runQuickPlotSimple

  • runScripts.runQuickPlotOne

  • runScripts.runPlotTopo

  • runScripts.runContourPlot

  • runScripts.runPlotAlongGivenProfile

  • runScripts.runPlotAreaRefDiffs

  • runScripts.runPlotContoursFromAsc

  • runScripts.runPlotProfile

  • runScripts.runRangeTimeDiagram

  • runScripts.runThalwegTimeDiagram

  • runScripts.runFetchPointValues

  • runScripts.runParticleAnalysisPlots

  • runScripts.runExportToCsv

Testing

  • runScripts.runDamBreak

  • runScripts.runAnalyzeDamBreak

  • runScripts.runSimilaritySol

  • runScripts.runAnalyzeSimilaritySol

  • runScripts.runTestFP

  • runScripts.runVariationsTestsCom1DFA

  • runScripts.runWriteDesDict

Update AvaFrame

To update go to your avaframe repository [YOURDIR]/Avaframe, pull the latest changes via:

git pull

Your next pixi call then runs the necessary updates. You might want to run pixi run rebuild to recompile the cython parts.