Logging with LIS Pro 3D

< Previous section    Next section >

Logging with LIS Pro 3D

You probably noticed that in all the examples given in this tutorial, we never used the standard print()-function of Python but only the special methods coming with LIS Pro 3D (lis_cmd).

Note

These methods ensure that our message outputs (standard out and error out) never get mixed up, even during parallel processing, where different processes write to the log files simultaneously. In fact, the process function wrappers collect the output of each process and write the output to the log files as soon as a process has finished. To achieve this, we cannot use simple Python outputs, but must use the C++ library (SAGA API) in the background.

Here you find a list of all the different logging functions provided by lis_cmd. Note how the purpose of each of these methods is slightly different. Each of these is appropriate in a specific context!

# output a message (stdout)
lis_cmd.Message_Print(msg)

# output an error message (stderr)
lis_cmd.Error_Print(msg)

# output an error message (stderr) and terminate the entire script
lis_cmd.Error_Exit(msg, starttime, exitLogging=True)

# output an error message (stderr) from a processing function and return "False" (terminate the function)
lis_cmd.Error_Return(msg)

Fill the run_script.py Script

If you consistently use the logging methods provided by LIS Pro 3D, the following script allows you to redirect nicely formatted logs of tool executions to log files. Create a new script called run_script.py and copy-paste the following code into it:

from subprocess import run
import os
import sys
import argparse

LOG_DIR = "../log"


def parse_args():
    parser = argparse.ArgumentParser(
        description="run a script with automated logging."
    )

    parser.add_argument(
        "scripts",
        type=str,
        nargs="+",
        help="the script(s) to execute, e.g. 'my_script.py' or 'my_script0.py my_script1.py my_script2.py'",
    )

    return parser.parse_args()


if __name__ == "__main__":
    args = parse_args()

    if not os.path.exists(LOG_DIR):
        os.makedirs(LOG_DIR, exist_ok=True)

    for script in args.scripts:
        scriptname, fileext = os.path.splitext(script)
        std_log = os.path.join(LOG_DIR, f"{scriptname}.std.log")
        err_log = os.path.join(LOG_DIR, f"{scriptname}.err.log")

        if fileext == ".py":
            command = (
                f"{sys.executable} -u  {script} > {std_log} 2> {err_log}"
            )
            print("running: " + command)
            run(command, universal_newlines=True, shell=True)
        else:
            print(f"Skipping {script} (unknown file extension)")

Execute the Processing Pipeline

Run the tool from the Miniforge Prompt using the following command:

python run_script.py data_processing.py

TipRecall

1. Search for Miniforge Prompt in the Windows start menu and open it.

2. After startup, you should be in the default (base) environment:

(base) C:\Users\laser>

3. Activate your project environment:

conda activate myenv

Your prompt should now look similar to this:

(myenv) C:\Users\laser>

4. Change to the scripts directory of your project:

cd C:\LiDAR_Data\project\scripts

You should now see something similar to:

(myenv) C:\LiDAR_Data\project\scripts>

5. Execute the processing pipeline:

python run_script.py data_processing.py

While the processing pipeline is running, you will not receive direct feedback in the terminal because all process information is now written to logfiles.

(myenv_10_12) C:\LiDAR_Data\project\scripts>python run_script.py data_processing.py
running: C:\Users\mb\miniforge3\envs\myenv_10_12\python.exe -u  data_processing.py > ../log\data_processing.std.log 2> ../log\data_processing.err.log
_

The command output shows where the logfiles are written:

project # top-level folder
├── data
├── scripts
├── processing
└── log
    ├── data_processing.err.log
    └── data_processing.std.log

After successful execution, the command finishes and the prompt is shown again.

(myenv_10_12) C:\LiDAR_Data\project\scripts>python run_script.py data_processing.py
running: C:\Users\mb\miniforge3\envs\myenv_10_12\python.exe -u  data_processing.py > ../log\data_pprocessing.std.log 2> ../log\data_processing.err.log

(myenv_10_12) C:\LiDAR_Data\project\scripts>

TipRun Multiple Scripts

Run multiple scripts with one command by typing:

python run_script.py data_preparation.py data_processing.py

Use the Tab key for auto-completion!

< Previous section    Next section >