In addition to the API and the GUI, VRE Middleware provides a command-line interface (CLI) to some of the functions available with the other two interfaces.

The wfengine command-line tool

The wfengine tool allows loading workflows from a launchpad or from a file and running them. If other workflow management functions are needed, then the API or the GUI should be used.

Syntax

The command wfengine --help yields this help:

usage: wfengine [-h] [-l LAUNCHPAD_FILE] [-q QADAPTER_FILE]
                [-n WORKER_NAME] [-d LAUNCHDIR] [-s SLEEP_TIME]
                [-c {all,interactive,batch}] [-f WORKFLOW_FILE]
                [--wf-query WF_QUERY]
                [--unique-launchdir | --no-unique-launchdir]
                [-r] [--loop | --no-loop]
                [--load-from-file LOAD_FROM_FILE]
                [--dump-to-file DUMP_TO_FILE] [--enable-logging]
                [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]

run a simple workflow engine from the command line

optional arguments:
  -h, --help            show this help message and exit
  -l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE
                        path to launchpad file
  -q QADAPTER_FILE, --qadapter-file QADAPTER_FILE
                        path to default qadapter file
  -n WORKER_NAME, --worker-name WORKER_NAME
                        worker name
  -d LAUNCHDIR, --launchdir LAUNCHDIR
                        path to launch directory
  -s SLEEP_TIME, --sleep-time SLEEP_TIME
                        sleep time for background evaluation in seconds
  -c {all,interactive,batch}, --category {all,interactive,batch}
                        category of nodes to run
  -f WORKFLOW_FILE, --workflow-file WORKFLOW_FILE
                        path to a workflow file
  --wf-query WF_QUERY   workflow query as JSON string
  --unique-launchdir, --no-unique-launchdir
                        use unique launchdir (default: True)
  -r, --autorun         run workflow nodes
  --loop, --no-loop     run nodes in an endless loop (default: True)
  --load-from-file LOAD_FROM_FILE
                        load a wfengine from file
  --dump-to-file DUMP_TO_FILE
                        dump the wfengine to file
  --enable-logging      enable logging messages
  --logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        logging level

Adding workflows

The wfengine program can be used to create an engine object and add workflows to it. Existing workflows on the launchpad can be added with the --wf_query option. For example:

Example:

wfengine --wf_query '{"metadata.uuid": "29f5f01a974c4c0b891c070d42a829c9"}'

The option -f, --workflow-file can be used to add a new workflow from a file to the negine and to the launchpad.

Example:

wfengine --workflow_file my_fireworks.yaml

The two latter flags can be combined to select workflows from the database and add one new workflow.

Running nodes

To activate the launcher loop, the -r, --autorun flag must be set. The launcher searches for nodes managed by the engine in READY state and runs those nodes of interactive category in a sequence immediately and submits the nodes of batch category as Slurm jobs. After that the launcher waits 30 seconds (that can be changed by using the -s, --sleep-time option) and then performs a new iteration. By default this loop is endless. The loop can be interrupted by pressing Ctrl+C or, if the process is running in background, by sending a SIGINT or SIGTERM signal. Then wfengine will terminate normally after a maximum delay of 30 seconds (or the time set by the -s, --sleep-time option). To only perform one iteration, use the flag --no-loop. Then wfengine will terminate itself. By default, both interactive and batch nodes are processed. One of these two node categories can be selected using the flag -c, --category.

Example: Add a workflow from a file and run only the interactive nodes in a loop with 10 seconds sleep time

wfengine --workflow_file my_fireworks.yaml -r -c interactive -s 10