VRE Middleware

This middleware package aims to create a platform for a Virtual Research Environment (VRE) for scientific computing. This platform integrates high performance computing (HPC) resources (currently via Slurm), workflow management systems (currently FireWorks) and MongoDB database, that are switched as transparent backends. The middleware provides three user interfaces:

• a Jupyter-oriented application programming interface (API);

• a graphical user interface (GUI) via Jupyter widgets;

• a command-line interface (CLI).

The middleware includes three major components:

• WFEngine: A workflow engine for managing FireWorks workflows and their nodes conveniently in a Jupyter notebook;

• WFQuery: A query tool for FireWorks workflows: retrieve data from workflows and their nodes;

• ResConfig: A resource configuration manager: manage Slurm queues, environment modules and much more.

Quick start

• Prerequisites
• Installation and setup
  • Installation from the pypi.org repository
  • Local installation from source
  • Development installation
  • Setting up FireWorks with Mongomock
  • Setting up FireWorks with a MongoDB instance
• Running the unit tests

Using the API

• Manage workflows with WFEngine
  • Configuration
  • Create a WFEngine object
  • Execute workflows
  • Add and remove workflows
  • Monitor workflows
  • Modify and rerun nodes
  • Cancel node execution
  • Add a workflow node
  • Further optional parameters for add_node()
  • Save and load a WFEngine object
  • Set and get the workflow IDs or the workflow query
• Glossary
• Query workflows and workflow nodes
  • The WFQuery class
  • Functions (methods) of the WFQuery class
  • Q&A
• Resource configuration
  • Location of resource configuration file
  • Create a resource configuration from scratch
  • Add a worker to an existing resconfig
  • Worker configuration
• How to create a LaunchPad object
• How to create a qadapter object
• Running a launcher on a remote computer
  • How to create a WFEngineRemote object
  • Using SSH to activate a remote launcher

Using the GUI

• How to manage workflows with the Ipywidgets GUI
  • Import the main object and main menu
  • Manage engine
  • Manage launcher
  • Manage workflows
  • Manage nodes

Using the CLI

• The wfengine command-line tool
  • Syntax
  • Adding workflows
  • Running nodes
• The resconfig command-line tool
  • Syntax
  • Behavior

Help

• Troubleshooting
  • Collecting console output
  • How to create an issue to report a problem
  • Support contact
• Frequently asked questions (FAQs)

Comprehensive documentation

• virtmat namespace
  • Packages
  • Module contents

Indices and tables

• Index

• Module Index

• Search Page