virtmat.middleware package

Subpackages

• virtmat.middleware.engine package
  • Submodules
  • virtmat.middleware.engine.wfengine module
    • WFEngine
      • WFEngine.add_node()
      • WFEngine.add_workflow()
      • WFEngine.append_wf_id()
      • WFEngine.cancel_job()
      • WFEngine.from_dict()
      • WFEngine.fw_ids
      • WFEngine.get_failed()
      • WFEngine.get_lost_jobs()
      • WFEngine.get_unreserved_nodes()
      • WFEngine.logger
      • WFEngine.name
      • WFEngine.qlaunch()
      • WFEngine.remove_wf_id()
      • WFEngine.remove_workflow()
      • WFEngine.rerun_node()
      • WFEngine.rlaunch()
      • WFEngine.show_launcher_status()
      • WFEngine.show_nodes_status()
      • WFEngine.show_wf_status()
      • WFEngine.start()
      • WFEngine.status_detail()
      • WFEngine.status_summary()
      • WFEngine.stop()
      • WFEngine.to_dict()
      • WFEngine.update_node()
      • WFEngine.update_rerun_node()
      • WFEngine.wf_ids
      • WFEngine.wf_query
  • virtmat.middleware.engine.wfengine_jupyter module
    • WFEnginejupyter
      • WFEnginejupyter.add_node_button_clicked()
      • WFEnginejupyter.add_nodes_button_clicked()
      • WFEnginejupyter.add_workflow_button_clicked()
      • WFEnginejupyter.cancel_launch_button_clicked()
      • WFEnginejupyter.commit_remove_workflow_button_clicked()
      • WFEnginejupyter.create_wf_id_select()
      • WFEnginejupyter.dump_engine_button_clicked()
      • WFEnginejupyter.func_name
      • WFEnginejupyter.job_category
      • WFEnginejupyter.jqadapter
      • WFEnginejupyter.logger
      • WFEnginejupyter.lpad_button_clicked()
      • WFEnginejupyter.manage_nodes_button_clicked()
      • WFEnginejupyter.manage_workflows_button_clicked()
      • WFEnginejupyter.new_engine_button_clicked()
      • WFEnginejupyter.node_id_select
      • WFEnginejupyter.nodes_status_button_clicked()
      • WFEnginejupyter.qadapter_button_clicked()
      • WFEnginejupyter.remove_workflow_button_clicked()
      • WFEnginejupyter.rerun_node_button_clicked()
      • WFEnginejupyter.resume_engine_button_clicked()
      • WFEnginejupyter.rows_inputs
      • WFEnginejupyter.rows_outputs
      • WFEnginejupyter.size_inp
      • WFEnginejupyter.size_out
      • WFEnginejupyter.start_launcher_clicked()
      • WFEnginejupyter.status_button_clicked()
      • WFEnginejupyter.status_detailed_button_clicked()
      • WFEnginejupyter.stop_launcher_clicked()
      • WFEnginejupyter.update_node_button_clicked()
      • WFEnginejupyter.update_rerun_node_button_clicked()
      • WFEnginejupyter.wf_id_select
      • WFEnginejupyter.wfe
    • add_workflow_method_changed()
    • clear_button_outputs()
    • clear_consoleoutput()
    • configure_button_clicked()
    • configure_engine_method_changed()
    • manage_launcher_button_clicked()
    • new_workflow_button_clicked()
    • remote_cluster_changed()
    • resconfig_button_clicked()
  • virtmat.middleware.engine.wfengine_remote module
    • WFEngineRemote
      • WFEngineRemote.check_jobcancel()
      • WFEngineRemote.exec_cancel()
      • WFEngineRemote.from_dict()
      • WFEngineRemote.launcher()
      • WFEngineRemote.logger
      • WFEngineRemote.setup_remote_configuration()
      • WFEngineRemote.setup_remote_fworker()
      • WFEngineRemote.setup_remote_launchpad()
      • WFEngineRemote.setup_remote_qadapter()
      • WFEngineRemote.slaunch()
      • WFEngineRemote.to_dict()
  • Module contents
• virtmat.middleware.query package
  • Submodules
  • virtmat.middleware.query.dbquery module
    • db_select()
    • get_launch()
  • virtmat.middleware.query.wfquery module
    • WFQuery
      • WFQuery.check_fw_ids()
      • WFQuery.get_data()
      • WFQuery.get_fw_ids()
      • WFQuery.get_fw_info()
      • WFQuery.get_i_data()
      • WFQuery.get_i_names()
      • WFQuery.get_nodes_providing()
      • WFQuery.get_o_data()
      • WFQuery.get_o_names()
      • WFQuery.get_task_info()
      • WFQuery.get_wf_ids()
      • WFQuery.get_wf_info()
    • io_info()
    • task_info()
  • Module contents
• virtmat.middleware.resconfig package
  • Submodules
  • virtmat.middleware.resconfig.cli module
    • setup_resconfig()
  • virtmat.middleware.resconfig.qadapter module
    • get_custom_qadapter()
    • get_default_qadapter()
    • get_pre_rocket()
    • get_qadapter()
    • get_worker_get_queue()
    • validate_resources()
  • virtmat.middleware.resconfig.resconfig module
    • CommandConfig
      • CommandConfig.cmd
      • CommandConfig.args
      • CommandConfig.commands
      • CommandConfig.args
      • CommandConfig.cmd
      • CommandConfig.commands
      • CommandConfig.get_command()
    • ModuleConfig
      • ModuleConfig.prefix
      • ModuleConfig.name
      • ModuleConfig.versions
      • ModuleConfig.path
      • ModuleConfig.get_command()
      • ModuleConfig.name
      • ModuleConfig.path
      • ModuleConfig.prefix
      • ModuleConfig.versions
    • QueueConfig
      • QueueConfig.name
      • QueueConfig.public
      • QueueConfig.resources
      • QueueConfig.accounts_allow
      • QueueConfig.accounts_deny
      • QueueConfig.groups_allow
      • QueueConfig.accounts_allow
      • QueueConfig.accounts_deny
      • QueueConfig.get_resource()
      • QueueConfig.groups_allow
      • QueueConfig.name
      • QueueConfig.public
      • QueueConfig.resources
      • QueueConfig.set_resource()
      • QueueConfig.validate_resource()
    • ResConfig
      • ResConfig.add_worker_from_scratch()
      • ResConfig.default_worker
      • ResConfig.from_scratch()
      • ResConfig.to_dict()
      • ResConfig.workers
    • ResConfigSerializable
      • ResConfigSerializable.from_dict()
      • ResConfigSerializable.to_dict()
    • ResourceConfig
      • ResourceConfig.name
      • ResourceConfig.default
      • ResourceConfig.maximum
      • ResourceConfig.minimum
      • ResourceConfig.name
    • WorkerConfig
      • WorkerConfig.name
      • WorkerConfig.type_
      • WorkerConfig.queues
      • WorkerConfig.accounts
      • WorkerConfig.modules
      • WorkerConfig.envvars
      • WorkerConfig.default_modules
      • WorkerConfig.default_envvars
      • WorkerConfig.default_venv
      • WorkerConfig.default_launchdir
      • WorkerConfig.default_commands
      • WorkerConfig.accounts
      • WorkerConfig.default_account
      • WorkerConfig.default_commands
      • WorkerConfig.default_envvars
      • WorkerConfig.default_launchdir
      • WorkerConfig.default_modules
      • WorkerConfig.default_queue
      • WorkerConfig.default_venv
      • WorkerConfig.envvars
      • WorkerConfig.from_scratch()
      • WorkerConfig.modules
      • WorkerConfig.name
      • WorkerConfig.queues
      • WorkerConfig.set_default_queue()
      • WorkerConfig.type_
    • configure()
    • get_default_resconfig()
    • get_inactive_envs()
    • get_inactive_modules()
    • get_resconfig_loc()
    • get_worker_config()
    • resource_type_checker()
    • set_defaults_from_guess()
  • Module contents

Submodules

virtmat.middleware.exceptions module

specific exceptions for the vre-middleware

exception virtmat.middleware.exceptions.ConfigurationException

Bases: CustomException

Describes errors in the configuration of the function or missing parameters

exception virtmat.middleware.exceptions.CustomException

Bases: Exception

Custom exception class used for all specific exceptions

exception virtmat.middleware.exceptions.InvalidStateException(msg, node_id)

Bases: CustomException

Describes errors while being in a wrong state

exception virtmat.middleware.exceptions.LmodError

Bases: CustomException

Exception to raise from functions detecting any problems with Lmod

exception virtmat.middleware.exceptions.ResourceConfigurationError

Bases: CustomException

Exception to raise from the resconfig module

exception virtmat.middleware.exceptions.SlurmError

Bases: RuntimeError

Exception to raise if Slurm is not responding or not available

exception virtmat.middleware.exceptions.TestingException

Bases: CustomException

Describes an exception while running a test

virtmat.middleware.utilities module

various utilities for the VRE middleware

virtmat.middleware.utilities.await_slurm_job_state(res_id, state, sleep_time=10)

return when a specified SLURM job state has been reached

virtmat.middleware.utilities.exec_cancel(res_id)

Execute the Slurm cancel command

Parameters:

res_id (int) – reservation ID (=Slurm JobID)

Raises:

SlurmError – if the command returns an error code

virtmat.middleware.utilities.format_warning_wrapper(func)

format user warnings, leave python warnings unchanged

virtmat.middleware.utilities.get_command_output(command)

run a slurm command (a string or list) and return the output

virtmat.middleware.utilities.get_env_modules(typ)

Return a list of Lmod modules with their versions

This interface is intended to determine the currently loaded modules or the modulefiles available in the system.

Parameters:

typ (str) – state of modules: either ‘loaded’ or ‘available’

Returns:

module attributes in a dictionary with ‘prefix’, ‘name’ and

’versions’ keys

Return type:

(list)

virtmat.middleware.utilities.get_logger(name, default_level=None)

return a custom logger object

virtmat.middleware.utilities.get_slurm_job_state(*args, method='json', **kwargs)

select the method to get the SLURM job state

virtmat.middleware.utilities.get_slurm_job_state_json(res_id, max_trials=3, trial_delay=3, trial=0)

extract the SLURM job state using the sacct command

virtmat.middleware.utilities.get_slurm_job_state_parse(res_id, max_trials=10, trial_delay=3, trial=0)

extract the SLURM job state using the sacct command

virtmat.middleware.utilities.get_unreserved_nodes(launchpad, fw_ids, expiration_time=1209600)

Detect reserved nodes, i.e. in ‘RESERVED’ state within FireWorks, that have not been updated for a while. Possible inconsistent states in SLURM are ‘CANCELLED’, ‘FAILED’, ‘COMPLETED’, ‘OUT_OF_MEMORY’, ‘BOOT_FAIL’, ‘TIMEOUT’ and ‘DEADLINE’

Parameters:

• launchpad (LaunchPad) – launchpad object

• fw_ids ([int]) – firework IDs

• expiration_time (int) – minimum time in seconds since the most recent update, default is 1209600 seconds

Returns:

a list of dictionaries containing the firework ids, the reservation ids, the SLURM states and the launch directories of such reserved nodes

Return type:

([dict])

virtmat.middleware.utilities.get_venv()

return a dictionary with venv type, name and path if activated

virtmat.middleware.utilities.lmod_env_module(command)

Process Lmod ‘module list’ and ‘module spider’ commands and return modules

Due to ambiguous format of provided input it is assumed that the last division is the module version in the case of more than one division.

Parameters:

command (str) – either ‘list’ or ‘spider’

Returns:

{prefix/…/name: [version1, …]} in case of more divisions

{‘name’: []} in case of one division, {} if no module loaded

None: if Lmod not available

Return type:

(dict)

virtmat.middleware.utilities.module_cmd(command, name=None, prefix=None, version=None, path=None)

Execute the Lmod ‘module’ command

This interface is intended for changing the environment with commands such as ‘load’, ‘add’, ‘unload’, remove’, ‘use’ etc. To read the current environment, with commands such as ‘avail’, ‘spider’, and ‘list’, the lmod_env_module function can be used.

Parameters:

• name (str) – a module name

• prefix (str) – the optional prefix; it should be used with name

• version (str) – the optional version; it should be used with name

• path (str) – a module path for the ‘use’ command; it may not be used with the other keyword arguments

Returns:

None

Raises:

LmodError – when the Lmod modules environment is not available

virtmat.middleware.utilities.module_load(name, *args, **kwargs)

Load an Lmod module

Parameters:

• name (str) – the module name

• args (list) – a list of the optional attributes prefix and version

• kwargs (dict) – the optional attributes prefix and version as keys

Returns:

None

virtmat.middleware.utilities.module_remove(name, *args, **kwargs)

Remove an Lmod module

Parameters:

• name (str) – the module name

• args (list) – a list of the optional attributes prefix and version

• kwargs (dict) – the optional attributes prefix and version as keys

Returns:

None

Module contents