virtmat.middleware package

Subpackages

• virtmat.middleware.engine package
  • Submodules
  • virtmat.middleware.engine.wfengine module
    • WFEngine
      • WFEngine.add_node()
      • WFEngine.add_workflow()
      • WFEngine.append_wf_id()
      • WFEngine.cancel_job()
      • WFEngine.exec_cancel()
      • WFEngine.from_dict()
      • WFEngine.fw_ids
      • WFEngine.get_failed()
      • WFEngine.get_lost_jobs()
      • WFEngine.get_unreserved_nodes()
      • WFEngine.launcher()
      • WFEngine.logger
      • WFEngine.name
      • WFEngine.qlaunch()
      • WFEngine.remove_wf_id()
      • WFEngine.remove_workflow()
      • WFEngine.rerun_node()
      • WFEngine.rlaunch()
      • WFEngine.show_launcher_status()
      • WFEngine.show_nodes_status()
      • WFEngine.show_wf_status()
      • WFEngine.start()
      • WFEngine.status_detail()
      • WFEngine.status_summary()
      • WFEngine.stop()
      • WFEngine.to_dict()
      • WFEngine.update_node()
      • WFEngine.update_rerun_node()
      • WFEngine.wf_ids
      • WFEngine.wf_query
  • virtmat.middleware.engine.wfengine_jupyter module
    • WFEnginejupyter
      • WFEnginejupyter.add_node_button_clicked()
      • WFEnginejupyter.add_nodes_button_clicked()
      • WFEnginejupyter.add_workflow_button_clicked()
      • WFEnginejupyter.cancel_launch_button_clicked()
      • WFEnginejupyter.commit_remove_workflow_button_clicked()
      • WFEnginejupyter.create_wf_id_select()
      • WFEnginejupyter.dump_engine_button_clicked()
      • WFEnginejupyter.func_name
      • WFEnginejupyter.job_category
      • WFEnginejupyter.jqadapter
      • WFEnginejupyter.logger
      • WFEnginejupyter.lpad_button_clicked()
      • WFEnginejupyter.manage_nodes_button_clicked()
      • WFEnginejupyter.manage_workflows_button_clicked()
      • WFEnginejupyter.new_engine_button_clicked()
      • WFEnginejupyter.node_id_select
      • WFEnginejupyter.nodes_status_button_clicked()
      • WFEnginejupyter.qadapter_button_clicked()
      • WFEnginejupyter.remove_workflow_button_clicked()
      • WFEnginejupyter.rerun_node_button_clicked()
      • WFEnginejupyter.resume_engine_button_clicked()
      • WFEnginejupyter.rows_inputs
      • WFEnginejupyter.rows_outputs
      • WFEnginejupyter.size_inp
      • WFEnginejupyter.size_out
      • WFEnginejupyter.start_launcher_clicked()
      • WFEnginejupyter.status_button_clicked()
      • WFEnginejupyter.status_detailed_button_clicked()
      • WFEnginejupyter.stop_launcher_clicked()
      • WFEnginejupyter.update_node_button_clicked()
      • WFEnginejupyter.update_rerun_node_button_clicked()
      • WFEnginejupyter.wf_id_select
      • WFEnginejupyter.wfe
    • add_workflow_method_changed()
    • clear_button_outputs()
    • clear_consoleoutput()
    • configure_button_clicked()
    • configure_engine_method_changed()
    • manage_launcher_button_clicked()
    • new_workflow_button_clicked()
    • remote_cluster_changed()
    • resconfig_button_clicked()
  • virtmat.middleware.engine.wfengine_remote module
    • WFEngineRemote
      • WFEngineRemote.check_jobcancel()
      • WFEngineRemote.exec_cancel()
      • WFEngineRemote.from_dict()
      • WFEngineRemote.launcher()
      • WFEngineRemote.logger
      • WFEngineRemote.setup_remote_configuration()
      • WFEngineRemote.setup_remote_fworker()
      • WFEngineRemote.setup_remote_launchpad()
      • WFEngineRemote.setup_remote_qadapter()
      • WFEngineRemote.slaunch()
      • WFEngineRemote.to_dict()
  • Module contents
• virtmat.middleware.query package
  • Submodules
  • virtmat.middleware.query.dbquery module
    • db_select()
    • get_launch()
  • virtmat.middleware.query.wfquery module
    • WFQuery
      • WFQuery.check_fw_ids()
      • WFQuery.get_data()
      • WFQuery.get_fw_ids()
      • WFQuery.get_fw_info()
      • WFQuery.get_i_data()
      • WFQuery.get_i_names()
      • WFQuery.get_nodes_providing()
      • WFQuery.get_o_data()
      • WFQuery.get_o_names()
      • WFQuery.get_task_info()
      • WFQuery.get_wf_ids()
      • WFQuery.get_wf_info()
    • io_info()
    • task_info()
  • Module contents
• virtmat.middleware.resconfig package
  • Submodules
  • virtmat.middleware.resconfig.cli module
    • setup_resconfig()
  • virtmat.middleware.resconfig.qadapter module
    • get_custom_qadapter()
    • get_default_qadapter()
    • get_pre_rocket()
    • get_qadapter()
    • get_worker_get_queue()
    • validate_resources()
  • virtmat.middleware.resconfig.resconfig module
    • ModuleConfig
      • ModuleConfig.prefix
      • ModuleConfig.name
      • ModuleConfig.versions
      • ModuleConfig.path
      • ModuleConfig.get_command()
      • ModuleConfig.name
      • ModuleConfig.path
      • ModuleConfig.prefix
      • ModuleConfig.versions
    • QueueConfig
      • QueueConfig.name
      • QueueConfig.public
      • QueueConfig.resources
      • QueueConfig.accounts_allow
      • QueueConfig.accounts_deny
      • QueueConfig.groups_allow
      • QueueConfig.accounts_allow
      • QueueConfig.accounts_deny
      • QueueConfig.get_resource()
      • QueueConfig.groups_allow
      • QueueConfig.name
      • QueueConfig.public
      • QueueConfig.resources
      • QueueConfig.set_resource()
      • QueueConfig.validate_resource()
    • ResConfig
      • ResConfig.add_worker_from_scratch()
      • ResConfig.default_worker
      • ResConfig.from_scratch()
      • ResConfig.workers
    • ResConfigSerializable
      • ResConfigSerializable.from_dict()
      • ResConfigSerializable.to_dict()
    • ResourceConfig
      • ResourceConfig.name
      • ResourceConfig.default
      • ResourceConfig.maximum
      • ResourceConfig.minimum
      • ResourceConfig.name
    • WorkerConfig
      • WorkerConfig.name
      • WorkerConfig.type_
      • WorkerConfig.queues
      • WorkerConfig.accounts
      • WorkerConfig.modules
      • WorkerConfig.default_modules
      • WorkerConfig.default_venv
      • WorkerConfig.default_launchdir
      • WorkerConfig.accounts
      • WorkerConfig.default_account
      • WorkerConfig.default_launchdir
      • WorkerConfig.default_modules
      • WorkerConfig.default_queue
      • WorkerConfig.default_venv
      • WorkerConfig.from_scratch()
      • WorkerConfig.modules
      • WorkerConfig.name
      • WorkerConfig.queues
      • WorkerConfig.set_default_account()
      • WorkerConfig.set_default_queue()
      • WorkerConfig.type_
    • configure()
    • get_default_resconfig()
    • get_env_modules()
    • get_resconfig_loc()
    • get_venv()
    • set_defaults_from_guess()
  • Module contents

Submodules

virtmat.middleware.exceptions module

specific exceptions for the vre-middleware

exception virtmat.middleware.exceptions.ConfigurationException

Bases: CustomException

Describes errors in the configuration of the function or missing parameters

exception virtmat.middleware.exceptions.CustomException

Bases: Exception

Custom exception class used for all specific exceptions

exception virtmat.middleware.exceptions.InvalidStateException(msg, node_id)

Bases: CustomException

Describes errors while being in a wrong state

exception virtmat.middleware.exceptions.ResourceConfigurationError

Bases: CustomException

Exception to raise from the resconfig module

exception virtmat.middleware.exceptions.SlurmError

Bases: RuntimeError

Exception to raise if Slurm is not responding or not available

exception virtmat.middleware.exceptions.TestingException

Bases: CustomException

Describes an exception while running a test

virtmat.middleware.utilities module

various utilities for the VRE middleware

virtmat.middleware.utilities.await_slurm_job_state(res_id, state, sleep_time=10)

return when a specified SLURM job state has been reached

virtmat.middleware.utilities.format_warning_wrapper(func)

format user warnings, leave python warnings unchanged

virtmat.middleware.utilities.get_logger(name, default_level=40)

return a custom logger object

virtmat.middleware.utilities.get_slurm_job_state(*args, method='parse', **kwargs)

select the method to get the SLURM job state

virtmat.middleware.utilities.get_slurm_job_state_json(res_id, max_trials=3, trial_delay=3, trial=0)

extract the SLURM job state using the sacct command

virtmat.middleware.utilities.get_slurm_job_state_parse(res_id, max_trials=10, trial_delay=3, trial=0)

extract the SLURM job state using the sacct command

virtmat.middleware.utilities.lmod_env_module(command)

Process lmod modules list and spider commands and return modules

Due to ambiguous format of provided input it is assumed that the last division is the module version in the case of more than one division.

Parameters:

command (str) – either ‘list’ or ‘spider’

Returns:

{prefix/…/name: [version1, …]} in case of more divisions

{‘name’: []} in case of one division

Return type:

(dict)

Module contents