virtmat.middleware.resconfig package

Submodules

virtmat.middleware.resconfig.cli module

command-line tools to configure available resources

virtmat.middleware.resconfig.cli.setup_resconfig(resconfig_loc=None): setup resource configuration interactively

virtmat.middleware.resconfig.qadapter module

utility functions dealing with fireworks qadapter

virtmat.middleware.resconfig.qadapter.get_custom_qadapter(cfg=None, w_name=None, q_name=None, g_name=None, lp_file=None, **kwargs)

Construct a custom queue adapter for the first matching resources always starting from the default resources

Parameters:

cfg (ResConfig) – resource configuration object, if None then it will
file (be loaded from resconfig)
w_name (str) – name of worker
q_name (str) – name of the queue, ignored if w_name is None
g_name (str) – group name to be used for accounting
lp_file (str) – a custom launchpad file to be specified in the
qadapter (rocket_launch section of)

Returns:

worker name, qadapter object in case of match, (None, None) otherwise

Return type:

tuple (str, CommonAdapter)

virtmat.middleware.resconfig.qadapter.get_default_qadapter(cfg=None, w_name=None, lp_file=None)

Construct a default qadapter from a resconfig object

Parameters:

cfg (ResConfig) – rsource configuration object, if None then it will
file (be loaded from resconfig)
w_name (str) – name of worker, if None then default_worker will be used
lp_file (str) – a custom launchpad file to be specified in the
qadapter (rocket_launch section of)

Returns:

a CommonAdapter object if worker type is not None, otherwise None

virtmat.middleware.resconfig.qadapter.get_pre_rocket(wcfg, **kwargs)

Return a pre_rocket string for qadapter

Parameters:

wcfg (WorkerConfig) – worker config
kwargs (dict) – required resources

Raises:

ResourceConfigurationError – in case of no match

virtmat.middleware.resconfig.qadapter.get_qadapter(wcfg, qcfg, g_name=None, lp_file=None, **kwargs)

Return a qadapter for specific worker, queue and optionally specified resources

Parameters:

wcfg (WorkerConfig) – worker configuration object
qcfg (QueueConfig) – queue configuration object
g_name (str) – group name to be used for accounting
lp_file (str) – a custom launchpad file to be specified in the
qadapter (rocket_launch section of)

Returns:

qadapter object None: if worker type not specified

Return type:

CommonAdapter

virtmat.middleware.resconfig.qadapter.get_worker_get_queue(cfg=None, w_name=None, q_name=None, g_name=None, **kwargs)

Return worker and queue matching a resource requirement

Parameters:

cfg (ResConfig) – resource configuration object, if None then it will
file (be loaded from resconfig)
w_name (str) – name of worker
q_name (str) – name of the queue, ignored if w_name is None
g_name (str) – group name to be used for accounting
kwargs (dict) – required resources

Returns:

worker and queue in case of match, (None, None) otherwise

Return type:

tuple(WorkerConfig, QueueConfig)

Raises:

ResourceConfigurationError – if resconfig (cfg) is not provided / found

virtmat.middleware.resconfig.qadapter.validate_resources(wcfg, qcfg, g_name=None, **kwargs)

Validate required resources for a specific worker and queue

Parameters:

wcfg (WorkerConfig) – worker configuration object
qcfg (QueueConfig) – queue configuration object
g_name (str) – group name to be used for accounting
kwargs (dict) – required resources

Raises:

ResourceConfigurationError – if the required resources are not valid

virtmat.middleware.resconfig.resconfig module

manage all available computing resources needed in fireworks

class virtmat.middleware.resconfig.resconfig.CommandConfig(cmd: str = None, args: list[str] = <factory>, commands: list[str] = <factory>)

Bases: ResConfigSerializable

Shell command configuration

cmd

a shell command

Type:: str

args

a list of command arguments, default: empty list

Type:: [str]

commands

a list of allowed commands

Type:: [str]

Raises:: ResourceConfigurationError – if a command is not allowed

args: list[str]

cmd: str = None

commands: list[str]

get_command(): return the command as a string

class virtmat.middleware.resconfig.resconfig.ModuleConfig(prefix: str = None, name: str = None, versions: list[str] = <factory>, path: str = None)

Bases: ResConfigSerializable

Module configuration

prefix

module prefix, if module name has prefix, else None

Type:: str

name

module name, mandatory

Type:: str

versions

available module versions, empty list if no versions

Type:: [str]

path

module path, if different from default path, else None

Type:: str

get_command(name, spec=None, raw=False)

Return the module load command

Parameters:

name (str) – name to match
spec (str) – a requirement specification (like ‘<=0.9.3’)
raw (bool) – if True return a dict (prefix, name, version) instead

Returns:

the shell command to load the module if matching else None

Return type:

(str|dict)

name: str = None

path: str = None

prefix: str = None

versions: list[str]

class virtmat.middleware.resconfig.resconfig.QueueConfig(name: str = None, public: bool = None, resources: list[~virtmat.middleware.resconfig.resconfig.ResourceConfig] = <factory>, accounts_allow: list[str] = <factory>, accounts_deny: list[str] = <factory>, groups_allow: list[str] = <factory>)

Bases: ResConfigSerializable

Queue configuration

name

queue name

Type:: str

public

True if publicly available, default None

Type:: bool

resources

list of configured resources

Type:: [ResourceConfig]

accounts_allow

list of usernames allowed to use the queue

Type:: [str]

accounts_deny

list of usernames disallowed to use the queue

Type:: [str]

groups_allow

list of groups allowed to use the queue

Type:: [str]

accounts_allow: list[str]

accounts_deny: list[str]

get_resource(res)

Get a specific computing resource configuration

Parameters:: res (str) – resource name
Returns:: ResourceConfig object if resource exists, None otherwise

groups_allow: list[str]

name: str = None

public: bool = None

resources: list[ResourceConfig]

set_resource(res, type_, val)

Set the value of a specific computing resource configuration

Parameters:

res (str) – resource name
type_ (str) – resource type: default, minimum, maximum
val (int|float) – value of the resource

Raises:

ValueError – if type_ has invalid value

validate_resource(name, val)

Check if resource value is between minimum and maximum

Parameters:

name (str) – resource name
val (int|float) – reource value

Raises:

ResourceConfigurationError – if resource value exceeds limits

class virtmat.middleware.resconfig.resconfig.ResConfig(workers: list[~virtmat.middleware.resconfig.resconfig.WorkerConfig] = <factory>, default_worker: ~virtmat.middleware.resconfig.resconfig.WorkerConfig = None)

Bases: ResConfigSerializable

collect and store all computing resources needed to use fireworks

add_worker_from_scratch(name=None, default=False): add a worker from scratch

default_worker: WorkerConfig = None

classmethod from_scratch(name=None): create a top-level resource configuration object from scratch

to_dict()

workers: list[WorkerConfig]

class virtmat.middleware.resconfig.resconfig.ResConfigSerializable

Bases: FWSerializable

base serialization class for resource classes

classmethod from_dict(*args, **kwargs)

to_dict(*args, **kwargs)

class virtmat.middleware.resconfig.resconfig.ResourceConfig(name: str, _minimum: int | float = None, _maximum: int | float = None, _default: int | float = None)

Bases: ResConfigSerializable

Computing resource

name

the name of the resource

Type:: str

property default

The default value of the resource

Returns:: the default value of the resource
Return type:: (float|int)

property maximum

The maximum value of the resource

Returns:: the maximum value of the resource
Return type:: (float|int)

property minimum: int | float

The minimum value of the resource

Returns:: the minimum value of the resource
Return type:: (float|int)

name: str

class virtmat.middleware.resconfig.resconfig.WorkerConfig(name: str = None, type_: str = None, queues: list[~virtmat.middleware.resconfig.resconfig.QueueConfig] = <factory>, accounts: list[str] = <factory>, modules: list[~virtmat.middleware.resconfig.resconfig.ModuleConfig] = <factory>, envvars: dict = <factory>, default_modules: list[~virtmat.middleware.resconfig.resconfig.ModuleConfig] = <factory>, default_envvars: list[str] = <factory>, default_venv: dict = None, default_launchdir: str = None, default_commands: list[~virtmat.middleware.resconfig.resconfig.CommandConfig] = <factory>, _default_queue: ~virtmat.middleware.resconfig.resconfig.QueueConfig = None, _default_account: str = None)

Bases: ResConfigSerializable

Worker configuration

Every computing cluster is mapped to a worker.

name

worker name

Type:: str

type_

type of batch queuing system, only ‘SLURM’ supported

Type:: str

queues

a list of configured queues

Type:: [QueueConfig]

accounts

a list of groups available for accounting

Type:: [str]

modules

([ModuleConfig]): a list of available environment modules

Type:: list[virtmat.middleware.resconfig.resconfig.ModuleConfig]

envvars

(dct): a dict of environment variables

Type:: dict

default_modules

([ModuleConfig]): a list of default environment modules

Type:: list[virtmat.middleware.resconfig.resconfig.ModuleConfig]

default_envvars

([str]): a list of default environment variables

Type:: list[str]

default_venv

(dict): a dict of default virtual environment

Type:: dict

default_launchdir

(str): default directory for launching jobs

Type:: str

default_commands

[CommandConfig]: a list of default commands

Type:: list[virtmat.middleware.resconfig.resconfig.CommandConfig]

accounts: list[str]

property default_account: str

The default group used for accounting

Returns:: the default group used for accounting if set, otherwise None
Return type:: str
Raises:: ResourceConfigurationError – if the group is not in the list of groups

default_commands: list[CommandConfig]

default_envvars: list[str]

default_launchdir: str = None

default_modules: list[ModuleConfig]

property default_queue: QueueConfig

The default queue

Returns:: the default queue configuration if set, otherwise None
Return type:: QueueConfig
Raises:: ResourceConfigurationError – if the queue is not in the list of queues

default_venv: dict = None

envvars: dict

classmethod from_scratch(name=None)

Extract all relevant available resources from slurm partitions

Parameters:: name (str) – optional worker name, if skipped one will be generated
Returns:: WorkerConfig object

modules: list[ModuleConfig]

name: str = None

queues: list[QueueConfig]

set_default_queue() → None: select the first configured queue as default queue

type_: str = None

virtmat.middleware.resconfig.resconfig.configure(resconfig_loc): create default resconfig and resconfig_loc path, and write resconfig

virtmat.middleware.resconfig.resconfig.get_default_resconfig(): return default resconfig

virtmat.middleware.resconfig.resconfig.get_inactive_envs(wcfg, **kwargs)

Return a list of required environment variables that need to be set

Parameters:

wcfg (WorkerConfig) – worker config
kwargs (dict) – required environment variables

Returns:

variables that need to be set

Return type:

(dict)

virtmat.middleware.resconfig.resconfig.get_inactive_modules(wcfg, **kwargs)

Return a list of required modules that are available but not activated

Parameters:

wcfg (WorkerConfig) – worker config
kwargs (dict) – required modules

Returns:

modules that have to be loaded, as tuples (prefix, name, version)

Return type:

(list)

Raises:

ResourceConfigurationError – if a resource is not available

virtmat.middleware.resconfig.resconfig.get_resconfig_loc(): return the resconfig path

virtmat.middleware.resconfig.resconfig.get_worker_config(cfg=None, name=None)

Get a worker config

Parameters:

cfg (ResConfig) – the resconfig to be used; if None: use default resconfig
name (str) – the name of worker, the default worker returned if None

Returns:

the matching worker config

Return type:

(WorkerConfig)

Raises:

ResourceConfigurationError – when no resconfig or no matching worker found

virtmat.middleware.resconfig.resconfig.resource_type_checker(func): decorate the menimum, maximum and default methods of ResourceConfig

virtmat.middleware.resconfig.resconfig.set_defaults_from_guess(wcfg)

Set defaults of specific resources from a guess

Parameters:: wcfg (WorkerConfig) – a worker configuration object

Module contents

export objects from resconfig, qadapter and cli modules