CommandRunnerPSIJ#

class maize.utilities.execution.CommandRunnerPSIJ(raise_on_failure: bool = True, working_dir: Path | None = None, validators: Sequence[Validator] | None = None, prefer_batch: bool = False, rm_config: ResourceManagerConfig | None = None)[source]#

Bases: object

Command running utility based on PSI/J.

Instantiate with preferred options and use a run method with your command.

Danger

It is not recommended to instantiate this class in the main process (i.e. outside of your nodes run() and prepare() methods). This is due to possible subtle threading problems from the interplay of maize and PSI/J.

Parameters:

  • raise_on_failure – Whether to raise an exception on failure, or whether to just return False.

  • working_dir – The working directory to use for execution, will use the current one by default.

  • validators – One or more Validator instances that will be called on the result of the command.

  • prefer_batch – Whether to prefer running on a batch submission system such as SLURM, if available

  • rm_config – Configuration of the resource manager

__init__(raise_on_failure: bool = True, working_dir: Path | None = None, validators: Sequence[Validator] | None = None, prefer_batch: bool = False, rm_config: ResourceManagerConfig | None = None) → None[source]#

Methods

__init__([raise_on_failure, working_dir, ...])

run(command[, verbose, working_dir, ...])

Run a command and validate.

run_only(command[, verbose, working_dir, ...])

Run a command locally and block.

run_parallel(commands[, verbose, n_jobs, ...])

Run multiple commands locally in parallel and block.

run_validate(command[, verbose, ...])

Run a command and validate.

validate(result)

Validate a process result.
run(command: list[str] | str, verbose: bool = False, working_dir: Path | None = None, command_input: str | None = None, config: JobResourceConfig | None = None, pre_execution: str | list[str] | None = None, timeout: float | None = None) → CompletedProcess[bytes]#

Run a command and validate.

Parameters:

  • command – Command to run as a single string, or a list of strings

  • verbose – If True will also log any STDOUT or STDERR output

  • working_dir – Optional working directory

  • command_input – Text string used as input for command

  • config – Resource configuration for jobs

  • pre_execution – Command to run directly before the main one

  • timeout – Maximum runtime for the command in seconds, or unlimited if None

Returns:

Result of the execution, including STDOUT and STDERR

Return type:

subprocess.CompletedProcess[bytes]

Raises:

ProcessError – If any of the validators failed or the returncode was not zero

run_only(command: list[str] | str, verbose: bool = False, working_dir: Path | None = None, command_input: str | None = None, config: JobResourceConfig | None = None, pre_execution: str | list[str] | None = None, timeout: float | None = None) → CompletedProcess[bytes][source]#

Run a command locally and block.

Parameters:

  • command – Command to run as a single string, or a list of strings

  • verbose – If True will also log any STDOUT or STDERR output

  • working_dir – Optional working directory

  • command_input – Text string used as input for command

  • config – Resource configuration for jobs

  • pre_execution – Command to run directly before the main one

  • timeout – Maximum runtime for the command in seconds, or unlimited if None

Returns:

Result of the execution, including STDOUT and STDERR

Return type:

subprocess.CompletedProcess[bytes]

Raises:

ProcessError – If the returncode was not zero

run_parallel(commands: Sequence[list[str] | str], verbose: bool = False, n_jobs: int = 1, validate: bool = False, working_dirs: Sequence[Path | None] | None = None, command_inputs: Sequence[str | None] | None = None, config: JobResourceConfig | None = None, pre_execution: str | list[str] | None = None, timeout: float | None = None) → list[CompletedProcess[bytes]][source]#

Run multiple commands locally in parallel and block.

Parameters:

  • commands – Commands to run as a list of single strings, or a list of lists

  • verbose – If True will also log any STDOUT or STDERR output

  • n_jobs – Number of processes to spawn at once

  • validate – Whether to validate the command execution

  • working_dirs – Directories to execute each command in

  • command_input – Text string used as input for each command, or None

  • config – Resource configuration for jobs

  • pre_execution – Command to run directly before the main one

  • timeout – Maximum runtime for the command in seconds, or unlimited if None

Returns:

Result of the execution, including STDOUT and STDERR

Return type:

list[subprocess.CompletedProcess[bytes]]

Raises:

ProcessError – If the returncode was not zero

run_validate(command: list[str] | str, verbose: bool = False, working_dir: Path | None = None, command_input: str | None = None, config: JobResourceConfig | None = None, pre_execution: str | list[str] | None = None, timeout: float | None = None) → CompletedProcess[bytes][source]#

Run a command and validate.

Parameters:

  • command – Command to run as a single string, or a list of strings

  • verbose – If True will also log any STDOUT or STDERR output

  • working_dir – Optional working directory

  • command_input – Text string used as input for command

  • config – Resource configuration for jobs

  • pre_execution – Command to run directly before the main one

  • timeout – Maximum runtime for the command in seconds, or unlimited if None

Returns:

Result of the execution, including STDOUT and STDERR

Return type:

subprocess.CompletedProcess[bytes]

Raises:

ProcessError – If any of the validators failed or the returncode was not zero

validate(result: CompletedProcess[bytes]) → None[source]#

Validate a process result.

Parameters:

result – Process result to validate

Raises:

ProcessError – If any of the validators failed or the returncode was not zero