PyDAG¶
Simple Python Interface to HTCondor and DAGMan
PyDAG allows you to create HTCondor submit descriptions, DAGMan nodes, and DAGs via Python. Any Python object that has a string representation that corresponds to a valid HTCondor command or DAGMan keyword can be added to the submit description. You can easily extend PyDAG by defining your own command/keyword classes; classes for pre/post-scripts and macros are already provided.
-
class
pydag.htcondor.
HTCondorSubmit
(filename, executable, queue=1, **kwargs)¶ HTCondor submit description
Represents a submit description of a HTCondor job.
Parameters: - filename (str) – HTCondor submit description file
- executable (str) – HTCondor executable file
- queue (int, optional) – Number of jobs to queue
- **kwargs – Submit description file commands
-
commands
¶ Mapping of describing submit description file commands to objects representing values
Type: dict(str, object)
Examples
Create HTCondor submit description for a Python script.
>>> import pydag >>> job = pydag.htcondor.HTCondorSubmit("example.submit", "example.py") >>> job.commands["initialdir"] = "$ENV(HOME)" >>> print(job) universe = vanilla executable = example.py initialdir = $ENV(HOME) queue 1 >>> job.dump() >>> print(job.written_to_disk) True
-
dump
()¶ Write HTCondor submit description to filename.
-
written_to_disk
¶ If True submit description was written to disk.
Type: bool
-
class
pydag.dagman.
DAGManJob
(filename, nodes)¶ DAGMan job
Specify a DAG’s input.
Parameters: - filename (str) – DAGMan input file
- nodes (list(DAGManNode)) – Sequence of DAGMan nodes
Examples
Create a DAG with one node; a macro
inputfile
is defined; the macro’s value is passed as an argument to the Python script that is specified as the executable in the node’s HTCondor submit description.>>> import pydag >>> job = pydag.htcondor.HTCondorSubmit("example.submit", "example.py") >>> job.commands["arguments"] = "$(inputfile)" >>> node = pydag.dagman.DAGManNode("example", job) >>> node["VARS"] = pydag.dagman.Macros(inputfile="example.txt") >>> dag = pydag.dagman.DAGManJob("example.dag", [node]) >>> print(dag) JOB example example.submit VARS example inputfile="example.txt" >>> dag.dump() >>> print(dag.written_to_disk) True
-
add_dependency
(parents, children)¶ Add dependency within the DAG.
Nodes are parents and/or children within the DAG. A parent node must be completed successfully before any of its children may be started. A child node may only be started once all its parents have successfully completed.
Parameters: - parents (tuple(str)) – Parent node names
- children (tuple(str)) – Children node names
Raises: ValueError
– If parents or children contains an unknown DAGMan node.
-
dependencies
¶ Dependencies within the DAG
Type: str
-
dump
()¶ Write DAGMan input file to filename.
-
written_to_disk
¶ If True DAGMan input file was written to disk.
Type: bool
-
class
pydag.dagman.
DAGManNode
(name, submit_description, **kwargs)¶ DAGMan node
Specify a DAG’s node.
Parameters: - name (str) – Uniquely identifies nodes within the DAGMan input file and in output messages
- submit_description (HTCondorSubmit, str) – HTCondor submit description; either a string specifying a submit description file or an HTCondorSubmit instance
- **kwargs – Node keywords
-
keywords
¶ Mapping of node keywords to objects representing values
Type: dict(str, object)
-
class
pydag.dagman.
DAGManScript
(executable, *args)¶ Pre-processing or post-processing
Specify a shell script/batch file to be executed either before a job within a node is submitted or after a job within a node completes its execution.
Parameters: - executable (str) – Specify the shell script/batch file to be executed.
- *args – Script/batch file arguments
-
arguments
¶ Sequence of objects representing script/batch file arguments
Type: list(object)
-
class
pydag.dagman.
Macros
¶ Define macros for DAGMan node.
Derived from dict; only __str__ is re-implemented to meet the DAGMan macro syntax.