DownloadThe q utility is a platform for creating and managing data analysis pipelines. It expands the value of your existing job scheduler - either Grid Engine or TORQUE PBS - through numerous functions that help you organize, submit, monitor, manage and share your informatics work.
Data processing pipelines require high-level organization and parallelization of work to optimize resource utilization and decrease the time to results. q (from queue) allows complex job sequences to be efficiently assembled and managed, including dependency tracking, parallelization, and pipeline-level monitoring, error recovery, and data protection. Pipelines are constructed from modular script files, with job definitions and results stored in easily retrieved job files. A web interface facilitates job submission and monitoring, with the complete pipeline exportable for full transparency.
Features
- Organization of work into higher order units from existing code pieces in any language.
- Pipelines are submitted and managed as a unit by calls to a single master file.
- Robust definition and tracking of job dependencies.
- Multiple modes of parallelization maximize resource utilization and minimize time to results.
- Admixture of submission-time and execution-time commands supports dynamic job configuration.
- Intelligent tracking of queued jobs supports incremental execution of new work.
- A few q-specific syntax elements create complex pipelines with minimal code.
- Highly modular structure facilities code reuse and sharing.
- Various features facilitate cross-platform implementation of pipelines.
- Extensive submission-time syntax checking helps prevent fractured pipelines.
- Enhanced job reporting and interfaces into job scheduler functions, including error handling.
- Write-protection and backup of output files integrated into pipeline definition files.
- History tracking of all work performed in a pipeline, with rollback.
- Web-based interface into job submission and management.
- Fully transparent sharing of work at publication via comprehensive HTML reports.
- End-user installation with no required management by system administrators.
Project Samples
