Consist documentation
Provenance tracking and run caching for scientific workflows
Consist records what your simulation runs consumed and produced, reuses prior outputs when nothing changed, and lets you trace any figure back to the inputs that made it. Built for scientific workflows where reproducibility is not optional.
Reference documentation for recording workflow runs, recovering outputs, and analyzing scenario lineage with DuckDB-backed metadata.
- Recorded state
- Code, config, inputs, outputs, and scenario metadata
- Cached execution
- Reuse unchanged runs and hydrate prior outputs on demand
- Queryable history
- Search past runs, recorded files, and scenario metadata in DuckDB
Start here
Read these pages first if you are new to Consist.
What Consist records
For each run, Consist records the execution context and artifact metadata needed to reproduce or inspect a result.
- the function or external tool that ran
- config and identity inputs that affect the run signature
- input and output artifacts, including file hashes and lineage
- scenario, stage, phase, and facet metadata for later querying
On a cache hit, Consist returns previous output artifact metadata instead of re-running the step. Load, hydrate, or materialize bytes only when the next tool needs them.
This matters when a reviewer asks which scenario produced a published figure, when a colleague needs to reproduce last quarter's result after the config has drifted, or when a parameter sweep wastes hours re-running preprocessing that didn't change.
Common reference needs
- Usage GuideChoose between
run,trace, andscenario. - Caching & HydrationUnderstand cache hits, misses, hydration, and materialization.
- First WorkflowPass artifacts between workflow steps.
- Historical RecoveryRestore historical outputs or stage inputs.
- RunSet AnalysisSelect, align, and compare scenario runs.
- Grouped ViewsCreate one analysis relation across schema-compatible artifacts.
- Container GuideRun Docker, Singularity, or Apptainer tools.
- Domain TrackerWrap Consist behind domain-specific verbs for shared codebases.
- GTFS SupportCanonicalize transit feeds into selected-service slices for Tracker or BEAM.
- Data Storage and IngestionIngest data for SQL analysis.
- Schema ExportGenerate SQLModel schemas from captured files.
- CLI ReferenceInspect runs from the terminal.
- DB MaintenanceRepair or compact the provenance database.
- TroubleshootingDebug a cache miss or missing output.
- API ReferenceLook up Python signatures.