Core Concepts Overview¶

This section establishes the mental model for Consist before covering API details.

Core Abstractions¶

Artifact: A file with provenance metadata: path, format, canonical fingerprint (artifact.hash), producing run, and ingestion status. Fingerprint semantics follow the active hashing strategy, so under fast hashing the value may be metadata-based rather than a strict byte-content digest.

Run: A single execution with tracked inputs, configuration, outputs, status, and timing. Each run has a signature computed from code, config, and inputs that enables cache reuse.

Scenario: A parent run grouping related child runs for multi-variant studies or iterative workflows.

Coupler: A helper that passes artifacts between steps in a scenario, linking lineage automatically.

How Caching Works¶

Consist computes a signature from code version, config, and input artifact hashes:

graph LR
    Code[Code Version] --> Hash[SHA256 Signature]
    Config[Configuration] --> Hash
    Inputs[Input Artifacts] --> Hash
    Hash --> Lookup{Cache Lookup}
    Lookup -->|Hit| Return[Return Cached Outputs]
    Lookup -->|Miss| Execute[Execute & Record New Run]

Same signature → return cached outputs. Different signature → execute and record new lineage.

On cache hits, Consist returns output artifact metadata without copying files. Load or hydrate outputs when you need bytes.

Provenance & Lineage¶

Provenance: The complete history of a result—code version, configuration, input data, and compute environment. Consist records provenance for every tracked run.

Lineage: The dependency chain showing which run created an artifact, which inputs that run consumed, and which runs produced those inputs.

graph TD
    Raw[Raw Data] --> Step1[Run: Clean]
    Config1[Threshold=0.5] --> Step1
    Step1 --> Art1[Artifact: Cleaned]
    Art1 --> Step2[Run: Analyze]
    Config2[GroupByKey=category] --> Step2
    Step2 --> Art2[Artifact: Summary]

Provenance answers three questions: Can I re-run this exactly? (reproducibility), Which config produced this figure? (accountability), and Why did this change? (debugging).

Example: You published a land-use forecast. A reviewer asks which scenario produced Figure 3. Run consist show <run_id> to see the code version (commit SHA), config parameters, input parcel data, and execution timestamp.

Canonical Terms (Quick Reference)¶

This page keeps one-line canonical definitions. Detailed behavior and policies live in the linked specialized pages.

For a full term index, see the Glossary.

How Inputs and Outputs Are Treated¶

Inputs are files or values that influence computation. File inputs are hashed by content or metadata depending on the hashing strategy (full vs fast).

Outputs are named artifacts registered via a function's dict[str, Path] return value together with declared outputs=[...], or explicitly via output_paths. Consist stores their paths and provenance metadata for lookup and querying.

Recommended pattern: file-based I/O¶

The recommended pattern is for functions to accept file paths as inputs, write output files, and return a dict[str, Path] mapping artifact keys to output paths. Declare the matching artifact keys in outputs=[...] when calling tracker.run(...); Consist then logs those returned paths as artifacts.

import pandas as pd
from pathlib import Path

def summarize_trips(trips_path: Path) -> dict[str, Path]:
    df = pd.read_parquet(trips_path)
    out = Path("./summary.parquet")
    df.groupby("mode")["distance_miles"].mean().to_frame().to_parquet(out)
    return {"summary": out}

result = tracker.run(
    fn=summarize_trips,
    inputs={"trips_path": trips_artifact},  # path resolved from artifact; hashed into the signature
    outputs=["summary"],
)

The function is a plain Python callable: testable without a tracker, honest about its I/O, and readable without framework knowledge.

When to use output_paths¶

If a function writes files but returns None (e.g., a legacy tool or subprocess wrapper), register outputs explicitly:

result = tracker.run(
    fn=run_legacy_model,
    inputs={"config": config_artifact},
    output_paths={"results": Path("./model_output.csv")},
)

Use ArtifactSpec when a declared output also needs schema, driver, facet, or file-profiling metadata:

from consist import ArtifactSpec

result = tracker.run(
    fn=run_legacy_model,
    inputs={"config": config_artifact},
    output_paths={
        "synthetic_firms": ArtifactSpec(
            path=Path("./synthetic_firms.csv"),
            schema=SyntheticFirms,
            profile_file_schema=True,
        )
    },
)

schema=... tags and persists the logical schema metadata. It does not validate the file contents; stricter validation remains a separate workflow. The declaration is applied when Consist logs the output. On cache hits, newly added schema or profiling metadata is not backfilled onto cached artifacts; force a rerun or change the cache version/epoch when you need that metadata recorded.

When to use output_sets¶

Use output_sets when one logical output is written as multiple files, such as annual partitions, thread chunks, or diagnostic bundles. For discovery-only sets, the smallest useful declaration still only needs a root directory and an include pattern:

from consist import OutputSet

result = tracker.run(
    fn=run_forecast,
    output_sets={
        "annual_outputs": OutputSet(
            root="annual",
            include="annual_*.csv",
        )
    },
)

Here run_forecast should write files such as annual/annual_2030.csv under the run output directory. Consist discovers every file matching include, sorts the members by relative path, records one logical parent artifact named annual_outputs, and records each file as a child artifact.

If a filename segment should become queryable metadata, use a capture-aware FilenamePattern instead of a raw glob string. In v1, each capture binds to a numbered wildcard explicitly, so the filename layout stays fail-closed and discovery-only patterns remain unchanged:

from consist import EnumCapture, FilenamePattern, IntCapture, OutputSet

yearly = OutputSet(
    root="annual",
    include=FilenamePattern.glob("annual_*.parquet").with_captures(
        IntCapture(name="year", wildcard=1)
    ),
)

purpose = OutputSet(
    root="trip_outputs",
    include=FilenamePattern.glob("trip_*.csv").with_captures(
        EnumCapture(
            name="purpose",
            allowed={"home", "work", "shopping"},
            wildcard=1,
        )
    ),
)

Capture-aware patterns match the output-set relative path, reject **, and require every wildcard to be bound. Captured values are stored with their real types on member artifacts, so year is queryable as an integer rather than a string.

The current capture surface is intentionally narrow: integer and enum-style captures cover the supported v1 cases. If you need another typed capture, open an issue first so the extension can be designed deliberately instead of growing ad hoc.

If a filename repeats the same facet more than once, bind both wildcards to the same capture name. Consist will keep one facet value and require the repeated values to match:

events = OutputSet(
    root="output",
    recursive=True,
    include=FilenamePattern.glob("IT.*/*.events.parquet").with_captures(
        IntCapture(name="iteration", wildcard=1),
        IntCapture(name="iteration", wildcard=2),
    ),
)

This matches paths like output/IT.3/3.events.parquet and yields one iteration=3 facet. If the two occurrences disagree, Consist raises during discovery instead of guessing which one to keep.

The optional expected_members and expected_count fields turn discovery into a completeness check. They are not required. Use them when the config tells you which files must exist and the run should fail if one is missing:

result = tracker.run(
    fn=run_forecast,
    config={"years": [2030, 2035]},
    output_sets={
        "annual_outputs": OutputSet(
            root="annual",
            include="annual_*.csv",
            expected_members=lambda config: [
                f"annual_{year}.csv" for year in config["years"]
            ],
        )
    },
)

Consist records one logical parent artifact plus member file artifacts linked by parent_artifact_id. Cache hydration restores the member files under the declared root.

If you pass schema=... on an OutputSet, Consist tags both the logical parent and discovered member artifacts with that schema. Member CSV/Parquet artifacts can also be profiled with OutputSet(profile_file_schema=True) or an enclosing Tracker.run(..., profile_file_schema=True) / ScenarioContext.run(..., profile_file_schema=True).

Output-set fields fall into two groups:

• Required: root, include.
• Optional discovery controls: exclude, recursive.
• Optional metadata: kind, schema, partition_key, facet, member_facets.
• Optional validation: expected_count, expected_members, validate.

For capture-aware sets, include accepts a FilenamePattern object. The wildcard= binding is the v1 mechanism for mapping each wildcard to exactly one capture; if the filename matches discovery but fails the typed capture, Consist raises during registration rather than silently skipping the file.

For v1, validate="manifest" is the default and applies the optional expected_* checks when provided. validate="exists" also requires at least one member. Stricter hash and schema validation modes are reserved for a future release.

Auto-loading inputs as DataFrames¶

If you prefer to receive a DataFrame directly instead of a path, use input_binding="loaded" and Consist will load the artifact for you before calling the function. This is convenient for short scripts but hides the I/O boundary — prefer input_binding="paths" for pipelines where the function boundary matters.

Next Steps¶

• Usage Guide — Practical patterns for moving from the mental model to working code
• Config Management — Understand the config vs. facet distinction and when to use each
• Caching & Hydration — Caching patterns and data recovery strategies
• Data Storage and Ingestion — When to ingest data and use hybrid views
• Grouped Views — Build one view across schema-matched artifacts
• Decorators & Metadata — Defaults, templates, and schema introspection