Skip to content

Public API (v0.1)

This page defines Consist's public API for the 0.1.x series. Items listed as Advanced are still public but are primarily targeted at advanced users and may be more verbose, lower-level, or easier to misuse.

For tiered guidance:

  • API Essentials for onboarding and preferred execution patterns
  • API Advanced for manual lifecycle, decorators, and lower-level APIs

When to use each run entry point

Entry point Use when Notes
Tracker.run You want explicit tracker ownership for one cache-aware function Preferred single-step entry point
Tracker.trace You need inline execution with explicit tracker control Block executes every time
consist.scenario You need multi-step workflows with shared scenario context Returns ScenarioContext
ScenarioContext.run You are inside a scenario and want cache-aware steps Step-level equivalent of Tracker.run; also accepts binding=BindingResult(...) for orchestrator-resolved inputs
ScenarioContext.map_runs You are inside a scenario and want one child run per sweep row Process-backed fan-out for independent parameter sweeps
ScenarioContext.trace You are inside a scenario and need inline step execution Step-level equivalent of Tracker.trace
consist.run You need the legacy context-resolving convenience wrapper Deprecated compatibility wrapper around Tracker.run
consist.trace You need the legacy context-resolving tracing wrapper Deprecated compatibility wrapper around Tracker.trace
consist.start_run You need the legacy context-resolving lifecycle wrapper Deprecated compatibility wrapper around Tracker.start_run

Relationship: consist.scenario, consist.run, and Tracker.run

  • Tracker.run(...) is the explicit class method for a single cache-aware step.
  • consist.scenario(...) creates a scenario context; inside it, call sc.run(...) for cache-aware steps or sc.trace(...) for inline blocks that execute each time. If an external planner has already resolved the step bindings, pass binding=BindingResult(...) to sc.run(...) instead of unpacking primitive input kwargs. That is the preferred scenario-level envelope for complex or externally orchestrated workflows.
  • consist.run(...) resolves the active/default tracker and then calls Tracker.run(...), but it is now a deprecated compatibility wrapper.

Top-level API policy

Consist keeps top-level consist.* helpers when they add one of two things:

  • Ambient run-scoped behavior: helpers that attach metadata, artifacts, or side effects to an already-active run without forcing tracker plumbing through nested workflow code. Examples include consist.log_artifact(...), consist.log_artifacts(...), consist.log_input(...), consist.log_output(...), consist.ingest(...), consist.log_meta(...), and consist.capture_outputs(...).
  • Pure/context utilities: helpers that do not create tracker-owned execution state, such as consist.ref(...), consist.refs(...), consist.output_path(...), and consist.output_dir(...).

Consist deprecates top-level helpers when they are only duplicate tracker-owning execution entry points:

  • consist.run(...)
  • consist.trace(...)
  • consist.start_run(...)

consist.scenario(...) remains a deliberate exception. It is still a top-level workflow entry point, but it carries meaningful API-level behavior beyond simple forwarding, including the enabled=False noop branch and the scenario ergonomics used throughout examples and multi-step workflows.

Run/trace identity kwargs (public surface)

Run/trace APIs share these public identity kwargs:

  • adapter=...
  • identity_inputs=[...]

This applies to:

  • consist.run(...) / consist.trace(...)
  • Tracker.run(...) / Tracker.trace(...)
  • ScenarioContext.run(...) / ScenarioContext.trace(...)

config_plan= and hash_inputs= are not accepted on these run/trace surfaces. Passing them raises TypeError (unexpected keyword argument ...).

For signature debugging, use run.identity_summary.

Minimal runnable comparison

from pathlib import Path
import consist
from consist import Tracker

tracker = Tracker(run_dir="./runs", db_path="./provenance.duckdb")

def write_result() -> Path:
    out = consist.output_path("result", ext="txt")
    out.write_text("ok\n")
    return out

# Preferred class API: explicit tracker ownership.
one = tracker.run(fn=write_result, outputs=["result"])

# Scenario API: grouped multi-step workflows.
with tracker.scenario("baseline") as sc:
    step = sc.run(fn=write_result, name="step_result", outputs=["result"])

Stable (intended for external users)

Scenario / workflow helpers

Scenario defaults like name_template and cache_epoch are configured via consist.scenario(...) and flow into ScenarioContext.run(...).

Parallel orchestration models

These models are returned by, or support, ScenarioContext.map_runs(...):

  • consist.BatchResult: aggregate result for a sweep/batch.
  • consist.RunSpecResult: per-spec result, including persisted_run_id when a run record exists.
  • consist.BatchRunSpec: serializable description of one child run.
  • consist.ExecutionAttempt: status details for one backend attempt.
  • consist.ExecutionSpec and consist.PythonCallableTarget: lower-level execution request objects used by the process backend.

Utilities and Introspection

Artifact logging and loading

Querying and views

Run lookup helpers treat stage and phase as first-class run dimensions, so you can filter historical runs with consist.find_runs(...), consist.find_latest_run(...), or Tracker.find_latest_run(...) without treating those values as opaque metadata. Use find_matching_run(...) when restart or reuse code needs the latest run matching a semantic target with optional cache_epoch= and run_scope=.

Tracker methods

This section groups commonly documented public Tracker methods. If you're new to Consist, start with the Core and Logging/Loading groups and reach for Advanced only as needed.

Core lifecycle

  • begin_run, start_run, run, trace, scenario, end_run
  • define_step, last_run, is_cached, cached_artifacts, cached_output
  • suspend_cache_options, restore_cache_options, capture_outputs, log_meta

Logging and loading

  • log_artifact, log_artifacts, log_input, log_output, log_dataframe
  • load, materialize, ingest

Querying and history

  • find_runs, run_set, find_run, find_latest_run, find_matching_run, find_matching_runs, get_latest_run_id
  • find_artifacts, get_artifact, get_child_artifacts, get_parent_artifact, get_artifacts_for_run
  • find_artifacts_by_params, get_artifact_kv, register_artifact_facet_parser
  • get_run, get_run_config, get_run_inputs, get_run_outputs
  • hydrate_run_outputs, hydrate_run_outputs_to_destinations, materialize_run_outputs, materialize_artifact
  • stage_artifact, stage_inputs
  • set_artifact_recovery_roots, register_artifact_recovery_copy, register_run_output_recovery_copies, archive_artifact, archive_run_outputs, archive_current_run_outputs, archive_run_output_files
  • get_artifact_lineage, print_lineage, history
  • diff_runs, get_config_facet, get_config_facets, get_run_config_kv
  • get_config_values, get_config_value, find_runs_by_facet_kv

Views and matrices

  • view, create_view, load_matrix, export_schema_sqlmodel
  • netcdf_metadata, openmatrix_metadata, spatial_metadata

Config canonicalization

  • canonicalize_config, canonicalize_gtfs, prepare_config, apply_config_plan, identity_from_config_plan

Format-specific logging

  • log_h5_container, log_h5_table, log_netcdf_file, log_openmatrix_file

log_h5_container(...) now supports child_specs={...} and child_selection="all" | "include_only" so callers can either customize discovered child artifacts or log only a selected subset of HDF5 dataset paths. It also accepts parent recovery policy kwargs such as container_recovery_unit="parent_file" and child_recovery_policy="descriptive_only" so downstream recovery tools can distinguish parent-file recovery from descriptive child-table artifacts. log_h5_table(...) accepts the same child-level semantic metadata surface as normal artifact logging: description, facet, facet_schema_version, and facet_index.

Advanced (power-user / lower-level)

These methods are still public, but are more low-level or easier to misuse.

  • engine, set_run_subdir_fn, run_artifact_dir, resolve_uri
  • run_query, get_run_record, resolve_historical_path, load_input_bundle
  • get_artifact_by_uri, get_run_artifact, load_run_output, find_matching_run
  • on_run_start, on_run_complete, on_run_failed

Stable, but optional extras

These APIs are part of the public surface, but require extra dependencies.