Documenting a Pipeline

Documenting a Pipeline#

A pipeline is a project that produces artifacts — parquet files, chart HTML, notebooks. ChartBook doesn’t run the pipeline; you build the artifacts however you like (doit, make, plain scripts) and chartbook.toml describes what they are, where they live, and how they fit together. chartbook build then renders that description into a website.

This page covers the describing. For the field-by-field reference, see Configuration Guide.

Layout#

The conventional layout (what chartbook init scaffolds):

my-pipeline/
├── chartbook.toml     # the manifest — this page's subject
├── README.md          # required; becomes the site's front page
├── dodo.py            # or any other way of running your pipeline
├── src/               # code that produces the artifacts
├── _data/             # parquet files ([dataframes] point here)
├── _output/           # chart HTML and executed notebooks ([charts] and [notebooks] point here)
└── docs_src/
    ├── charts/        # chart write-ups (referenced via docs_path)
    ├── dataframes/    # dataframe write-ups
    └── site/          # optional extra site pages, merged in automatically

Only chartbook.toml and README.md are fixed names. The rest is convention — each manifest entry carries its own path, so artifacts can live wherever your build puts them.

Project metadata#

Every [project] field is optional with sensible defaults (name defaults to the directory name, copyright to the current year, maintainer to the first contributor; the pipeline id’s scope is inferred from the git origin remote when id and repo_url are unset):

[project]
name = "Yield Curve"
description = "Zero-coupon yield curve estimates and repo rates"
maintainer = "Jane Doe"
build = """
conda activate myenv
doit
"""

build records how to regenerate the artifacts. It has shell-script semantics — a multi-line value is one script, so environment activation carries into later lines. It’s displayed on the pipeline’s page (and reserved for a future chartbook run).

Charts#

A chart is an HTML file plus an entry describing it. The HTML usually comes from Plotting, but any HTML works — ChartBook embeds it in an iframe.

[charts.repo_rates]
name = "Repo Rates"
description = "SOFR and EFFR overnight rates"
dataframe = "repo_public"                     # which [dataframes.*] entry feeds it
path = "./_output/repo_rates.html"
docs_path = "./docs_src/charts/repo_rates.md"
tags = ["Repo", "Short Term Funding"]
frequency = "Daily"
units = "Percent"

Two rules:

  • dataframe names the entry in [dataframes] that feeds the chart; referencing a missing one is an error.

  • Exactly one of docs_path (a markdown file) or docs (inline markdown) supplies the write-up.

The write-up is the part only you can provide: what the chart shows, how to read it, caveats. The generated chart page assembles the interactive chart, your write-up, and auto-generated spec tables (chart metadata, dataframe manifest, pipeline manifest). Fields you leave out simply don’t appear; the full metadata vocabulary — publication tracking, clearance dates, release timing — is in Configuration Guide.

Dataframes#

A dataframe is a parquet artifact plus provenance:

[dataframes.repo_public]
name = "Public Repo Data"
description = "Overnight repo rates and volumes"
sources = ["FRED", "Office of Financial Research"]
path = "./_data/repo_public.parquet"
date_col = "date"
docs_path = "./docs_src/dataframes/repo_public.md"

The same docs_path/docs rule applies. The generated page shows your write-up, a glimpse (schema and row count read from the parquet at build time), and a manifest table. Provenance fields — providers, license, access_types, pull_method, and friends — exist for governance-minded catalogs; see the reference.

For hive-partitioned datasets, path takes a glob:

path = "./_data/market_data/**/*.parquet"

Glob-path dataframes load as polars LazyFrames only, with partition columns appearing automatically; see Catalogs and Data Access.

Notebooks#

[notebooks.exploration]
description = "Exploratory analysis of curve-fitting residuals"
path = "_output/01_exploration.ipynb"
publishable = true

Notebooks render as pages, pre-executed — ChartBook does not run them. The title comes from the notebook’s first # Heading; set name to override. publishable = true includes the notebook when you run chartbook publish.

Notes and custom pages#

[notes] entries add standalone markdown documents to the site:

[notes.methodology]
path = "./docs_src/methodology.md"

And any markdown placed in docs_src/site/ merges into the site automatically (that’s the default site_dir; auto-generated pages live in a separate cb/ directory, so nothing collides). Write-ups and site pages are MyST markdown — regular markdown plus admonitions, math, and cross-references; see the MyST syntax guide.

Finding project paths from code#

Pipeline scripts often need paths that work from any working directory, including notebooks. chartbook.env helps:

import chartbook

BASE_DIR = chartbook.env.get_project_root()   # searches upward for .git / pyproject.toml / .env
DATA_DIR = BASE_DIR / "_data"
API_KEY = chartbook.env.get("FRED_API_KEY", default="")  # .env file or environment

Checking your work#

chartbook build is strict by default: if a path or docs_path points at a file that doesn’t exist, the build fails and names both the missing file and the manifest entry referencing it. --no-strict downgrades this to a warning that skips the affected pipeline. Two more inspection tools:

  • Unknown [project] keys warn with a did-you-mean suggestion rather than becoming silently dead metadata.

  • chartbook create-data-glimpses scans your dodo.py targets and writes data_glimpses.md — schema, null rates, sample rows, and summary stats for every data file your tasks produce.

Once the pipeline builds cleanly, register it in a catalog: Catalogs and Data Access.

Contents