How the Build Works

How the Build Works#

The chartbook build command uses a two-stage pipeline to generate your documentation site:

Markdown generation — reads your chartbook.toml manifest and project files, then produces intermediate Markdown files (and copies static assets) into a temporary _docs/ directory.
Sphinx build — runs sphinx-build to convert those intermediate Markdown files into a static HTML site.

By default, the temporary directories are deleted when the build finishes — whether it succeeded or failed. Using the --keep-build-dirs flag preserves them so you can inspect the intermediate output, which is exactly what you want when debugging a failed build.

Understanding these internals is useful for:

Debugging build failures — examining the exact Markdown that Sphinx tried to process
Verifying metadata — checking that manifest values rendered correctly in spec tables
Customizing templates — overriding the default Jinja2 templates used during generation
Understanding the output — seeing how your chartbook.toml configuration maps to generated pages

The Two-Stage Build Process#

The following diagram shows the full build flow:

chartbook build [OUTPUT_DIR]
    |
    |-- Stage 0: Template Preparation
    |   Copy theme templates into _docs_src/
    |   Copy logo + favicon
    |
    |-- Stage 1: Markdown Generation
    |   Copy charts, notebooks, parquets into _docs/
    |   Generate diagnostics CSV
    |   Render Jinja2 templates into .md files
    |   Copy remaining _docs_src/ files to _docs/
    |
    |-- Stage 2: Sphinx Build
    |   Render conf.py.j2 -> conf.py
    |   Run: sphinx-build -M html _docs _docs/_build
    |   Copy _docs/_build/html/ to OUTPUT_DIR
    |
    |-- Cleanup (unless --keep-build-dirs)
        Delete _docs/ and _docs_src/

Stage 0: Template Preparation (`_docs_src/`)#

Before any Markdown is generated, ChartBook assembles a temporary _docs_src/ directory containing all the Sphinx source templates and configuration:

Theme selection: Based on the resolved project type — inferred from your chartbook.toml structure (a [pipelines] table means catalog), or set explicitly via [project] type — the package’s built-in theme templates are copied from either docs_src_pipeline/ (for single pipelines) or docs_src_catalog/ (for multi-pipeline catalogs).
Assets: Your logo and favicon (from the [project] logo and favicon paths in chartbook.toml, or package defaults) are copied into _docs_src/_static/ and _docs_src/assets/.

To override a template, place a file with the same base name in your project root — that’s the first entry on the Jinja search path (see Template resolution order below). Your project’s own docs_src/ directory is not copied into _docs_src/.

The resulting _docs_src/ directory looks like this:

_docs_src/
├── _static/
│   ├── logo.png
│   ├── favicon.ico
│   ├── custom.css
│   └── chartbook-custom.js          # catalog theme only
├── _templates/
│   ├── chart_entry_top.md            # chart page header + iframe
│   ├── chart_entry_bottom.md         # chart specs table + manifests
│   ├── dataframe_entry_top.md        # dataframe page heading
│   ├── dataframe_manifest.md         # dataframe metadata table
│   ├── pipeline_manifest.md          # pipeline metadata table
│   └── layout.html                   # catalog theme only
├── assets/
│   ├── logo.png
│   └── favicon.ico
├── conf.py.j2                        # Sphinx config (Jinja2 template)
├── index.md                          # site index template
├── charts.md                         # charts listing page
├── dataframes.md                     # catalog theme only
├── pipelines.md                      # catalog theme only
├── diagnostics.md                    # catalog theme only
└── contributing.md                   # catalog theme only

Stage 1: Markdown Generation (`_docs/`)#

This is the core of the build. The build_all() function in markdown_generator.py orchestrates all file copying and Markdown generation:

All generated content is namespaced under a cb/ subdirectory, keeping it separate from your custom site pages (which are copied to the docs root) so the two can never collide. In file paths, {pipeline_id} is the scoped ID with / replaced by -- (e.g. acme/yield_curve → acme--yield_curve) so IDs stay filesystem-safe.

1. File alignment (copying source files into position):

Source	Destination in `_docs/`	Purpose
Chart HTML files	`_static/{pipeline_id}/{chart_id}.html`	Displayed in `<iframe>` on chart pages
Chart HTML files	`cb/download_chart/{pipeline_id}/{chart_id}.html`	“Full Screen” download link
Parquet data files	`cb/download_dataframe/{pipeline_id}/{dataframe_id}.parquet`	Data download (when `enable_data_download = true`)
Notebook `.ipynb` files	`cb/notebooks/{pipeline_id}/{notebook_filename}`	Rendered by myst-nb during Sphinx build (copied under the basename of the notebook’s `path`, not renamed to its ID)

2. Diagnostics CSV: A metadata completeness report is generated at _docs/_static/diagnostics/chartbook_metadata_diagnostics.csv. This CSV lists every pipeline, dataframe, and chart with whether its metadata is complete and which fields are missing.

3. Per-pipeline Markdown generation:

For each pipeline defined in your manifest:

Dataframe pages (cb/dataframes/{pipeline_id}/{dataframe_id}.md): Generated by rendering your user-authored dataframe documentation (from the dataframe’s docs_path file, or inline docs) together with a generated heading and Jinja2 template partials for the DataFrame Glimpse, dataframe manifest table, and pipeline manifest table.
Chart pages (cb/charts/{pipeline_id}.{chart_id}.md): Generated by combining chart_entry_top.md (ABlog frontmatter, heading, iframe embed) + your user-authored chart docs (from the chart’s docs_path file, or inline docs) + chart_entry_bottom.md (specs table, manifest tables).
Pipeline READMEs (cb/pipelines/{pipeline_id}_README.md): Catalog theme only. Includes the pipeline’s README content with a manifest header.

4. Top-level pages:

index.md — rendered from the theme’s index template with pipeline/dataframe/chart listings; lives at the docs root
cb/charts.md, and for the catalog theme cb/dataframes.md, cb/pipelines.md, cb/diagnostics.md — listing pages over the generated content

5. Remaining files: All other files from _docs_src/ (CSS, JS, templates, etc.) are copied into _docs/.

Stage 2: Sphinx Build#

Once all Markdown files are in place:

conf.py rendering: The conf.py.j2 Jinja2 template is rendered with your project metadata (name, maintainer, and copyright from [project] in chartbook.toml, plus the Sphinx theme chosen by project type) and written as _docs/conf.py. The .j2 template is then deleted.
Sphinx execution: ChartBook runs:
```
sphinx-build -M html _docs _docs/_build
```
This processes all .md files using extensions like myst-nb (notebook rendering), ablog (blog-style chart pages), sphinx_design (grid cards), and sphinx_copybutton.
Output copy: The HTML output from _docs/_build/html/ is copied to your output directory (default: ./docs). A .nojekyll file is added for GitHub Pages compatibility.

Cleanup#

By default, both _docs/ and _docs_src/ are deleted when the build finishes, whether it succeeded or failed. To keep them — for example, to inspect the intermediate output after a failed build — pass --keep-build-dirs:

chartbook build --keep-build-dirs

The command prints the path to the preserved directory:

Keeping temporary build directory: /path/to/project/_docs

Inspecting Intermediate Files#

Using `--keep-build-dirs`#

The simplest way to inspect intermediate files:

chartbook build --keep-build-dirs

This preserves both _docs/ (generated Markdown + assets) and _docs_src/ (resolved templates) after the build.

Custom Build Directory Paths#

You can control where intermediate files are placed:

chartbook build --docs-build-dir ./my_build --temp-docs-src-dir ./my_src --keep-build-dirs

This is useful if you want to compare outputs from different builds side by side.

Development Workflow#

A typical debugging workflow:

# 1. Build with intermediate files preserved
chartbook build --keep-build-dirs

# 2. Inspect the generated Markdown
cat _docs/cb/charts/acme--yield_curve.repo_rates.md

# 3. Check the rendered Sphinx config
cat _docs/conf.py

# 4. Look at diagnostics
cat _docs/_static/diagnostics/chartbook_metadata_diagnostics.csv

# 5. Browse the final HTML locally
python -m http.server -d ./docs

Complete `_docs/` Directory Structure#

After a build with --keep-build-dirs, the _docs/ directory contains:

_docs/
├── index.md                              # Main site index (rendered from template)
├── conf.py                               # Rendered Sphinx configuration
├── contributing.md                       # catalog theme only
├── *.md                                  # your custom site pages (from site_dir)
│
├── cb/                                   # ── all generated content ──
│   ├── charts.md                         # Charts listing page (toctree)
│   ├── dataframes.md                     # catalog theme only
│   ├── pipelines.md                      # catalog theme only
│   ├── diagnostics.md                    # catalog theme only
│   │
│   ├── pipelines/                        # catalog theme only
│   │   └── {pipeline_id}_README.md       # Pipeline README with manifest header
│   ├── dataframes/
│   │   └── {pipeline_id}/
│   │       └── {dataframe_id}.md         # Dataframe doc with spec table + glimpse
│   ├── charts/
│   │   └── {pipeline_id}.{chart_id}.md   # Chart doc with iframe + metadata
│   ├── notebooks/
│   │   └── {pipeline_id}/
│   │       └── {notebook_filename}       # Copied notebook (keeps its path basename)
│   ├── download_chart/
│   │   └── {pipeline_id}/
│   │       └── {chart_id}.html           # Chart HTML for download link
│   └── download_dataframe/               # only when enable_data_download = true
│       └── {pipeline_id}/
│           └── {dataframe_id}.parquet    # Parquet file for download link
│
├── _static/
│   ├── logo.png
│   ├── favicon.ico
│   ├── custom.css
│   ├── {pipeline_id}/
│   │   └── {chart_id}.html              # Chart HTML for iframe display
│   └── diagnostics/
│       └── chartbook_metadata_diagnostics.csv
│
├── _templates/                           # Jinja2/Sphinx templates
│   ├── chart_entry_top.md
│   ├── chart_entry_bottom.md
│   ├── dataframe_entry_top.md
│   ├── dataframe_manifest.md
│   └── pipeline_manifest.md
│
└── _build/
    └── html/                             # Final Sphinx HTML output
        ├── index.html
        ├── cb/
        └── ...

(Directory names like {pipeline_id} use the slugged form, acme--yield_curve.)

What Each Generated File Contains#

Dataframe pages#

Path: cb/dataframes/{pipeline_id}/{dataframe_id}.md

Each dataframe page is assembled from your user-authored documentation (specified by the dataframe’s docs_path in chartbook.toml, or inline docs) plus auto-generated sections:

# Dataframe: `acme/yield_curve:repo_public` - Public Repo Data

[Your user-authored documentation from docs_path]

## DataFrame Glimpse

  Rows: 16580
  Columns: 31
  $ SVENY01    <f64> 4.216397633330158
  $ SVENY02    <f64> 5.006153097049543
  ...

## Dataframe Manifest

| Dataframe Name  | Public Repo Data                       |
|-----------------|----------------------------------------|
| Dataframe ID    | repo_public                            |
| Data Sources    | FRED, Office of Financial Research     |
| Data Providers  | FRED, Office of Financial Research     |
| ...             | ...                                    |

## Pipeline Manifest

| Pipeline Name   | Yield Curve                            |
|-----------------|----------------------------------------|
| Pipeline ID     | acme/yield_curve                       |
| ...             | ...                                    |

The DataFrame Glimpse is generated by reading a single sample row of the parquet file with Polars — the last row for normal-sized files (and glob/partitioned paths), or the first row for files above the size threshold (50 MB by default, where reading the head avoids a full scan). The total row count shown is always the true count, corrected after sampling. If the file is missing, an error message is shown instead.

Chart pages#

Path: cb/charts/{pipeline_id}.{chart_id}.md

Each chart page includes ABlog frontmatter (for blog-style categorization), an interactive chart iframe, and metadata tables:

---
date: 2026-03-02
tags: FRED, Office of Financial Research
category: Short Term Funding, Repo
---

# Chart: Repo Rates
SOFR, the Effective Funds Rate, and the Fed's Target Range

## Chart
  (iframe embedding the interactive HTML chart)

Full Screen Chart (download link)

[Your user-authored chart documentation from docs_path]

## Chart Specs
| Chart Name     | Repo Rates       |
|----------------|------------------|
| Data Frequency | Daily            |
| Units          | Percent          |
| ...            | ...              |

## Dataframe Manifest
(auto-generated table)

## Pipeline Manifest
(auto-generated table)

Index page#

Path: index.md

The index page differs by theme:

Pipeline theme: Contains toctrees organized by Notebooks, Charts, and Dataframes, plus the pipeline manifest table and README content.
Catalog theme: Contains grid cards linking to Charts, Dataframes, Pipelines, Diagnostics, and Contributing pages.

Diagnostics page (catalog only)#

Path: cb/diagnostics.md

Shows metadata completeness for every object in the catalog. Generated from the diagnostics CSV at _static/diagnostics/chartbook_metadata_diagnostics.csv.

Sphinx configuration#

Path: conf.py

The rendered Sphinx configuration, with the name, maintainer, and copyright values from [project] in chartbook.toml substituted in (the Sphinx theme follows from the project type). Includes all extension settings for myst_nb, ablog, sphinx_design, etc.

Template System and Customization#

Template Resolution Order#

During Markdown generation, Jinja2 searches for templates in the following order (first match wins):

Project root — {project_dir}/
Temp docs source — _docs_src/
Temp docs templates — _docs_src/_templates/
Package templates — chartbook/templates/ (inside the installed package)
Theme directory — chartbook/docs_src_pipeline/ or chartbook/docs_src_catalog/
Theme templates — chartbook/docs_src_{theme}/_templates/

This means you can override any built-in template by placing a file with the same base name in your project root — search path #1, e.g. {project_dir}/chart_entry_bottom.md. Note that your project’s own docs_src/_templates/ directory is not on this list (only the temporary _docs_src/ copy is), so a template placed there has no effect; use the project root.

Overridable Templates#

Template File	What It Controls
`chart_entry_top.md`	Chart page header: ABlog frontmatter, title, iframe embed, sources line, download link
`chart_entry_bottom.md`	Chart page footer: specs table, dataframe manifest, pipeline manifest
`dataframe_manifest.md`	Dataframe metadata table (shown on dataframe and chart pages)
`pipeline_manifest.md`	Pipeline metadata table (shown on multiple page types)
`index.md`	Main site index layout and toctree structure
`conf.py.j2`	Sphinx configuration (advanced — override with care)

Example: Customizing the Chart Footer#

To add a custom disclaimer to all chart pages, create chart_entry_bottom.md in your project root and include your additions:

## Chart Specs

| Chart Name             | {{name}}                          |
|------------------------|-----------------------------------|
| Data Frequency         | {{frequency}}                     |
| Units                  | {{units}}                         |
| HTML Chart             | [HTML](../download_chart/{{pipeline_id | replace('/', '--')}}/{{chart_id}}.html) |

## Dataframe Manifest
{% include "dataframe_manifest.md" %}

## Pipeline Manifest
{% include "pipeline_manifest.md" %}

## Disclaimer
This chart is for informational purposes only.

You can use any Jinja2 template variables that are available in the original template. Inspect _docs_src/_templates/ (with --keep-build-dirs) to see the default templates and available variables.

Debugging Tips#

Chart display issues#

Verify the chart HTML was copied: check that _docs/_static/{pipeline_id}/{chart_id}.html exists
Check the generated Markdown: open _docs/cb/charts/{pipeline_id}.{chart_id}.md and verify the iframe src path
Compare with the download copy: _docs/cb/download_chart/{pipeline_id}/{chart_id}.html

Missing or incorrect data#

Check the DataFrame Glimpse section in _docs/cb/dataframes/{pipeline_id}/{dataframe_id}.md
If the glimpse shows an error, the parquet file may be missing or corrupted
Verify most_recent_data_min and most_recent_data_max values in the dataframe manifest table

Template issues#

Compare _docs_src/_templates/ (the resolved templates) with your project’s docs_src/_templates/ overrides
Check that your override template includes all necessary {% include %} directives
Verify the rendered output in _docs/ matches expectations

Sphinx build errors#

Check _docs/conf.py for configuration issues (e.g., missing extensions, incorrect theme name)
Sphinx error messages reference paths inside _docs/ — use --keep-build-dirs to examine those files
Look at _docs/_build/ for Sphinx build artifacts and logs

Metadata completeness#

Open _docs/_static/diagnostics/chartbook_metadata_diagnostics.csv to see which fields are missing
For catalog projects, check _docs/cb/diagnostics.md for the rendered completeness report

Next Steps#

Documenting a Pipeline — the manifest that drives all of this
CLI Reference — full option reference for chartbook build
Catalogs and Data Access — combine multiple pipelines into a catalog