chartbook.index

The index: one JSON description of everything a build contains.

chartbook docs generate produces a website and, beside it, index.json: the resolved, composed view of the manifests, as a flat list of self-describing records. Every pipeline and every asset is in it once, with a fully qualified ID, a content hash, and what it was made from. A renderer, a search box, an exporter or a diff of two builds can read that one file and need nothing else.

The format is specified in docs_src/design/index-schema.md. This module writes Catalog, Pipeline, Dataframe, Series, Chart, Notebook, Note and Model records, and each pipeline’s declared dependencies, with what chartbook.lock pins of them. Build records come with the part of the asset graph that produces them.

Two forms of the same index are written:

• the full index, in the build directory, in which each location also has the absolute resolved_path on this machine;

• the published index, beside the HTML, with every machine-specific path removed (:func:published), since it goes wherever the site goes.

::

from chartbook import index
from chartbook.manifest import load_manifest

idx = index.build_index(load_manifest("."))
index.write_index(index.published(idx), "docs/index.json")

The index schema is singular by design: there is one writer, this one.

Module Contents

Functions

apply_code_times	Set each Pipeline record’s code_changed_at from the manifest’s code times.
build_index	Build the index for a loaded manifest, pipeline or catalog.
code_changed_at	When the pipeline’s code last changed, as a docs build worked it out, in UTC.
dtype_name	Name a column’s type the way the index writes it.
pipeline_records	The records for one pipeline: itself, then its assets.
published	The index as it may leave this machine: no local absolute paths.
record_kind	The record kind for an asset kind: chart is recorded as Chart.
schema_path	The JSON Schema file for a record kind, or for the envelope.
write_index	Write an index as JSON.

Data

ASSET_KINDS
INDEX_FILE
RECORD_VERSION
SCHEMA
SCHEMAS_DIR

API

chartbook.index.ASSET_KINDS

(‘dataframe’, ‘chart’, ‘notebook’, ‘note’, ‘model’)

chartbook.index.INDEX_FILE

‘index.json’

chartbook.index.RECORD_VERSION

‘chartbook/v1’

chartbook.index.SCHEMA

‘chartbook/index/v1’

chartbook.index.SCHEMAS_DIR

None

chartbook.index.apply_code_times(index: dict, manifest: dict) → dict

Set each Pipeline record’s code_changed_at from the manifest’s code times.

For a build that works the times out after the index is made, as an incremental build does once it knows which pipelines it reuses.

chartbook.index.build_index(manifest: dict, *, cache: Optional[chartbook.hashing.HashCache] = None, skip_pipelines: Iterable[str] = (), generated_at: Optional[datetime.datetime] = None) → dict

Build the index for a loaded manifest, pipeline or catalog.

Same manifests and same files give the same index, apart from generated_at: records are in a fixed order and so are their keys. A catalog’s index also lists the snapshots in this machine’s store, as a model’s versions are read from it.

A declared asset whose file does not exist still has a record; it has no content_hash. The index says what the manifest declares, and chartbook check is what complains about missing files.

Parameters:

• manifest – The manifest, from :func:chartbook.manifest.load_manifest.

• cache – A hash cache, so unchanged files are not read again.

• skip_pipelines – Pipelines left out of this build altogether.

• generated_at – The time to record; defaults to now.

Returns:

The full index, with resolved_path in each location. Pass it through :func:published before it leaves this machine.

chartbook.index.code_changed_at(pipeline_manifest: dict) → Optional[str]

When the pipeline’s code last changed, as a docs build worked it out, in UTC.

Present only when :func:chartbook.manifest.apply_code_change_times has run on the manifest, which a docs build does once per pipeline and remembers across incremental builds: the last commit that touched the source where it is committed and unmodified, else the newest modification time. An index built from a manifest as loaded says nothing here, rather than passing off a file’s modification time as the code’s history. The index asks git for nothing itself.

chartbook.index.dtype_name(dtype: Any) → str

Name a column’s type the way the index writes it.

The name is the Polars type, lower-cased, with its parameters in square brackets: float64, int32, string, boolean, date, datetime[ns], datetime[us, UTC], duration[ms], decimal[10,2], list[int64], categorical, struct. Polars is what data.load() returns, so this is the type a reader gets.

chartbook.index.pipeline_records(pipeline_id: str, pipeline_manifest: dict, cache: Optional[chartbook.hashing.HashCache] = None, known_pipelines: Optional[Mapping[str, pathlib.Path]] = None) → list[dict]

The records for one pipeline: itself, then its assets.

Assets come in the order dataframes, charts, notebooks, notes, models, each by name, and each dataframe is followed by its series.

Parameters:

• pipeline_id – The pipeline’s ID in this build (a catalog’s key).

• pipeline_manifest – Its loaded manifest.

• cache – A hash cache, so unchanged files are not read again.

• known_pipelines – Every pipeline in this build, each ID and its directory (:func:chartbook.manifest.get_pipeline_dirs), so that another one of them is written by the ID it has here.

chartbook.index.published(index: dict) → dict

The index as it may leave this machine: no local absolute paths.

Drops resolved_path from every asset’s location, and the value of a pipeline’s location when it is a local directory, which goes for each of its dependencies too. Everything else, hashes included, is unchanged.

chartbook.index.record_kind(kind: str) → str

The record kind for an asset kind: chart is recorded as Chart.

chartbook.index.schema_path(name: str) → pathlib.Path

The JSON Schema file for a record kind, or for the envelope.

Parameters:

name – A record kind ("Dataframe"), or "index" for the file’s envelope.

chartbook.index.write_index(index: dict, path: Union[str, pathlib.Path], only_if_changed: bool = False) → pathlib.Path

Write an index as JSON.

Parameters:

only_if_changed – If True, and the file already holds this index in everything but generated_at, leave the file as it is. A build that finds nothing different then touches nothing, and generated_at stays the time the index last changed.

Returns:

The path of the file.