Skip to content

Python API

中文

The package root intentionally exports only Lumberjack and Document. Import other public components from their dedicated modules.

Orchestration

Lumberjack

Orchestrate the complete document-to-chunks pipeline.

saw 

saw(
    document,
    *,
    format="auto",
    document_title=None,
    metadata_overrides=None,
    source_path=None,
)

Parse, split, and finalize one raw document into final chunks.

saw_many 

saw_many(inputs, *, fail_fast=False)

Stream per-document outcomes without preloading inputs into memory.

Results preserve input order. Components are reused sequentially; this is deliberate because optional tokenizer implementations are not thread-safe. Set fail_fast to re-raise the first document failure.

trace 

trace(
    document,
    *,
    format="auto",
    document_title=None,
    metadata_overrides=None,
    source_path=None,
)

Run one document and return explicit parser, splitter, and finalizer views.

Models

models

Document dataclass

Raw document and provenance waiting to be parsed into a structured document.

DocTree dataclass

Parsed document with a normalized section tree, source, and metadata.

Attributes:

Name Type Description
title str

Document title. Priority: user-provided document_title, then front matter title field, then first level-1 heading, then "Anonymous".
source str

Original text for Markdown and HTML inputs. Binary parsers may leave this empty or provide a normalized textual representation.
root SectionNode

Root section node of the heading tree.
source_path str | None

Original file path or caller-supplied source provenance.
metadata dict[str, Any]

Semantic document metadata parsed from the source and merged with caller-provided overrides.
reference_definitions dict[str, dict[str, str]]

Link/image reference definitions ([label]: url).
topology DocumentTopology

Structural semantics of the root. "records" is a flat, ordered sequence of atomic records rather than a heading tree.

DocumentBlock dataclass

Format-neutral block node in the canonical rendered representation.

Attributes:

Name Type Description
kind BlockKind | str

Block type (e.g. "paragraph", "heading", "code_fence", "blockquote", "list", "list_item", "table", "html_block", "math_block").
text str

Canonical rendered text consumed by splitters. Markdown input is normalized Markdown; HTML and DOCX input is converted to the same Markdown-like representation. It is not guaranteed to be a source slice.
start_line int | None

1-based line number where the block begins.
end_line int | None

1-based line number where the block ends.
children tuple[DocumentBlock, ...]

Nested child blocks for container types ("blockquote", "list", "list_item"). Empty for leaf blocks.
inlines tuple[DocumentInline, ...]

Normalized inline nodes parsed from the block content. Populated for "paragraph" and "heading" blocks.
attrs dict[str, Any]

Additional attributes (e.g. heading level, list style, code language).

DocumentInline dataclass

Format-neutral inline node normalized by an input parser.

Attributes:

Name Type Description
kind str

Inline node type (e.g. "text", "link", "code_inline", "strong", "em", "image").
text str

Rendered plain text of this node.
children tuple[DocumentInline, ...]

Nested inline children (e.g. emphasis wrapping text).
attrs dict[str, Any]

Additional attributes ("href", "src", "title", etc.).

SectionNode dataclass

Heading-tree node representing a section and its children.

Attributes:

Name Type Description
level int

Heading level.
title str

Plain-text heading title.
path HeadingPath

Tuple of (level, title) pairs from root to this section.
blocks list[DocumentBlock]

Block-level content directly under this section (not in sub-sections).
children list[SectionNode]

Child sections (sub-headings nested within this section).
index int

Position of this section among its siblings (0-based).
start_line int | None

1-based line number where the section heading begins.
title_inlines tuple[DocumentInline, ...]

Normalized inline nodes parsed from the heading text.

add_block 

add_block(block)

Append a block (roughly one paragraph) to this section.

ChunkDraft dataclass

Intermediate split result holding grouped entries and a token estimate.

Args: entries: List of entries to be merged into the chunk, with heading context and body. headings: Full external heading path excluded from the rendered body. own_heading: Optional final heading identifying the chunk itself.

       ``# H1
H2.1

Content1``, headings=[(1, "H1"), (2, "H2.1")].

       ``# H1
H2.1

Content1 ## H2.2

Content2``, headings=[(1, "H1")].

   headings_token_count: The token count for the chunk's full heading path.
   body_token_count: The token count for the chunk body (sum of entry body_token_count plus separator deltas).
   token_count: Split-time sum of heading tokens, the external-heading
       separator, and body tokens.
   split_origin: The split operation that produced this draft, for debugging/analysis.
   chunk_type: The draft content type (e.g. "paragraph", "code_block"), used for metadata.

Chunk dataclass

Final chunk payload with separated heading/body content and token counts.

Attributes:
    chunk_id: Unique identifier for this chunk.
    chunk_type: Origin block type (e.g. ``"paragraph"``, ``"heading"``,
        ``"code_fence"``, ``"document"``).
    body: Rendered chunk body. Ancestor and own headings are never rendered
        here; headings needed to represent merged internal sections remain.
    token_count: Sum of heading tokens, ``tokenizer.count("

"), and body tokens. estimated_token_count: Split-time running estimate (additive + separator-delta window).token_countis the authoritative final total. The two may differ slightly for incremental splitters due to join approximations. headings_token_count: Token count of the canonical Markdown rendering of the complete heading path. body_token_count: Token count ofbody. ancestor_headings: Tuple of(level, title)pairs representing the chunk's ancestor heading path. own_heading: The chunk's own(level, title)heading, orNone`` when the chunk represents multiple merged sibling sections.

    section_level: Deepest heading level in this chunk.

        ``section_level`` is derived from the full section paths covered by
        the chunk, not from the ancestor-only ``ancestor_headings`` metadata.

    document_title: Title of the source document.
    document_path: File path of the source document, if split from a file.
    start_line: 1-based line number where this chunk begins in the source.
    end_line: 1-based line number where this chunk ends in the source.

SplitResult dataclass

Document-level result of one complete splitting pipeline run.

Built-in components

parser

Public parsers and automatic parser selection.

AutoParser

Select a built-in parser from document provenance or content.

DelimitedTextParser

Parse CSV or TSV as atomic rows, keeping header schema in row metadata.

DocTreeBuilder

Incrementally construct a validated, format-neutral document tree.

The builder deliberately permits blocks directly on the root. Parsers for flat records (CSV, JSONL, logs) must use that form instead of inventing a heading hierarchy solely to satisfy a section-oriented API.

current_section property 

current_section

The section receiving subsequent blocks.

add_block 

add_block(
    kind,
    text,
    *,
    locations=(),
    children=(),
    inlines=(),
    attrs=None,
)

Add one canonical rendered block to the current section.

add_field_value 

add_field_value(field, value, *, locations=(), attrs=None)

Add one scalar field/value unit without turning its key into a heading.

add_record 

add_record(text, *, kind='record', locations, attrs=None)

Add one ordered, atomic record with explicit record provenance.

add_section 

add_section(level, title, *, location=None, inlines=())

Add a heading section and select it as the current section.

build 

build()

Return the completed tree after validating every section path.

declare_block_kind 

declare_block_kind(kind)

Declare an extension block kind before emitting it.

DocxParser

Bases: ParserProtocol

Parse DOCX documents into DocTree.

Maps DOCX structural elements to the same DocTree model used by the Markdown parser, enabling reuse of all existing splitters.

Block kind mapping
  • OOXML outline levels → SectionNode hierarchy
  • Normal paragraphs → paragraph
  • Tables → table
  • OOXML numbering → list with list_item children

block_kinds property 

block_kinds

Block kinds this parser can produce.

parse 

parse(
    document,
    *,
    document_title=None,
    metadata_overrides=None,
    source_path=None,
)

Parse DOCX binary data into a DocTree.

Parameters:

Name Type Description Default
document Document | bytes

A Document or raw DOCX file content.

 required
document_title str | None

Optional override for the document title.

 None
metadata_overrides dict[str, object] | None

Semantic metadata that overrides DOCX core properties.

 None
source_path str | Path | None

Optional source provenance stored separately from metadata.

 None

HTMLParser

Bases: ParserProtocol

Parse HTML documents into the shared DocTree model.

It mirrors the public parser shape used by Markdown and DOCX: it exposes block_kinds and returns a heading-tree DocTree so the existing splitters can operate on HTML input without a separate path.

block_kinds property 

block_kinds

Block kinds this parser can produce.

parse 

parse(
    document,
    *,
    document_title=None,
    metadata_overrides=None,
    source_path=None,
)

Parse raw HTML text into a DocTree.

Parameters:

Name Type Description Default
document Document | str

A Document or raw HTML source.

 required
document_title str | None

Optional override for the document title.

 None
metadata_overrides dict[str, object] | None

Semantic metadata that overrides values parsed from HTML metadata tags.

 None
source_path str | Path | None

Optional source provenance stored separately from metadata.

 None

Raises:

Type Description
TypeError

If the resolved document source is not a str.

JSONLinesParser

Parse JSON Lines as one canonical JSON value per atomic record.

JSONParser

Bases: _StructuredDataParser

Parse JSON into scalar records with JSON-path provenance.

LogParser

Parse each non-empty log line as one atomic, ordered record.

MarkdownBlockContext dataclass

Context passed to custom Markdown block handlers.

MarkdownBlockSpec dataclass

Declare how MarkdownIt token types map to lumberjack block kinds.

MarkdownItParser

Bases: ParserProtocol

Parse Markdown with markdown-it-py and normalize tokens into lumberjack's document model.

block_kinds property 

block_kinds

Block kinds this parser instance can produce, based on active rules.

find_matching_close 

find_matching_close(tokens, index)

Return the matching close-token index for custom block handlers.

parse 

parse(
    document,
    *,
    document_title=None,
    metadata_overrides=None,
    source_path=None,
)

Parse raw Markdown text into a DocTree with section tree and reference definitions.

Parameters:

Name Type Description Default
document Document | str

A Document or raw Markdown text to parse.

 required
document_title str | None

Optional override for the document title.

 None
metadata_overrides dict[str, object] | None

Semantic metadata that overrides values parsed from front matter.

 None
source_path str | Path | None

Optional source provenance stored separately from metadata.

 None

Raises:

Type Description
TypeError

If the resolved document source is not a str.

parse_child_blocks 

parse_child_blocks(tokens, start, end, source_lines)

Parse child block tokens for custom block handlers.

NotebookParser

Parse Jupyter cells as ordered, language-aware atomic records.

SQLParser

Parse semicolon-delimited SQL statements as ordered atomic records.

SQLiteParser

Parse SQLite table rows as atomic records.

Byte inputs are loaded into an in-memory database either through sqlite3.Connection.deserialize (Python 3.11+) or a temporary-file extraction (Python 3.10), so every supported runtime accepts the same SQLite file format.

SourceCodeParser

Parse source files into symbol-aware records without treating code as prose.

When the code-parsing extra is installed, Tree-sitter selects top-level declarations and retains valid results from malformed source. Otherwise, Python uses ast and JavaScript/TypeScript use the built-in fallback.

TOMLParser

Bases: _StructuredDataParser

Parse TOML into scalar records with key-path provenance.

TextParser

Parse plain text as root-level paragraph or line blocks.

XMLParser

Parse XML leaf elements as ordered records with element-path provenance.

XlsxParser

Parse each non-empty XLSX row as an atomic, sheet-aware record.

Requires pip install lumberjack-py[spreadsheets].

YAMLParser

Bases: _StructuredDataParser

Parse YAML into scalar records with key-path provenance.

splitter

Public structure-aware splitters.

ExactSectionSplitter

Bases: ExactCountingMixin, SectionTopologyMixin

Per-heading section splitter without subtree-collapse.

Emits one draft per heading section's direct body and recurses into children. This variant:

  1. Never collapses an entire subtree into a single draft (no subtree-collapse short-circuit — see :class:ExactSubtreeSplitter for that topology).
  2. Merges only adjacent same-heading paragraph tails, bottom-up, according to merge_below_ratio. Non-text block drafts remain isolated.

Oversized section bodies are still split by token budget respecting block_options (standalone isolation, splittable kinds, per-block budgets). Every budget decision fully recounts the rendered candidate text.

Publicly exposed as ExactSectionSplitter and selected by the exact-section CLI/Web integration name. Works with any tokenizer.

ExactSubtreeSplitter

Bases: ExactCountingMixin, SubtreeTopologyMixin

Subtree-first splitter using exact rendered-text budget decisions.

RecordSplitter

Bases: BaseSplitter

Pack complete root-level records while preserving their input order.

A record is atomic: when it alone exceeds the budget, the emitted draft is marked protected instead of splitting its text or claiming compliance.

Counting is exact but cache-free: like the exact splitters, identical strings within one split (the emitted body re-counted after packing) are deduplicated by a per-split memo instead of the tokenizer cache. The part-sum join bound assumes the join-counting property documented on TokenizerProtocol.

tokenizer

ApproxByteTokenizer

Bases: TokenizerProtocol

Approximate tokenizer using len(text.encode("utf-8")) // 3 tokens.

Assumes an average of 3 UTF-8 bytes per token, which is a better fit for mixed ASCII / CJK text than the older chars // 4 heuristic.

TiktokenTokenizer

Bases: TokenizerProtocol

Tokenizer backed by the tiktoken library.

with_fresh_cache 

with_fresh_cache()

Share the encoding backend while allocating a new request-local cache.

TransformersTokenizer

Bases: TokenizerProtocol

Tokenizer backed by a Hugging Face fast tokenizer.

with_fresh_cache 

with_fresh_cache()

Share the model backend while allocating a new request-local cache.

block

Public block kinds and per-block splitting configuration.

BlockConfig dataclass

Splitting behavior for one built-in block kind.

BlockKind

Bases: StrEnum

Built-in block kinds emitted by lumberjack parsers.

CustomBlockConfig dataclass

Sawing behavior for a parser plugin's custom block kind.

HTMLTableConfig dataclass

Splitting behavior for HTML tables.

MarkdownTableConfig dataclass

Splitting behavior for Markdown pipe tables.

default_block_config 

default_block_config(kind)

Return the default config for a built-in or custom block kind.

normalize_block_options 

normalize_block_options(options)

Validate block configs and index them by normalized kind.

finalizer

ChunkFinalizer

Normalize, transform, measure, and finish drafts into chunks.

render_draft_body 

render_draft_body(entries, external_headings)

Render draft entries while keeping external headings as metadata.

normalizer

TextNormalizer

Conservatively stabilize text without removing document markup.

transformer

PlainTextTransformer

Bases: TextTransformer

Remove common Markdown and HTML surface syntax while retaining readable text.

TextTransformer

Normalize line endings and block spacing while retaining markup.

Extension protocols

protocols

ExtractionParserProtocol

Bases: Protocol

Optional parser capability for exposing an upstream extraction view.

ParserProtocol

Bases: Protocol

Parse a raw document into the shared structured representation.

SplitterProtocol

Bases: Protocol

Split a structured document into unfinished drafts.

TextNormalizerProtocol

Bases: Protocol

Stabilize rendered draft text before transformation.

TextTransformerProtocol

Bases: Protocol

Normalize or simplify text before final chunk creation.

TokenizerProtocol

Bases: Protocol

Measure text units used by splitters and finalizers.

Implementations used with the exact splitters (Exact*Splitter, RecordSplitter) should satisfy the join-counting property: for the separators used when rendering chunks ("\n\n" and "\n"),

``count(a + separator + b) >= count(a) + count(b) - 2``

— joining two texts cannot save more than two tokens per join. The built-in tokenizers satisfy this property, and the exact splitters rely on it to skip provably overflowing candidates without encoding them. A custom tokenizer that collapses joined text (for example, one that maps any string containing a blank line to a near-constant count) violates the contract: exact-mode pruning may then reject a join that a full recount would accept, producing smaller chunks than unbounded counting would. Chunk content is never lost either way.