xberg-io/xberg

# Alef-Generated Bindings

Xberg specifics on top of the shared Alef workflow skill (generic regenerate/verify/commit loop).

- Check `alef.toml` before touching anything in `packages/*/` or a binding crate.
- Regenerate with `alef all --clean --skip-compile --skip-snippet-validation` (after
  `mise run dart:format:context`); it formats via poly. There is no `alef:generate` or `alef:format` task.
- `mise run alef:verify` runs `alef verify --exit-code`.
- Run `alef docs` directly to generate API documentation.
- The `alef:*` namespace is exactly: `verify`, `build`, `verify:readme-paths`, `verify:ffi-formats`,
  `verify:fields`.

## Freshness check

# API Server & MCP Protocol

**Axum server for document extraction, plus the rmcp Model Context Protocol surface**

Locations: `crates/xberg/src/api/` (`router.rs`, `handlers.rs`, `startup.rs`, `types.rs`,
`error.rs`, `jobs.rs`) and `crates/xberg/src/mcp/`. There is no `api/server.rs`.

## REST routes

Registered in `api/router.rs`, all on one `Router`:

| Route | Handler |
| --- | --- |
| `POST /extract` | `extract_handler` — multipart files, URL fields, or JSON; builds `ExtractInput` |

# Benchmark workflow

The benchmark system lives in `tools/benchmark-harness/`; the GitHub workflow is
`.github/workflows/benchmarks.yaml`. The workflow is dispatch-only, so it does not run on push or gate merges. Treat
a result as evidence for its exact commit SHA and inputs, not for newer local work.

## Ground-truth integrity

- Never use Xberg's own extractor output as benchmark ground truth. Use an independent source and record it in the
  fixture's `ground_truth.source` field (`manual`, `vision`, `pdf_text_layer`, `pandoc`, `python-docx`, and similar).
- Before blaming ground truth for a score, render or otherwise inspect the source document. If the derived `.md` or
  `.txt` disagrees with the source, fix the ground truth; if it agrees, investigate the extractor or metric.
- Use the fixture schema in `tools/benchmark-harness/README.md`, the generator at
  `tools/benchmark-harness/scripts/generate_markdown_gt.py`, and the harness `validate-gt` command implemented in

# Chunking & Embeddings

**Text splitting, ONNX/static embedding generation, RAG pipeline integration**

Locations: `crates/xberg/src/chunking/` and `crates/xberg/src/embeddings/` (both directories,
not single files).

## Chunking

`ExtractionConfig.chunking: Option<ChunkingConfig>` drives it. The standalone entry points are
`chunking::chunk_text(text, &ChunkingConfig, page_boundaries) -> Result<ChunkingResult>`
(`chunking/core.rs`) and `chunking::rag::chunk_for_rag(text, &ChunkingConfig)`
(`chunking/rag.rs`), which upgrades `ChunkerType::Text` to `Markdown` and fills each chunk's
`heading_path`.

# Configuration Loading & Precedence

## CLI Mode Precedence (highest to lowest)

1. Individual CLI flags (`--ocr`, `--output-format`, `--chunk`)
2. Inline JSON config (`--config-json` or `--config-json-base64`)
3. Config file (`--config path.toml`)
4. Auto-discovered config (`xberg.{toml,yaml,yml,json}` in cwd/parents, then the user config dir)
5. Default values

## Server/MCP Mode Precedence

1. CLI arguments (`--host`, `--port`)
2. Environment variables (`XBERG_HOST`, `XBERG_PORT`)

# Crate Structure

Version source of truth: root `Cargo.toml` `[workspace.package] version`.

## Workspace members (`crates/`)

- `xberg` — core library: extraction engine, MIME detection, plugin system, OCR, chunking, embeddings, API/MCP server
- `xberg-cli` — CLI binary. Defines no `cli` feature of its own; its `core-cli` feature is what forwards to the core `xberg/cli`
- `xberg-ffi` — C FFI layer; opaque handles, cbindgen headers. Consumed by Go (cgo), C# (P/Invoke), Zig (C ABI), Swift (swift-bridge shim, which injects the dependency itself) and Java (Panama FFM)
- `xberg-jni` — JNI bindings backing `packages/kotlin-android`. The Java package does **not** go through this: `alef.toml [crates.java]` sets `ffi_style = "panama"`, i.e. Panama FFM over `xberg-ffi`
- `xberg-node` — NAPI-RS Node.js/TypeScript bindings
- `xberg-py` — PyO3 Python bindings
- `xberg-php` — ext-php-rs PHP bindings
- `xberg-wasm` — wasm-bindgen WASM bindings; `wasm-target` feature set

# xberg E2E Fixtures

One JSON object per file under `fixtures/<dir>/`. The directory is the default category; an explicit `category` field
overrides it and names the generated test file (e.g. `fixtures/plugin_api/list_reranker_backends.json` sets
`reranker_backend_management`, so its Python test lands in `e2e/python/tests/test_reranker_backend_management.py`).

- Fields: `id` (unique snake_case, becomes the test name), `call`, `description`, `docs` (`topic`, `title`,
  `description`, `side_effects`), `assertions`; optional `category`, `tags`, `input`, `config`, `args`, `skip`.
- `call`: `extract` (most fixtures), `extract_batch`, `pdf_page_count`, `extract_with_external_redaction`, and the
  plugin-registry calls (`list_*`, `register_*`, `unregister_*`, `clear_*` for OCR, embedding, tokenizer, reranker
  backends, renderers, validators, post-processors; `ocr_backend_supports_language`).
- `input`: `extract_input` (`kind: "uri"` with `$mock_url/...`, or `bytes` + `filename` + `mime_type`) plus
  `mock_responses` (`path`, `status_code`, `headers`, optional `method`, and a body: `body_file` resolved against
  a `responses` subdirectory of `fixtures/` (none exists yet), then `fixtures/` itself, so `../test_documents/...` is

# Extraction Pipeline Patterns

**Format detection → extractor routing → post-processing over the `FORMATS` registry**

The full-registry counts are verified against published claims by
`scripts/sync_supported_counts.py verify`. Runtime `SUPPORTED_FORMAT_COUNT` and
`SUPPORTED_EXTENSION_COUNT` values are derived from the full static `FORMATS` registry.

## Layout

- `crates/xberg/src/core/pipeline/` — orchestration (`mod.rs`, `cache.rs`, `execution.rs`,
  `features.rs`, `format.rs`, `initialization.rs`, `page_markers.rs`)
- `crates/xberg/src/core/mime.rs`, `core/formats.rs` — detection and the `FORMATS` registry
- `crates/xberg/src/extractors/` — one module per format, each implementing

# Feature-flag conventions

Shared concerns use the **same flag name with the same semantics** in every library. A consumer that learns `otel` or `mcp` once should get identical behavior across every library and its bindings. Repo-specific flags stay local — this contract governs only shared concerns.

## Standard flags

| Flag | Meaning | Default | Notes |
|------|---------|---------|-------|
| *(none)* `tracing` | Structured tracing is **always-on**, never a feature. | — | `tracing` is a non-optional dependency. Do not gate span emission. Un-gate any legacy optional `tracing` feature. |
| `otel` | Opt-in OpenTelemetry / OTLP export layer, on top of always-on tracing. | off | Pulls `tracing-opentelemetry` + the `opentelemetry*` stack. This is the only observability feature. Reconcile any always-on OTLP deps behind it. |
| `mcp` | Embedded Model Context Protocol server (stdio transport). | off | |
| `mcp-http` | MCP Streamable HTTP transport; implies `mcp`. | off | `mcp-http = ["mcp", ...]`. |
| `mimalloc` | Select the mimalloc global allocator. | off | Mutually exclusive with `jemalloc`; see the allocator rule below. |
| `jemalloc` | Select the jemalloc global allocator. | off | Mutually exclusive with `mimalloc`. |

# Format-Specific Extraction Workflows

## Office XML (DOCX/PPTX/ODT)

```text
ZIP archive → SecurityBudget → XML parsing → Text + tables + metadata
```

1. `let budget = SecurityBudget::from_config(config);` (`extractors/security.rs`), plus
   `config.security_limits…max_files_in_archive` as the member cap. The Office path does
   **not** use `ZipBombValidator` — that is the archive/iWork/HWPX path.
2. Open with `zip::ZipArchive::new(cursor)` and read the parts
   (`word/document.xml`, `ppt/slides/*.xml`, `content.xml`).
3. Parse with `quick-xml::Reader` (streaming), threading `&mut budget` through the recursive

# MIME Detection & Routing

## Detection Flow

```text
Policy -> content and/or extension evidence -> validate_mime_type -> registry.get(mime) -> extractor
```

## Key Functions

| Function | Location | Behaviour |
| --- | --- | --- |
| `detect_mime_type(path, check_exists: bool)` | `core/mime.rs` | **Path-based only** — never reads bytes. Lowercased extension → `EXT_TO_MIME`, then tree-sitter extension detection (feature `tree-sitter`), then `mime_guess::from_path`. `check_exists` gates a file-existence check, not content inspection. |
| `detect_mime_type_from_bytes(bytes)` | `core/mime.rs` | Magic-number detection via the `infer` crate. The only content-sniffing entry point. |

# OCR pipeline and quality

OCR flows through preprocessing, backend execution, structured conversion, page acceptance, and caching. Backend
outputs are not interchangeable measurements.

## Backends and execution

- All backends implement `OcrBackend`. Tesseract is the default and is bound through the in-repo
  `crates/xberg-tesseract` C FFI crate; there is no `leptess` dependency.
- `OcrBackendType` is `Tesseract | PaddleOCR | Candle | Custom`. Sceptre is selected by name through `Custom`, not a
  dedicated enum variant.
- Run blocking OCR work through `tokio::task::spawn_blocking`, minimize runtime/FFI lock duration, and respect backend
  resource limits.
- Check `PageOrientationHandling` before assuming a backend handles rotated input.

# PDF backends

`PdfConfig.backend` selects `PdfBackend::Native` (default) or `PdfBackend::Pdfium`. Backend choice is explicit: never
silently return native output when Pdfium was requested.

## Native backend

- The native engine is `xberg_native_pdf` and supports the full Xberg PDF pipeline: text and structure extraction,
  page rendering/OCR fallback, tables, annotations, images, attachments, and richer metadata.
- `config.force_ocr` forces the whole document; `config.force_ocr_pages` selects pages. Otherwise, pages without usable
  native text can enter the OCR stage inside `PdfExtractor`; OCR is not a second PDF extractor in the registry.
- `PdfRenderSession` in `pdf/render.rs` opens document bytes once and exposes page count and page rendering. Use it for
  repeated page renders instead of reopening the same document for each page. It is a Rust primitive and need not be
  exposed by every binding.

# Plugin Architecture & Registration

## Plugin Types

| Type | Trait | Location |
| --- | --- | --- |
| Document extractor (binding-facing) | `DocumentExtractor: Plugin` | `plugins/extractor/trait.rs` |
| Document extractor (in-crate) | `InternalDocumentExtractor: Plugin` | `plugins/extractor/trait.rs` |
| OCR backend | `OcrBackend: Plugin` | `plugins/ocr.rs` (a file, not a directory) |
| Post processor | `PostProcessor: Plugin` | `plugins/processor/trait.rs` |
| Validator | `Validator: Plugin` | `plugins/validator/trait.rs` |
| Embedding backend | `EmbeddingBackend: Plugin` | `plugins/embedding.rs` |
| Reranker backend | `RerankerBackend: Plugin` | `plugins/reranker.rs` |
| Tokenizer backend | `TokenizerBackend: Plugin` | `plugins/tokenizer.rs` |

# Polyrepo boundaries

Each sibling under the workspace root is an independent Git repository with its own history, release, CI, and
governance. Pushing Xberg does not publish a sibling dependency.

## Ownership

- `xberg`: reusable Rust document-intelligence primitives, public Rust API, CLI/server surfaces, bindings, and Xberg
  CI/release packaging.
- `xberg-enterprise`: product and deployment behavior that consumes Xberg. Check it before removing or reshaping Rust
  primitives, even when the symbol is not exposed in language bindings.
- `alef`: binding, documentation, snippet, and e2e generation defects. Work around a generator defect in Xberg only
  when necessary; report and fix the reusable cause in Alef.
- `liter-llm`: reusable LLM client behavior, provider request construction, endpoint validation, and network-security

# Test corpus

`test_documents` is a git submodule (`.gitmodules` → `xberg-io/test_documents`). Its binary
fixtures (about 1000 objects, ~0.6 GiB; see `test_documents/corpus.lock.json`) are **not committed**.
They are materialised from a public GCS bucket over anonymous HTTPS, with no credentials and no
SDK.

```bash
python3 test_documents/scripts/fetch_corpus.py                   # everything
python3 test_documents/scripts/fetch_corpus.py --include 'pdf/**'
```

Fetched files are gitignored inside the submodule, so neither the submodule nor the
superproject goes dirty. CI does the same thing with `xberg-io/actions/fetch-test-documents@v1`,

# WASM Build Constraints

## Overview

WASM target lives in `crates/xberg-wasm/`, built with wasm-bindgen over sync-only internal
APIs. Note that `crates/xberg-wasm/src/lib.rs` is Alef-generated — do not hand-edit it.

## Feature Flags

```toml
# crates/xberg/Cargo.toml
wasm-target = [
    "no-ort-target",
    "excel-wasm",

# Xberg release

Repo specifics for the shared `release-workflow` skill: how the version propagates, and the readiness gate before a
push or release.

## Versioning

### Single source of truth

The root `Cargo.toml` `[workspace.package] version` is the one authoritative version (including any
`-rc.N` pre-release suffix). Everything else is derived from it — never hand-edit
a version in a package manifest.

### `mise run version:sync` runs three steps

# Xberg TypeScript toolchain

These repository-specific facts override generic TypeScript conventions where they differ.

- Lint and format with `poly lint .` and `poly fmt .`. Poly embeds oxlint and oxfmt; do not install or invoke oxlint,
  oxfmt, Prettier, ESLint, or Biome directly, and do not add per-package lint scripts.
- Root `tsconfig.json` enables strict checking options and is a project-reference root for `e2e/node` and `e2e/wasm`.
  Confirm that the package being changed is included before treating a root typecheck as evidence.
- Testing uses Vitest with `@vitest/coverage-v8`. No coverage threshold is enforced unless one is added to current
  configuration.
- No runtime schema-validator dependency is guaranteed. Use existing boundary guards or propose a dependency
  explicitly rather than assuming Zod is available.
- The pnpm workspace uses the root lockfile. Packages under `integrations/node/` are outside that workspace and use
  their own npm lockfiles. Determine ownership before installing or updating dependencies.

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents
- Performs OCR on scanned documents or images
- Batch-processes multiple files
- Configures extraction options (output format, chunking, OCR, language detection)
- Implements custom plugins (post-processors, validators, OCR backends)

> If the `xberg` MCP server is registered in this session, prefer its tools over shelling out to the CLI — they expose the same extraction surface with structured arguments and results.

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

## Basic usage

```bash
# Glob expands to many paths; results come back in a JSON envelope (default)
xberg batch *.pdf

# Mixed formats, markdown content for LLM ingestion

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

## Inline during extraction

Turn on chunking with `--chunk` and the chunks appear on the structured
result under `chunks`:

```bash

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

## Keywords (YAKE / RAKE)

Keyword extraction is configured via the `[keywords]` config block (or
inline JSON) — there is no single `--keywords` CLI flag. When enabled,
extracted keywords appear on `result.extracted_keywords` (`extractedKeywords`
in Node.js; the CLI JSON field is `extracted_keywords`). Two algorithms are
available:

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

## Basic usage

```bash
# Markdown tables embedded in the content stream
xberg extract report.pdf --layout --content-format markdown

# Structured JSON output, tables appear under result.tables

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

## When to force OCR

- Extraction returned an empty `content` field, but the file opens visually.
- The PDF text layer is junk (copy-paste from a viewer produces gibberish).
- You want consistent output across mixed scanned + digital PDFs.

```bash

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |
| `--format`          | How the CLI prints the result                     | `text`, `json`, `toon`                 | `text` (`extract`), `json` (`batch`) |
| `--content-format`  | How extracted content is rendered inside `result` | `plain`, `markdown`, `djot`, `html`, `json`, `doctags` | `plain`     |
| `--token-reduction` | Strip whitespace / boilerplate for LLM contexts   | `off`, `light`, `moderate`, `aggressive`, `maximum` | `off`  |

`--format json` returns an envelope wrapping the `ExtractedDocument` — the
document lives under `.result` for `extract` and under `.results[]` for
`batch`, with `content`, `metadata`, `tables`, and `images` as fields of

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:b1718303d659167ff376b4e1d4baf546d3e74f8f0804698a3486c0a1d2e6c913
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:b1718303d659167ff376b4e1d4baf546d3e74f8f0804698a3486c0a1d2e6c913
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:b1718303d659167ff376b4e1d4baf546d3e74f8f0804698a3486c0a1d2e6c913
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:b1718303d659167ff376b4e1d4baf546d3e74f8f0804698a3486c0a1d2e6c913
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:b1718303d659167ff376b4e1d4baf546d3e74f8f0804698a3486c0a1d2e6c913
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting with OCR

Use this when a document is image-based: scanned PDFs, photographed pages,
screenshots, JPEG/PNG/TIFF with text. Xberg auto-OCRs raster images and
auto-detects PDFs that lack a text layer. Force it on when extraction
returned empty/garbled text from a PDF that "looks" textual.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:a2f204c34c2eb303812f0b7b0a9acf29c879edd1985b8cc3312555b347378e1e
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Picking a format

Xberg has two orthogonal format knobs. Get them right up front and the
downstream code stays simple.

| Knob                | What it controls                                  | Values                                 | Default          |
| ------------------- | ------------------------------------------------- | -------------------------------------- | ---------------- |

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:5c39696cf46f9dabada4de547cf7b610394f57c4002ed5d2055fc83521fa3141
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Xberg Document Extraction

Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats.

Use this skill when writing code that:

- Extracts text or metadata from documents

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:324433f2b10c7c9cdc1c06b98c1e427c8cb0ca9ccf034d336e28d49706add520
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Batch extraction

Use this when processing a directory or glob of documents in one pass.
`xberg batch` shares one extraction config across every file, runs
extractions concurrently, and returns a JSON envelope — failures on
individual files do not abort the run.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:aaaffebaf030b1c214ffc0a869f0a63366360a5a568638f6996b98e75901d121
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's `chunks` field), or standalone via the `chunk` command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:491d3cbcd4f6128d43e32b39c13d03a023b397f97c71da48ed5a02d6be00ae28
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated `embed` command.

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:80d26904ff70da39df4ee2b35bcfc06a186b30c0d2566cf4b60832cd42e475ef
Source-Hash: blake3:a6076da32e6bfe3750e87d00b1c57c1dafffe1958ab6f8b6768e4a94fae3b20a
Schema-Version: v1
-->

# Extracting tables

Use this when the user wants structured tabular data — financial
statements, scientific tables, invoices, spreadsheet-style PDFs. Xberg
detects tables via a layout model (RT-DETR v2) and reconstructs cell
structure with a configurable table model.