---
title: "hbo-text-tokenization — Mova code model"
description: "Mova code model of tools/hbo/src/text.rs: components with their labels, obernants and the model text."
url: "https://groupants.elamur.ai/free/en/mova/tools/hbo/src/text.rs.groupant/tokenization.groupant/"
---
Elamur: [Mova](https://md.mova.elamur.ai/free/en/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · **Groupants**

This page as HTML: <https://groupants.elamur.ai/free/en/mova/tools/hbo/src/text.rs.groupant/tokenization.groupant/>

# hbo-text-tokenization

The groupant beside tools/hbo/src/text.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

## Facts

- **kind:** groupant_fragment
- **status:** partial source bound model
- **source:** [`tools/hbo/src/text.rs`](https://github.com/ElamurAI/mova-free/blob/main/tools/hbo/src/text.rs)
- **source SHA-256:** `a123d548879eaa25…`
- **covers:** NFC then chunk scan emits maqaf-left and punctuation tokens; second pass closes sentences and consumes paragraph markers. (bytes 3755–5287)
- **part of:** [hbo-text](https://md.groupants.elamur.ai/free/en/mova/tools/hbo/src/text.rs.groupant/)
- **repository folder:** [`tools/hbo/src/text.rs.groupant/tokenization.groupant`](https://github.com/ElamurAI/mova-free/tree/main/tools/hbo/src/text.rs.groupant/tokenization.groupant)
- **repository commit:** `855da7bd4a7b`

> **Implementation material.** A Mova code model describes this code at one source revision: partial models, finite checks and their stated scope. It is not a reviewed general exposition, and its labels are shown as written, never raised.

## Components

| component | label | where |
|---|---|---|
| **approximant** | partial | [approximant](#approximant) |
| **ast** | partial | `../syntax.json` |
| **cost** | symbolic source derived | [invariant-and-cost](#invariant-and-cost) |
| **descriptant** | present | [descriptant](#descriptant) |
| **invariant** | partial | [invariant-and-cost](#invariant-and-cost) |
| **tests** | executed bounded checks | [tests](#tests) |

## Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

| id | type | label | where |
|---|---|---|---|
| `behavior_complement` | behavior_complement | partial source derived model | [behavior_complement](#behavior-complement) |
| `reverse_reconstruction` | reverse_reconstruction | partial source derived model | [reverse_reconstruction](#reverse-reconstruction) |
| `maqaf_left` | maqaf_left | partial source derived model | [maqaf_left](#maqaf-left) |
| `paragraph_attach` | paragraph_attach | partial source derived model | [paragraph_attach](#paragraph-attach) |
| `empty_input` | empty_input | partial source derived model | [empty_input](#empty-input) |
| `nfc_boundary` | nfc_boundary | partial source derived model | [nfc_boundary](#nfc-boundary) |

## Model text


### descriptant

#### Descriptant

2026-10-31, prospective epoch: inspected source before edits and predicted NFC then chunk scan emits maqaf-left and punctuation tokens; second pass closes sentences and consumes paragraph markers.

Measured epoch: bounded supplied fixtures passed twice in debug and release; six behavioral mutants were rejected after compilation, each restored exactly. These observations support the scoped equations without extending them to corpus accuracy, semantic understanding or persistent learning. Production code bytes are unchanged; only isolated tests appended. No benchmark or percentage surrogate was introduced.

Retained caveats: Finite supplied-input checks only; no corpus, training, external annotator, GPU, throughput or semantic-understanding claim. Allocation exhaustion and malformed arbitrary model arithmetic are not exhaustively exercised. No useful numerical surrogate is implemented; future caps may only be 1/2/5/10% with a declared metric.

### approximant

#### Forward relation

NFC then chunk scan emits maqaf-left and punctuation tokens; second pass closes sentences and consumes paragraph markers.

Domain: Token indices refer to normalized tokens; raw whitespace/normalization history discarded.

Source snapshot SHA-256: a123d548879eaa2575f9ca17c5836bab9068a0615ce5dd11f996afaa579e44b7. This is a partial source-derived mathematical relation, not an executable surrogate. Tests exercise actual production helpers with synthetic inputs.

### models

## Typed model catalog

Subject: hbo-text-tokenization at SHA-256 a123d548879eaa2575f9ca17c5836bab9068a0615ce5dd11f996afaa579e44b7. Domain and fixed state: Token indices refer to normalized tokens; raw whitespace/normalization history discarded.. Observation: return values, stated mutations and failure order of this scope. Models are partial source-derived relations; finite tests can falsify them but do not prove general completeness. Behavior complement is a scoped exclusion claim in the admitted transition universe. Reverse reconstruction retains compatible antecedents, not a unique historical cause. Specialized entries hold this domain fixed and isolate their named observation. Oracle: named actual-code fixtures in tests.md; unexecuted effect boundaries remain static hypotheses.

### behavior_complement

Apostrophe inside a chunk is not split by tokenize; colon does end a sentence.

### reverse_reconstruction

All whitespace and canonically equivalent raw variants yielding same tokens are compatible.

### maqaf_left

Append maqaf and emit current token; repeated maqaf can yield maqaf-only token.

### paragraph_attach

Following פ/ס tokens attach only after an ending token.

### empty_input

No chunks implies no tokens and no sentences.

### nfc_boundary

NFC occurs before character dispatch, so output slices are newly owned normalized strings.

### tests

#### Bounded evidence

Primary actual-code fixture: text::wave32::coordinates_thinning_and_sentence_edges. Run using cargo test --offline --locked -p hbo --lib text::wave32::coordinates_thinning_and_sentence_edges -- --exact --test-threads=1; repeat with --release. The external wave32 Hbo receipt set contains all 14 named outcomes twice per profile, numeric exit statuses, six compiled behavioral source negatives and byte-exact restoration hashes, and two identical source syntax projections. Source-local primary tests include assertions and caught expected panics. Fragments share these source witnesses; no claim of one test per branch.

Finite supplied-input checks only; no corpus, training, external annotator, GPU, throughput or semantic-understanding claim. Allocation exhaustion and malformed arbitrary model arithmetic are not exhaustively exercised. No useful numerical surrogate is implemented; future caps may only be 1/2/5/10% with a declared metric.

### invariant-and-cost

#### Invariant and cost

Token indices refer to normalized tokens; raw whitespace/normalization history discarded.

##### Cost

O(B+NFC reorder work+T) time and O(B+T) space.

Resources must permit allocations; cost variables are source bytes, units, words and table entries as explicitly stated. No measured speed or universal error claim.