---
title: "langseg-engine-tokenize — Mova code model"
description: "Mova code model of tools/langseg/src/lib.rs: components with their labels, obernants and the model text."
url: "https://groupants.elamur.ai/free/en/mova/tools/langseg/src/lib.rs.groupant/tokenize.groupant/"
---
Elamur: [Mova](https://md.mova.elamur.ai/free/en/) · [Dictionary](https://md.dictionary.elamur.ai/free/en/) · [Research](https://md.research.elamur.ai/free/en/) · **Groupants**

This page as HTML: <https://groupants.elamur.ai/free/en/mova/tools/langseg/src/lib.rs.groupant/tokenize.groupant/>

# langseg-engine-tokenize

The groupant beside tools/langseg/src/lib.rs in the Mova repository: what the code computes, what it never does, how to go back from its outputs, its invariants, cost and checks.

## Facts

- **kind:** groupant_fragment
- **status:** partial source bound model
- **source:** [`tools/langseg/src/lib.rs`](https://github.com/ElamurAI/mova-free/blob/main/tools/langseg/src/lib.rs)
- **source SHA-256:** `33431b9d2b514927…`
- **covers:** Scan char_indices; markers toggle italic ids, short tags skipped, script starts at strong char then consumes matching continuations. (bytes 4889–7602)
- **part of:** [langseg-engine](https://md.groupants.elamur.ai/free/en/mova/tools/langseg/src/lib.rs.groupant/)
- **repository folder:** [`tools/langseg/src/lib.rs.groupant/tokenize.groupant`](https://github.com/ElamurAI/mova-free/tree/main/tools/langseg/src/lib.rs.groupant/tokenize.groupant)
- **repository commit:** `855da7bd4a7b`

> **Implementation material.** A Mova code model describes this code at one source revision: partial models, finite checks and their stated scope. It is not a reviewed general exposition, and its labels are shown as written, never raised.

## Components

| component | label | where |
|---|---|---|
| **approximant** | partial | [approximant](#approximant) |
| **ast** | partial | `../syntax.json` |
| **cost** | symbolic source derived | [invariant](#invariant) |
| **descriptant** | present | [descriptant](#descriptant) |
| **invariant** | partial | [invariant](#invariant) |
| **tests** | executed bounded checks | [tests](#tests) |

## Obernants

The typed relations around the forward model: the behaviour complement (what the code never does), the reverse reconstruction (every input compatible with an output) and specialised views.

| id | type | label | where |
|---|---|---|---|
| `behavior_complement` | behavior_complement | partial source derived model | [behavior_complement](#behavior-complement) |
| `reverse_reconstruction` | reverse_reconstruction | partial source derived model | [reverse_reconstruction](#reverse-reconstruction) |
| `utf8` | utf8 | partial source derived model | [utf8](#utf8) |
| `italic` | italic | partial source derived model | [italic](#italic) |
| `apostrophe` | apostrophe | partial source derived model | [apostrophe](#apostrophe) |
| `suffix_cost` | suffix_cost | partial source derived model | [suffix_cost](#suffix-cost) |

## Model text


### descriptant

#### Descriptant

##### Prospective epoch

2026-10-31 source review predicted: Scan char_indices; markers toggle italic ids, short tags skipped, script starts at strong char then consumes matching continuations. Resource model: O(B²) worst-case '<'^n suffix find scans; ordinary bounded tags O(B); O(B) chars/lower buffers. External prospective.md predates test additions and preserves APIs, effect order, hazards and interaction hypotheses.

##### Observed epoch

Bounded real helper checks in tests.md constrain these models; syntax is a partial lexical projection with no macro expansion, name resolution or proof. Production expressions unchanged; five inline test modules added. Compiled mutants are temporary and exactly restored. Existing malformed-input and nonfinite domains remain explicit, not silently repaired. Parent composes this scope with siblings without adding overlapping costs twice.

Exact byte scope; finite correspondence only. Untested branches source-derived.

### approximant

#### Forward relation

Scan char_indices; markers toggle italic ids, short tags skipped, script starts at strong char then consumes matching continuations.

Domain: Original UTF-8 offsets, omitted neutral prefix; italics heuristic not nested grammar.

### invariant

#### Invariant and cost

##### Invariant

Original UTF-8 offsets, omitted neutral prefix; italics heuristic not nested grammar.

##### Cost

O(B²) worst-case '<'^n suffix find scans; ordinary bounded tags O(B); O(B) chars/lower buffers.

##### Limits

Exact byte scope; finite correspondence only. Untested branches source-derived. No percentage surrogate is implemented: categorical parsing/effect observations lack an admitted numerical error metric. Any future profile must define its metric/domain and use only1/2/5/10 percent caps.

### models

## Typed source-bound relations

Subject: exact source SHA-256 in manifest. Domain and resource premises are in invariant.md. Each model is partial, source-derived, over a single helper invocation or the described finite trace. Complements are proposed inner exclusions under those premises; reverse relations retain all compatible antecedents, with no unique inverse claim. The oracle is the named checks in tests.md; unexecuted cases remain hypotheses. No numerical approximation certificate.

### behavior_complement

Neutral-only input cannot start token; multiplication/division excluded from Latin.

### reverse_reconstruction

Spans admit arbitrary skipped gaps/markup consistent with byte positions.

### utf8

Starts char_indices; ends next char byte or text.len; ascii lowercase preserves offsets.

### italic

Repeated same marker toggles once; entry increments span id; HTML opens separately.

### apostrophe

Hebrew inner doublequote needs next Hebrew; apostrophe accepted only when next non-Hebrew.

### suffix_cost

k<12 bounds skip only after full suffix.find('>'); unmatched openers cause quadratic sum.

### tests

#### Checks and evidence

Named source/helper checks: script_runs_and_transliteration, wave31::empty_ties_and_switch_objective.

Wave31 on2026-10-31:30 distinct exact tests passed twice debug and twice release, with equal sorted named outcomes. This node claims only its listed subset; empty subset means source review only. Six actual source-behavior mutants compiled, failed assertions, restored exact bytes/hash, and passed focused green reruns. No assertion-only mutants; existing swapped-input controls are separate. Receipt bundle: $MOVA_DATA/runs/octopus-wave31-corpus-2026-10-31 (native-commands.json, source-test-map.json, mutation-results.json, *.status/*.log). No live corpus, teacher, frozen assessment, download or production data used.

Exact byte scope; finite correspondence only. Untested branches source-derived.