lojban-rs implementation plan

The plan is split into focused documents under plan/. Code is the source of truth for implemented features; the plan tracks gaps and documents architecture.

The document references @cll-md/ reference grammar book.

Quick start for agents

1. Read plan/RULES.md before implementing any feature.
2. Consult the CLL section(s) cited in the open ticket row.
3. Add or update #[jbo_ref(...)] in code; keep plan/OPEN.md rows for non-ready work.
4. Run bash scripts/regenerate_plan.sh (or at minimum sync_plan_ready.py + ref_audit.py --strict).

Regenerate

bash scripts/regenerate_plan.sh

Plan rules & agent workflow

This document must be kept up to date. Update the plan after implementing or fixing any features/tasks from it.

The plan is split under plan/ — see PLAN.md for the index. The document references @cll-md/ reference grammar book.

Rule: Read Before Implement

Before implementing any feature, you MUST read the corresponding CLL section(s) listed in the reference column. CLL sections contain the authoritative grammar definitions, examples, and edge cases needed for correct implementation. Do not implement from the feature description alone — always consult the source material first.

When implementing or fixing a feature: (1) add or update #[jbo_ref(...)] (Rust) or an @ref: comment block (.egg, Cargo.toml) with full metadata in code, (2) if still AD HOC / PARTIAL / MISSING, keep or update the row in OPEN.md, (3) run python3 scripts/sync_plan_ready.py and python3 scripts/ref_audit.py.

Reference Conventions

Code is the source of truth for implemented features. Each @ref: block in source holds: Feature, Status, CLL, PlanSection, and optional Deviation, Notes.

Code-first workflow

When you implement or fix a feature:

1. Read the CLL section(s) cited in the open ticket row (if any).
2. Add or update #[jbo_ref(...)] on the Rust item (or a const marker for module-level gaps). Use comment blocks only in .egg / Cargo.toml.
3. If the feature is still AD HOC, PARTIAL, or MISSING, keep or update the matching OPEN.md row.
4. Run python3 scripts/sync_plan_ready.py to regenerate READY.md from code.
5. Run python3 scripts/ref_audit.py --strict and JBO_REF_STRICT=1 cargo build.

Migrating remaining gaps: use python3 scripts/migrate_plan_refs.py for templates. Completed features live in #[jbo_ref] (Rust), comment blocks (.egg/Cargo.toml), or src/plan_implemented.rs for bulk registry entries — then run sync_plan_ready.py.

See SCRIPTS.md for the full maintenance script reference.

Slug naming

Slugs follow layer.name and are stable across refactors:

Grammar rule alts use semantic names {cll_nonterminal}_{variant}. Tree extraction uses greedy disambiguation: longest-match for left-grouped ... repetition (CLL §21 rule 7), longest-alt-name preference for equal-span ties (left-recursive expansions over base cases, quantifier parses over bare selbri), and ParseTree::Ambig parse-forest nodes for remaining structural ambiguity.

Anchor formats

Rust (required — use #[jbo_ref], not hand-written /// @ref: comments):

#[jbo_ref(
    slug = "semantic.parse_sumti_atom",
    feature = "All sumti atom types",
    status = "IMPLEMENTED",
    cll = "§6.2, §6.6",
    plan_section = "2.4",
)]
fn parse_sumti_atom(...) { ... }

Egg / TOML comment block:

; @ref:grammar.sumti_6_zo
; Feature: ZO single-word quotation
; Status: IMPLEMENTED
; CLL: §19.10
; Notes: sumti_6_zo = ZO_clause any_word frees; word-1100 skips indicator merge after ZO
; PlanSection: 2.4

Validate with python3 scripts/sync_plan_ready.py, python3 scripts/ref_audit.py --strict, and JBO_REF_STRICT=1 cargo build after any change.

Status Legend

• ✅ IMPLEMENTED — fully working in lojban-rs
• ⚠️ PARTIAL — partially implemented or behind feature flag
• ⚠️ AD HOC — works, but implemented via hardcoded cmavo/special-case rather than generic rule
• ❌ MISSING — not implemented yet
• ✅ N/A — Not needed for lojban-rs (reference/design feature)
• ⏳ NEEDS EVAL — feature needs a math evaluator for full semantics
• 📝 DOC-ONLY — documentation/reference section with no parser gap

Plan maintenance scripts

Companion to RULES.md. Run bash scripts/regenerate_plan.sh to refresh the auto-generated sections and audit reports.

Detecting gaps and discrepancies

Regenerate everything

bash scripts/regenerate_plan.sh

This runs, in order:

1. python3 scripts/sync_plan_ready.py — regenerate READY.md from code #[jbo_ref] / @ref: blocks
2. python3 scripts/ref_audit.py — consistency report at scripts/ref_audit_report.md
3. python3 scripts/cll_coverage.py — CLL section coverage at scripts/cll_coverage_report.md
4. python3 scripts/grammar_audit.py — CLL Ch 21 non-terminal coverage at scripts/grammar_audit_report.md

CI on main runs the same pipeline and commits updated plan/READY.md plus audit reports when they drift.

After any feature change

python3 scripts/sync_plan_ready.py
python3 scripts/ref_audit.py --strict
JBO_REF_STRICT=1 cargo build

Architecture & tooling

Human-readable description of the parser pipeline, semantic layers, and subsystems. Open gaps are tracked in OPEN.md; completed features are auto-listed in READY.md.

Pure CFG Earley

The word-level Earley recognizer is a pure CFG: earley_word_schema.egg saturates all chart items with no ordered-choice pruning.

Behaviour

Tree extraction policy (src/earley_extract.rs)

1. Longest match — maximize child_end at each dot (CLL §21 rule 7, left-grouped ...; not PEG).
2. Greediness — among equal-span alternatives, prefer the longest alt name string (child_alt.len()). alt names follow the convention {cll_nonterminal}_{variant} (see plan/RULES.md), so the left-recursive expansion variant is longer than its base case (e.g. selbri_3_selbri_3_selbri_4 vs selbri_3_selbri_4). This is a name-length heuristic, not a separate left-recursion algorithm; it also prefers quantifier parses over bare selbri parses (CLL §6.8).
3. Remaining equal-span ties — emit ParseTree::Ambig { alternatives } (parse forest).

AST resolution (src/earley_to_syntax.rs): resolve_ambig_first_ok tries each alternative at root and hot spots (sumti, paragraph, lerfu_word) until conversion succeeds.

Verification

• cargo test --test earley_integration — 85/85 pass (including indefinite-description, FA scrambling, semantics pipeline).
• cargo test --test earley_ambiguity_probe — corpus accepted; report at scripts/grammar_ambiguity_report.json.
• python3 scripts/grammar_audit.py — 374 egg rules vs 90 CLL non-terminals.
• No peg_alt_priority / grammar_alt_priority references remain in the codebase.

Comprehensive Feature Gap Analysis: CLL vs lojban-rs

Our parser must follow the grammar of @cll-md/chapter-grammars.md so no rule or part of it can be omitted. If there is or planned a deviation from that it must be done through a feature flag and reported by editing this document. Thus no simplficatons to the formal grammar are allowed. Fix related issues if any.

The codebase must be clean, stramlined, avoid duplication, avoid hardcoding where needed. the codebase must try to be generic where needed.

Ad hoc / architecture principles (avoid when implementing fixes):

1. No string comparisons on cmavo names — use selma'o checks, generated tables, or AST tags instead.
2. No hand-enumerating optional-position variants — prefer optional ([]) and repetition (...) in grammar; expand at generation time.
3. No layer crossing — morphology should not hide grammar structure (e.g., ZO/ZOI must consume quoted content) unless CLL explicitly says so.
4. No manual semantic tables in parser code — move tables like indicator_bridi_tail or ZAhO pairings into typed_dictionary.rs or external data files.

Gaps, deviations, and missing features are tracked once each in the section tables below (grouped by CLL topic). Do not maintain a separate duplicate index.

14. SEMANTIC RULES (egglog_rules.egg)

14.0 Rewrite rule taxonomy (3a–3d)

Tersmu/camxes-rs groups rewrite rules into four categories. lojban-rs implements the same rules under the §1–§23 section names below.

14.1 Implemented Rules (Internal egglog rules — no direct CLL mapping)

14.2 Testing Strategy (CLL Ch 2 — examples throughout)

• Add test cases for each new feature in the relevant test file
• Add CLL-referenced test sentences for each missing feature
• Ensure all 20 example tests pass before and after changes
• Add dedicated test files for complex features (mekso, abstractions, intervals)
• Add mekso-specific test suite with CLL examples 18.2–18.148

15. QUANTIFIER SCOPE & naku BOUNDARIES — IMPLEMENTATION

15.1 Problem Statement

egglog rewrite rules are unconditional pattern matches. They can match structural patterns like (PAnd p q) but cannot express predicate conditions like "x is not free in q". This blocks:

• Partial distribution: ∃x.(P∧Q) → (∃x.P)∧Q when x∉FV(Q)
• Vacuous quantification: ∀x.P → P when x∉FV(P)
• Quantifier over implication: ∀x.(P→Q) → (∃x.P→Q) when x∉FV(Q)
• naku partial distribution: naku (∃x.P ∧ Q) ↔ (∀x.(naku P) ∨ naku Q) when x∉FV(Q)

The negation push-through rule (¬∃x.P ↔ ∀x.¬P) already handles the core naku boundary case, but partial distribution under quantifiers was the gap.

15.2 Solution: Two-Phase Architecture

The solution uses two complementary phases that together cover all cases:

Phase 1 (Rust pre-lowering)          Phase 2 (egglog saturation)
─────────────────────────            ──────────────────────────
simplify_prop() on JboProp           FreeIn facts + conditional rules
    │                                     │
    ├─ Partial distribution               ├─ Partial distribution
    ├─ Vacuous quantifier removal         ├─ Vacuous quantification
    │                                     ├─ Quantifier over implication
    │                                     │
    ▼                                     ▼
  Lowered to egglog ◄──────────── FreeIn(BoundVar(v), subprop) = 0/1

Phase 1 handles the initial parse form before saturation. It's fast, deterministic, and handles the common case.

Phase 2 handles forms discovered during equality saturation. When egglog rewrites produce new quantified forms, the conditional rules can simplify them.

15.3 Phase 1: Rust Pre-Lowering Simplification

File: jbo_prop.rs — pub fn simplify_prop(prop: &JboProp) -> JboProp

Core mechanism: A recursive contains_bound_var(v, prop) function traverses JboProp (including HOAS closures) to check if BoundVar(v) appears in a subformula. When it doesn't, the quantifier is vacuous or partial distribution applies.

Rules implemented (6 distribution + 1 vacuous):

Hook point: egglog_lower.rs:lower_texticule() calls simplify_prop() before lowering each TexticuleProp.

Helper functions (all in jbo_prop.rs):

• contains_bound_var(v, prop) — main checker
• contains_bound_var_in_rel(v, rel) — relation traversal
• contains_bound_var_in_term(v, term) — term traversal
• contains_bound_var_in_mex(v, mex) — mex expression traversal
• contains_bound_var_in_operator(v, op) — operator traversal
• contains_bound_var_in_tag(v, tag) — tag traversal
• contains_bound_var_in_tag_unit(v, tu) — tag unit traversal

Tests: 6 unit tests in jbo_prop::simplify_tests covering vacuous removal, partial distribution, and negative cases.

15.4 Phase 2: egglog FreeIn Facts + Conditional Rules

Schema (egglog_schema.egg):

(function FreeIn (JboTerm JboProp) i64 :merge (max old new))

• FreeIn(BoundVar(v), prop) = 1 means v is free in prop
• FreeIn(BoundVar(v), prop) = 0 (default) means v is not free (or not checked)
• Merge: max(old, new) — once set to 1, stays 1

Lowering (egglog_lower.rs):

• generate_free_in_facts(prop, lines) — entry point, called from lower_texticule()
• collect_free_in_facts(bound_var_idx, prop, lines) — recursive traversal
• For each PQuant(v, body), evaluates body(1) and asserts (set (FreeIn (BoundVar 1) subprop) 1) for every subprop containing BoundVar(1)
• Helper functions: term_contains_bound_var, prop_contains_bound_var, rel_contains_bound_var, mex_contains_bound_var, op_contains_bound_var, tag_contains_bound_var, tag_unit_contains_bound_var

Conditional rules (egglog_rules.egg §7c) — 12 rules:

; ∃x.(P∧Q) → (∃x.P)∧Q when x∉FV(Q)
(rule ((PQuant (Exists) restr (PAnd p q) v)
       (= (FreeIn (BoundVar v) q) 0))
      ((union (PQuant (Exists) restr (PAnd p q) v)
              (PAnd (PQuant (Exists) restr p v) q))))

Full list:

1. ∃-conjunction: (P∧Q) with x∉FV(Q) and x∉FV(P)
2. ∀-disjunction: (P∨Q) with x∉FV(Q) and x∉FV(P)
3. ∀-conjunction: (P∧Q) with x∉FV(Q) and x∉FV(P)
4. ∃-disjunction: (P∨Q) with x∉FV(Q) and x∉FV(P)
5. ∀-implication: (P→Q) with x∉FV(Q)
6. ∃-implication: (P→Q) with x∉FV(Q)
7. Vacuous ∀: any body with x∉FV(body)
8. Vacuous ∃: any body with x∉FV(body)

Key design decision: Using (= (FreeIn ...) 0) instead of (not (FreeIn ...)) because egglog's not requires built-in Bool type, but FreeIn returns i64. The = 0 check achieves the same semantics.

15.5 How the Two Phases Interact

Input: ∃x.(P(x) ∧ Q)
         │
         ▼
Phase 1: simplify_prop()
  - contains_bound_var(1, Q) = false
  - Apply ∃-conjunction-left: (∃x.P(x)) ∧ Q
         │
         ▼
Lowering: egglog_lower.rs
  - Lower (∃x.P(x)) ∧ Q to egglog
  - Generate FreeIn facts:
    - FreeIn(BoundVar(1), P(BoundVar(1))) = 1  [P contains x]
    - FreeIn(BoundVar(1), Q) = 0  [Q does not contain x]
         │
         ▼
Phase 2: egglog saturation
  - If equality saturation produces ∃x.(P∧Q) again
  - §7c rule fires: (= (FreeIn (BoundVar v) q) 0) → union with (∃x.P)∧Q

15.6 Status Updates

§14.1: Added §7c Quantifier Scope Conditional Rules to implemented rules list.

§15.7 naku Negation Boundaries — How They're Solved

The CLL §16.9 rule "naku ro da P ↔ su'o da naku P" is semantically identical to ¬∀x.P ↔ ∃x.¬P, which was already implemented in §7 (negation push-through). The partial distribution rules in §7c complete the picture:

• naku (∃x.P ∧ Q) = ¬∃x.(P ∧ Q)
• By §7: ¬∃x.(P ∧ Q) = ∀x.¬(P ∧ Q) = ∀x.(¬P ∨ ¬Q)
• If x∉FV(Q): by §7c: ∀x.(¬P ∨ ¬Q) → (∀x.¬P) ∨ ¬Q = (∀x.naku P) ∨ naku Q

This is the composition of existing §7 rules + new §7c rules — no separate naku-specific rules needed.

16. OLOG INTEGRATION (dzaske)

Ologs (Ontology Logs) are a category-theoretic knowledge-representation format. lojban-rs embeds the dzaske olog system with bidirectional integration: extract schemas from parsed Lojban, validate instance data, run relational queries, and compose with the egglog e-graph.

Theory & sources: docs/ologs.md, docs/olog/sources/ (Spivak & Kent 2012, Patterson 2017, Lambert & Patterson 2024).

Note: Olog integration uses Cargo features olog and olog-proofs (both default-on). See @ref:feature.olog.

16.1 Core extraction

Extraction mapping:

16.2 Validation & querying

16.3 Bundled corpus

16.4 Egglog composition (Layers A/B/C)

Tests: cargo test --test camxes_olog_egglog.

17. PROLOG EXPORT

Prolog source generation from Lojban semantics via a target-neutral Prolog IR (src/prolog/: AST + tokenizer + precedence-climbing parser + dialect-aware pretty-printer). Two emission dialects behind PrologDialect:

• Swi — executable SWI-Prolog source (:-, ?-, \+, is/2, =.., lists, member/2).
• Tersmu — tersmu/lojysamban parity mode matching examples/*.loj (Skolem constants c0/c1, --> implication, non-veridical: side-facts, ju'a nai rewrites).

The legacy SWI-dialect string API (prop_to_prolog, semantic_results_to_prolog, eval_text_to_prolog) is preserved for backward compatibility; new code uses the _dialect variants. References: docs/prolog/references.md.

18. SEMANTIC EQUIVALENCE & OUTPUT FORMATS

18.1 Semantic equivalence

18.2 Output formats

18.3 Future CLI (tersmu parity)

lojban-rs has no main.rs today. tersmu/camxes-rs CLI flags for reference when a binary is added: --json, -E/--egglog, --equiv=S, -O/--olog, --olog-schema=F, -P/--prolog, -l/--loj, -j/--jbo.

Open Tickets

Manual tracking for MISSING, PARTIAL, AD HOC, NEEDS EVAL, and @ref:plan.* rows without a code anchor. Completed work moves to code #[jbo_ref] entries and appears in READY.md after sync_plan_ready.py.

3. TENSES & MODALS (CLL Ch 10)

Simple tense cmavo from CLL §10.2–10.11 (PU/ZI/FAhA/VA/ZAhO/ZEhA/VEhA/VIhA/MOhI/TAhE/ROI/re'u/FEhE/KI/CUhE/CAhA and compounds) are implemented — see auto-generated §3.1 Spatial Tenses, §3.2 Temporal Tenses, and §8.1–8.2. Remaining Ch 10 gaps are tracked in §3.6 below (§10.23 tense vs modals). §10.26–10.28 are reference/summary (N/A for parser).

3.6 Complex Tense Constructs (remaining)

5. NEGATION (CLL Ch 15)

5.2 Scalar Negation (CLL Ch 15)

5.6 Metalinguistic Negation (CLL Ch 15)

7. ABSTRACTIONS (CLL Ch 11)

7.3 Event Contour Relationship (CLL Ch 11)

9. MEKSO / MATHEMATICS (CLL Ch 18)

9.7 Mekso Selbri & Conversions (CLL §18.7, §18.11, §18.18–18.19, §18.21)

Completed null-operand/operator work (tu'o / ge'a, CLL §18.14) lives in READY.md under @ref:semantic.is_null_mex.

10. LETTERALS (CLL Ch 17)

CLL §17.14 auxiliary lerfu-word cmavo (this chapter):

Not letterals (were incorrectly listed under the old §9.8 “Acronyms” ticket): la'u (BAI modal “with quantity”, §9.6/§10.25), te'u (TEhU mekso/vector terminator, §18.15/§18.18), ce'u (KOhA abstraction focus, §7.11) — implemented elsewhere; see auto-generated §7.11, §9.6, §18.15.

Chapter 17 letterals are implemented — see auto-generated READY §10 / §9.8 lerfu entries (@ref:semantic.lau_by_shift_stack, @ref:semantic.se_e_unicode_evaluation, @ref:semantic.acronyms_lerfu_names, @ref:semantic.mathematical_lerfu_strings, @ref:grammar.lerfu_word_lau_chain).

11. ATTITUDINALS & INDICATORS (CLL Ch 13, 21)

11.2 Miscellaneous (CLL Ch 13, 19)

17. PROLOG EXPORT — tersmu parity gaps

The Prolog IR (src/prolog/), dialect flag (PrologDialect::Swi / Tersmu), AST lowerer, tersmu display-format path (eval_text_to_tersmu_lines, behind tersmu-parity), and the rendering/AST fixes (connective normalization, is/2 arithmetic goals, AbsPred/AbsProp/Moi lowering, tanru seltau in convert_selbri_3) are implemented — see auto-generated §17 in READY.md.

The tersmu display-format path (tersmu-parity feature flag) produces the tersmuLines output format: per-line > framing, ----- separators, non-veridical: side-facts, [Fragment: ...] rendering, and error messages (Parse error: / Morphology error: with context pointer). examples/1.loj achieves 100% byte-parity; examples/2.loj 99.6%.

Remaining examples/*.loj byte-parity gaps are upstream morphology/parser/semantic differences between our pipeline and tersmu's Pappy pipeline — not output-format issues. Completed morphology/lexer and tanru/gek logshow work lives in READY.md (e.g. morph.deterministic_cll_lexer, prolog.logshow_tanru_nesting).

Open upstream gaps (see also §18):

• Parser acceptance after identical tokenization (Earley vs Pappy).
• Morphology differential corpus: separate token/class mismatches from parser mismatches across examples/*.jbo — §18.2.2 / @ref:morph.differential_corpus.

18. GRAMMAR DEVIATIONS: CLL EBNF / tersmu Pappy vs Earley/egglog

Comparison of our Earley/egglog grammar with (a) the CLL Chapter 21 EBNF grammar and (b) tersmu's Pappy PEG parser. The golden test (tersmu_lines_golden_all_char) shows 1738/14476 lines match (12.0%) across all examples (with 1s per-line timeout; lines exceeding timeout produce morphology-error placeholders).

Completed items (gek forethought, deterministic CLL lexer, nested tanru logshow, abstraction side-fact rendering, fragment lerfu cia) are in READY.md — do not re-list them here.

18.2 PEG ordered choice vs CFG ambiguity

tersmu uses PEG / (ordered alternation): first match wins, no backtracking. Our Earley CFG explores all alternatives in parallel. resolve_ambig_first_ok / resolve_ambig_prefer_statement emulate PEG ordering at the AST conversion level. Many golden test mismatches stem from this fundamental difference: our parser accepts inputs tersmu rejects (and vice versa), and ambiguous inputs resolve differently.

18.2.1 How the CLL EBNF handles this

CLL Chapter 21 is a formal word/selma'o grammar, not the complete character-to-AST algorithm used by tersmu:

1. Upper-case symbols are terminals that morphology has already classified (§21.1 notation rule 2). Morphological ambiguity must therefore be resolved before this grammar is applied.
2. | is unordered EBNF alternation. The CLL does not assign PEG first-alternative priority to it.
3. ... has an explicit left-grouping convention (§21.1 rule 7). This is a disambiguation rule outside ordinary CFG notation and explains our longest-match handling for left-recursive repetitions.
4. //...// marks an elidable terminator which may be omitted only “if no grammatical ambiguity results” (§21.1 rule 10). This is a contextual admissibility condition, not a context-free production. A conforming parser must implement it with guards, grammar transformation, parse-forest validation, or an equivalent mechanism.
5. Pappy's / and ! are tersmu's operational implementation choices. Some negative lookaheads implement CLL terminator/elision constraints; others encode preparsing, magic-word, morphology, or tersmu-specific precedence. They must be audited individually rather than copied on the assumption that every ! is itself a CLL rule.

Therefore an Earley parser can implement Chapter 21, but a pure unguarded CFG recognizer alone is insufficient for the CLL's contextual elision rule. The recognizer may remain CFG-based if a separate deterministic constraint layer enforces those conditions.

18.2.2 Morphology differential corpus (remaining Phase 0 follow-up)

18.2.3 Syntax alignment options after Phase 0

All options preserve the CLL grammar unless explicitly placed behind a tersmu-parity feature. They may be combined.

Chosen primary policy — maximum token coverage from the forest. After Earley saturation builds a parse forest, select trees that consume the maximum number of input tokens. This generalizes the existing longest-child heuristic in earley_extract.rs (max_end among equal-dot candidates) to a global forest-ranking rule:

1. Prefer root candidates whose covered span end is maximal.
2. Prefer alternatives whose leaf-token coverage is maximal (fewest unused tokens / fewest empty elisions that shrink coverage).
3. Only after equal maximum coverage remains, apply secondary policies (statement-over-fragment, audited guards, PEG priority metadata, semantic convertibility).

Rationale: CLL §21.1 rule 7 already requires left-grouped longest attachment for ...; maximum coverage is the natural CFG-side analogue of “take as much input as the grammar can legitimately claim,” without importing PEG ordered choice as the primary rule. It also avoids prematurely choosing a shorter Fragment tree when a full Statement covers the same line.

What max-coverage does not fix alone:

• residual morphology mismatches (see §18.2.2 differential corpus);
• over-acceptance of inputs that should fail CLL/tersmu guards;
• cases where multiple full-coverage trees remain (gek subsentence Ambig, tense attachment, etc.) — those still need secondary policy.

Option A — explicit PEG alternative-priority metadata. Keep Earley recognition of every CFG parse, but attach Pappy declaration order to alternatives and use it as a secondary tie-breaker after maximum token coverage. Replace lexicographic alt_path_key, alt-name-length, and scattered “first OK” order with named, generated priorities.

Option B — systematic negative-lookahead/guard layer. Audit every Pappy ! against CLL and encode justified cases as declarative guards. Extend the existing Phase B NegLookaheadPos/NegInhibited mechanism for terminal and nonterminal lookahead. This fixes over-acceptance and reduces forests.

Option C — CFG factoring and token-set difference. Rewrite selected rules into disjoint alternatives, or generate complement terminal classes such as NA_NOT_BEFORE_JA and I_NOT_BEFORE_JEK_JOIK. This puts decisions in recognition and can improve performance, but causes grammar expansion and is appropriate only where the excluded language is regular and finite-state.

Option D — parse-forest constraints. Keep the full shared packed parse forest, then eliminate trees violating CLL elision, grouping, or audited Pappy constraints before selecting a tree. Maximum token coverage runs on the remaining forest. This most clearly separates recognition from disambiguation and can report true ambiguity, but requires richer forest storage than the current local ParseTree::Ambig reconstruction.

Option E — semantic-first forest pruning. Try AST/semantic conversion for each surviving max-coverage alternative and select the first valid one according to an explicit policy. Useful fallback for conversion failures after max-coverage selection, but must not outrank coverage or CLL guards.

Option F — tersmu-style preparser. Port textHead, wholeStatement, free stripping, quote handling, and I/NIhO boundaries into a deterministic Rust phase before Earley. This can substantially reduce top-level ambiguity but duplicates grammar concerns unless generated from one source.

Option G — generate Earley metadata/guards from Pappy. Build a pappy_to_earley generator that emits CFG productions, alternative priorities, lookahead guards, and an audit of semantic actions that cannot be translated. This minimizes long-term drift but is a large tooling project; PEG lookahead and ordered choice cannot always be represented by CFG productions alone.

Option H — dual parser/shadow oracle. Run tersmu or a Rust PEG port beside Earley in tests and report acceptance/tree differences. This is strong differential QA but does not resolve which behavior is CLL-authoritative.

Option I — replace the word parser with a Rust PEG port. This offers the closest parse parity but abandons the pure Earley recognizer as the primary path, retains semantic-port differences, and conflicts with the project's “No PEG dependency” architecture unless explicitly approved.

18.2.4 Recommended sequence

1. Build the morphology differential corpus report (§18.2.2 / @ref:morph.differential_corpus). Do not attribute remaining tokenization mismatches to PEG-vs-CFG behavior until that report exists.
2. Implement maximum token coverage as the primary forest selection rule (root + equal-span ties). Keep statement-over-fragment only as a secondary preference among equal-coverage trees.
3. Add an audit that extracts Pappy alternative order and negative-lookahead sites, classifying each as CLL-required, morphology/preparser, or tersmu-specific.
4. Implement Option B (audited guards) and Option A (priority metadata) as secondary tie-breakers after max coverage.
5. Use Option D for remaining genuine CFG ambiguity and Option E only as a conversion fallback.
6. Apply Option C or F selectively to measured performance/ambiguity hot spots.
7. Keep Option H as continuous differential testing. Consider G or I only if maintaining hand-aligned metadata proves more expensive than generation or replacement.

Success metrics must be reported separately:

• morphology token/class parity;
• syntax accept/reject parity on identical tokens;
• selected-tree/AST parity for jointly accepted inputs;
• semantic/logshow byte parity;
• parse-forest size, saturation batches, and timeout rate.

18.2.5 Worked mismatch examples (lojban-rs vs examples/*.loj)

Reference outputs below are from examples/*.loj (canonical tersmu-lines corpus). “Got” outputs were captured with eval_text_to_tersmu_lines under tersmu-parity (2026-07-17). Pipeline stages:

raw text
  → deterministic morphology (run_egglog_morphology / deterministic_cll_lex / …)
  → WordToken + WordClass facts
  → Earley egglog saturation (all competing Items)
  → Phase B NegInhibited (sparse today)
  → tree extraction / Ambig forest (longest-child + alt-name heuristics)
  → AST conversion (resolve_ambig_first_ok / prefer_statement)
  → jbo_parse semantics
  → logshow / tersmuLines framing

Example C — mi sutra-zei-tavla (morphology ZEI)

Expected (tersmu-lines / OPEN D4): sutra-zei-tavla(mi) (or spaced sutra zei tavla(mi) in some tersmu builds).

Got:

tra-zei-tavla()
tra-zei-tavla

Step-by-step:

1. Input is one orthographic word with internal -zei-.
2. Morphology/ZEI extraction consumes a suffix starting mid-rafsi (tra-zei-tavla) instead of the full pre-ZEI word sutra.
3. The resulting brivla token has no leading sumti attachment in the chosen tree, so semantics print tra-zei-tavla() with empty args.
4. Max-token forest selection cannot repair a wrong token stream.

Example E — coi djan. (vocative / empty output)

Expected (OPEN D5 / tersmu Vocative handling): a vocative side-fact such as coi(james) / name rendering.

Got: empty prolog/rewrite sections (only framing).

Step-by-step:

1. Morphology yields COI + CMENE.
2. Earley places the vocative in free / text-head structure rather than a bridi statement.
3. eval_text produces no main proposition from frees-only text.
4. tersmuLines therefore prints no logical body. Coverage selection does not create a proposition that was never in the semantic AST.

Example F — mi noi nanmu cu klama (relative clause / timeout)

Expected (OPEN D3):

nanmu(mi)
klama(mi)

Got: Morphology error: placeholder (timeout or acceptance failure under the current parse-timeout path).

Step-by-step:

1. Intended morphology: KOhA + NOI + BRIVLA + CU + BRIVLA.
2. Earley chart for noi relative clauses can explode or fail to accept within LOJBAN_PARSE_TIMEOUT_MS.
3. parse_line_with_timeout returns None.
4. eval_text_to_tersmu_lines maps that to format_morphology_error, which is a mislabeled timeout/reject, not necessarily a morphology classifier bug.
5. After reliability fixes, forest selection should prefer the full statement covering all tokens over a shorter fragment.

Example G — cases that already match (coverage/heuristic working)

These currently match examples/*.loj style and are useful negatives:

• mi pu klama → (pu)(). klama(mi) / pu ku zo'u mi klama (same shape as examples/10.loj mi pu klama le zarci).
• mi ki tavla → (ki)(). tavla(mi) (same shape as examples/9.loj).
• le re nanmu → non-veridical: (nanmu(c0) /\ 2 mei(c0)) + [Fragment: c0] (matches examples/4.loj; older notes about (re) vs 2 are stale for this path).
• pu broda gi'e brode → ((pu)(). broda() /\ brode()) (lazy term scoping; §18.3).

How max-token coverage should change the selection algorithm

Current extraction (simplified):

1. Among child edges at a dot, keep those with maximal child_end.
2. Among remaining equal-span ties, prefer quant names, else longest alt name, else emit Ambig.
3. AST conversion iterates Ambig with first_ok / prefer_statement.

Proposed:

1. Score every complete candidate tree by token coverage (end - start at root, then sum of consumed terminals).
2. Discard candidates below the maximum score.
3. Among survivors, prefer statement over fragment, then apply audited guards / PEG priority, then semantic convertibility.
4. If still tied, keep Ambig and fail loudly in tersmu-strict rather than silently picking lexicographic alt_path_key.

18.3 Lazy vs greedy term parsing (resolved)

With the gi'e connective parsing fix and tense_modal → DecoratedTagUnit conversion fix, pu broda gi'e brode now parses as (pu broda) gi'e brode and pu ku broda gi'e brode parses as pu (broda gi'e brode). The tests/debug_lazy_terms.rs cases assert these scopings:

• pu broda → Modal(pu, Rel(broda))
• broda gi'e brode → Connected(And, Rel(broda), Rel(brode))
• pu broda gi'e brode → Connected(And, Modal(pu, Rel(broda)), Rel(brode))
• pu ku broda gi'e brode → Modal(pu, Connected(And, Rel(broda), Rel(brode)))

This matches tersmu's lazy term scoping behavior for these inputs. The remaining open concern is term = (tag | FA #) /KU#/ elision in the Earley grammar, which currently only produces BareTag terms when KU is explicit (term_tag_ku). If broader tag-as-term-without-KU cases are needed, this should be handled by extending term grammar alternatives and relying on resolve_ambig_prefer_statement to prefer sentence_bridi_tail (where the tag attaches to the first selbri) over sentence_terms_bridi_tail (where the tag is a bare term).

18.4 Negative lookahead (PEG-specific)

tersmu uses PEG negative lookahead ! extensively (e.g. !JA on fragment NA, !MOI on quantifier, !jek !joik on I-separators). A plain CFG production cannot directly express arbitrary negative lookahead, but our overall Earley pipeline can implement equivalent constraints through audited Phase B guards, finite token-set complements/grammar factoring, or parse-forest validation. See §18.2.1–18.2.4. Do not move these decisions into ad hoc AST string checks.

18.6 Concrete discrepancies (tested sequentially on examples/*.jbo)

Running each example line through eval_text_to_tersmu_lines and comparing with examples/*.loj reference. Overall: 1738/14476 lines match (12.0%).

D3. Relative clauses with noi (MEDIUM)

Input: mi noi nanmu cu klama Expected: nanmu(mi)\nklama(mi) Got: Morphology error

Root cause: Our parser may not handle noi relative clauses correctly in all positions, or the cu separator interacts badly with the relative clause.

Approximate fix: Verify relativeClause grammar in earley_word_grammar.egg handles noi subsentence KUhO correctly. Check convert_relative_clause in earley_to_syntax.rs.

D4. ZEI compound word extraction (MEDIUM)

Input: mi sutra-zei-tavla Expected: sutra-zei-tavla(mi) Got: tra-zei-tavla() (missing su prefix and argument)

Root cause: The ZEI compound sutra-zei-tavla is being split incorrectly during morphology. The first syllable su is being consumed by a preceding word classification.

Approximate fix: Check earley_morph_extract.rs ZEI handling — ensure the full pre-ZEI word is captured, not just the tail.

D5. Vocative handling (LOW)

Input: coi djan. Expected: coi(james) (or similar vocative output) Got: Empty (no prolog/rewrite sections)

Root cause: Vocatives (COI DOI CMENE) are parsed as frees/vocatives but eval_text doesn't produce a proposition from them. tersmu has special Vocative handling in JboParse.hs:56-58.

Approximate fix: In eval_text or convert_paragraph, when a vocative is detected, emit a side-fact like coi(name).

D7. SE conversion on selbri (LOW)

Input: mi se klama le zdani Expected: klama(zdani(c0),mi) (x1/x2 swapped) Got: klama(c0,mi) (missing non-veridical: zdani(c0) in correct position)

Root cause: The SE conversion is working (arguments are swapped) but the non-veridical side-fact for le zdani is not being attached to the correct argument position.

Approximate fix: Check convert_selbri_3 SE handling and ensure le descriptions produce non-veridical side-facts at the correct argument position after SE swap.

D8. NUhA/forethought mex morphology (LOW)

Input: li nu'a su'i pa re du li ci Expected: su'i(1,2,3) Got: Morphology error

Root cause: nu'a (NUhA) is not being recognized as a valid cmavo in the morphology pass, or pe'o (PEhO) is not recognized.

Approximate fix: Check earley_word_grammar.egg for NUhA and PEhO terminal rules. Verify cmavo_to_terminal mapping includes these selma'o.

Priority order for fixes

1. Morphology differential corpus (§18.2.2) — regenerate mismatch reports on the deterministic lexer
2. Maximum token coverage forest selection (§18.2.3) — primary CFG disambiguation policy
3. D4 ZEI compounds — affects all ZEI usage
4. D3 Relative clauses — affects noi/poi usage (also timeout labeling)
5. D7 SE conversion — affects argument ordering
6. D5 Vocatives — low frequency
7. D8 NUhA/PEhO — low frequency

Implemented Features (auto-generated)

1.1 Phonology (CLL Ch 3)

1.2 Word Classes (CLL Ch 4)

1.3 Lujvo Formation & Decomposition (CLL Ch 4, 12)

1.4 Fu'ivla (CLL Ch 4)

1.5 Cmevla (CLL Ch 4)

2.1 Text Structure (CLL Ch 19)

2.2 Sentence Structure (CLL Ch 9, 19)

2.3 Terms (CLL Ch 9, 14, 16)

2.4 Sumti (CLL Ch 6, 8, 14)

2.5 Relative Clauses (CLL Ch 8)

2.6 Selbri (CLL Ch 5, 11)

2.7 Selbri Place Structure (CLL Ch 9)

2.8a SE conversion AST + type inference (CLL §9.4, §6.8)

2.8 Pro-sumti and Pro-bridi (CLL Ch 7)

3.1 Spatial Tenses (CLL Ch 10)

3.2 Temporal Tenses (CLL Ch 10)

3.3 Aspect & Modality (CLL Ch 10)

3.4 BAI Modal Tags (CLL Ch 9)

3.5 Modal Connections (CLL Ch 9)

3.6 Complex Tense Constructs (remaining)

4.1 Logical Connectives (CLL Ch 14)

4.2 Gihek (Bridi-tail connection) (CLL Ch 14)

4.3 Non-Logical Connectives (CLL Ch 14)

4.4 Termset Connection (CLL Ch 14)

4.5 Tensed Connectives (CLL Ch 10, 14)

4.6 Connective Questions (CLL Ch 14)

5.1 Bridi Negation (CLL Ch 15)

5.2 Scalar Negation (CLL Ch 15)

5.3 Sumti Negation (CLL Ch 15, 16)

5.4 nai Suffix Negation (CLL Ch 15)

5.5 Affirmation (CLL Ch 15)

6.1 Variables (CLL Ch 16)

6.2 Prenex (CLL Ch 16)

6.3 Quantifiers (CLL Ch 16)

6.4 Scope & DeMorgan (CLL Ch 16)

7.1 Abstractor Types (CLL Ch 11)

7.2 Abstractor Features (CLL Ch 7, 11)

7.3 Event Contour Relationship (CLL Ch 11)

8.1 Tense Conversion (CLL Ch 10)

8.2 Complex Tense Constructs (CLL Ch 10)

9.1 Numbers & Punctuation (CLL §18.2–18.4)

9.2 Operators & Infix (CLL §18.5, §18.14, §18.24)

9.3 Forethought & Reverse Polish (CLL §18.6, §18.16)

9.4 Non-decimal & Compound Bases (CLL §18.10)

9.5 Vectors & Matrices (CLL §18.15)

9.6 Indefinite & Approximate Numbers (CLL §18.8–18.9)

9.7 Mekso Selbri & Conversions (CLL §18.7, §18.11, §18.18–18.19)

9.8 Mekso Evaluator (NEEDED for full semantics)

11.1 Emotion Indicators (CLL Ch 13)

11.2 Scale System (CLL Ch 13)

11.3 Emotional Categories (CLL Ch 13)

11.5 Vocatives (CLL Ch 6, 13)

11.6 Miscellaneous (CLL Ch 13, 19)

12. TEXT STRUCTURE (CLL Ch 19)

12b.4 Status

13.1 Selma'o Catalogue (CLL Ch 20)

13.2 Formal Grammar (CLL Ch 21)

14.1 Implemented Rules (Internal egglog rules — no direct CLL mapping)

17. PROLOG EXPORT — tersmu parity

10.1

10.2

10.3

16

16.2

16.3

16.4

16.6

16.7

16.8

18

18.2.2

18.4