# Configurable parsing and retained document data

The current execution priority is the [optimization and simplification
plan](OPTIMIZATION_PLAN.md). It preserves the feature scope while replacing
the accumulated cache/overlay graph. The contracts below remain evidence and
correctness inputs; they do not require retaining every implementation layer.

Design direction recorded on 2026-10-25. The three parsing presets and the
initial GEOS block-cache path are implemented on the development branch.
Source checkpoint before implementation:
`dad569e` on the local branch `supereditor/reu-cache-redesign`. Generated media,
build outputs, logs, external font extracts and local working records are
excluded from Git.

The current runtime implements Plain, Markdown and Full, retained block
classification, selective reuse of sparse inline events, and retained lexical
colors for the first eligible long code line per baseline. Narrow edit-aware semantic
reuse is now registered through C41/R34; general edits and retained layout remain
unfinished. See the [current rebase contract](https://github.com/ReadyOS-C64/UltimateExperiments/blob/main/SuperEditor/src/cache/edit-rebase-integration/README.md). The paired native BASIC/assembler
warm repaint uses about28% fewer PAL cycles with identical screenshots, although
it still costs roughly17 million cycles. See [lexical cache scope](SYNTAX_CACHE.md)
for admission, cancellation, source-identity limits and the complete measurements.
The native 4,102-byte styled-navigation Bottom command now takes 4,087,703 PAL
cycles including repaint, versus 306,428,091 in its control. Its real dispatcher
trace drops thousands of repeated inline requests. See
[performance measurements](PERFORMANCE.md) for the exact scope and evidence.

Current runtime `69720da` also keeps required fold operations ahead of optional
analysis. A real allocation failure may retire a fully checked type53 object
and retry the type46 allocation once. This preserves document, history,
clipboard, fold choices and the observer ticket/incarnation/ring. Cache TRIED
remains set to prevent repeated allocation/reparse attempts under pressure;
normal parsing continues until a later explicit baseline policy or document
incarnation permits rebuilding. All 28 fold frames, 14 ownership/resource cases
and native Copy/Cut/Undo pass. See the performance notes for scope and evidence.

The cold filename catalog now uses two bytes per entry and frees 47 startup
bytes without changing names, order or the tutorial. The catalog remains 54
assets with 58 cold bytes free. R21 also computes vertical clipping once per
strip; full diverse GEOS repaint is 10.150M PAL, still 6.7% above the older path.
Neither change closes edit-aware cache, lexical retention or general speed work.

Cache admission must account for total guard, DMA and overlay costs. Private
lexical measurements already show that a 25-byte code block is cheaper to
parse directly, whereas a roughly 1 KiB block benefits from retained spans.
The promoted navigation path retains ordinary movement when the new lookup
would add overhead. The integrated lexical policy uses the long-line threshold
and a single bounded attempt per baseline; short or inapplicable lines keep the
existing parser path.

The navigation implementation adds C26 operations6/7 and an exact generated
size/CRC witness checked by C23 before using them. Only an uncached E47 reveal
enters this selector. With no projected folds, a successful C35 query moves the
scan to the target's authenticated logical start. Refusal restores the original
query and any real parser checkpoint byte for byte. Subsequent metric windows
can reuse the live page proof and actual block bounds; checked arena writes
revoke that proof before DMA. No guessed carry or replacement source identity
is introduced. Source changes and missing-worker errors keep ordinary cleanup.

C23 links the existing readonly geometry micromodule, omitting unused OPEN and
WRITE code, and fits3,899 bytes; C26 fits4,000. The resident changes one branch
displacement so completed queries return to their actual navigation or paint
caller. There is no new BSS, zero page, allocation or asset. Native14 and the
packaged/public/fault gates are recorded in the integration checkpoint. This
does not skip softwrapped metric work within the target block or finish general
redraw and typing performance.

The `fbcc8f6` checkpoint connects bounded C7 admission, the R33 REU writer,
a retained C34 semantic producer, and C35 cached block classification. A full
GEOS repaint uses the stored classifications without block parser feeds;
independent pixel, source, lease and native display checks pass. This is an
initial baseline and classification implementation. Inline/lexical reuse,
edit-chain rebasing and retained layout are still work in progress.

Performance qualification remains open. The first cached path took26.376 million
PAL cycles versus9.512 million before it. Reusing a checked CR-page proof within
a viewport reduces native Full GEOS redraw to18.448 million cycles in the first
qualified integration. Each checked cache write revokes that proof before DMA;
fresh viewports, parser fallback and the painter transition also revoke it.
The subsequent forward-query capsule retains the checked page cursor and starts
one full query per page rather than one per logical line. Its integrated native
Full redraw takes12.268 million PAL cycles; Markdown takes11.747 million. This
still exceeds the old9.512-million-cycle Full path, so performance remains open.
Initial root validation and repeated source-state exchange still cost work.

C37 also skips inline interpretation only when its checked CR certifies a
completed inline pass, zero events and a canonical null inline reference. Missing
or inconsistent metadata keeps the conservative parser path. This reduces native
Full redraw to10.977 million PAL cycles and Markdown to10.456 million, with the
same independently verified pixels. Full remains15.4% above the older path.
This is an empty-result shortcut, not reuse of nonempty inline style events.

Skipping decoded glyph scanlines with no ink and no rendering flags further
reduces Full redraw to10.608 million PAL cycles and Markdown to10.087 million.
Selection, caret and inverse flags retain the original rendering path. The
4,094-byte renderer and fixed font addresses stay unchanged in size and placement;
all11 native screenshots match. Full remains11.5% above the older path. This
raster optimization does not reduce overlay loads or close the cache redesign.

The next raster change computes the intersecting color-tile range once and
unrolls the histogram reset, retaining the same4,094-byte payload and font
addresses. Together with committed-only source admission, the verified native
Full redraw is10.443 million PAL cycles (all11 screenshots unchanged). This
remains above the older9.512-million-cycle path.

A fresh block query can now accept a changed hot-window geometry only when
both snapshots have no pending edits, their29-byte canonical source prefix and
merged lengths agree, and the real current reader and typed cache authority
pass validation. It copies the actual current59-byte identity into the local
query; every subsequent reader guard stays exact. Pending edits, REKEY and
SPLICE still use the existing safe fallback.

This admission is correctness-qualified, with a remaining small-case regression:
after a window move, the251-byte case takes1.958 million instructions versus
1.749 million for the previous parser fallback. The first implementation also
regressed the66-KiB case: its C35 seed still requested inline parsing for an
empty result. C35 now uses the same checked completed-empty predicate as C37.
Safe publisher compaction fits this into4,090 bytes while the disabled profile
remains byte-identical. The change removes198,000 inline-parser feeds and lowers
the same-window navigation interval from92.066 to56.358 million instructions,
including actual caret reveal. Pixels, source and foreign REU remain unchanged.
These are CPU instruction measurements; initial baseline construction is separate.
The long sparse inline path described below is now integrated; retained layout
remains a private experiment.

The enabled C35 shortcut passes actual editor, foreground handoff, parser-bypass,
incremental, stack and native11 gates. Native Full redraw is10.283 million PAL
cycles, Markdown9.762 million, with all11 screenshots unchanged. Full remains
8.1% above the older path. Native overlay loads fall153→149 (Full) and147→143
(Markdown); no overall responsiveness or completion claim follows.

C39 now consumes completed inline events for eligible source continuations.
C26 admits bodies of at least 4,096 bytes only with a live checked viewport,
no queued edits, and the existing source continuation. C39 verifies the cached
record and accepts at most eight events; larger or unsuitable results return to
C26 through an explicit fallback operation, without attempting C39 again.
Short and dense bodies retain the parser path. The worker fits 4,093 bytes,
adds no resident state, and preserves the existing source-reader cleanup ABI.
Actual public editor comparisons and fault cases check the complete dispatcher,
source maps, pixels, history and exact REU boundaries. The 54-asset native build
also passes the eleven display captures and full startup import. The diverse
short-document Full redraw remains 10.283 million PAL cycles, so this does not
close general performance qualification or slow styled navigation. The long-case
instruction measurements include repaint, rather than initial baseline creation.

File/Open now retires the optional semantic-cache object after successful file
preflight and size validation, before reserving the two staging objects. The
File phase uses the ordinary checked object service and verifies type `$53`; it
never borrows the cache workspace while File owns that memory. Successful
retirement drops the capability and cache proofs, preserving the observer ticket,
incarnation and unread edit ring. Stale, foreign or active capabilities are
refused. Source, history and clipboard remain intact. This fixes object-slot
pressure even when physical REU banks are available. If physical banks are still
exhausted, Open retains its existing safe refusal and cancellation behavior.
An unsuccessful Open may therefore discard derived analysis while preserving
the document. The new C12 image is 4,031 bytes, with no new BSS or zero page.

C35 seeds an untrusted slot after final source validation. C37 validates the
page certificate, current source/capability tuple and the selected record's CRC
and canonical START before using that slot. A bad slot, backward query or page
boundary hands off to C35 BEGIN on the next foreground slice. The resident
preserves that operation0; no C-to-C call is introduced. Actual foreground
rendering and a corrupted-slot handoff pass independent pixel/source/full16MiB
checks. The53-asset build also passes observer42, parser55, incremental15,
cleanup9, stack68 and native11 gates. Generated media and logs remain ignored.

The dirty-heading incremental regression is fixed using an explicit cached
outgoing-carry type. It consumes authenticated CR data and never fabricates a
legacy parser checkpoint. All15 incremental editor cases pass, alongside the
cached-paint, parser-bypass and stack checks. Inline/lexical and edit-rebase work
remains open. Zero parser feeds or fewer overlay loads alone are not speed
results; the native cycle measurement above remains slower than the earlier path.

The user's requirements are configurable highlighting with a real parsing
bypass, reuse of unchanged document analysis and layout, efficient REU use,
and reconsideration of overlay residency. The particular cache representation
below is a proposal, rather than a user-mandated implementation.

The repeated work is measured: a private retained-style probe still feeds the
same 857-byte BODY six times for an 863-byte test document. Its inline worker
accounts for about 16% of that run's instructions, while the raster worker
accounts for about 54%. These are CPU instruction counts from an unpromoted
probe, not native cycle timings. Both the analysis pipeline and the drawing
path need attention; retaining syntax results alone is insufficient.

## Separate the work that can be disabled

Provide session controls and host build defaults, as with the existing font
settings. Do not add disk configuration persistence in this phase.

| Source preset | Structure and inline formatting | BASIC/assembler token coloring | Raw-source behavior |
| --- | --- | --- | --- |
| Plain | Disabled | Disabled | Literal bytes, uniform body font/pitch and colors; normal wrapping, navigation, selection and editing |
| Markdown | Enabled | Disabled | Headers, emphasis and code-block layout; code has a single configured color |
| Full | Enabled | Enabled | Markdown presentation plus language token colors |

The implementation should keep structural formatting and code-token coloring
as separate capabilities, even if presets make them convenient to select.
Turning off only code-token coloring skips the lexical pass. Plain source
skips automatic block, inline and lexical parsing altogether: no speculative
background cache fill and no formatting-driven bold/italic metrics or code
font substitution. A low-color screen that still runs the parser is not this
bypass.

F7 accepts `syntax plain` (also `syntax off`), `syntax markdown`, and `syntax
full` (also `syntax on`). Settings H cycles Plain → Markdown → Full. R restores
both the font settings and the configured preset. Host defaults use a separate
`[editor] source_preset = "full"` field in `config/editor-defaults.toml`; the
existing 64-byte font settings format is unchanged. Configurations that omit
the editor section retain Full. Nothing is saved to disk by these controls.

Plain keeps literal markers visible, uses the ordinary fixed grid font in
40/60/80 views and unstyled BSW/9 throughout editable GEOS source. Code fences
therefore do not select native8 pitch in Plain. The separate body/heading/code
font choices remain formatted-preview settings, as before. Switching presets
invalidates the derived viewport/style caches before publishing the new preset.

Preview is an explicit consumer of document structure and inline formatting.
Code-token coloring can still be disabled in preview. Plain source does not
promise active semantic folding without structural analysis: it presents the
expanded literal document and rejects explicit fold operations. Stored fold
choices are retained for returning to Markdown or Full. Formatted preview is
an explicit formatting operation even when the source preset is Plain; it
retains its selected fonts and bold/italic/underline. Preview code is uniform
in Plain/Markdown and token-colored in Full. Its retained bitmap key includes
the preset, preventing reuse of colors from a previous setting.

## Cache semantic blocks independently of the viewport

Use CR-delimited logical lines as the basic semantic records. A fenced code
section or explicit region is a dependency across these records, not a single
large unit whose tokens must all be regenerated after every interior edit.
Current BASIC/assembler lexical state resets at CR, so an ordinary code edit
usually needs one line's lexical runs once its language is known.

A completed record contains:

- Source identity, canonical START/BODY/END/NEXT and CR-versus-EOF reason.
- Incoming and outgoing block-parser state, including fence, language, region
  depth and region-overflow state. The current CR-boundary carry is six bytes;
  mid-line parsing uses the existing larger continuation state.
- Block kind, heading level and control-marker information.
- Completed inline events or spans, plus language-token runs when requested.
- Its own version and a final publication marker.

Store event/token positions relative to the logical block. An insertion before
an unchanged block should rebase an anchor through a complete edit history,
rather than rewrite every token's absolute position. An indexed anchor or piece
reference must also prove that the unchanged text is still the same text.

Keep layout records separate: wrap boundaries, glyph advances, source/glyph
mapping, height/baseline and resolved font/style runs. Keep color and bitmap
projection separate again. Scrolling, a palette change or a caret move should
not invalidate the document's parsed meaning.

## Invalidate from edits, and stop at proven convergence

Record every visible mutation, including unflushed hot-gap typing. Document
generation alone is insufficient: many pending edits share that generation.
The cache root must identify the visible revision, source graph, provider epoch
and parser ABI. Migration to a newer revision requires a complete ordered edit
history. The existing bounded fold journal is conditional on a valid fold
object; it is not a universal edit journal.

For an edit, restart at the affected certified logical boundary. Include the
preceding line when removing a CR joins two lines. Invalidate the entire
affected inline BODY: a newly matched closing marker can change styling before
the insertion point. Inserted/deleted CRs split or merge records.

Propagate structural parsing forward until an unchanged suffix has both an
exact unchanged-source proof and matching incoming parser state. A fence or
language change can propagate to EOF. Equal offsets, similar visible colors or
a checksum alone do not prove convergence. Ordinary prose edits should usually
converge after the affected logical line; the stopping rule must establish it.

Cache complete inline analysis once per logical BODY, then supply windows from
that result. The present repeated whole-BODY scans for successive 160-character
requests are a replacement target. A long line must use chained chunks or a
retained continuation; a small result-buffer limit must not cause repeated
complete parsing or silently truncate the result.

| Change | Recompute |
| --- | --- |
| Text edit | Affected inline/lexical records; structural suffix until convergence; dependent layouts and damaged paint |
| Parser/dialect change | Semantic records and dependents |
| Enable code coloring | Missing lexical runs for needed blocks |
| Palette/foreground/background change | Color projection and paint; retain parsing and applicable metrics |
| Font, size or metric-affecting style change | Dependent layout and paint; retain parsing |
| Column width or wrapping change | Layout and paint; retain parsing |
| Top, pan, selection, caret or fold visibility | View projection and damage; retain parsing |

Old records remain available until their replacements are complete. Publish
validity last. A missing/overflowed edit history, stale provider or unverifiable
record must cause explicit invalidation and bounded rebuilding, never a guessed
hit. Stop/error recovery must preserve the pending document edit.

## Reorganize residency around jobs

The current root loop draws on damage requests; it does not repaint continuously
on every idle VIC frame. However, content/layout jobs do make many overlay calls
and transfers. The redesign should reduce those transitions architecturally.

1. Consume or build semantic records for a batch of logical blocks while that
   worker remains loaded. Retain continuation state across bounded yields.
2. Derive layout from those completed records while the layout worker remains
   loaded. Styled advances are required before wrapping can be finalized.
3. Paint damage in consecutive font/style runs, preserving draw order and the
   C64 8x8 color-cell rules. Retain the compositor and active expanded glyphs
   for the paint job, rather than swapping helpers for each glyph or packet.

Cold file dialogs, settings, import/export and recovery are suitable overlay
work. Frequently used source access, cache lookup, caret and drawing primitives
need a measured residency budget. Preserve the assembler micromodule approach;
link and schedule modules according to task lifetime instead of making every
micro-operation a runtime overlay boundary.

Do not append another supervisor layer to the current graph without measuring
the result. Budget a replacement hot path and compare overlay loads, DMA bytes,
parsing feeds and total cycles on the same documents. After adding the presets,
resident code/constants end at `$3EFD`, with two bytes before the CRC table;
resident BSS uses 1,534 bytes, with two before the ReadyOS reservation. A new
resident/phase map is required; additional hot storage cannot be assumed.

The [implemented observer foundation](REU_CACHE_OBSERVER.md) appends a 17-byte
descriptor and three 32-byte records to existing work segments. OBJECTS now
occupies 454 of 512 bytes and DOCSTATE 997 of 1,024 bytes. Existing data fields
retain their addresses; ordinary BSS and zero page do not grow. The existing
clear routine covers `$9A00–$9FFF`, and the lifecycle adapter copies that range.
Cache query revocation on restoration remains an integration requirement.

The chosen producer records edits in RAM inside the already loaded mutation
images; it performs no REU DMA on warm local edits. A separate edit ticket
observes hot edits and R2 byte/reference publication, with REKEY for already
visible flushes and incarnation revocation on successful new H_INIT. It does
not redefine `se_r_revision`. Typed arena transport and sequential semantic
production are separately tested components; renderer reuse and lifecycle
integration remain open. Allocation failure must only invalidate disposable
cache data.

Checks belong at ownership/transaction entry, actual yield/resume boundaries,
resource changes and publication. Work under a still-valid, non-yielding lease
should not repeat the complete source-graph or font-header validation for every
glyph. Establish those lifetime guarantees before removing redundant checks.

## REU capacity and RAM contracts

Allocate optional cache pages through typed logical handles. Do not hard-code
physical bank numbers. Make semantic, lexical, layout and decoded-glyph pages
independently evictable; document, clipboard and undo correctness takes priority.
Cache pressure may evict or stream derived data, never reject a document edit.

Measure both the normal large REU profile and the existing small profile. The
1MiB profile already reaches 15 allocated banks in an earlier acceptance run;
an always-reserved extra cache bank is not a safe assumption. Use an explicit
quota and graceful pressure behavior rather than claiming unlimited cache space.

Existing FE row, checkpoint, font and preview extents are occupied and have
published contracts. Likewise, $5000 is an armed staging/font/code window and
$8D00 is a shared ownership phase. Neither is general spare memory. A new map
must prove every persistent and temporary lifetime, including IRQ behavior.
Preserve $C600-$C9FF, define cache revocation on suspend/resume/provider changes,
and keep the REU provider boundary suitable for a later ReadyOS adapter.

## Implementation gates

1. Inventory hot-path residency and produce a replacement memory/call budget.
   Specify the independent source-presentation controls and session ABI.
2. Implement Plain bypass first. Instrument actual emitted-code parser/lexer
   entries: off-mode source editing must make zero such calls, including after
   warm/cold mode changes and foreground idle. Verify pixels, text, history,
   selection and file operations in all four source views.
3. Implement typed cache pages, complete edit tracking and semantic records.
   Prove edits, pending input, CR splits/joins, distant delimiter closure, fence
   propagation, eviction, cancellation and resume. A viewport change over valid
   cached text should add no semantic parser feeds.
4. Add layout reuse and the batched paint path. Prove independent pixel/source
   maps, fonts and color-cell results; test settings/palette invalidation.
5. Compare cold open, ordinary/repeated edits, scrolling and full redraw under
   normal PAL/NTSC timing. Record parsing feeds, cache hits/misses, overlay loads
   and DMA bytes alongside total cycles. A cache hit must eliminate the work,
   not merely skip showing its result. Keep the original usability targets.

Existing private style, producer and wrapped-line experiments remain evidence
and possible reusable components. They are not prerequisites that must all be
promoted before this architecture can replace the expensive path. Rendering is
also a separate measured bottleneck; semantic caching alone cannot promise to
solve full-page latency.
