Documentation · launch release

Documentation

Everything below is checkable against the repository: the commands are the ones the README ships, the component names are the ones in the source tree, and every number is reported with its corpus, its denominator and its basis.

Install, fetch a model, run the benchmark

TekMyra is a Python library and benchmark harness. The package that ships is tekmyra-core; the import name is tekmyra. The README leads with the from-source install, and so does this page: it is the path that reproduces the published figures.

Install from source (primary)

Dependencies are declared as ranges so TekMyra can live inside your application; ranges float, so the published figures were measured against the exact versions recorded in constraints-reproduce.txt. Install against the constraints file if you want to reproduce the numbers rather than only install the library:

install: from a clone, pinned for reproduction
pip install -c constraints-reproduce.txt -e .
install: with the test dependencies
pip install -e '.[dev]'   # the library plus pytest, to run the suite below

If you do not want the test dependencies, pip install -e . installs the library alone.

Core dependencies are click, numpy, pydantic, scikit-learn, scipy, tiktoken and pyyaml. tiktoken is a core dependency, not an extra: token-reduction figures are unobtainable without it.

Install from PyPI

install: from the package index
pip install tekmyra-core

Getting a model

Trained model artifacts are not in the repository and never will be: they are the commercial half. The reference artifact bundle is published as a release asset, tekmyra-reference-artifacts.tar.gz, with its SHA-256 printed in the release notes. Download it, check the digest before extracting, and unpack it at the root of your clone:

fetch and verify the reference artifact bundle
curl -L -O https://github.com/laconiq-ai/tekmyra/releases/latest/download/tekmyra-reference-artifacts.tar.gz
shasum -a 256 tekmyra-reference-artifacts.tar.gz   # compare against the release notes
tar xzf tekmyra-reference-artifacts.tar.gz         # unpacks .cache/compressor/1.0.0 and .cache/router/1.0.0

The artifact directories are resolved in this order, and a failure prints every location it tried: $TEKMYRA_ARTIFACT_ROOT/.cache/… if that variable is set, then .cache/… under your working directory, then .cache/… beside the installed package. Details on what the bundle contains and what it does and does not unlock are in the reference bundle section.

Run the benchmark

reproduce what the repository can reproduce
python -m tekmyra.benchmark compressor --fixture-dir benchmarks/fixtures/synthetic
python -m tekmyra.benchmark compressor --fixture-dir benchmarks/fixtures/long_context_v1
python -m pytest tests/test_public_corpus_notice.py   # public bytes + attribution
python -m pytest tests                    # the suite, green without a model
python scripts/verify_open_core.py        # exits 6 on a clean clone: seam proven, numbers await the bundle

The test suite passes without a model: the tests that need a trained artifact skip, and each skip names every location that was searched.

Without a model, the benchmark refuses to give you a number

That is deliberate rather than a bug, and it follows the same rule as the verifier:

what refusing looks like
Error: Learned router artifact did not load — refusing to produce a benchmark
number. artifact_dir='<your clone>/.cache/router/1.0.0' exists=False
backend='none'.
could not resolve '.cache/router/1.0.0'; searched, in order:
  absent  <your clone>/.cache/router/1.0.0
Set $TEKMYRA_ARTIFACT_ROOT to the directory holding .cache/ if your artifacts
live elsewhere. Re-run with --allow-rules-only to record an explicitly
rules-only run; such a run cannot emit a headline compression ratio.

Take the escape hatch it offers and the run records exactly what it was: the headline is withheld (not zero and not one, withheld), token reduction is 0.00%, and 68 of 68 locked spans are still preserved across 28 reached fixtures. The safety machinery works without a model. The compression does not.

The numbers, with their denominators

With the reference artifact bundle, on the corpora in the repository, via the joblib/scikit-learn path (no onnxruntime installed):

corpus fixtures mean size headline ratio tokens locked spans
synthetic 28 eligible of 301 shipped (28 reached, 0 refused) 250 chars (over 301 shipped) 0.7409 +25.86% 68/68
long_context_v1 40 eligible of 40 shipped (26 reached, 14 refused) 23,644 chars 0.2929 +48.44% corpus / +70.40% on the prose route 704/704

How to read that table Ratio on reached fixtures is the share of content kept, averaged over the fixtures the compressor reached; it is a reached-set diagnostic, not a headline. Tokens is corpus-wide token reduction with refusals in the denominator: a refusal saves nothing and is counted as saving nothing. The 14 refusals on long_context_v1 are refusals by design. This build cannot resolve anchors (see recovery), so it refuses fixtures the commercial build completes, and it produces the identical 0.2929 on the 26 fixtures both builds reach. Locked spans: name the build before quoting the ratio, because both builds report 1.000, over different denominators (704/704 here, 1785/1785 commercial). Both rows also share a property that the benchmark prints beside every number: routing is blind, and some fixture labels name a codec that was declared but never built (20 of synthetic’s 28 eligible, 20 of long_context_v1’s 40). Those fixtures were compressed by prose_token_compressor rather than the codec their labels specify. The report’s routing_caveat field states it per corpus, derived from the fixtures at run time.

Decide before you remove

Most compression tools shorten first and repair after. TekMyra inverts that: it works out what must survive before it removes anything, and when it cannot guarantee survival it declines. A decline is a recorded outcome, not an error, and it counts against every published number. This is a condensation of the repository’s ARCHITECTURE.md, in the order a request meets the pieces. In deployment terms the gateway is a reverse proxy for model traffic: an LLM proxy your application points at, which passes each request onward once the stages below have run.

  1. Routing: what kind of content is this?

    rule_router.py classifies deterministically by content shape; learned_router/ is the learned classifier; hybrid_router.py combines the two. The learned router may abstain from a decision. The rule router’s decision then stands and the record carries LEARNED_ABSTAINED with a reason. Disagreements resolve conservatively: the learned router may raise the risk tier but never lower it, and a lossy route proposed at elevated risk is overridden to a reversible one. router_provenance.py records which router decided and on what basis.

  2. Route classes: not every route compresses

    Requests land on one of sixteen route classes, and a route name does not tell you whether compression happened. At the time of writing the sixteen split six working codecs, five that pass through untouched (SAFE_TERMINAL_ROUTES), and five unbuilt. The unbuilt routes (UNIMPLEMENTED_CODEC_ROUTES) fail closed: they are roadmap entries, not working compressors. Both sets are derived from the codec map, so you can count them in the source rather than take this sentence on trust.

  3. Protection: what must not change?

    protected_spans.py detects spans whose exact wording carries the meaning: identifiers, amounts, dates, quoted terms. Detected spans are pinned before any rewriting is considered; every later stage receives them as constraints. Detection is regex- and rule-driven inside the module itself; the vocabulary modules that sit beside it serve fixture generation and measurement, not detection.

  4. Compression: shorten only what is safe

    Two engine families, selected by route. The deterministic codecs in tekmyra/codecs/ (json_codec.py, log_codec.py, code_codec.py, diff_codec.py, stacktrace_codec.py) give the same output for the same input every time, with no model involved. The learned compressor in tekmyra/learned_compressor/ runs segmenter.pyfeatures.pymodel.pyselector.py under a budget from compression_budget.py. The model is a versioned release artifact with a fingerprint, fetched separately; see the reference bundle.

  5. Verification: prove the spans survived

    verifier.py re-checks every pinned span against the compressed output before anything is emitted: each protected span must be represented exactly once, either verbatim, as an approved typed redaction marker, or as a token that resolves to the original. entity_fidelity.py and numeric_fidelity.py hold the span-class checks; sentence_boundaries.py guards structural cuts. safety_gate.py classifies risk and safety_merge.py applies the merge that locks spans. The 68/68 and 704/704 locked-span counts in the quickstart table are this machinery, measured.

  6. Declining: three distinct mechanisms

    “It abstains” is one phrase covering three mechanisms with three owners. The router can abstain from classifying (LEARNED_ABSTAINED; the rule router stands). The codec can decline to transform: SAFE_TERMINAL_ROUTES share a passthrough codec, and the prose codec returns its input untouched on HIGH and CRITICAL risk tiers. Verification can fail: on failure the runtime retries through a fallback route and re-verifies, and if the fallback also fails it raises TerminalVerificationError. It refuses to construct a response at all rather than ship output the verifier rejected. The benchmark counts refusals and abstentions separately, from disjoint populations, and prints them on separate lines.

  7. Recovery: what this build cannot do

    anchor_interface.py defines the anchor contract: content removed on a reversible route gets an addressable identifier rather than simply being gone. The store this repository ships, NonResolvingAnchorStore, mints identifiers and then cannot resolve them: get() returns None for every identifier, including ones it minted itself. The encrypted, tenant-partitioned, retention-bounded, retrieval-audited store is the commercial half. The consequence is a behaviour, not a gap: the open build cannot ask “was that dropped span recoverable?”, so it treats the drop as unrecoverable and refuses more than the commercial build. The 14 refusals in the quickstart table are exactly this.

  8. The open/closed seam: provable, not asserted

    What is published is decided by an allowlist over paths, computed rather than maintained: scripts/open_surface.py is that rule, and it ships, including the exclusion list naming every withheld module and why. scripts/verify_open_core.py builds the published surface from the allowlist, proves every closed module is unimportable before measuring anything, runs the suite inside the built tree, and refuses to print a number if any of that fails. Not in the repository, stated as design: the trained artifacts and training stack, the resolving anchor store, the economics gate, paid-provider adapters, tenancy and audit, and the run-telemetry emitter and console.

What the artifact bundle is, and is not

tekmyra-reference-artifacts.tar.gz is a release asset containing the two trained-artifact directories the benchmarks resolve: .cache/compressor/1.0.0 and .cache/router/1.0.0. It is verifiable as a receipt (compare its digest) and it is not reproducible from source: the training corpus and the trainer are deliberately not published, under the same rule that keeps trained artifacts closed. No tool in the payload claims otherwise.

Verify before extracting

sha-256 check: the launch release’s digest, from its release notes
shasum -a 256 tekmyra-reference-artifacts.tar.gz
9c185a524b24693213cd2a226021e466da9905fb29ab5582c8cc7480abb69883  tekmyra-reference-artifacts.tar.gz

The digest above is the one printed in the launch release’s notes (the asset is 5,879,184 bytes). The authoritative record is always the notes of the release you downloaded from: the digest is published beside the asset rather than pinned in the repository, so that you check the artifact you actually have against the record for that release, not against a constant a later release would silently falsify.

What reproduces without it

The four public corpora (benchmarks/fixtures/public_{domain,code,diff,json}_v1) ship in the repository. A clean clone, with no bundle, can verify that their fixture bytes match the committed baseline and that every third-party source is covered by NOTICE:

no bundle needed: corpus bytes and attribution
python -m pytest tests/test_public_corpus_notice.py

What needs it

The headline public-corpus figure is 62.1458% effective byte reduction over 138 fixtures (2,708,761 → 1,025,379 bytes), with 6 refusals (4.35% of fixtures, 3.55% of bytes) contributing zero saving; the on-accepted diagnostic is 64.4356%. The headline is a byte-basis measurement and is not comparable to the ratio and token figures in the quickstart table. Reproducing it requires the reference artifacts: the published bundle carries the versioned 1.0.0 directories the code loads (the baseline artifact, tests/baselines/public_corpus_baseline.json, additionally records a digest over the full private artifact directory; the README's evidence section states exactly which of its digests verify against what you have). The ratchet suite proves the rate reproduces from the bundle byte-exactly. With the bundle unpacked, run:

bundle required: the 62.1458% rate
python -m pytest tests/test_public_corpus_ratchet.py

A clean checkout has no artifacts, so the ratchet’s rate tests skip, visibly and with the fetch instructions in the skip reason: never a silent pass, and never a failure for following the README. The corpus-integrity tests still run and still bind the fixture bytes. The headline benchmark ratios in the quickstart table need the bundle for the same reason; without it the benchmark refuses to emit a headline rather than printing a flattering 1.000.

What the bundle does not change You cannot retrain these artifacts from the repository, and the bundle does not change what the anchor store can do: NonResolvingAnchorStore still resolves nothing (see recovery). The bundle unlocks measurement, not the commercial half.

Every number here can name its run

The figures on this page are the README’s, each carrying its corpus, its denominator, its basis, and which execution path produced it. The commands above are how you re-derive the ones a clone can re-derive. Figures on different bases are not comparable and are never put in one table. The paper reports the same results with the method written out.