Documentation · launch release
Documentation
Everything below is checkable against the repository: the commands are the ones the README ships, the component names are the ones in the source tree, and every number is reported with its corpus, its denominator and its basis.
Quickstart
Install, fetch a model, run the benchmark
TekMyra is a Python library and benchmark harness. The package that ships is
tekmyra-core; the import name is tekmyra. The README leads
with the from-source install, and so does this page: it is the path that reproduces
the published figures.
Install from source (primary)
Dependencies are declared as ranges so TekMyra can live inside your application;
ranges float, so the published figures were measured against the exact versions
recorded in constraints-reproduce.txt. Install against the constraints
file if you want to reproduce the numbers rather than only install the library:
pip install -c constraints-reproduce.txt -e .
pip install -e '.[dev]' # the library plus pytest, to run the suite below
If you do not want the test dependencies, pip install -e . installs the
library alone.
Core dependencies are click, numpy, pydantic,
scikit-learn, scipy, tiktoken and
pyyaml. tiktoken is a core dependency, not an extra:
token-reduction figures are unobtainable without it.
Install from PyPI
pip install tekmyra-core
Getting a model
Trained model artifacts are not in the repository and never will be: they are the
commercial half. The reference artifact bundle is published as a release asset,
tekmyra-reference-artifacts.tar.gz, with its SHA-256 printed in the
release notes. Download it, check the digest before extracting, and unpack it at the
root of your clone:
curl -L -O https://github.com/laconiq-ai/tekmyra/releases/latest/download/tekmyra-reference-artifacts.tar.gz
shasum -a 256 tekmyra-reference-artifacts.tar.gz # compare against the release notes
tar xzf tekmyra-reference-artifacts.tar.gz # unpacks .cache/compressor/1.0.0 and .cache/router/1.0.0
The artifact directories are resolved in this order, and a failure prints every
location it tried: $TEKMYRA_ARTIFACT_ROOT/.cache/… if that
variable is set, then .cache/… under your working directory, then
.cache/… beside the installed package. Details on what the bundle
contains and what it does and does not unlock are in the
reference bundle section.
Run the benchmark
python -m tekmyra.benchmark compressor --fixture-dir benchmarks/fixtures/synthetic
python -m tekmyra.benchmark compressor --fixture-dir benchmarks/fixtures/long_context_v1
python -m pytest tests/test_public_corpus_notice.py # public bytes + attribution
python -m pytest tests # the suite, green without a model
python scripts/verify_open_core.py # exits 6 on a clean clone: seam proven, numbers await the bundle
The test suite passes without a model: the tests that need a trained artifact skip, and each skip names every location that was searched.
Without a model, the benchmark refuses to give you a number
That is deliberate rather than a bug, and it follows the same rule as the verifier:
Error: Learned router artifact did not load — refusing to produce a benchmark
number. artifact_dir='<your clone>/.cache/router/1.0.0' exists=False
backend='none'.
could not resolve '.cache/router/1.0.0'; searched, in order:
absent <your clone>/.cache/router/1.0.0
Set $TEKMYRA_ARTIFACT_ROOT to the directory holding .cache/ if your artifacts
live elsewhere. Re-run with --allow-rules-only to record an explicitly
rules-only run; such a run cannot emit a headline compression ratio.
Take the escape hatch it offers and the run records exactly what it was: the headline is withheld (not zero and not one, withheld), token reduction is 0.00%, and 68 of 68 locked spans are still preserved across 28 reached fixtures. The safety machinery works without a model. The compression does not.
The numbers, with their denominators
With the reference artifact bundle, on the corpora in the repository, via the
joblib/scikit-learn path (no onnxruntime installed):
| corpus | fixtures | mean size | headline ratio | tokens | locked spans |
|---|---|---|---|---|---|
synthetic |
28 eligible of 301 shipped (28 reached, 0 refused) | 250 chars (over 301 shipped) | 0.7409 | +25.86% | 68/68 |
long_context_v1 |
40 eligible of 40 shipped (26 reached, 14 refused) | 23,644 chars | 0.2929 | +48.44% corpus / +70.40% on the prose route | 704/704 |
How to read that table
Ratio on reached fixtures is the share of content kept, averaged over the
fixtures the compressor reached; it is a reached-set diagnostic, not a headline. Tokens is corpus-wide token reduction with refusals in
the denominator: a refusal saves nothing and is counted as saving nothing. The
14 refusals on long_context_v1 are refusals by design. This build cannot
resolve anchors (see recovery), so it
refuses fixtures the commercial build completes, and it produces the identical 0.2929
on the 26 fixtures both builds reach. Locked spans: name the build before quoting the
ratio, because both builds report 1.000, over different denominators (704/704 here,
1785/1785 commercial). Both rows also share a property that the benchmark prints
beside every number: routing is blind, and some fixture labels name a codec that was
declared but never built (20 of synthetic’s 28 eligible, 20 of
long_context_v1’s 40). Those fixtures were compressed by
prose_token_compressor rather than the codec their labels specify. The
report’s routing_caveat field states it per corpus, derived from
the fixtures at run time.
Architecture
Decide before you remove
Most compression tools shorten first and repair after. TekMyra inverts that: it works
out what must survive before it removes anything, and when it cannot
guarantee survival it declines. A decline is a recorded outcome, not an error, and it
counts against every published number. This is a condensation of the repository’s
ARCHITECTURE.md, in the order a request meets the pieces.
In deployment terms the gateway is a reverse proxy for model traffic: an LLM proxy
your application points at, which passes each request onward once the stages below
have run.
-
Routing: what kind of content is this?
rule_router.pyclassifies deterministically by content shape;learned_router/is the learned classifier;hybrid_router.pycombines the two. The learned router may abstain from a decision. The rule router’s decision then stands and the record carriesLEARNED_ABSTAINEDwith a reason. Disagreements resolve conservatively: the learned router may raise the risk tier but never lower it, and a lossy route proposed at elevated risk is overridden to a reversible one.router_provenance.pyrecords which router decided and on what basis. -
Route classes: not every route compresses
Requests land on one of sixteen route classes, and a route name does not tell you whether compression happened. At the time of writing the sixteen split six working codecs, five that pass through untouched (
SAFE_TERMINAL_ROUTES), and five unbuilt. The unbuilt routes (UNIMPLEMENTED_CODEC_ROUTES) fail closed: they are roadmap entries, not working compressors. Both sets are derived from the codec map, so you can count them in the source rather than take this sentence on trust. -
Protection: what must not change?
protected_spans.pydetects spans whose exact wording carries the meaning: identifiers, amounts, dates, quoted terms. Detected spans are pinned before any rewriting is considered; every later stage receives them as constraints. Detection is regex- and rule-driven inside the module itself; the vocabulary modules that sit beside it serve fixture generation and measurement, not detection. -
Compression: shorten only what is safe
Two engine families, selected by route. The deterministic codecs in
tekmyra/codecs/(json_codec.py,log_codec.py,code_codec.py,diff_codec.py,stacktrace_codec.py) give the same output for the same input every time, with no model involved. The learned compressor intekmyra/learned_compressor/runssegmenter.py→features.py→model.py→selector.pyunder a budget fromcompression_budget.py. The model is a versioned release artifact with a fingerprint, fetched separately; see the reference bundle. -
Verification: prove the spans survived
verifier.pyre-checks every pinned span against the compressed output before anything is emitted: each protected span must be represented exactly once, either verbatim, as an approved typed redaction marker, or as a token that resolves to the original.entity_fidelity.pyandnumeric_fidelity.pyhold the span-class checks;sentence_boundaries.pyguards structural cuts.safety_gate.pyclassifies risk andsafety_merge.pyapplies the merge that locks spans. The 68/68 and 704/704 locked-span counts in the quickstart table are this machinery, measured. -
Declining: three distinct mechanisms
“It abstains” is one phrase covering three mechanisms with three owners. The router can abstain from classifying (
LEARNED_ABSTAINED; the rule router stands). The codec can decline to transform:SAFE_TERMINAL_ROUTESshare a passthrough codec, and the prose codec returns its input untouched on HIGH and CRITICAL risk tiers. Verification can fail: on failure the runtime retries through a fallback route and re-verifies, and if the fallback also fails it raisesTerminalVerificationError. It refuses to construct a response at all rather than ship output the verifier rejected. The benchmark counts refusals and abstentions separately, from disjoint populations, and prints them on separate lines. -
Recovery: what this build cannot do
anchor_interface.pydefines the anchor contract: content removed on a reversible route gets an addressable identifier rather than simply being gone. The store this repository ships,NonResolvingAnchorStore, mints identifiers and then cannot resolve them:get()returnsNonefor every identifier, including ones it minted itself. The encrypted, tenant-partitioned, retention-bounded, retrieval-audited store is the commercial half. The consequence is a behaviour, not a gap: the open build cannot ask “was that dropped span recoverable?”, so it treats the drop as unrecoverable and refuses more than the commercial build. The 14 refusals in the quickstart table are exactly this. -
The open/closed seam: provable, not asserted
What is published is decided by an allowlist over paths, computed rather than maintained:
scripts/open_surface.pyis that rule, and it ships, including the exclusion list naming every withheld module and why.scripts/verify_open_core.pybuilds the published surface from the allowlist, proves every closed module is unimportable before measuring anything, runs the suite inside the built tree, and refuses to print a number if any of that fails. Not in the repository, stated as design: the trained artifacts and training stack, the resolving anchor store, the economics gate, paid-provider adapters, tenancy and audit, and the run-telemetry emitter and console.
Reference bundle
What the artifact bundle is, and is not
tekmyra-reference-artifacts.tar.gz is a release asset containing the two
trained-artifact directories the benchmarks resolve:
.cache/compressor/1.0.0 and .cache/router/1.0.0. It is
verifiable as a receipt (compare its digest) and it is not reproducible
from source: the training corpus and the trainer are deliberately not published, under
the same rule that keeps trained artifacts closed. No tool in the payload claims
otherwise.
Verify before extracting
shasum -a 256 tekmyra-reference-artifacts.tar.gz
9c185a524b24693213cd2a226021e466da9905fb29ab5582c8cc7480abb69883 tekmyra-reference-artifacts.tar.gz
The digest above is the one printed in the launch release’s notes (the asset is 5,879,184 bytes). The authoritative record is always the notes of the release you downloaded from: the digest is published beside the asset rather than pinned in the repository, so that you check the artifact you actually have against the record for that release, not against a constant a later release would silently falsify.
What reproduces without it
The four public corpora
(benchmarks/fixtures/public_{domain,code,diff,json}_v1) ship in the
repository. A clean clone, with no bundle, can verify that their fixture bytes match
the committed baseline and that every third-party source is covered by
NOTICE:
python -m pytest tests/test_public_corpus_notice.py
What needs it
The headline public-corpus figure is 62.1458% effective byte
reduction over 138 fixtures (2,708,761 → 1,025,379 bytes), with
6 refusals (4.35% of fixtures, 3.55% of bytes) contributing zero
saving; the on-accepted diagnostic is 64.4356%. The headline is a byte-basis
measurement and is not comparable to the ratio and token figures in the quickstart
table. Reproducing it requires the reference artifacts: the published bundle carries
the versioned 1.0.0 directories the code loads (the baseline artifact,
tests/baselines/public_corpus_baseline.json, additionally records a
digest over the full private artifact directory; the README's evidence section states
exactly which of its digests verify against what you have). The ratchet suite proves
the rate reproduces from the bundle byte-exactly. With the bundle unpacked, run:
python -m pytest tests/test_public_corpus_ratchet.py
A clean checkout has no artifacts, so the ratchet’s rate tests skip, visibly and with the fetch instructions in the skip reason: never a silent pass, and never a failure for following the README. The corpus-integrity tests still run and still bind the fixture bytes. The headline benchmark ratios in the quickstart table need the bundle for the same reason; without it the benchmark refuses to emit a headline rather than printing a flattering 1.000.
What the bundle does not change
You cannot retrain these artifacts from the repository, and the bundle does not
change what the anchor store can do: NonResolvingAnchorStore still
resolves nothing (see recovery). The bundle unlocks
measurement, not the commercial half.
Check us
Every number here can name its run
The figures on this page are the README’s, each carrying its corpus, its denominator, its basis, and which execution path produced it. The commands above are how you re-derive the ones a clone can re-derive. Figures on different bases are not comparable and are never put in one table. The paper reports the same results with the method written out.