Quinn Avila

Tokenization engine for agentic sessions. https://quinnavila.github.io/tokmachine/ Agentic sessions are growing in context size with millions of tokens repeatedly sent and resent for re-tokenization. If we only re-encode new text plus a short tail and stitch the results. We can achieve more efficient tokenization. CPU Serial, parallel, GPU https://docs.google.com/presentation/d/1189cka65F5juo5I8xDHVbr-deTwK2dk0mCqGuALxfcA/edit?usp=sharing