https://samarthya.me/tags/tokenomics/Recent content in Tokenomics on SamarthyaHugo -- gohugo.ioen© Samarthya 2018-2026. All rights reserved.Mon, 26 Oct 2026 18:20:00 -0400https://samarthya.me/posts/ai-in-practice-3-tokens-and-tokenomics/Mon, 26 Oct 2026 18:20:00 -0400https://samarthya.me/posts/ai-in-practice-3-tokens-and-tokenomics/<div class="lead text-neutral-500 dark:text-neutral-400 !mb-9 text-xl"> Language models don&rsquo;t read words, and they don&rsquo;t bill by the request. They read and charge in tokens. Once you can count them, both the model&rsquo;s limits and the monthly invoice stop being surprises. </div> <h2 class="relative group">What a token is <div id="what-a-token-is" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-a-token-is" aria-label="Anchor">#</a> </span> </h2> <p>Before a model sees your text, a <strong>tokenizer</strong> cuts it into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Common words are one token; rarer words are split into pieces; spaces and punctuation are tokens too. Most modern tokenizers use variants of <em>byte-pair encoding</em>, which learns the most frequent fragments from training text.</p>