https://samarthya.me/tags/tokens/Recent content in Tokens on SamarthyaHugo -- gohugo.ioen© Samarthya 2018-2026. All rights reserved.Mon, 26 Oct 2026 18:20:00 -0400https://samarthya.me/posts/ai-in-practice-3-tokens-and-tokenomics/Mon, 26 Oct 2026 18:20:00 -0400https://samarthya.me/posts/ai-in-practice-3-tokens-and-tokenomics/<div class="lead text-neutral-500 dark:text-neutral-400 !mb-9 text-xl"> Language models don&rsquo;t read words, and they don&rsquo;t bill by the request. They read and charge in tokens. Once you can count them, both the model&rsquo;s limits and the monthly invoice stop being surprises. </div> <h2 class="relative group">What a token is <div id="what-a-token-is" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-a-token-is" aria-label="Anchor">#</a> </span> </h2> <p>Before a model sees your text, a <strong>tokenizer</strong> cuts it into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Common words are one token; rarer words are split into pieces; spaces and punctuation are tokens too. Most modern tokenizers use variants of <em>byte-pair encoding</em>, which learns the most frequent fragments from training text.</p>https://samarthya.me/posts/pat-personal-access-token/Fri, 24 Feb 2023 12:00:00 +0000https://samarthya.me/posts/pat-personal-access-token/<figure><img class="my-0 rounded-md" loading="lazy" decoding="async" fetchpriority="auto" alt="Steps for a Git personal access token. Generate it, keep it safe, configure git, use the macOS keychain credential helper, then push" width="1200" height="630" src="https://samarthya.me/posts/pat-personal-access-token/an-lint_hu_23adda14bfe86d6e.png" srcset="https://samarthya.me/posts/pat-personal-access-token/an-lint_hu_23adda14bfe86d6e.png 800w, https://samarthya.me/posts/pat-personal-access-token/an-lint.png 1280w" sizes="(min-width: 768px) 50vw, 65vw" data-zoom-src="https://samarthya.me/posts/pat-personal-access-token/an-lint.png"><figcaption>Recreated in October 2026 from the post; the original image was lost</figcaption></figure> <p>Using Git Personal Access Tokens (PATs) is a secure and convenient way to authenticate with Git service providers. In this blog post, we will discuss how to use Git PATs on both Mac and CentOS 7 systems.</p>