“What is a token, and how is it used?”
Someone asked me that at a party on Saturday. It’s a fair question, and I wish more business owners asked it before signing an AI contract. The answer explains how AI tools are priced. It also explains why they seem to forget what you told them twenty minutes ago.
Here’s the plain-English version, and the part that matters if you run a business.
What a Token Is
AI models don’t read words the way you and I do. They break text into small chunks called tokens. A common word is usually one token. A long word, a product code, or a company name might be split into several pieces.
A useful rule of thumb for English: 100 tokens is roughly 75 words. A one-page memo runs somewhere around 500 to 700 tokens.
The model also writes in tokens. When you watch a chatbot type out its answer piece by piece, you’re watching it predict one token at a time, each one based on everything that came before it.
How Tokens Are Used
Tokens do two jobs that matter to a business owner. One gets all the attention. The other causes most of the frustration.
They’re the Unit You Pay For
Most AI providers charge by the token, for both what you send in and what the model sends back. That’s the part people hear about, and I covered it in AI Got 280x Cheaper. Your Bill Didn’t. I won’t repeat it here.
They Limit What the AI Can See
Every model has a context window. That’s the maximum number of tokens it can work with at one time.
Everything competes for that space. Your question. The instructions the vendor wrote behind the scenes. Every earlier message in the conversation. Any document you attached. And the answer the model is about to write.
When the window fills, something gets cut. Depending on the tool, older messages are dropped, compressed into a summary, or trimmed without notice. The model keeps answering from whatever is left.
That’s what people describe as the AI “forgetting.” Nothing was forgotten. It just isn’t in view anymore.
A Shift Handoff, Not a Memory
The best comparison I have comes from running global operations at Kyndryl, with more than 200 engineers spread across time zones.
Work moved between shifts through handoff notes. The incoming team knew what the note said and nothing else. If a detail didn’t make it into the handoff, it didn’t exist for the next eight hours, no matter how important it was to the person who went home.
A context window works the same way. Every time you send a message, the model gets a fresh handoff. Whatever made it into the window, it can use. Whatever didn’t, it never sees.
Bigger Windows Help Less Than You’d Expect
Context windows have grown a lot. Some models can now take in the equivalent of several books at once. That helps, but it doesn’t solve the problem.
A 2023 study led by Stanford researchers found that language models use information at the beginning and end of a long input far better than information buried in the middle. More room doesn’t mean equal attention to every page.
Size also costs money. Everything in the window gets processed on every request. Paste in a 200-page manual to ask one question about returns, and you pay for all 200 pages, every time you ask.
Why Uploading Every Document Rarely Works
This is where a party question turns into a business problem.
A common first move goes like this: load every policy, procedure, contract, and spreadsheet into an AI tool and start asking questions. It feels like handing the AI the whole company. But the model can still only work with what fits in the window for that one question.
Tools that handle large document collections deal with this using a step called retrieval. Before the model answers, a search system pulls the few passages that look most relevant and puts only those in the window. It’s the right design. It also means your answer now depends on whether that search grabbed the right pages.
What the Search Step Finds in a Real Business
Across our active clients, the same mess shows up. Three versions of a pricing policy with no label saying which one is current. A customer address that differs between the CRM and the billing system. A procedure that lives only in one manager’s sent folder. File permissions nobody has reviewed in years.
The search step can’t tell which pricing policy is current. It grabs whichever passage best matches the words in your question. If last year’s version matches better, last year’s version is what the model reads. You get a confident, well-written answer based on the wrong document.
The model did exactly what it was built to do. Nobody had decided which page was right. I made the broader case for this in You Don’t Have an AI Problem. You Have a Data Problem. Tokens are the mechanics underneath it.
Before You Blame the Tool
Most of the fixes here aren’t technical. They’re housekeeping and habits.
Clean Up What the AI Will Read
Pick one current version of each policy, price list, and procedure, and archive or clearly label the rest. It’s tedious work. It will improve your AI answers more than upgrading to a bigger model.
Do the same with permissions before you connect anything. An assistant linked to your shared drives can pull up whatever that connection can reach. If someone in the warehouse can technically open the salary spreadsheet today, assume the AI can surface it tomorrow.
Ask the Vendor Two Questions
First: what happens when a conversation or document set is larger than the context window? Second: can I see which sources each answer came from?
The second one matters most. If you can see the sources, your team can catch the outdated policy before it reaches a customer. If you can’t, you’re trusting the search step blind.
Change One Habit on Your Team
When the topic changes, start a new conversation. A thread that has wandered through five subjects is full of material that no longer applies, and it crowds out what does. A fresh chat with the three key facts restated at the top will usually give a better answer than message forty of an old one.
Back to Saturday
So, what is a token and how is it used? It’s a small piece of text. AI reads it, writes it, and bills by it. And the number of tokens a model can hold at once decides what it can see when it answers you.
That last part is the one worth acting on. If your AI tools keep getting things wrong, look hard at what you’re putting in front of them before you go shopping for a smarter model.
Not sure what your AI would find in your files? Our AI Infrastructure Assessment maps your data sources, integration gaps, and permission risks before you commit to a tool. Start the conversation →
See how we work. From assessment through implementation and adoption, every engagement starts with the foundation. Explore our services →
Keep reading. More practical guidance for business leaders is on the Summit AI blog →
Russell Love is the Founder & CEO of Summit AI Business Solutions, based in Browns Summit, NC. With 20+ years of enterprise transformation experience at IBM and Kyndryl, Russell helps businesses build the foundations that make AI actually work.