What Is a Large Language Model, and How Does It Predict Text?

Large language model basics: tokens, training and prediction

A large language model works on tokens, which are words or word pieces. During training it reads huge amounts of text and keeps adjusting billions of internal numbers, called parameters, to guess the next token better. When you ask a question, it uses those settings to produce a likely reply, token by token.

Take the phrase “The Taj Mahal is in”. A trained model gives “Agra” a high probability as the next token, because it saw that pattern many times.

Nothing is looked up at that moment. The answer comes from patterns stored in the parameters during training.

That is both the power and the weakness. Patterns generalise well, but a pattern is not a fact check.

When Digital Magma explains this to clients, we use their own brand as the example.

Ask a chat assistant about the company with search switched off, and whatever it says comes purely from those stored patterns.

Context windows set a limit too. A model can only weigh a certain number of tokens at once, which is why very long documents get trimmed or summarised first.

LLM transformer architecture in plain words

LLM designs today mostly follow the transformer, introduced in 2017 by a team mostly from Google in the paper “Attention Is All You Need”. Its key idea is attention: when processing a word, the model weighs every other word in the passage to judge which ones matter for meaning, all in parallel.

The paper by Vaswani and colleagues proposed an architecture “based solely on attention mechanisms”. It dropped the older step-by-step designs.

The authors reported that it trained faster and translated better than earlier systems.

In “The bank raised its interest rate”, attention helps link “its” to “bank” and read “bank” as finance, not a riverside.

Stacking many attention layers is what makes a model “large”. More layers and parameters let it hold more patterns.

LLM training: pretraining, fine-tuning and alignment

LLM training happens in three broad stages. Pretraining teaches general language by predicting tokens across a huge text collection. Fine-tuning trains it further on narrower examples, such as question-and-answer pairs. Alignment adjusts its behaviour using human feedback, so replies are more helpful and safer.

• Pretraining: the longest and most expensive stage. The model learns grammar, facts and style from public and licensed text.
• Fine-tuning: shorter training on curated examples of the tasks it should do.
• Alignment: people rate answers, and the model is tuned towards the better ones.

Web pages feed the first stage. That is where training crawlers come in.

OpenAI’s crawler overview says GPTBot crawls content that “may be used in training” its foundation models.

Anthropic’s help article describes ClaudeBot the same way. Both companies honour robots.txt opt-outs for training.

Large language model examples, including models built in India

Large language model examples include OpenAI’s GPT models behind ChatGPT, Google’s Gemini, Anthropic’s Claude and Meta’s Llama. Model names and versions change every few months, so the family names are more useful than any version number. India has several of its own efforts, focused on Indian languages.

In India, Sarvam calls itself “India’s Full-Stack Sovereign AI Platform”, with models built for Indian languages.

BharatGen, linked to TIH IIT Bombay, describes multilingual LLMs for Indian languages. It says it will release some models and weights as open source.

For UPSC aspirants, the useful framing is sovereign AI. These projects aim to keep data, compute and language coverage within India.

LLM vs chatbot vs AI search engine vs foundation model

LLM is often used as a catch-all, but it names only the underlying model. A chatbot is an app wrapped around a model. An AI search engine adds live retrieval and citations. A foundation model is the wider category of large pre-trained models, which includes LLMs and models for images or audio.

Generative AI is the broadest label of all. It covers any AI that produces new content, from text to images and music.

The distinction matters in practice. Digital Magma can influence what an AI search engine retrieves about a client, but it cannot change a model’s weights.

Large language models for a business: why they matter

Large language models matter to a business because they now sit between customers and search results. When someone asks an assistant which supplier, clinic or software to pick, an LLM writes the shortlist. What it says about your brand comes from training data and from pages it retrieves live.

The training side is slow. A model only learns from text collected before its knowledge cutoff.

The retrieval side is faster. AI search tools fetch pages at question time, so current, clear pages can shape the answer this month.

Digital Magma’s work focuses on the second side, because it can be checked and changed.

LLM limits: hallucination, knowledge cutoff and bias

LLM limits come from how they work. Because they predict likely text, they can state false things fluently, which is called hallucination. They know nothing after their knowledge cutoff unless they retrieve it. And they can repeat biases present in their training text, about people, places or brands.

• Hallucination: a confident, wrong answer. Grounding in sources reduces it but does not end it.
• Knowledge cutoff: the date after which the model saw no training text.
• Bias: patterns from the training data, including stereotypes and outdated views.

Google’s Gemini API documentation frames search grounding as the fix for the cutoff problem. It lets the model “cite verifiable sources beyond its knowledge cutoff”.

Digital Magma treats a stale answer as a symptom to trace. For a brand, the cutoff explains many of them.

An old address or retired product can linger in a model’s memory for a long time.

Large language models in our client work

Large language model answers about Digital Magma’s clients are something we check, not assume. For a Toronto ENT clinic, the risk we guard against is mixed-up details, such as services or doctor names. Plain, consistent pages across the site give retrieval tools the right facts to quote.

A Canadian welding equipment manufacturer we have worked with for three years now appears in ChatGPT and AI Overviews answers. It also holds 40+ keywords in Google’s top 10.

In both cases the lever is the same. Clear public pages, crawlable by the right bots, give the models better material to work from.

Large language model: frequently asked questions

Large language model questions from students and business owners, answered briefly.

Large language model: related terms, sources and author

Large language model topics connect to these entries in the Digital Magma knowledge base:

• Retrieval-augmented generation: giving a model live documents
• LLM optimization: improving how models describe a brand
• Grounding: tying answers to sources
• ChatGPT search: an LLM with web search
• How AI chooses which brands to mention: the business follow-up

Sources: Vaswani et al. on arXiv (1706.03762), OpenAI’s crawler overview, Anthropic’s crawler help article, Google’s Gemini API grounding documentation, the Sarvam and BharatGen websites, and Wikipedia’s large language model article.

All were checked on 22 October 2026.