HOW TO BUILD YOUR OWN LLM FROM SCRATCH IN 5 STAGES
Don't want to figure this out alone? I walk members through every step inside the community. Join the Skool → skool.com/raycfu
The exact pipeline behind GPT and Claude
I spent way too many hours pulling apart how models like GPT and Claude actually get built, and I compressed the entire pipeline into one map. By the end of this, you'll understand the exact five-stage path that takes raw internet text and turns it into something that can hold a conversation with you.
Here's the thing that surprised me most. Most people think building an LLM comes down to the architecture, the neural network design, the secret sauce in the transformer. That's not where the real work happens. That part is basically the same across every major lab. If architecture were the moat, everyone would already have GPT-4.
What actually separates a good model from a mediocre one is data, evaluation, and systems. Not architectural tweaks. The best models aren't just trained. They're engineered.
So that's how I'm laying this out. Five stages. Architecture gets one short section inside Stage 1. The other four are where models are actually won and lost.
Teaching the model language itself
It starts with one simple objective: predict the next word. That's it. Given a sequence of words, the model learns the probability of what comes next.
Do that across enough text and the model starts absorbing grammar, facts, and reasoning, not because anyone taught it those things directly, but because predicting the next word well requires all of it.
Before any of that happens, the text gets broken into tokens using something called Byte-Pair Encoding. That step shapes everything that comes after it.
And the architecture? The model is a transformer. That's basically the whole section, and that's the point. You don't win by inventing a smarter transformer. The data even backs this up with scaling curves: transformers just have a better constant and slope than older approaches like LSTMs. You pick the standard tool and move on to the parts that matter.
Where models are actually won
If architecture matters least, data matters most. This is the stage that separates the good models from the forgettable ones, and it's the one most people completely underestimate.
It starts with Common Crawl, a scrape of the public web so massive it's measured in petabytes. We're talking 250 billion pages and over a million gigabytes. But raw web data is a mess. Turning it into something usable is a brutal multi-step process:
Pull the text out of the HTML, including special cases like math and boilerplate
Filter out anything harmful, NSFW, or containing personal data
Deduplicate by URL, document, and line, because the web repeats itself constantly with headers, footers, and menus
Run heuristic filters to cut low-quality docs based on word count and weird tokens
Use a model to predict whether a page is good enough to be referenced by Wikipedia
Classify everything into categories like code, books, and entertainment, then reweight based on scaling laws
Here's the line worth remembering: data quality beats data quantity, every time. Collecting good data is the actual key to a usable LLM, and it's the most guarded secret in the entire field. Closed datasets dwarf the open ones. LLaMA 3 trained on 15 trillion tokens. GPT-4 reportedly trained on around 13 trillion.
Spending compute the smart way
Picture this. You've got 10,000 GPUs for a month. Do you train something bigger, or train on more data? Guess wrong and you waste millions. Scaling laws exist so you don't have to guess.
The finding is simple: more data and bigger models reliably mean better results, and you can actually predict how a model will perform based on its size and training data before you ever run it. The modern approach is to tune everything on small models first, then extrapolate up to the one massive final run.
The famous Chinchilla rule says about 20 tokens of training data per parameter is the sweet spot for training cost. But that's only half the picture. Once you factor in the cost of actually running the model afterward, that ratio jumps past 150 tokens per parameter. You train a smaller model on way more data because you're going to run it millions of times after.
And the bigger lesson underneath all of it, the one people call the "bitter lesson," is this: don't overcomplicate things. Do the simple stuff and scale it up. In the long run, the only thing that actually matters is how well you leverage compute.
Turning a predictor into something that can actually assist you
After pretraining, you've got something powerful but completely useless for a conversation. It can complete text, but it has no idea it's supposed to answer you. Ask it a question and it might just respond with three more questions, because that's a perfectly reasonable next-word continuation to it.
This is where Supervised Fine-Tuning comes in. You show the model thousands of examples, a prompt followed by a strong response, and it learns to copy that pattern. This is the step that took GPT-3 and turned it into ChatGPT.
The part that surprised me is how little data this actually takes. A few thousand examples is enough, because SFT isn't teaching the model new knowledge. That's already baked in from pretraining. It's just teaching it the shape of a good answer. The Alpaca project even generated its training data using another LLM, just 52,000 instruction-response pairs, and used it to turn a LLaMA 7B into a genuinely capable assistant.
But SFT has three problems. It's limited by how good the humans writing the examples are, it teaches the model to hallucinate by cloning answers it doesn't actually know, and writing ideal answers at scale is expensive.
That's where RLHF comes in. Instead of imitation, you optimize for preference. The model produces two answers, a human picks the better one, and those preferences train a reward model that the LLM then optimizes against, classically using something called PPO.
A simpler, more modern version called DPO gets you to comparable quality using plain supervised learning, and it's become the standard in the open-source world.
STAGE 5: EVALUATION AND SYSTEMS
Proving it works and making it possible at all
These two wrap around everything else. Skip either one and you don't actually have a real model.
On evaluation: during pretraining, the metric is perplexity, basically how many tokens the model is "torn between" at each step. Between 2017 and 2023, the best models went from hesitating among roughly 70 tokens down to fewer than 10. But perplexity stops being useful once you've done alignment, so evaluation shifts to benchmarks and head-to-head comparisons:
MMLU and HELM test the model across many domains with known correct answers. MMLU is the most trusted pretraining benchmark out there
Chatbot Arena has real humans blind-compare two models and vote, with over 300,000 votes feeding into an Elo leaderboard
AlpacaEval uses an LLM to judge other LLMs. It correlates with Chatbot Arena at 98%, takes under three minutes, and costs less than $10, but it has its own biases, like favoring longer responses
Here's the honest truth. Evaluating an aligned model is genuinely hard, and no single number tells the whole story. The exact same model can score 0.637 or 0.488 on MMLU depending purely on how the prompt is formatted.
On systems: everyone in this space is bottlenecked by compute. GPUs are expensive, hard to get, and limited by how fast they can talk to each other. A 7B parameter model needs roughly 112GB just to train without optimization. So the systems layer isn't a nice-to-have, it's what makes any of this possible:
Using lower precision, like 16-bit instead of 32-bit, which cuts memory in half and speeds things up
Fusing operations and tiling memory access, where something like FlashAttention alone gives you a roughly 1.7x speedup
Splitting the dataset across GPUs, sharding the optimizer state to save memory
Splitting the model itself across GPUs, either by layer or by matrix
Using sparsity through Mixture of Experts, where you get more parameters without more compute by only activating a subset per token
WHAT THIS WHOLE THING ACTUALLY TEACHES YOU
Walk back through all five stages and the pattern is obvious. Architecture, the thing everyone obsesses over, gets the least attention here. Data, scaling, alignment, evaluation, and systems are where every real decision actually gets made.
That's exactly why two labs can use the same architecture and end up with completely different models. The architecture is shared. Everything that actually matters is not.
THE MISTAKES THAT QUIETLY SINK MOST LLM PROJECTS
Obsessing over architecture, the most copied and least differentiating part of the whole stack
Treating data like a commodity, when dirty data caps your ceiling no matter how much compute you throw at it
Skipping the Chinchilla math and ending up with a model that's too big for the data it was trained on
Stopping at SFT, which only teaches imitation. Without RLHF or DPO, the model never actually learns what people prefer
Trusting perplexity after alignment, when post-training has already changed the distribution and the number stops meaning anything
A great model isn't trained. It's engineered.
Most people are going to keep believing this comes down to architecture, keep reading the same transformer explainers, and keep missing where the actual work happens.
The people who get this will see it clearly. Language modeling, then clean data, then smart scaling, then alignment, then honest evaluation running on efficient systems. Five stages. Architecture is one short section inside one of them.
Pick the stage you've been ignoring. For most people, that's data or evaluation. Go deep on just that one. That's where the real difference lives.
You just read the full playbook. Most people will close this tab and never implement it. The ones who do usually hit a wall around the technical setup and quit.
Inside the Skool, I walk you through the exact build step-by-step, troubleshoot your setup live in the community, and share the scripts and templates I use to actually land paying clients.
If you want the shortcut instead of the long way around: