How to Build an AI Picture Book
1 · Call a model APIhours · a prompt and a few lines of code 2 · Add your data (RAG, tools)days · retrieval, function calls, evals 3 · Fine-tune a modelweeks · labelled data, GPUs, evals 4 · Train from scratchmillions of dollars more cost, control, risk
PLATE 0 — Start at the bottom of the ladder. Climb only when you must.
A picture book for grown-ups · with code

How to Build an AI

Nine spreads, from how machines learn, to training a small model yourself, to building an app on a large language model with your own data, tools, and tests.

Say it like a pro

"We start with a hosted model and a clear task, ground it with retrieval and tools, measure it with an eval set, and only fine-tune if the numbers say we must."

Examples use Python. Use the tabs or the ← → keys.

Traditional programming Rules Data Answers Machine learning Data Answers Rules (model) Three kinds of learning supervised: labelled examples (spam / not spam, house price) unsupervised: find structure (clusters) · reinforcement: learn from reward
PLATE 1 — Instead of writing the rules, show examples and let the computer find them.
Chapter 1 · How machines learn

A model is a function learned from examples

A model has parameters (numbers). Training adjusts them to reduce a loss: how wrong the predictions are on known examples. Then you use the model on new data (inference).

TaskTypeExample
predict a numberregressionhouse price
pick a categoryclassificationfraud / not fraud
group similar thingsclusteringcustomer segments
generate text/imagesgenerativechat, summaries
$ python -m venv .venv && source .venv/bin/activate
$ pip install numpy pandas scikit-learn matplotlib jupyter
feature
an input column the model can use
label
the answer you want it to predict
overfitting
memorizing training data instead of learning the pattern
Say it like a pro

"It's supervised classification. We minimize loss on training data and judge it on held-out data."

All your labelled data Train · 70% (learn from) Val15% Test15% 🔒 tune choicesfinal grade, once Garbage in, garbage out data quality matters more than model choice leakage: test info sneaking into training makes scores a lie bias: if the data under-represents a group, the model will too privacy: remove or protect personal data before you train
PLATE 2 — Keep an untouched test set. It's the only honest score.
Chapter 2 · Data

Collect, clean, split

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv("customers.csv")
print(df.shape); print(df.head()); print(df.isna().sum())      # look at it first!

df = df.dropna(subset=["churned"])                              # need labels
X = df.drop(columns=["churned"])
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)           # keep class balance
print(y_train.mean(), y_test.mean())
  1. Define the target precisely: what exactly is being predicted, and when?
  2. Explore: distributions, missing values, duplicates, outliers.
  3. Clean and encode: fill or drop gaps; turn categories into numbers.
  4. Split before you scale or select features, to avoid leakage.
  5. Version your data (DVC, or dated files) so runs are reproducible.
Say it like a pro

"Stratified split, no leakage, a locked test set, and a baseline to beat."

1 predict 2 loss 3 adjust repeat train val gap opens → overfitting → stop earlier Fixing overfitting more (or better) data simpler model regularization · dropout early stopping cross-validation Fixing underfitting more features bigger / smarter model train longer Always beat a dumb baseline first
PLATE 3 — Learning is a loop: guess, check, nudge, repeat.
Chapter 3 · Train and evaluate

Your first real model, in 15 lines

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.dummy import DummyClassifier
from sklearn.metrics import classification_report, roc_auc_score

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("baseline acc", baseline.score(X_test, y_test))       # the bar to beat

model = make_pipeline(StandardScaler(), RandomForestClassifier(n_estimators=300, random_state=42))
model.fit(X_train, y_train)

pred = model.predict(X_test)
print(classification_report(y_test, pred))                   # precision, recall, F1
print("AUC", roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

import joblib; joblib.dump(model, "churn.joblib")            # save it
accuracy
misleading when classes are imbalanced (99% "no fraud")
precision / recall
of flagged items, how many were right / of real ones, how many did you catch
cross-validation
average over several splits for a steadier score
Say it like a pro

"We beat the baseline on held-out data, and we picked the metric that matches the cost of mistakes."

inputshidden layersoutput forward pass → prediction ← backpropagation: send the error back, nudge each weight each neuron: weighted sum → activation
PLATE 4 — Layers of simple units, trained by passing the error backwards.
Chapter 4 · Neural networks

Stack simple units, learn by gradient descent

A neuron computes activation(w·x + b). Stack thousands of them in layers and the network can learn very complex patterns (images, speech, language). Training uses gradient descent: nudge every weight in the direction that lowers the loss.

pip install torch   # PyTorch

import torch, torch.nn as nn
Xt = torch.tensor(X_train.values, dtype=torch.float32)
yt = torch.tensor(y_train.values, dtype=torch.float32).unsqueeze(1)

net = nn.Sequential(nn.Linear(Xt.shape[1], 32), nn.ReLU(),
                    nn.Linear(32, 16), nn.ReLU(),
                    nn.Linear(16, 1))
opt  = torch.optim.Adam(net.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()

for epoch in range(50):
    opt.zero_grad()
    loss = loss_fn(net(Xt), yt)       # forward + loss
    loss.backward()                   # backpropagation
    opt.step()                        # gradient descent step
    if epoch % 10 == 0: print(epoch, loss.item())
tabular data
gradient-boosted trees (XGBoost, LightGBM) usually beat neural nets
images / text / audio
neural nets win; start from a pretrained model, don't train from zero
GPU
does the matrix math in parallel; Colab and Kaggle give free ones
Say it like a pro

"Trees for tables, pretrained networks for perception and language, and transfer learning before training from scratch."

"The catsat on" tokens[464, 2415…] vectorsembeddings Transformerattention × N layers next-token odds the 41% a 20% mat 9%… pick one, append, repeat What that means in practice it predicts likely text, so it can be fluent and wrong (hallucination) attention lets every token look at every other token in the context window temperature controls randomness: low = focused, high = creative it knows only its training data plus what you put in the prompt
PLATE 5 — An LLM is a very good next-token predictor, run in a loop.
Chapter 5 · Large language models

How chatbots actually work

An LLM is a transformer neural network trained on huge amounts of text to predict the next token. Then it's tuned to follow instructions and be helpful and safe. You rarely train one; you use one.

TermMeaning
tokena chunk of text, roughly ¾ of a word
context windowhow many tokens it can read at once (prompt + reply)
system promptstanding instructions for role, tone, rules
temperaturerandomness of sampling
embeddinga vector that captures meaning; similar text → nearby vectors
fine-tuningextra training on your examples to change style or skill
limits
can be confidently wrong, has a knowledge cutoff, can't see your private data unless you provide it
cost
billed per input and output token; shorter prompts and caching save money
Say it like a pro

"It's a next-token predictor, so I give it the facts it needs in context and verify the output."

Browser Your serverholds the API key 🔑 Model APIClaude · others stream tokens back A request has four parts model · which model to use system · role, rules, format ("You are a support agent for Acme…") messages · the conversation so far (you resend it each time) max_tokens · cap on the reply length Server-side only never ship API keys to the browser · rate-limit and log per user
PLATE 6 — The model is a service. Your code supplies the context and handles the answer.
Chapter 6 · Build with an LLM API

A working assistant in minutes

$ pip install anthropic
$ export ANTHROPIC_API_KEY="..."      # from console.anthropic.com; never commit it
import anthropic
client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY

SYSTEM = "You are a concise support assistant for Acme. If unsure, say so. Answer in under 120 words."
history = []

def ask(user_text):
    history.append({"role": "user", "content": user_text})
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=500,
        system=SYSTEM,
        messages=history,
    )
    text = resp.content[0].text
    history.append({"role": "assistant", "content": text})
    return text

print(ask("How do I reset my password?"))

# streaming for a snappy UI
with client.messages.stream(model="claude-sonnet-5-5", max_tokens=500,
                            messages=[{"role": "user", "content": "Explain RAG in two lines"}]) as s:
    for chunk in s.text_stream: print(chunk, end="", flush=True)
prompting
be specific, give examples, say the output format, and put the task last in long inputs
structured output
ask for JSON matching a schema, then validate it before using it
costs
trim history, cache repeated prefixes, use a smaller model for simple steps
Say it like a pro

"System prompt for behaviour, streamed responses, validated structured output, and the key stays on the server."

Index (once) docs chunk embed vector DB Answer (every question) question embed top-k similar chunks prompt = question + chunks"answer only from these sources" LLM → answer + citations gives the model fresh, private facts without retraining · cites sources you can checkquality lives in chunking, retrieval, and the "say I don't know" instruction
PLATE 7 — Don't retrain the model on your docs. Look the relevant bits up and hand them over.
Chapter 7 · Your data: embeddings & RAG

Make it answer from your documents

# minimal RAG with an in-memory index (swap in pgvector / Chroma / Qdrant at scale)
import numpy as np
from sentence_transformers import SentenceTransformer     # pip install sentence-transformers
emb = SentenceTransformer("all-MiniLM-L6-v2")

chunks = [c for doc in load_docs() for c in split(doc, size=800, overlap=100)]
vecs = emb.encode(chunks, normalize_embeddings=True)

def retrieve(q, k=4):
    qv = emb.encode([q], normalize_embeddings=True)[0]
    top = np.argsort(vecs @ qv)[::-1][:k]                  # cosine similarity
    return [chunks[i] for i in top]

def answer(q):
    ctx = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(retrieve(q)))
    return client.messages.create(
        model="claude-sonnet-5-5", max_tokens=600,
        system="Answer ONLY from the sources. Cite like [1]. If they don't contain the answer, say you don't know.",
        messages=[{"role": "user", "content": f"Sources:\n{ctx}\n\nQuestion: {q}"}],
    ).content[0].text
chunking
too big dilutes meaning; too small loses context; overlap helps
hybrid search
combine keyword (BM25) and vector search, then rerank
permissions
filter retrieved chunks by what the asking user may see
Say it like a pro

"Retrieval first, a grounded prompt that allows 'I don't know', citations, and per-user access filters."

1 model thinks 2 asks for tool 3 your code runs repeat until"final answer" search database email code The model proposes. Your code disposes.validate arguments, limit permissions, confirm risky actions
PLATE 8 — Give the model tools, run them yourself, feed back results.
Chapter 8 · Tools & agents

From answering to doing

tools = [{
  "name": "get_order_status",
  "description": "Look up an order's status by its ID.",
  "input_schema": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}]

messages = [{"role": "user", "content": "Where is order A1042?"}]
while True:
    r = client.messages.create(model="claude-sonnet-5-5", max_tokens=800, tools=tools, messages=messages)
    messages.append({"role": "assistant", "content": r.content})
    if r.stop_reason != "tool_use":
        print(r.content[0].text); break
    results = []
    for block in r.content:
        if block.type == "tool_use":
            out = get_order_status(block.input["order_id"])   # YOUR code, YOUR checks
            results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(out)})
    messages.append({"role": "user", "content": results})
  • Least privilege: read-only tools by default; require human approval for anything destructive.
  • Treat tool output and documents as untrusted: text inside them may try to give the model instructions (prompt injection).
  • Cap the loop: max steps, max tokens, timeouts, budgets.
  • Log every step so you can replay what happened.
Say it like a pro

"Narrow, validated tools, human approval for risky actions, a step budget, and full traces."

Eval set50–500 cases Your systemprompt+RAG+tools Scorerules · LLM judge · human change → re-run → compare Before you ship ☐ accuracy on a held-out eval set you wrote from real questions ☐ tested on edge cases: empty, hostile, off-topic, multi-language ☐ prompt-injection and data-leak tests · PII handling ☐ fallback when the model errors or is unsure ☐ cost per request, latency (p95), rate limits ☐ logging + a thumbs up/down to grow the eval set ☐ humans in the loop for high-stakes decisions
PLATE 9 — If you can't measure it, you can't improve it, and you can't trust it.
Chapter 9 · Evaluate, secure & ship

Test it like software, because it is

# a tiny eval harness: run, score, compare versions
import json
cases = json.load(open("evals.json"))      # [{"q": "...", "must_include": ["30 days"], "must_not": ["guarantee"]}]

def run_eval(answer_fn):
    passed = 0
    for c in cases:
        a = answer_fn(c["q"]).lower()
        ok = all(s in a for s in c.get("must_include", [])) and not any(s in a for s in c.get("must_not", []))
        passed += ok
        if not ok: print("FAIL:", c["q"])
    print(f"{passed}/{len(cases)} passed")
run_eval(answer)                           # re-run after every prompt or retrieval change
  1. Write the eval set first, from real questions and known-bad cases.
  2. Change one thing at a time (prompt, chunk size, model) and re-score.
  3. Guard the edges: validate inputs and outputs, filter sensitive data, keep keys server-side.
  4. Monitor in production: sample real conversations, track failures, and feed them back into the eval set.
  5. Be honest with users that it's AI, and show sources where you can.
next steps
fine-tuning only if evals show a gap prompts can't close · model routing · prompt caching · human review queues
Say it like a pro

"An eval suite gates every change, we log and sample production, and risky actions keep a human in the loop."