AI Chatbot

Learning Objectives

By the end of this project, you will be able to:

• Call a chat completion API and explain every field in the request and response (model, messages, system, max_tokens, temperature, stop_reason, usage).
• Implement multi-turn conversation by correctly maintaining and resending the messages array, given that the API is stateless.
• Explain why context windows exist and predict what happens when a conversation exceeds one.
• Stream tokens to the UI using server-sent events instead of waiting for the full response.
• Calculate the exact cost of any API call from usage and per-MTok pricing, and project costs at scale.
• Design a system prompt that reliably changes model behavior, and distinguish it from a user message.
• Switch LLM providers without changing application logic, using LiteLLM's unified completion() interface.
• Identify when a chatbot is the right architecture — and when a single-shot call or a different pattern is better.

1. Motivation

Why This Exists

A raw language model is a function: text in, next-token-probabilities out. It has no concept of "a conversation," no memory, and no notion of who is speaking. The messages array standardized dialogue into a structured, role-tagged format that every major provider now speaks — so you stop hand-formatting prompt strings and re-parsing output every turn.

The Solution (and Its Consequences)

The fix is counterintuitive: you give the model memory by resending the entire conversation every single time. The model re-reads the whole history on every turn. This one fact explains why long chats get slow, why they get expensive, and why context windows matter.

Real-World Stakes

Every RAG system, coding copilot, and agent is a chatbot underneath. Cursor, Claude.ai, ChatGPT, support bots — all manage a messages array, a system prompt, and a context budget. A team that doesn't track tokens ships a product that quietly costs 10× what it should.

2. Mental Model

Explain Like I'm 12

Imagine a brilliant friend with total amnesia. Every time you talk, they forget everything the instant the conversation ends. To have a real conversation, you hand them a notebook with the entire chat so far written down. They read the whole notebook, say one new thing, and forget again. To continue, you write their new sentence into the notebook and hand the whole thing back. The notebook is the messages array. The friend is the model.

Explain Like I'm a Software Engineer

The chat API is a pure function over conversation state: f(messages, system, params) → next_message. No session, no cookie, no server-side state. You maintain the state client-side as an append-only list of {role, content} dicts. Each request sends the full list; the response is one assistant message you append before the next user turn.

This is the same shape as a reducer: state = reduce(state, action) where actions alternate user/assistant. The model is trained on strictly alternating roles. The system prompt is not a message in the array — it's a separate top-level field that conditions every turn.

Real-World Analogy

How It Works (Diagram)

Turn 1 request:                      Turn 2 request (resends everything):
  system: "You are helpful."           system: "You are helpful."
  messages:                            messages:
    [user] "Capital of France?"          [user] "Capital of France?"
                                         [assistant] "Paris."     ← appended from turn 1
  →  [assistant] "Paris."                [user] "Its population?"  ← new
                                       →  [assistant] "About 2.1M."

Each turn re-sends all prior turns. input_tokens grows monotonically.

3. Technical Explanation

Formal Definition

A chat completion is a request to POST /v1/messages containing a model, a max_tokens cap, and a messages array of {role: "user"|"assistant", content} objects — optionally a top-level system string and sampling params (temperature, top_p, stop_sequences). The response is a Message with content, a stop_reason, and a usage object reporting input_tokens and output_tokens. [anthropic-messages-api]

How It Works, Step by Step

1. You assemble messages (full history) + system + params.
2. The provider tokenizes the entire input. The token count = input_tokens.
3. The model generates output tokens one at a time until a natural stop, max_tokens, or a stop_sequence — recorded in stop_reason.
4. Non-streaming: the whole Message at once. Streaming: tokens arrive as SSE events (message_start → content_block_delta* → message_stop).
5. You read usage to compute cost, append the assistant message to history, and wait for the next user turn.

The Math — Why Long Chats Get Expensive

cost = input_tokens/1e6 * input_price_per_MTok
     + output_tokens/1e6 * output_price_per_MTok

Across an N-turn conversation where each turn adds ~t tokens, turn k resends ~k·t tokens, so total input ≈ t·N(N+1)/2 — quadratic-ish. This is why long conversations get disproportionately expensive and why context management exists.

Key Concepts

4. Guided Examples

Example 1: Simplest Case — one stateless call

A single call carries no memory. The usage fields are your cost ground-truth.

import litellm

response = litellm.completion(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=100,
)
print(response.choices[0].message.content)        # "The capital of France is Paris."
print(response.usage.prompt_tokens,                # input_tokens
      response.usage.completion_tokens)            # output_tokens

Example 2: Real-World Case — multi-turn loop with system prompt

The second question only works because the assistant's prior answer was appended to history.

import litellm

SYSTEM = "You are a concise assistant. Answer in one sentence."
history = [{"role": "system", "content": SYSTEM}]   # LiteLLM puts system in messages

def ask(user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    resp = litellm.completion(
        model="claude-sonnet-4-6", messages=history, max_tokens=200, temperature=0.7,
    )
    answer = resp.choices[0].message.content
    history.append({"role": "assistant", "content": answer})   # CRITICAL: persist the turn
    return answer

print(ask("What is the capital of France?"))   # "Paris."
print(ask("What is its population?"))          # resolves "its" -> Paris, because history was resent

Example 3: Edge Case — streaming + cost tracking

Text appears incrementally — that's the SSE content_block_delta stream.

import litellm

PRICES = {"claude-sonnet-4-6": (3.0, 15.0)}   # ($/MTok in, out) — src: anthropic-pricing

def stream_ask(history, model="claude-sonnet-4-6"):
    stream = litellm.completion(model=model, messages=history, max_tokens=500, stream=True)
    parts = []
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)     # token-by-token to the terminal
            parts.append(delta)
    print()
    return "".join(parts)

5. Reflection Before Building

After filling in UNDERSTANDING.md, use the AI mentor pattern in docs/meta/learning-flow.md to get feedback. Record the feedback in UNDERSTANDING_FEEDBACK.md.

6. Project Assignment

See PROJECT.md for the overview and source/project.md for the full engineering spec.

Starter Code & Workflow

Setup is solved for you; the core logic is not. code/ ships labeled starter files:

Core Requirement

A CLI chatbot that: (a) holds a multi-turn conversation with correct history management, (b) applies a configurable system prompt, (c) streams responses token-by-token, and (d) reports per-turn and cumulative token cost. You must be able to explain every line — no copied magic.

7. Project Milestones

Work through these in order. Each milestone should produce runnable code before moving on.

8. Self-Evaluation

After building, honestly evaluate your implementation against these criteria. Record your answers in EVALUATION.md.

9. Common Mistakes

10. Connections

Builds On

This is Project 1 — the foundation. Everything downstream builds on the messages/cost/context mental model established here.

Enables

• P2 (Tokens & Embeddings): deepens the "what is a token" idea introduced here.
• P4 (RAG): retrieved documents get injected into messages — the same array, now with stuffed context.
• P5 (Memory): an explicit answer to "history grows unbounded" — summarize/store instead of resend.
• P8 (Agent): the loop becomes a tool-use loop; stop_reason == "tool_use" replaces end_turn.

Production Patterns

Real products manage the array with a sliding window (drop oldest turns), summarization (compress old turns into a synthetic note), and prompt caching (cache the stable prefix so repeated history costs ~0.1× input). Model selection (Haiku→Sonnet→Opus) is a first-order cost lever.

StarcallOS Relevance

Sources

See source/resources.md for the complete annotated source list.