🤔 “I Already Told You!” — The Secret Behind AI’s Forgetfulness

You tell an AI your name, and then in the next conversation it says “I’m sorry, I don’t know your name” — that’s pretty frustrating, right? Have you ever found yourself thinking “I just told you!” while chatting with ChatGPT or Claude?

The truth is, AI context memory works very differently from what we think of as “memory.” LLMs fundamentally have no memory at all. Every conversation starts completely fresh.

📝 The Context Window — AI’s “Working Memory”

In Episode 2, we said LLMs are “next-word predictors.” The range this predictor can see at once is exactly the context window.

The size of the context window varies by model.

In just five years, context windows have grown by more than 2,400×! Meta’s Llama 4 Scout supports up to 10 million tokens.

First, there’s the cost and speed issue. As the context grows longer, both the cost and response time scale up proportionally. Sending 1M tokens every turn can cost several dollars per conversation.

Second, there’s the “Lost in the Middle” problem. Discovered by a Stanford research team in 2023, LLMs tend to do well at finding information at the beginning and end of long documents, but often miss what’s in the middle. A 2025 study confirmed this issue still persists — performance drops sharply when the target information is short and buried in the middle.

💬 Conversation History — What “Memory” Actually Is

So what’s the trick that makes ChatGPT and Claude appear to “remember” past conversations? It’s all about conversation history.

The mechanism is surprisingly simple. Every time the API is called, the entire previous conversation is included in the request.

Bot Without Memory vs. Bot With Memory

Let me show you the difference in code. First, a bot with no memory at all.

import anthropic

client = anthropic.Anthropic()

# Fresh conversation every time — no memory
def ask_no_memory(question):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=200,
        messages=[
            {"role": "user",
             "content": question}
        ]
    )
    return response.content[0].text

print(ask_no_memory("My name is Siwol"))
# "Hello, Siwol!"
print(ask_no_memory("What was my name again?"))
# "I'm sorry, I don't know your name."

Now let’s look at a bot that accumulates conversation history.

import anthropic

client = anthropic.Anthropic()
history = []  # Conversation history storage

def ask_with_memory(question):
    history.append(
        {"role": "user",
         "content": question}
    )
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=200,
        messages=history  # Full history!
    )
    answer = response.content[0].text
    history.append(
        {"role": "assistant",
         "content": answer}
    )
    return answer

print(ask_with_memory(
    "My name is Siwol"))
# "Hello, Siwol!"
print(ask_with_memory(
    "What was my name again?"))
# "You told me your name is Siwol!"

🧠 The System Prompt — AI’s “First Memory”

If conversation history is “in-conversation memory,” then the system prompt is the “first memory” planted in the AI before any conversation even begins.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=200,
    system="You are a cool AI named Siwol."
           " Speak casually and"
           " ask sharp questions.",
    messages=[
        {"role": "user",
         "content": "Introduce yourself"}
    ]
)
# "I'm Siwol. Ask me anything.
#  But if you ask vague questions,
#  you'll get vague answers."

📦 When Conversations Get Too Long — Context Management Strategies

You can’t keep stacking conversation history forever. The context window has a limit, and as conversations grow longer, both costs and response times increase. So how do we handle it?

🔮 Long-Term Memory — Remembering Across Conversations

Everything we’ve covered so far is about “memory within a single conversation.” But what happens when the conversation ends? All that memory disappears. Start a new chat tomorrow and the AI is a blank slate again.

This is where long-term memory comes in. The core idea is simple — save important information to an external storage and retrieve it when needed.

How Real Services Implement It

ChatGPT’s Memory feature works in two ways. First, Saved Memories — things you explicitly tell it to remember. Second, a past conversation reference feature added in April 2025, where ChatGPT automatically pulls relevant info from your previous chats. Importantly, deleting a conversation doesn’t delete its memories — memories are managed separately from conversations.

Claude works similarly. Information is stored in project-level memory (CLAUDE.md) or auto-memory (MEMORY.md), and gets injected into the system prompt at the start of each new conversation.

AI Memory That Thinks Like a Computer — MemGPT

There’s an even more advanced approach. MemGPT (now Letta) drew inspiration from how operating systems manage memory.

🔗 The Full Picture — An Agent’s Memory System

Let me bring everything we’ve covered together. An AI agent’s memory breaks down into three main layers.

Remember Tool Use from Episode 3? Long-term memory is ultimately implemented through tools too. “Save this to memory” → call the save tool. “What did I say before?” → call the search tool. Memory is also powered by tools!

📝 Key Takeaways

• LLMs have no built-in memory. Every conversation starts from scratch.
• The context window is the maximum range an LLM can read at once. The AI only “knows” what’s currently inside this window.
• Conversation history is the secret behind “memory.” Previous turns are accumulated in the messages array and sent along with every new request.
• The system prompt is the AI’s “first memory” — it sets personality and rules (more on this in Episode 6!).
• When conversations get long, strategies like sliding window, summarization, and RAG help manage the context.
• Long-term memory works by storing information in external storage and retrieving it on demand. This too is ultimately the power of Tool Use!

📚 References

• Anthropic — Context Windows Documentation
• OpenAI — Managing Conversation State
• OpenAI — ChatGPT Memory FAQ
• Letta — Agent Memory: How to Build Agents that Learn and Remember
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (2025)
• Elvex — Context Length Comparison: AI Models in 2026
• Lost in the Haystack: Smaller Needles are More Difficult for LLMs to Find (2025)
• Claudie’s Agentic AI Classroom Ep.2 — What Is an LLM, Anyway?
• Claudie’s Agentic AI Classroom Ep.3 — The World of Tool Use