How to Add Memory to Your AI Chatbot (No Vector Database Needed)

A practical guide to giving your LLM a working memory — using tools you probably already have

There’s a specific kind of frustration that comes from building an AI chatbot and realizing it’s completely useless the moment the conversation ends.

I built a nutrition coaching bot. Smart replies, great personality, really helpful in the moment. But the second a user started a new session? Gone. It had no idea they were lactose intolerant. It forgot that their homemade dal recipe is 180 calories a bowl or something they’d mentioned twice. It asked them for their goals again. And again.

The bot was the conversational equivalent of a goldfish.

I knew I needed to give it memory. What I didn’t want was to over-engineer it - spinning up a vector database, learning embeddings, building a whole RAG pipeline - for a feature that, at its core, just needed to remember a few important things about a user.

So I built something simpler. And it works really well. Here’s how.

⚠️ A quick heads-up before we dive in: Everything in this article is based on my current scale - which is small. I’m running this on free tiers: Supabase’s free plan, Upstash Redis’s free tier, and a modest LLM API budget. This approach works really well at that scale, and I haven’t hit any walls yet. But if you’re building for thousands of concurrent users, you’ll likely need to think harder about things like Redis memory limits, DB connection pooling, and the cost of extra LLM calls for fact extraction. I’m not there yet, so I can’t speak to those tradeoffs from experience. Take this as “what worked for me at early-stage scale,” not a battle-tested enterprise blueprint.

First, understand why the AI forgets in the first place

This isn’t a bug. It’s just how LLM APIs work.

Every time you call an API like OpenAI or Claude, it’s completely stateless. The model doesn’t know who you are, what you said yesterday, or even what you said five minutes ago in a different tab. The only thing it knows is what you put inside the messages array in that specific request.

The naive fix is: save every message ever, and send the whole thing every time. Simple, right?

Not really. Two problems:

Problem 1: Cost. Every token you send costs money. A user who’s been chatting for three months is now sending a massive wall of text on every single new message - most of which is completely irrelevant to what they just asked.

Problem 2: Limits. Models have a maximum context window. Send too much and it literally won’t fit.

So the naive approach breaks down fast. You need something smarter.

The insight that makes this tractable

Think about how you remember things.

You don’t replay every conversation you’ve ever had word-for-word. You remember the main points. You remember facts that stuck. “Oh right, she’s vegetarian.” “He mentioned he’s training for a marathon.” The exact back-and-forth from three weeks ago? Gone. But the conclusion? Still there.

That’s exactly the model I used:

• Keep the last 12 messages verbatim - because the exact wording of what just happened still matters
• Compress everything older into short, durable facts - because for old conversations, all that matters is what was established, not how you got there

Two types of memory. Two different storage strategies. Simple.

Where the data actually lives

I’m using two things I already had in my stack:

Supabase (Postgres) - the permanent record. Two tables:

• chatHistoryevery message, ever. This never gets deleted or compressed. It's what powers the "scroll up and see old conversations" feature in the UI.
• userMemoryFacts every extracted fact, ever. Same deal - permanent storage.

Redis (Upstash) — the fast cache that sits in front of what the AI actually needs right now. Specifically:

• The last 12 messages
• The most recent 50 extracted facts

Here’s the key separation to understand: the AI’s working memory is allowed to be lossy and capped. The database record is not. A user can scroll back through their full chat history in the app. But the bot itself only needs to “actively remember” the last dozen messages plus a handful of facts. That’s all it needs to be helpful.

Reading memory before every reply

Before the bot responds to anything, it assembles context. Here’s the pattern:

1. Check Redis for recent messages
2. If not in cache, pull the last 12 from the database, then populate the cache
3. Do the same for facts

const getContext = async (userId, supabase) => {
  let recentMessages = await getCache(Keys.chatHistory(userId));  
if (!recentMessages) {
    const { data } = await supabase
      .from("chatHistory")
      .select("role, content")
      .order("created_at", { ascending: false })
      .limit(12);    
recentMessages = (data || []).reverse(); // flip back to chronological
await setCache(Keys.chatHistory(userId), recentMessages);
  }  // same pattern for facts...  return { facts, recentMessages };
};

Building the prompt

Once we have context, assembling the prompt looks like this:

const messages = [
  { role: "system", content: systemPrompt },            // static persona/rules
  { role: "system", content: `Known facts about this user: ${facts.join("\n")}` },
  ...recentMessages,                         
  { role: "user", content: message },                   // the new message
];

A few deliberate choices here:

The static system prompt comes first and stays the same every call. This lets OpenAI’s prompt caching kick in - if the prefix is byte-identical across requests, it gets served from cache, which is cheaper and faster. Mixing variable content into the static prompt would break that.

Facts go in a separate system message after the static prompt. Same reason - the cached prefix stays clean. Only this one block changes as facts evolve.

Writing memory after every reply

After the bot responds, we write the new turn back. This is where the compression logic lives:

const MAX_RECENT = 12;
const TRIGGER_AT = 18; // compress once we're 6 over the limit 

const appendTurn = async ({ userId, userMessage, assistantMessage, recentMessages }) => {
  // 1. Always write to the DB first — permanent record
  await supabase.from("chatHistory").insert([
    { user_id: userId, role: "user", content: userMessage },
    { user_id: userId, role: "assistant", content: assistantMessage },
  ]);  

const updated = [...recentMessages, 
    { role: "user", content: userMessage }, 
    { role: "assistant", content: assistantMessage }
  ];  

if (updated.length > TRIGGER_AT) {
    const overflow = updated.slice(0, updated.length - MAX_RECENT); // the old stuff
    const kept = updated.slice(updated.length - MAX_RECENT);         // the fresh 12    
    await extractFacts(userId, overflow); // compress old → facts
    await setCache(Keys.chatHistory(userId), kept);
    return;
  }  

await setCache(Keys.chatHistory(userId), updated);
};

Why trigger at 18 instead of 12? Because running an LLM call to extract facts on every single message would be wasteful. Instead, we let the buffer grow to 18 and then compress once every ~3 conversation turns. Same outcome, way less overhead. (You can increase it to more)

The fact extraction: a separate LLM call

This is the magic step. When old messages get pushed out of the working window, we send them to a separate, focused LLM call with a very specific prompt:

From this conversation, extract only concrete, durable facts worth remembering:
custom food calorie/macro values, allergies, dietary restrictions, stated goals.Ignore small talk and anything not explicitly stated.
Do NOT invent or infer facts that weren't directly stated.Respond with strict JSON only: {"facts": ["...", "..."]}

A few things I’m deliberately doing here:

It’s a separate call, not bolted onto the main response. The main bot already does tool calling, multi-step reasoning, all that. Mixing “also compress old messages” into that would tangle two unrelated jobs. This call is narrow and cheap - low temperature, JSON format, small token budget.

The prompt explicitly forbids inference. This matters a lot. If the model extracts a “fact” that it hallucinated or inferred incorrectly, that fact gets injected into every future conversation as ground truth. A hallucination here isn’t one bad reply - it’s permanent corruption of the user’s profile. So: only extract what was directly stated.

The rule that makes the whole thing user-friendly

const { content } = await generateResponse(messages);
try {
  await appendTurn({ ... }); // save memory
} catch (memoryError) {
  console.log("Memory write failed:", memoryError);
}
return content; // user gets their answer either way

The user gets their reply before we even try to save memory. If saving memory fails - database hiccup, malformed JSON from the extraction call, Redis is down - it gets logged and swallowed. The user never sees an error.

Should you use this pattern?

Yes, if:

• You’re building something users come back to across sessions - a coach, assistant, support bot.
• The stuff worth remembering is a handful of discrete facts (preferences, restrictions, goals, custom data)
• You already have a relational DB and a cache - this adds zero new infrastructure

Known limitation I haven’t solved yet:

As users keep chatting over weeks and months, the facts table keeps growing. Right now, I cap the prompt at the 50 most recent facts, so older facts quietly stop appearing in context even though they’re still in the database. That’s mostly fine, but it’s not great. A user’s oldest preferences could silently get dropped.

Look at vector databases instead, if:

• You need to recall semantically related content — “what did we discuss that’s relevant to this question?”, across hundreds of documents
• You need similarity search, not just a recency-ordered list

Skip memory entirely, if:

• It’s a single-turn, transactional bot (“summarize this doc”, FAQ bot) where continuity doesn’t matter

The one-line version

Keep recent messages verbatim, compress old ones into durable facts, cache the hot path, and never let memory failures block the response the user is waiting for.

That’s it. No embeddings. No new infrastructure. Just two tables, a cache, and one extra LLM call.

If you’re building something where continuity actually matters to your users, I genuinely think this is the right starting point before reaching for something heavier. It’s been running in production and it works.

P.S.- As the facts grows the memory of facts also grows and leads too many facts for LLM.