When you open the chat bubble on this site and ask "what does OMZ Tech Solutions charge for SEO?", you get a direct, specific answer in about two seconds. There's no hallucination about prices that don't exist. There's no generic "I'm just an AI" dodge. It answers accurately because I told it exactly what to say — just not in the way most people think.
This post walks through the full implementation: the RAG pattern, the streaming SSE setup, the Anthropic API integration, and the decisions I made along the way.
The problem with a plain LLM
The naive approach is to dump your site content into a system prompt and call messages.create. That works for a demo. It breaks in production for three reasons:
- Cost. Every message re-sends the entire context. For a site assistant that handles hundreds of conversations, this adds up fast.
- Accuracy. LLMs hallucinate. Without grounding, the model will invent pricing tiers, fabricate service names, and confidently tell a potential client something that isn't true.
- Control. You want the assistant to represent your business correctly. A plain system prompt without a knowledge base gives the model too much latitude to improvise.
The right pattern is RAG — Retrieval-Augmented Generation. For a personal site at this scale, the "retrieval" part is trivial: a single markdown file that gets injected wholesale into the system prompt. No vector database, no embeddings, no semantic search. Just a well-structured document.
The knowledge base
The knowledge base lives at src/content/knowledge-base.md. It's a plain markdown file I can edit like documentation:
# OMZ Tech Solutions — Knowledge Base
## Services
### Website Design & Development
We build custom Next.js sites for small businesses...
## Pricing
### Starter Plan — $899 one-time
- Up to 4 responsive pages
- Core on-page SEO setup
...The file covers services, pricing, process, FAQs, tech stack, and contact information. It's the single source of truth for everything the assistant is allowed to say.
The API route
The entire backend is one file: src/app/api/ai/assistant/route.ts. Here's the core of it:
import Anthropic from "@anthropic-ai/sdk";
import { readFileSync } from "fs";
import { join } from "path";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
let knowledgeBase = "";
try {
knowledgeBase = readFileSync(
join(process.cwd(), "src/content/knowledge-base.md"),
"utf-8"
);
} catch {
knowledgeBase = "";
}
const SYSTEM_PROMPT = `You are the AI assistant for OMZ Tech Solutions...
Use only the knowledge base below — do not invent details.
---
${knowledgeBase}`;Two things worth noting:
The knowledge base is read at module load time, not per-request. In a serverless environment like Vercel, this means it's read once when the function cold-starts and cached in memory until the next cold start. This is fine — the knowledge base rarely changes, and if it does, redeploying is the right way to push updates.
The fallback to empty string means the assistant degrades gracefully if the file is missing. It'll still respond, just without business-specific context. Better than a 500.
Streaming with SSE
The response is streamed using the Anthropic SDK's stream() method and sent back as Server-Sent Events:
const stream = client.messages.stream({
model: "claude-haiku-4-5",
max_tokens: 1024,
system: SYSTEM_PROMPT,
messages: anthropicMessages,
});
const readable = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder();
try {
for await (const event of stream) {
if (
event.type === "content_block_delta" &&
event.delta.type === "text_delta"
) {
const chunk = JSON.stringify(event.delta.text);
controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
}
}
controller.enqueue(encoder.encode("data: [DONE]\n\n"));
} catch (err) {
console.error("Streaming error:", err);
controller.enqueue(encoder.encode("data: [DONE]\n\n"));
} finally {
controller.close();
}
},
});
return new Response(readable, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
},
});The SSE format is simple: each chunk is data: <json-encoded string>\n\n. The client reads these with an EventSource-like pattern and appends each token to the displayed message. The [DONE] sentinel tells the client the stream is complete.
Why SSE instead of WebSockets? SSE is one-directional (server → client) and works over standard HTTP. No upgrade handshake, no persistent connection management, no special infrastructure. Perfect for a chat response where the client sends one message and the server streams one reply.
The chat bubble component
On the client side, the bubble is a floating button that expands into a panel. The key part is the streaming read loop:
async function sendMessage(userMessage: string) {
const newMessages = [...messages, { role: "user", content: userMessage }];
setMessages([...newMessages, { role: "assistant", content: "" }]);
setIsStreaming(true);
const response = await fetch("/api/ai/assistant", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messages: newMessages }),
});
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let assistantContent = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
const text = decoder.decode(value);
const lines = text.split("\n");
for (const line of lines) {
if (line.startsWith("data: ")) {
const data = line.slice(6);
if (data === "[DONE]") {
setIsStreaming(false);
return;
}
try {
const token = JSON.parse(data) as string;
assistantContent += token;
setMessages((prev) => {
const updated = [...prev];
updated[updated.length - 1] = {
role: "assistant",
content: assistantContent,
};
return updated;
});
} catch {
// partial chunk, skip
}
}
}
}
setIsStreaming(false);
}The assistant message starts as an empty string and gets updated on every token. React re-renders on each state update, which produces the typewriter effect. This is more efficient than it sounds — the updates are small and React batches them effectively.
Model choice: Haiku over Sonnet or Opus
I used claude-haiku-4-5 deliberately. For a customer-facing site assistant with a bounded knowledge base, the bottleneck isn't model intelligence — it's the grounding document. Haiku follows the knowledge base instructions well, responds in under two seconds, and costs a fraction of Sonnet. If I needed the model to reason across ambiguous questions or synthesize complex information, I'd upgrade. Here, it's overkill.
What I'd do differently at scale
The current implementation works well for a personal site. At higher volume, I'd change a few things:
Chunked retrieval instead of full injection. Right now the entire knowledge base goes into every request. At 5,000 tokens of knowledge base content, that's not a problem. At 50,000 tokens (a proper product knowledge base), you'd want to embed the knowledge base and retrieve only the relevant chunks per query.
Rate limiting on the API route. There's currently no protection against someone hammering the endpoint. Adding a per-IP rate limit with Vercel's edge middleware would be the right first step.
Conversation history truncation. Long conversations accumulate tokens. A simple fix is to keep only the last N messages, or summarize older turns. Right now there's a 2,000-character cap per message, which prevents the worst case but isn't a full solution.
The interesting thing about this approach is how much it can do with how little infrastructure. No vector database, no embedding pipeline, no model training. Just a markdown file, one API route, and 200 lines of React. For a bounded domain like "answer questions about my business", that's the right amount of complexity.