Chat & RAG System

Overview

Afunana's chat is a retrieval-augmented, multi-agent assistant that answers questions about the live legacy estate — and, on request, produces a validated change plan. Every factual answer is grounded in the analyzed code and cited to the exact source line (PROGRAM:LINE), rendered as a clickable badge that jumps to the code. Answers are grounded, not guessed.

The chat draws on the same knowledge graph and hybrid index the build pipeline produces (see AI & Analysis Pipeline). What sets it apart from a chat window over a context window: it queries a persistent, cross-referenced map of the whole system — call graphs, file usage, field lineage — not just the snippet in front of it.

The implementation is a multi-agent loop: PlannerAgent → ExecutorAgent → iterative AnswerBuilder.

Multi-agent loop

A question flows through three cooperating agents:

1. PlannerAgent classifies the request and emits a single JSON plan — intent, complexity, and the tool calls to run. Intent/plan classification is folded into the planner rather than run as a separate up-front model call, so simple and complex questions are triaged in one step (backed by the chat_planner role; the smallest model, chat_classifier, handles the lightest triage).
2. ExecutorAgent runs the planned tool calls against the knowledge graph and the hybrid index, assembling an evidence pack.
3. AnswerBuilder composes an answer from the evidence and streams it. It is agentic: if the evidence is thin, it can request another round of retrieval, up to a configurable maximum (default 6). Most questions resolve in 1–3 rounds.

Routing by complexity keeps cost down: simple questions are answered by chat_answer_simple; complex, multi-step ones go to chat_answer, a reasoning model that manages its own extended thinking (no per-role thinking-token budget applied — consistent with AI & Analysis Pipeline). Any of these roles can be repointed to Anthropic, OpenAI, an Azure deployment, or a local Ollama model by editing its fallback chain — see AI & Analysis Pipeline.

Query planning

The planner produces a structured plan describing which tools to call, what evidence to gather, and in what order. The executor is not limited to one lookup per round — a plan can fan out across several tools, and the answer loop can re-plan when it finds a gap.

Evidence tools available to the executor:

Hybrid retrieval

Retrieval combines two complementary methods, built locally during the build pipeline — no external embedding or search service is involved.

• Vector (ChromaDB, intfloat/multilingual-e5-large) — semantic similarity; strong on natural-language and conceptual queries; multilingual.
• BM25 — term-frequency lexical scoring; strong on exact names and technical identifiers.

Fusion. Results from both are merged with weighted reciprocal-rank fusion (RRF) — the vector/lexical weights are configurable — then deduplicated. Because RRF blends rank positions rather than raw scores, a hit that both methods rank highly rises to the top even when the two scoring scales differ.

Program-level re-rank. The fused hits are assembled into an evidence pack and re-ranked at the program level, so the agent reasons over whole programs and their relationships rather than isolated chunks — this is where the knowledge graph, not just similarity, drives what the agent sees.

Context assembly

Before calling the answer model, the executor assembles a bounded context window from the evidence pack:

Line counts are configurable, and total context size is tracked to stay within the model's window. Section-based chunking and sequence→display-line mapping (from the build) mean the lines shown match the lines cited.

Streaming answers

Answers stream to the client over Server-Sent Events (SSE) with live tool-status progress, so the user sees the agent planning, searching, and answering in real time.

A chat request runs the full multi-round agent. The answer can be returned in a single blocking response or streamed incrementally over SSE as it is produced.

Inline markers in the SSE stream carry tool-status progress, signal when a change plan was saved or failed validation, and mark stream completion or error.

A request carries the user's question, any prior conversation, the desired language, and session identifiers.

Citations & validation

Every factual answer is cited to PROGRAM:LINE. Because chunking preserves sequence→display-line mapping, a citation points at the line a developer actually sees in the source, and the UI renders it as a badge that opens the code there.

For change plans, citation is not cosmetic — the plan is structurally validated (the required JSON shape and typed steps), and a failed validation triggers a corrective retry rather than returning a broken plan. An invalid plan is reported back as failed; a valid one is persisted and reported as saved.

Query modes

Ask mode

Read-only question-and-answer. The agent retrieves evidence and composes a cited answer. No side effects. This is the default.

Plan mode

The user asks for a change ("add a field to this file," "stop cancelling policies past the grace date"). The agent analyzes the request and the affected programs, then hands off to the code_developer developer agent, which produces a validated, style-matched change plan grounded in the live codebase. The plan is saved for review.

Producing the plan is the core value — "discovering and understanding what is wrong and knowing how to fix it." Applying it is a separate, approval-gated step run from the admin UI, gated by configuration. The full plan → approve → execute workflow, the fixed-format COBOL style guard, and the honest caveats (compile step stubbed; execution targets the live IBM i; the VS Code extension only renders plans) are documented in AI & Analysis Pipeline → Close-the-loop code generation.

Bring Your Own Documents

Existing specs, manuals, wiki pages, and analysts' notes fold into the same searchable, cited index as the code-derived knowledge. Uploaded PDF / DOCX / TXT / MD (up to 20 MB) are chunked and added to both ChromaDB and BM25, so they are retrieved in chat exactly like any program documentation — and cited the same way.

• Collection-wide: uploaded into the collection's supporting-documents area, with list, fetch, and remove operations.
• Program-scoped: a document can be attached to a specific program.

This is distinct from spec-doc generation, which produces new specifications rather than ingesting existing ones. See the API Overview.

Semantic Q→A cache

To cut cost and latency, answered questions are cached semantically — a later question close in meaning to an earlier one can reuse the prior answer without re-running the full agent.

Two rules keep the cache safe:

• Change plans are never cached — plan mode always runs fresh against the current code.
• The cache is wiped on rebuild — after a build or a delta refresh, cached answers that could reference now-changed code are cleared, so answers never drift from the code.

Chat sessions

Conversation history is persisted, scoped per user and per collection. Sessions can be created, listed, renamed, and deleted through the API (see the API Overview); prior messages in a session are supplied as context for follow-up questions.

Audit trail

Every LLM interaction during chat is logged: the role and model used, the full prompt and response, input/output/thinking token counts, latency, and the session and message IDs. Each call's token usage and cost are also written to the cost & usage ledger (see AI & Analysis Pipeline → Cost & usage metering), so spend is attributable per query and per model. Combined with the security audit log (chat queries are recorded as audit events), this gives a complete, reviewable record of what was asked, what the model saw, and what it cost.