The build transforms raw legacy source into searchable, AI-generated documentation and a persistent knowledge graph. It runs in sequential stages, each building on the previous one, and it is designed to run unattended -- start it and come back when it finishes.
Afunana is multi-platform, and the build adapts to the source. IBM i (COBOL, RPG, CL, DDS) is the worked example here. Oracle (PL/SQL and the Oracle data dictionary) and plain SQL feed the same pipeline through a shared extraction contract — all supported. z/OS is architected for but not yet analyzed -- it is on the near-term roadmap.
Pipeline at a Glance
Five stages take source to a fully documented, searchable collection.
| Stage | Name | Input | Output |
|---|---|---|---|
| 1 | Extract | Source platform (IBM i / Oracle) | Normalized extract CSVs + source members |
| 2 | Parse & Map | Extracted source | Structural model: call tree, cross-reference, field catalog |
| 3 | AI Documentation | Source + structure | Per-program, per-file, per-SQL docs; collection artifacts; quality checks |
| 4 | Embeddings & Index | All docs | Local vector store + keyword index |
| 5 | Auto-Tag | All docs + structure | Business-concept taxonomy applied to programs and files |
Build duration is dominated by Stage 3 (LLM documentation). Parsing and cross-referencing are fast even for large collections; overall time is driven by the AI documentation stage.
Stage 1: Extract (Platform Dispatch)
Extraction is where the platforms differ. Each produces the same normalized CSV knowledge-graph contract, so every downstream stage is platform-agnostic.
IBM i. A server-side extraction agent (TTDOC) is submitted as a batch job via SBMJOB. It reads the configured source libraries on the box and produces the normalized CSV set (programs, files, fields, cross-references), which Afunana pulls back over FTP/IFS. Programs, physical/logical files, copybooks, display files, printer files, and CL commands are all captured, with metadata (library, source file, member name, type, modification date) alongside the source text.
Oracle. A read-only connection via python-oracledb in thin mode (no Oracle client install required) reads the Oracle data dictionary -- packages, procedures, functions, triggers, tables, columns -- and writes it into the same normalized contract the IBM i pipeline consumes. Nothing is modified on the Oracle side.
Offline / no connectivity. For air-gapped or restricted environments, a source snapshot can be exported to CSV and uploaded through the web UI or API, in the same layout. The rest of the pipeline runs identically.
Output. Extracted source lands in a working extract folder as individual members, ready for parsing.
Stage 2: Parse & Map
Deterministic, language-specific parsers build a structural model of the entire application. No AI is involved -- the parsers apply grammar rules, so the same source always yields the same model. Parsing dispatches per member by language: COBOL, RPG, and CL/DDS on IBM i; PL/SQL on Oracle.
Parsers extract:
| Element | Description |
|---|---|
| Divisions / sections | IDENTIFICATION, ENVIRONMENT, DATA, PROCEDURE (COBOL); package/procedure structure (PL/SQL) |
| Paragraphs / subroutines | PERFORM targets, RPG subroutines, PL/SQL blocks |
| Field / variable definitions | PIC clauses, data types, sizes, decimal positions, level numbers; PL/SQL declarations |
| CALL statements | Static and dynamic call targets with parameter lists |
| File / table I/O | READ, WRITE, REWRITE, DELETE, START; SQL DML |
| COPY / INCLUDE references | Copybooks resolved to their full expanded definitions |
| Screen / report definitions | Display-file fields, printer-file layouts |
| SQL statements | Embedded SQL with table references |
From this, parsing produces three artifacts, all available through the UI and API before any AI runs:
- Call tree -- root programs down through every callee, recursively; circular calls detected and flagged.
- Cross-reference matrix -- every program mapped to every file it uses, with access type (read/write/update/delete).
- Field catalog -- all field definitions aggregated into a unified data dictionary.
You get a complete structural map from parsing alone. See Data Dictionary & Cross-Reference.
Stage 3: AI Documentation
The LLM generates the readable knowledge, and the deterministic quality checks run against the structural model. This is the most time-consuming stage. Documentation calls dispatch by language -- COBOL/RPG/CL programs use the COBOL-aware prompt path; PL/SQL uses the PL/SQL-aware path.
Program documentation. Every program is documented at three levels (Business Specification, Systems Analysis, Program Specification) in a single call per program to keep the levels consistent. Programs are ordered leaves-to-roots by call depth and processed in parallel per level, so callee context is available when a caller is documented. Each request includes the full source, parsed metadata, cross-reference context, copybook expansions, and the prompt template. Chunking is section-based, call-tree-ordered, with semantic slicing for large programs; copybook rows are excluded and sequence numbers mapped to display lines so citations line up.
File and SQL documentation. File documents (purpose, fields, keys, access paths, using programs) and SQL-member documentation are generated next, using the cross-reference and program docs as added context.
Collection artifacts. With every program and file documented, the pipeline assembles the collection-wide artifacts: the system overview (a two-pass narrative of subsystems, batch flows, online transactions, data stores, and risks), the compiled data dictionary (field, table, and application-group entries), the finalized cross-reference, and per-file change-impact reports.
Quality checks (deterministic, no LLM). The interface/parameter checks and the full quality catalog run here against the structural model, honoring the per-check severities in CHECK_SETTINGS, and a build-quality report is stored with the build record. These are the silent-failure checks -- cross-program parameter/byte-size mismatch, MOVE truncation, unhandled I/O status, and (for Oracle) the PL/SQL rule family. See Code Quality Analysis.
Fallback chains. Every LLM call routes through the One AI Hub gateway. If the primary provider fails (rate limit, timeout, error), the system falls back to the next configured provider automatically. Four providers are supported -- Anthropic, OpenAI, Azure OpenAI, and Ollama (local) -- switchable live with no rebuild. Failed programs are logged and skipped; the build continues and they can be retried in continue mode.
Stage 4: Embeddings & Index (Nothing Leaves the Box)
This stage builds the search indexes that power the AI chat, entirely on-premise.
Local vector embeddings. All documentation is chunked into semantic units and embedded with a local embedding model into an on-premise vector store, with metadata linking each chunk to its source program. There is no external embedding API -- embeddings are computed on the box and never leave the customer network. This is what makes fully offline, air-gapped operation real.
BM25 keyword index. A lexical index is built alongside the vector store to catch exact matches (program names, field names, error codes) that semantic search may rank lower. Chat fuses both indexes at query time. See Chat Capabilities.
Stage 5: Auto-Tag
The LLM derives a business-concept taxonomy (up to 15 labels, up to 5 per program/file) and applies tags across the collection -- business domain, subsystem, program role, and similar dimensions. Auto-tagging is additive: tags created or assigned manually by an administrator are preserved and never overwritten. The resulting tag pills appear on the Programs and Files lists and drive tag-based filtering.
REFRESH -- Incremental Content-Hash Delta Rebuild
Documentation stays current without a full rebuild. The changes build mode is the REFRESH phase of Afunana's four-phase model: it keeps an already-built collection current by re-analyzing only what actually changed.
Crucially, the admin does not pick which members changed -- the pipeline extracts the entire current source (exactly like a full build) and computes the delta itself by content hash.
When triggered (Admin Panel → Collections → Build → Build Changes), the pipeline:
- Extracts and pulls the full current snapshot into
Data/{collection}/CHANGED_SOURCE/, leaving the live collection untouched during the pull. - Detects changed programs, copybooks, and files by comparing the source content of each snapshot member against the live collection. The comparison uses only the source column (
SRCDTA/SRC) --SEQ,SRCSEQ,LINE,DATmetadata and compiler ruler rows are ignored -- so a program counts as changed only when its real code differs (or it is new). Deletions are found by diffing the old inventory against the new metadata. - Backs up the prior state (
info-from-as400-backup/and a timestamped doc-JSON backup). - Applies only the changed members into the collection and swaps in the new metadata. Unchanged programs keep their existing source.
- Removes documentation, source, and embeddings for deleted objects.
- Regenerates docs only for the members whose source actually changed -- there is no dependency cascade. (Because the
programs_csvlisting already contains expanded copybook lines, a program affected by a changed copybook is itself detected as changed and rebuilt.) - Fully refreshes the collection-wide artifacts (system overview, data dictionary, cross-reference) and
DOC_PROGRAMS.csv. - Updates ChromaDB by deleting and re-adding only the affected documents, refreshes the BM25 index, and clears the chat answer cache.
If the collection has never been built, every member is treated as new, so changes degrades gracefully to a first full build. A full snapshot can also be placed at CHANGED_SOURCE/ manually (same layout) for environments without direct connectivity; the content diff works the same way.
Build Modes
| Mode | Behavior |
|---|---|
full |
Complete rebuild from scratch. All stages run. Existing documentation replaced. |
continue |
Resume an interrupted or partial build. Skips already-documented members; retries previously failed programs. |
programs_only |
Runs extraction, parsing, and program documentation only. |
files_only |
Regenerates file documentation only. Requires a prior parsed build. |
system_only |
Regenerates collection artifacts, indexes, and auto-tags only, without re-documenting programs. |
changes |
Incremental content-hash delta update -- the REFRESH phase described above. |
Build Configuration
Every LLM call routes through the One AI Hub gateway (Anthropic, OpenAI, Azure OpenAI, Ollama), configured under the Admin Panel's AI Config and Config screens; per-check severities under Check Settings. Adaptive parameter-stripping lets any gateway or local model work, and per-call token telemetry makes cost measurable.
Pause and Resume
A running build can be paused; the system completes in-flight LLM calls and stops. No work is lost -- completed programs are saved immediately after each response. Resume with a continue build, which skips already-documented members. Useful for freeing LLM capacity during business hours, recovering from provider outages, or splitting long builds across time windows.
Monitoring Builds
- Admin Panel → Collections & builds -- all builds with status, progress, current stage, elapsed and estimated time; full log and quality report per build.
- Live log -- real-time LLM call status: in-flight requests, completed count, error count, throughput.
- Collection Dashboard -- current build status and documentation coverage for the active collection.
- Per-program status -- queued, parsing, generating, embedding, complete, or failed.
- REST API -- build status for CI/CD or external monitoring.
Build errors are non-fatal by default: a failed program is logged and skipped, and the build continues. Failed programs can be retried in a continue build or rebuilt individually.
Overall build time is driven almost entirely by the AI documentation stage, and scales with the number of programs and the throughput of the configured model. Smaller models are faster but produce less detailed documentation. For per-role model defaults and the gateway, see AI & Analysis Pipeline.