> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brainstormer.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Knowledge Base System

> RAG pipeline architecture: indexing and search flow, content connectors, knowledge graph, citation system, and configuration.

## Overview

The Knowledge Base System is a comprehensive RAG (Retrieval-Augmented Generation) platform that enables agents to ground their responses in custom knowledge sources. It supports multiple document formats, API connectors, social media sources, and a knowledge graph for structured entity queries.

**Service:** Knowledge Service (Port 4005)

## Architecture

### Multi-Loader Strategy

| Loader               | Purpose                                                           |
| -------------------- | ----------------------------------------------------------------- |
| LlamaIndex (v0.12.0) | Primary parser for complex documents via LlamaParse cloud service |
| LangChain Loaders    | TextLoader, CSVLoader, WebBaseLoader for simple formats           |
| Unstructured.io      | Fallback parser for 40+ file types                                |
| Custom Connectors    | YouTube, Instagram, Twitter, LinkedIn, RSS, blog platforms        |

### Technology Stack

| Component           | Technology                                            |
| ------------------- | ----------------------------------------------------- |
| Document Processing | LlamaIndex v0.12.0                                    |
| RAG Orchestration   | LangChain v0.3.30                                     |
| Vector Storage      | ChromaDB (single vector-store port; Pinecone removed) |
| Sparse / keyword    | Postgres full-text (`tsvector` + GIN) on chunk text   |
| Reranker            | OpenRouter cross-encoder (`cohere/rerank-v3.5`)       |
| Embeddings          | Gemini Embedding 2 (3072 dimensions, multimodal)      |
| Vision Extraction   | OpenRouter vision models                              |
| Transcription       | OpenRouter Whisper                                    |
| Async Processing    | BullMQ v5.34.4 + Redis                                |

## Unified Content Processing Pipeline

All KB source ingestion flows through a unified pipeline built on the **ContentConnector** pattern.

### ContentConnector Interface

Every connector implements:

```typescript theme={null}
interface ContentConnector {
  name: string;                    // e.g., "youtube", "instagram", "document"
  detect(input): boolean;          // Returns true if this connector handles the input
  fetch(input, options): Promise<NormalizedContent[]>;  // Fetches and normalizes content
}
```

### ConnectorRegistry Resolution

```mermaid theme={null}
graph TD
    A[Input: URL or File] --> B{Explicit sourceType?}
    B -->|Yes, not 'url'| C[Match by sourceType]
    B -->|No| D[Try each connector.detect]
    C --> E[Connector Found]
    D --> E
    D -->|No match| F[Error: Unknown source]
    E --> G["fetch() -> NormalizedContent[]"]
```

### Registered Connectors

| Connector       | Source Types                                               |
| --------------- | ---------------------------------------------------------- |
| `youtube`       | YouTube video/channel/playlist URLs                        |
| `instagram`     | Instagram profile/post URLs                                |
| `twitter`       | Twitter/X profile/post URLs                                |
| `rss`           | RSS/Atom feed URLs                                         |
| `blog-platform` | Substack, Medium, Ghost, other blog platforms              |
| `url`           | Generic web URLs (fallback)                                |
| `document`      | Uploaded files (PDF, CSV, TXT, DOCX, images, audio, video) |

### NormalizedContent Shape

```typescript theme={null}
interface NormalizedContent {
  externalId: string;           // Dedup key
  contentHash: string;          // SHA-256 change detection
  title: string;
  textContent: string;
  media: MediaAttachment[];     // Images, video, audio
  source: {
    connector: string;
    url: string;
    platform: string;
    author: string;
    publishedAt: Date;
  };
  thumbnailUrl?: string;
  engagementMetrics?: object;
  mediaType?: string;
}
```

## Indexing Flow

<Steps>
  <Step title="Upload / Add Source">
    Document uploaded or URL source added. Frontend sends to Knowledge Service via Gateway.
  </Step>

  <Step title="Connector Resolution">
    `ConnectorRegistry` resolves input to the appropriate `ContentConnector`.
  </Step>

  <Step title="Content Normalization">
    Connector fetches and returns `NormalizedContent[]` with dedup keys and content hashes. For social media: one `kb_document` per post.
  </Step>

  <Step title="Media Extraction">
    Vision text extraction (OpenRouter) for images/PDFs. Transcription for audio/video.
  </Step>

  <Step title="Chunking">
    Text chunking with RecursiveCharacterTextSplitter (1000 chars, 200 overlap).
  </Step>

  <Step title="Embedding">
    Gemini Embedding 2 generates multimodal embeddings (3072 dimensions).
  </Step>

  <Step title="Vector Storage">
    Vectors stored in ChromaDB (single vector-store port for all environments), with a sparse full-text index (`tsvector` + GIN) maintained on the chunk text for hybrid retrieval.
  </Step>

  <Step title="Graph Extraction">
    Async via BullMQ: entity and relationship extraction with entity seeding from existing KB entities.
  </Step>

  <Step title="Document Summary">
    Per-document summaries generated for KB map overview.
  </Step>
</Steps>

## Retrieval Stack

Retrieval is a multi-stage pipeline. Everything below the fusion step is on by default and independently toggleable via config.

<Steps>
  <Step title="Hybrid retrieval">
    Two channels run in parallel: **dense** (Gemini vector similarity via ChromaDB) and **sparse** (Postgres full-text `ts_rank_cd` over the chunk `tsvector`). `unified-search.service` also folds in image and graph-informed discovery. Toggle: `HYBRID_SEARCH_ENABLED`.
  </Step>

  <Step title="Reciprocal Rank Fusion">
    The ranked channels are fused by RRF (`score = Σ 1/(k + rank)`, `k = 60`), which combines by **rank** rather than raw score — robust to the dense-cosine vs. sparse-`ts_rank` scale mismatch. Lexical-only hits get a calibrated pass-through score so exact matches still clear the downstream relevance gate. Pure impl: `rank-fusion.utils`.
  </Step>

  <Step title="Engagement prior (learning loop)">
    A precomputed per-`(org, document)` boost from accumulated engagement (`resource_engagement_prior`) is applied to the fused scores — empirical-Bayes shrunk and floor-gated, so it only ever helps and never moves low-evidence resources. Global kill-switch `LEARNING_PRIOR_ENABLED`; per-KB switch `knowledge_bases.learning_prior_enabled`.
  </Step>

  <Step title="Cross-encoder rerank (bot side)">
    The agent path re-scores fused candidates with an OpenRouter cross-encoder (`cohere/rerank-v3.5`). Falls back to a lexical blend (60% vector + 25% text relevance + 15% importance weight) when `RERANKER_ENABLED=false`.
  </Step>

  <Step title="Context assembly (MMR + token budget)">
    Post-rerank chunks are diversified with **MMR** (lexical Jaccard, `λ = 0.6`) to drop near-duplicates, then capped by a **token budget** (`CONTEXT_TOKEN_BUDGET`, \~4000) so retrieval can't crowd out the prompt/history. Pure impls: `mmr.utils`, `token-budget.utils`.
  </Step>
</Steps>

## Outcome Attribution & Learning Loop (Workstream H)

The moat: capture what an agent surfaced, what the customer did with it, and feed that back so retrieval compounds.

* **Signal capture** — `retrieval_events` (per-turn candidate set + injected chunks + final order), `message_feedback` (thumbs), and `resource_interactions` (impression / click / conversion, via a signed first-party redirect). All keyed by a `signal_source` enum whose values include `end_user_thumbs`, `click`, `conversion`, `hitl_correction`, and the reserved `creator_training` (future training mode).
* **Attribution** — `analytics.service` joins `retrieval_events` (unnest `injected_chunk_ids`) ⋈ feedback ⋈ interactions, scoped to a KB via chunk → document, to produce per-document performance, feedback-driven knowledge gaps, and the impression → click → conversion funnel (analytics `section=resource-performance|feedback-gaps|funnel`).
* **Learning** — `learning-loop.service` mines the signals into an engagement prior (refreshed by a repeatable BullMQ job) and a **training-readiness** score with weighted per-source authority (`creator_training` > `hitl_correction` > thumbs/conversion > click). Surfaced at analytics `section=learning`. The reranker fine-tune / learned-weight replacement is a later phase that consumes this substrate once signal volume accrues.

## Knowledge Graph Pipeline

Three-phase extraction:

<Tabs>
  <Tab title="Entity Extraction">
    Text, image, and video content analyzed by LLM to extract named entities (people, orgs, concepts, products) and relationships. Extraction prompts are **seeded with existing KB entities** so the LLM reuses canonical names.
  </Tab>

  <Tab title="Entity Resolution">
    LLM-powered duplicate detection via name similarity (Levenshtein, substring, abbreviation matching). Confirmed merges consolidate relationships, aliases, and mention counts.
  </Tab>

  <Tab title="Cross-Document Inference">
    After document-level extraction, an inference pass examines document summaries and entity lists to discover relationships between entities from different documents.
  </Tab>
</Tabs>

**API:** `POST /knowledge/kb/:id/enhance-graph` triggers entity resolution + cross-document inference as queued jobs.

## Database Schema

| Table                  | Purpose                                                    |
| ---------------------- | ---------------------------------------------------------- |
| `knowledge_bases`      | Core KB metadata, organization-scoped, visibility settings |
| `kb_sources`           | Data source connectors with sync scheduling                |
| `kb_documents`         | Ingested documents with processing status                  |
| `kb_document_chunks`   | Text chunks with vector DB ID mapping                      |
| `kb_agent_links`       | Many-to-many KB-agent relationship with priority           |
| `kb_analytics`         | Usage tracking and query relevance scores                  |
| `kg_document_registry` | Per-document summaries and topics                          |
| `kg_entities`          | Extracted entities (name, type, aliases, mention\_count)   |
| `kg_relationships`     | Entity relationships (source, target, type, weight)        |
| `kg_communities`       | Community detection groups                                 |

## Citation UI System

The citation system provides interactive source attribution in chat:

### Backend

* Chat response includes `sources` array with numbered citations
* Each source has: `number`, `kbName`, `source`, `relevanceScore`, `text`
* Citations are transient per message (not persisted to database)

### Frontend Components

| Component                  | Purpose                                                 |
| -------------------------- | ------------------------------------------------------- |
| `CitationTooltip.tsx`      | Interactive tooltips on `[1]`, `[2]` markers (Radix UI) |
| `MessageWithCitations.tsx` | Parse and display citations in messages                 |
| `SourcesPanel.tsx`         | Slide-in sidebar with grouped KB sources                |
| `SourcesToggleButton.tsx`  | Floating button for panel visibility                    |

### Response Format

```json theme={null}
{
  "message": "Based on our documentation [1], the feature works by... [2]",
  "sources": [
    {
      "number": 1,
      "kbName": "Product Documentation",
      "source": "user_guide.pdf",
      "relevanceScore": 0.89,
      "text": "The feature works by processing user input..."
    }
  ]
}
```

## Configuration

```bash theme={null}
# Knowledge Service
KNOWLEDGE_SERVICE_PORT=4005

# LlamaParse (optional)
LLAMA_CLOUD_API_KEY=your-llama-cloud-api-key

# Vector Database
PINECONE_API_KEY=your-pinecone-api-key
PINECONE_ENVIRONMENT=us-east-1
PINECONE_INDEX_NAME=brainstormer-kb

# Redis (required for BullMQ)
REDIS_HOST=localhost
REDIS_PORT=6379

# Embeddings
GOOGLE_GEMINI_API_KEY=your-gemini-api-key
EMBEDDING_MODEL=gemini-embedding-exp-03-07

# Processing
CHUNK_SIZE=1000
CHUNK_OVERLAP=200
MAX_CONCURRENT_JOBS=5
```

## Cost Estimates (Monthly)

| Component                       | Cost                              |
| ------------------------------- | --------------------------------- |
| LlamaParse                      | \~$300 (100k pages @ $0.003/page) |
| ChromaDB                        | self-hosted (compute only)        |
| Reranker (`cohere/rerank-v3.5`) | usage-based via OpenRouter        |
| Gemini Embeddings               | \~\$10 (usage-based)              |
| Redis Cloud                     | \~\$30 (queue + cache)            |
| S3 Storage                      | \~\$23 (1TB)                      |
| **Total**                       | **\~\$430/month**                 |

<Tip>
  Brainstormer uses self-hosted ChromaDB as its single vector store (Pinecone has been removed) and can run Redis locally, which significantly reduces costs for development and small deployments.
</Tip>