Knowledge Base System

Overview

The Knowledge Base System is a comprehensive RAG (Retrieval-Augmented Generation) platform that enables agents to ground their responses in custom knowledge sources. It supports multiple document formats, API connectors, social media sources, and a knowledge graph for structured entity queries. Service: Knowledge Service (Port 4005)

Architecture

Multi-Loader Strategy

Loader	Purpose
LlamaIndex (v0.12.0)	Primary parser for complex documents via LlamaParse cloud service
LangChain Loaders	TextLoader, CSVLoader, WebBaseLoader for simple formats
Unstructured.io	Fallback parser for 40+ file types
Custom Connectors	YouTube, Instagram, Twitter, LinkedIn, RSS, blog platforms

Technology Stack

Component	Technology
Document Processing	LlamaIndex v0.12.0
RAG Orchestration	LangChain v0.3.30
Vector Storage	ChromaDB (single vector-store port; Pinecone removed)
Sparse / keyword	Postgres full-text (`tsvector` + GIN) on chunk text
Reranker	OpenRouter cross-encoder (`cohere/rerank-v3.5`)
Embeddings	Gemini Embedding 2 (3072 dimensions, multimodal)
Vision Extraction	OpenRouter vision models
Transcription	OpenRouter Whisper
Async Processing	BullMQ v5.34.4 + Redis

Unified Content Processing Pipeline

All KB source ingestion flows through a unified pipeline built on the ContentConnector pattern.

ContentConnector Interface

Every connector implements:

interface ContentConnector {
  name: string;                    // e.g., "youtube", "instagram", "document"
  detect(input): boolean;          // Returns true if this connector handles the input
  fetch(input, options): Promise<NormalizedContent[]>;  // Fetches and normalizes content
}

ConnectorRegistry Resolution

Registered Connectors

Connector	Source Types
`youtube`	YouTube video/channel/playlist URLs
`instagram`	Instagram profile/post URLs
`twitter`	Twitter/X profile/post URLs
`rss`	RSS/Atom feed URLs
`blog-platform`	Substack, Medium, Ghost, other blog platforms
`url`	Generic web URLs (fallback)
`document`	Uploaded files (PDF, CSV, TXT, DOCX, images, audio, video)

NormalizedContent Shape

interface NormalizedContent {
  externalId: string;           // Dedup key
  contentHash: string;          // SHA-256 change detection
  title: string;
  textContent: string;
  media: MediaAttachment[];     // Images, video, audio
  source: {
    connector: string;
    url: string;
    platform: string;
    author: string;
    publishedAt: Date;
  };
  thumbnailUrl?: string;
  engagementMetrics?: object;
  mediaType?: string;
}

Indexing Flow

Upload / Add Source

Document uploaded or URL source added. Frontend sends to Knowledge Service via Gateway.

Connector Resolution

ConnectorRegistry resolves input to the appropriate ContentConnector.

Content Normalization

Connector fetches and returns NormalizedContent[] with dedup keys and content hashes. For social media: one kb_document per post.

Media Extraction

Vision text extraction (OpenRouter) for images/PDFs. Transcription for audio/video.

Chunking

Text chunking with RecursiveCharacterTextSplitter (1000 chars, 200 overlap).

Embedding

Gemini Embedding 2 generates multimodal embeddings (3072 dimensions).

Vector Storage

Vectors stored in ChromaDB (single vector-store port for all environments), with a sparse full-text index (tsvector + GIN) maintained on the chunk text for hybrid retrieval.

Graph Extraction

Async via BullMQ: entity and relationship extraction with entity seeding from existing KB entities.

Document Summary

Per-document summaries generated for KB map overview.

Retrieval Stack

Retrieval is a multi-stage pipeline. Everything below the fusion step is on by default and independently toggleable via config.

Hybrid retrieval

Two channels run in parallel: dense (Gemini vector similarity via ChromaDB) and sparse (Postgres full-text ts_rank_cd over the chunk tsvector). unified-search.service also folds in image and graph-informed discovery. Toggle: HYBRID_SEARCH_ENABLED.

Reciprocal Rank Fusion

The ranked channels are fused by RRF (score = Σ 1/(k + rank), k = 60), which combines by rank rather than raw score — robust to the dense-cosine vs. sparse-ts_rank scale mismatch. Lexical-only hits get a calibrated pass-through score so exact matches still clear the downstream relevance gate. Pure impl: rank-fusion.utils.

Engagement prior (learning loop)

A precomputed per-(org, document) boost from accumulated engagement (resource_engagement_prior) is applied to the fused scores — empirical-Bayes shrunk and floor-gated, so it only ever helps and never moves low-evidence resources. Global kill-switch LEARNING_PRIOR_ENABLED; per-KB switch knowledge_bases.learning_prior_enabled.

Cross-encoder rerank (bot side)

The agent path re-scores fused candidates with an OpenRouter cross-encoder (cohere/rerank-v3.5). Falls back to a lexical blend (60% vector + 25% text relevance + 15% importance weight) when RERANKER_ENABLED=false.

Context assembly (MMR + token budget)

Post-rerank chunks are diversified with MMR (lexical Jaccard, λ = 0.6) to drop near-duplicates, then capped by a token budget (CONTEXT_TOKEN_BUDGET, ~4000) so retrieval can’t crowd out the prompt/history. Pure impls: mmr.utils, token-budget.utils.

Outcome Attribution & Learning Loop (Workstream H)

The moat: capture what an agent surfaced, what the customer did with it, and feed that back so retrieval compounds.

Signal capture — retrieval_events (per-turn candidate set + injected chunks + final order), message_feedback (thumbs), and resource_interactions (impression / click / conversion, via a signed first-party redirect). All keyed by a signal_source enum whose values include end_user_thumbs, click, conversion, hitl_correction, and the reserved creator_training (future training mode).
Attribution — analytics.service joins retrieval_events (unnest injected_chunk_ids) ⋈ feedback ⋈ interactions, scoped to a KB via chunk → document, to produce per-document performance, feedback-driven knowledge gaps, and the impression → click → conversion funnel (analytics section=resource-performance|feedback-gaps|funnel).
Learning — learning-loop.service mines the signals into an engagement prior (refreshed by a repeatable BullMQ job) and a training-readiness score with weighted per-source authority (creator_training > hitl_correction > thumbs/conversion > click). Surfaced at analytics section=learning. The reranker fine-tune / learned-weight replacement is a later phase that consumes this substrate once signal volume accrues.

Knowledge Graph Pipeline

Three-phase extraction:

Entity Extraction
Entity Resolution
Cross-Document Inference

Text, image, and video content analyzed by LLM to extract named entities (people, orgs, concepts, products) and relationships. Extraction prompts are seeded with existing KB entities so the LLM reuses canonical names.

API: POST /knowledge/kb/:id/enhance-graph triggers entity resolution + cross-document inference as queued jobs.

Database Schema

Table	Purpose
`knowledge_bases`	Core KB metadata, organization-scoped, visibility settings
`kb_sources`	Data source connectors with sync scheduling
`kb_documents`	Ingested documents with processing status
`kb_document_chunks`	Text chunks with vector DB ID mapping
`kb_agent_links`	Many-to-many KB-agent relationship with priority
`kb_analytics`	Usage tracking and query relevance scores
`kg_document_registry`	Per-document summaries and topics
`kg_entities`	Extracted entities (name, type, aliases, mention_count)
`kg_relationships`	Entity relationships (source, target, type, weight)
`kg_communities`	Community detection groups

Citation UI System

The citation system provides interactive source attribution in chat:

Backend

Chat response includes sources array with numbered citations
Each source has: number, kbName, source, relevanceScore, text
Citations are transient per message (not persisted to database)

Frontend Components

Component	Purpose
`CitationTooltip.tsx`	Interactive tooltips on `[1]`, `[2]` markers (Radix UI)
`MessageWithCitations.tsx`	Parse and display citations in messages
`SourcesPanel.tsx`	Slide-in sidebar with grouped KB sources
`SourcesToggleButton.tsx`	Floating button for panel visibility

Response Format

{
  "message": "Based on our documentation [1], the feature works by... [2]",
  "sources": [
    {
      "number": 1,
      "kbName": "Product Documentation",
      "source": "user_guide.pdf",
      "relevanceScore": 0.89,
      "text": "The feature works by processing user input..."
    }
  ]
}

Configuration

# Knowledge Service
KNOWLEDGE_SERVICE_PORT=4005

# LlamaParse (optional)
LLAMA_CLOUD_API_KEY=your-llama-cloud-api-key

# Vector Database
PINECONE_API_KEY=your-pinecone-api-key
PINECONE_ENVIRONMENT=us-east-1
PINECONE_INDEX_NAME=brainstormer-kb

# Redis (required for BullMQ)
REDIS_HOST=localhost
REDIS_PORT=6379

# Embeddings
GOOGLE_GEMINI_API_KEY=your-gemini-api-key
EMBEDDING_MODEL=gemini-embedding-exp-03-07

# Processing
CHUNK_SIZE=1000
CHUNK_OVERLAP=200
MAX_CONCURRENT_JOBS=5

Cost Estimates (Monthly)

Component	Cost
LlamaParse	~ $300 (100k pages @$ 0.003/page)
ChromaDB	self-hosted (compute only)
Reranker (`cohere/rerank-v3.5`)	usage-based via OpenRouter
Gemini Embeddings	~$10 (usage-based)
Redis Cloud	~$30 (queue + cache)
S3 Storage	~$23 (1TB)
Total	~$430/month

Brainstormer uses self-hosted ChromaDB as its single vector store (Pinecone has been removed) and can run Redis locally, which significantly reduces costs for development and small deployments.

Overview

Services

Architecture

Agent Provisioning API

Deployment

Design System (Neo)

Overview

Architecture

Multi-Loader Strategy

Technology Stack

Unified Content Processing Pipeline

ContentConnector Interface

ConnectorRegistry Resolution

Registered Connectors

NormalizedContent Shape

Indexing Flow

Retrieval Stack

Outcome Attribution & Learning Loop (Workstream H)

Knowledge Graph Pipeline

Database Schema

Citation UI System

Backend

Frontend Components

Response Format

Configuration

Cost Estimates (Monthly)

​Overview

​Architecture

​Multi-Loader Strategy

​Technology Stack

​Unified Content Processing Pipeline

​ContentConnector Interface

​ConnectorRegistry Resolution

​Registered Connectors

​NormalizedContent Shape

​Indexing Flow

​Retrieval Stack

​Outcome Attribution & Learning Loop (Workstream H)

​Knowledge Graph Pipeline

​Database Schema

​Citation UI System

​Backend

​Frontend Components

​Response Format

​Configuration

​Cost Estimates (Monthly)

Overview

Architecture

Multi-Loader Strategy

Technology Stack

Unified Content Processing Pipeline

ContentConnector Interface

ConnectorRegistry Resolution

Registered Connectors

NormalizedContent Shape

Indexing Flow

Retrieval Stack

Outcome Attribution & Learning Loop (Workstream H)

Knowledge Graph Pipeline

Database Schema

Citation UI System

Backend

Frontend Components

Response Format

Configuration

Cost Estimates (Monthly)