Chat Attachments & RAG

Overview

When a user sends a message with file attachments in chat, the system processes them through multiple parallel paths:

File Upload and Storage — files are stored and made accessible
RAG Retrieval — the user’s message (and images) search the knowledge base
Knowledge Graph Enhancement — entity-based context enrichment
LLM Invocation — the model receives the message, files, KB context, and graph context

Architecture

Step 1: File Upload and Storage

User selects files (max 5, max 50MB each)
  -> POST /api/bot/attachments/upload (multipart)
  -> FileUploadService.uploadFile()
    1. Validate: size, type, org limits
    2. Store to disk: /storage/attachments/
    3. Upload to S3 (if enabled)
    4. Generate 64-char access token
    5. Create DB record (message_attachments)
    6. Record billing event (if S3)
  -> Returns: { id, accessToken, filename, contentType }

Supported types:

Images: JPEG, PNG, WebP
Documents: PDF
Text: .txt, .md
Audio: MP3, WAV

Step 2: Chat Request Processing

POST /api/bots/{botId}/chat
Body: { message, conversationId, attachments: [{id, accessToken}] }

1. Validate each attachment (fetch from DB, verify accessToken)
2. Separate by type:
   - imageAttachments[] -> used for RAG + sent to LLM
   - pdfAttachments[]   -> sent to LLM only (NOT RAG)
   - textAttachments[]  -> sent to LLM only
   - audioAttachments[] -> sent to LLM only

Step 3: RAG Retrieval (Parallel)

For each linked knowledge base, four parallel operations run:

Text Search
Image Search
KB Map
Graph Search

POST /knowledge/kb/{kbId}/search
Body: { query: enhancedQuery, topK: 3 }
-> Generates text embedding (Gemini 3072d)
-> Vector similarity search
-> Returns chunks with scores + metadata

POST /knowledge/kb/{kbId}/search/image
Body: { imageBase64, mimeType, topK: 3 }
-> Generates image embedding (Gemini multimodal)
-> Visual similarity search across KB
-> Deduplicates against text search results

GET /knowledge/kb/{kbId}/map
-> Returns document summaries, topics, types
-> Formatted as "AVAILABLE KNOWLEDGE" section

POST /knowledge/kb/{kbId}/graph/search
Body: { query: enhancedQuery }
-> Extract entities from query
-> Match against kg_entities table
-> Recursive CTE traversal (1-2 hops)
-> Formatted as "KNOWLEDGE GRAPH CONTEXT"

Query Enhancement

The search query is enhanced with conversation context:

Fetch last 5 messages from conversation
Build: "Recent conversation: ...\nCurrent query: ..."

Merge and Rerank

Results from all KBs are merged:

Hybrid scoring: 60% vector similarity + 25% text relevance + 15% importance weight
Top 5 chunks returned with citations

Formatted Context

---AVAILABLE KNOWLEDGE---
KB "Name": N documents
- "Doc Title" [topics] -- summary snippet
---END AVAILABLE KNOWLEDGE---

---KNOWLEDGE BASE CONTEXT---
[1] Source: KB Name (filename) | Relevance: 85%
chunk text...
---END KNOWLEDGE BASE CONTEXT---

---KNOWLEDGE GRAPH CONTEXT---
Related entities: EntityA (type) -- desc; EntityB...
Connections: EntityA --[rel]--> EntityB: desc
---END KNOWLEDGE GRAPH CONTEXT---

Step 4: LLM Invocation

Size limits:

Images: 2MB each (base64 data URL)
PDFs: 2MB each (base64 file)
Total multimodal content: 5MB
Large files (>2MB): “[File too large: name (size)]“

Step 5: Response

{
  "message": "Based on the knowledge base [1], collagen...",
  "conversationId": "uuid",
  "sources": [
    {
      "number": 1,
      "source": "KB Name (file.pdf)",
      "relevanceScore": 0.85,
      "text": "chunk preview..."
    }
  ],
  "usage": { "inputTokens": 1200, "outputTokens": 450, "totalTokens": 1650 }
}

File Type Processing Matrix

File Type	Stored	Sent to LLM	RAG Text Search	RAG Image Search	Graph Search
Images (JPEG, PNG, WebP)	Yes	Yes (base64)	No	Yes (visual similarity)	Yes (if entities match)
PDFs	Yes	Yes (base64)	No	No	No
Text (.txt, .md)	Yes	Yes (text)	No	No	No
Audio (MP3, WAV)	Yes	Yes (if model supports)	No	No	No

RAG search is driven by the user’s text message and image embeddings. PDF content is NOT extracted for search — the PDF is sent directly to the LLM. Only documents ingested into the Knowledge Base via the KB upload flow are searchable via RAG.

Knowledge Graph Enhancement

The Knowledge Graph adds three layers of context beyond traditional RAG:

Layer	Purpose
KB Map	Document registry showing what documents exist, their summaries, and topics
Vector Search	Chunk-level similarity with summary-guided filtering
Knowledge Graph	Entity extraction from query, graph traversal (1-2 hops), related concepts and connections

Key Integration Points

Component	File	Purpose
File upload UI	`apps/web/src/components/chat/FileUpload.tsx`	File picker, validation, upload progress
Chat interface	`apps/web/src/components/chat/EnhancedChatInterface.tsx`	Combines message + attachments
File storage	`services/bot/src/services/file-upload.service.ts`	Disk + S3 storage, access tokens
Chat handler	`services/bot/src/services/bot.service.ts`	Attachment retrieval + RAG orchestration
LLM invocation	`services/bot/src/services/langchain-chat.service.ts`	Multimodal content construction
Graph routes	`services/knowledge/src/routes/graph.routes.ts`	Graph search, KB map
Graph extraction	`services/knowledge/src/services/graph-extraction.service.ts`	Multimodal entity extraction

Overview

Services

Architecture

Agent Provisioning API

Deployment

Design System (Neo)

Overview

Architecture

Step 1: File Upload and Storage

Step 2: Chat Request Processing

Step 3: RAG Retrieval (Parallel)

Query Enhancement

Merge and Rerank

Formatted Context

Step 4: LLM Invocation

Step 5: Response

File Type Processing Matrix

Knowledge Graph Enhancement

Key Integration Points

​Overview

​Architecture

​Step 1: File Upload and Storage

​Step 2: Chat Request Processing

​Step 3: RAG Retrieval (Parallel)

​Query Enhancement

​Merge and Rerank

​Formatted Context

​Step 4: LLM Invocation

​Step 5: Response

​File Type Processing Matrix

​Knowledge Graph Enhancement

​Key Integration Points

Overview

Architecture

Step 1: File Upload and Storage

Step 2: Chat Request Processing

Step 3: RAG Retrieval (Parallel)

Query Enhancement

Merge and Rerank

Formatted Context

Step 4: LLM Invocation

Step 5: Response

File Type Processing Matrix

Knowledge Graph Enhancement

Key Integration Points