> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brainstormer.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Chat Attachments & RAG

> File processing pipeline for chat attachments, multimodal LLM invocation, RAG retrieval with knowledge graph enhancement.

## Overview

When a user sends a message with file attachments in chat, the system processes them through multiple parallel paths:

1. **File Upload and Storage** -- files are stored and made accessible
2. **RAG Retrieval** -- the user's message (and images) search the knowledge base
3. **Knowledge Graph Enhancement** -- entity-based context enrichment
4. **LLM Invocation** -- the model receives the message, files, KB context, and graph context

## Architecture

```mermaid theme={null}
graph TD
    A["User sends message + attachments"] --> B[Frontend: Upload files]
    B --> C["POST /api/bot/attachments/upload<br/>Returns: attachment ID + token"]
    A --> D["POST /api/bots/:id/chat<br/>{message, attachments: [{id, token}]}"]
    D --> E[Gateway -> Bot Service]

    E --> F[Attachment Retrieval]
    E --> G[RAG Retrieval]
    E --> H[Knowledge Graph]

    F & G & H --> I["LLM Invocation (LangChain)<br/>SystemMessage + KB context + graph<br/>HumanMessage + images + PDFs"]
    I --> J[Response with citations]
```

## Step 1: File Upload and Storage

```
User selects files (max 5, max 50MB each)
  -> POST /api/bot/attachments/upload (multipart)
  -> FileUploadService.uploadFile()
    1. Validate: size, type, org limits
    2. Store to disk: /storage/attachments/
    3. Upload to S3 (if enabled)
    4. Generate 64-char access token
    5. Create DB record (message_attachments)
    6. Record billing event (if S3)
  -> Returns: { id, accessToken, filename, contentType }
```

**Supported types:**

* Images: JPEG, PNG, WebP
* Documents: PDF
* Text: .txt, .md
* Audio: MP3, WAV

## Step 2: Chat Request Processing

```
POST /api/bots/{botId}/chat
Body: { message, conversationId, attachments: [{id, accessToken}] }

1. Validate each attachment (fetch from DB, verify accessToken)
2. Separate by type:
   - imageAttachments[] -> used for RAG + sent to LLM
   - pdfAttachments[]   -> sent to LLM only (NOT RAG)
   - textAttachments[]  -> sent to LLM only
   - audioAttachments[] -> sent to LLM only
```

## Step 3: RAG Retrieval (Parallel)

For each linked knowledge base, four parallel operations run:

<Tabs>
  <Tab title="Text Search">
    ```
    POST /knowledge/kb/{kbId}/search
    Body: { query: enhancedQuery, topK: 3 }
    -> Generates text embedding (Gemini 3072d)
    -> Vector similarity search
    -> Returns chunks with scores + metadata
    ```
  </Tab>

  <Tab title="Image Search">
    ```
    POST /knowledge/kb/{kbId}/search/image
    Body: { imageBase64, mimeType, topK: 3 }
    -> Generates image embedding (Gemini multimodal)
    -> Visual similarity search across KB
    -> Deduplicates against text search results
    ```
  </Tab>

  <Tab title="KB Map">
    ```
    GET /knowledge/kb/{kbId}/map
    -> Returns document summaries, topics, types
    -> Formatted as "AVAILABLE KNOWLEDGE" section
    ```
  </Tab>

  <Tab title="Graph Search">
    ```
    POST /knowledge/kb/{kbId}/graph/search
    Body: { query: enhancedQuery }
    -> Extract entities from query
    -> Match against kg_entities table
    -> Recursive CTE traversal (1-2 hops)
    -> Formatted as "KNOWLEDGE GRAPH CONTEXT"
    ```
  </Tab>
</Tabs>

### Query Enhancement

The search query is enhanced with conversation context:

```
Fetch last 5 messages from conversation
Build: "Recent conversation: ...\nCurrent query: ..."
```

### Merge and Rerank

Results from all KBs are merged:

* Hybrid scoring: 60% vector similarity + 25% text relevance + 15% importance weight
* Top 5 chunks returned with citations

### Formatted Context

```
---AVAILABLE KNOWLEDGE---
KB "Name": N documents
- "Doc Title" [topics] -- summary snippet
---END AVAILABLE KNOWLEDGE---

---KNOWLEDGE BASE CONTEXT---
[1] Source: KB Name (filename) | Relevance: 85%
chunk text...
---END KNOWLEDGE BASE CONTEXT---

---KNOWLEDGE GRAPH CONTEXT---
Related entities: EntityA (type) -- desc; EntityB...
Connections: EntityA --[rel]--> EntityB: desc
---END KNOWLEDGE GRAPH CONTEXT---
```

## Step 4: LLM Invocation

```mermaid theme={null}
graph TD
    A[Check Model Capabilities] --> B{Supports multimodal?}
    B -->|Yes| C["Build multimodal content<br/>Images: base64 data URL<br/>PDFs: base64 file<br/>Text: inline content"]
    B -->|No| D["Fallback message:<br/>'This model does not support files'"]
    C --> E["Construct Messages:<br/>SystemMessage (prompt + KB context)<br/>Previous messages (history)<br/>HumanMessage (text + images + PDFs)"]
    E --> F["Invoke via OpenRouter<br/>Returns response + usage stats"]
```

**Size limits:**

* Images: 2MB each (base64 data URL)
* PDFs: 2MB each (base64 file)
* Total multimodal content: 5MB
* Large files (>2MB): "\[File too large: name (size)]"

## Step 5: Response

```json theme={null}
{
  "message": "Based on the knowledge base [1], collagen...",
  "conversationId": "uuid",
  "sources": [
    {
      "number": 1,
      "source": "KB Name (file.pdf)",
      "relevanceScore": 0.85,
      "text": "chunk preview..."
    }
  ],
  "usage": { "inputTokens": 1200, "outputTokens": 450, "totalTokens": 1650 }
}
```

## File Type Processing Matrix

| File Type                    | Stored | Sent to LLM             | RAG Text Search | RAG Image Search            | Graph Search            |
| ---------------------------- | ------ | ----------------------- | --------------- | --------------------------- | ----------------------- |
| **Images** (JPEG, PNG, WebP) | Yes    | Yes (base64)            | No              | **Yes** (visual similarity) | Yes (if entities match) |
| **PDFs**                     | Yes    | Yes (base64)            | No              | No                          | No                      |
| **Text** (.txt, .md)         | Yes    | Yes (text)              | No              | No                          | No                      |
| **Audio** (MP3, WAV)         | Yes    | Yes (if model supports) | No              | No                          | No                      |

<Note>
  RAG search is driven by the user's **text message** and **image embeddings**. PDF content is NOT extracted for search -- the PDF is sent directly to the LLM. Only documents ingested into the Knowledge Base via the KB upload flow are searchable via RAG.
</Note>

## Knowledge Graph Enhancement

The Knowledge Graph adds three layers of context beyond traditional RAG:

| Layer               | Purpose                                                                                    |
| ------------------- | ------------------------------------------------------------------------------------------ |
| **KB Map**          | Document registry showing what documents exist, their summaries, and topics                |
| **Vector Search**   | Chunk-level similarity with summary-guided filtering                                       |
| **Knowledge Graph** | Entity extraction from query, graph traversal (1-2 hops), related concepts and connections |

## Key Integration Points

| Component        | File                                                          | Purpose                                  |
| ---------------- | ------------------------------------------------------------- | ---------------------------------------- |
| File upload UI   | `apps/web/src/components/chat/FileUpload.tsx`                 | File picker, validation, upload progress |
| Chat interface   | `apps/web/src/components/chat/EnhancedChatInterface.tsx`      | Combines message + attachments           |
| File storage     | `services/bot/src/services/file-upload.service.ts`            | Disk + S3 storage, access tokens         |
| Chat handler     | `services/bot/src/services/bot.service.ts`                    | Attachment retrieval + RAG orchestration |
| LLM invocation   | `services/bot/src/services/langchain-chat.service.ts`         | Multimodal content construction          |
| Graph routes     | `services/knowledge/src/routes/graph.routes.ts`               | Graph search, KB map                     |
| Graph extraction | `services/knowledge/src/services/graph-extraction.service.ts` | Multimodal entity extraction             |