> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brainstormer.io/llms.txt
> Use this file to discover all available pages before exploring further.

# System Architecture

> High-level architecture overview of the Brainstormer V2 platform, including layer breakdown, service map, and data flow diagrams.

## Overview

Brainstormer V2 is a modern AI chatbot platform built with a microservices architecture, using Next.js 15 for the frontend and Fastify-based Node.js microservices for the backend.

## System Architecture

```mermaid theme={null}
graph TB
    subgraph "Client Layer"
        CL[Web Browsers / Mobile]
    end

    CL -->|HTTPS| FL

    subgraph "Frontend Layer"
        FL["Next.js 15 (Port 3000)<br/>React 19, TailwindCSS, TypeScript<br/>SSR, Turbopack"]
    end

    FL -->|HTTP/REST| GW

    subgraph "API Gateway Layer"
        GW["Fastify Gateway (Port 4000)<br/>Routing, CORS, Rate Limiting<br/>Health Checks"]
    end

    GW --> AUTH["Auth Service<br/>Port 4001"]
    GW --> BOT["Bot Service<br/>Port 4002"]
    GW --> AUDIO["Audio Service<br/>Port 4003"]
    GW --> KB["Knowledge Service<br/>Port 4005"]
    GW --> HITL["HITL Service<br/>Port 4006"]

    subgraph "External Services"
        OR[OpenRouter API]
        EL[ElevenLabs API]
        S3[AWS S3 Storage]
        PC[Pinecone]
    end

    BOT --> OR
    AUDIO --> EL
    BOT --> S3
    KB --> PC

    AUTH --> INFRA
    BOT --> INFRA
    AUDIO --> INFRA
    KB --> INFRA
    HITL --> INFRA

    subgraph "Infrastructure Layer"
        INFRA["PostgreSQL 15 (5432)<br/>Redis 7 (6379)<br/>ChromaDB (8000)"]
    end
```

## Layer Breakdown

### 1. Frontend Layer (Next.js 15)

<Columns>
  <Card title="Technology" icon="code">
    Next.js 15, React 19, TypeScript 5, TailwindCSS
  </Card>

  <Card title="Port" icon="plug">
    3000
  </Card>
</Columns>

**Key Features:**

* Server-Side Rendering (SSR) for optimal performance
* Turbopack for fast development builds
* Component-based architecture with reusable UI components
* Multi-path agent onboarding (`classic` and `creator` flows)

**Main Pages:**

| Route                           | Purpose                           |
| ------------------------------- | --------------------------------- |
| `/`                             | Landing page                      |
| `/auth/login`, `/auth/register` | Authentication                    |
| `/agents`                       | Agent management dashboard        |
| `/agents/create?path=creator`   | Creator-first onboarding (3-step) |
| `/agents/create?path=classic`   | Step-by-step setup                |
| `/agents/[id]`                  | Individual agent interface        |
| `/conversations`                | Conversation history              |

### 2. API Gateway (Fastify)

**Port:** 4000

**Responsibilities:**

* Route requests to appropriate microservices
* CORS configuration and security
* Rate limiting and request validation
* Health check aggregation

### 3. Microservices Layer

| Service           | Port | Purpose                                                                   |
| ----------------- | ---- | ------------------------------------------------------------------------- |
| Auth Service      | 4001 | User authentication, JWT, organizations, RBAC                             |
| Bot Service       | 4002 | Agent CRUD, AI chat (OpenRouter/LangChain), RAG context, billing          |
| Realtime Audio    | 4003 | Voice cloning (ElevenLabs), STT, LiveKit (Coming Soon)                    |
| Knowledge Service | 4005 | KB management, RAG pipeline, content connectors, knowledge graph          |
| HITL Service      | 4006 | Human-in-the-loop escalation routing, operator dashboard, approval queues |

### 4. Infrastructure Layer

| Component     | Port | Purpose                                  |
| ------------- | ---- | ---------------------------------------- |
| PostgreSQL 15 | 5432 | Primary relational database              |
| Redis 7       | 6379 | Caching, BullMQ job queue, rate limiting |
| ChromaDB      | 8000 | Vector database (local dev)              |

### 5. External Services

| Service            | Purpose                                                 |
| ------------------ | ------------------------------------------------------- |
| OpenRouter         | 300+ AI models, unified API, auto-sync every 60 minutes |
| ElevenLabs         | Voice cloning and synthesis                             |
| AWS S3             | File storage for message attachments                    |
| Pinecone           | Production vector database                              |
| Gemini Embedding 2 | Multimodal embeddings (3072 dimensions)                 |

## Data Flow Diagrams

### User Authentication Flow

```mermaid theme={null}
sequenceDiagram
    participant U as User
    participant FE as Frontend (3000)
    participant GW as Gateway (4000)
    participant AUTH as Auth Service (4001)
    participant DB as PostgreSQL

    U->>FE: Submit login form
    FE->>GW: POST /api/auth/login
    GW->>AUTH: Forward request
    AUTH->>DB: Validate credentials
    DB-->>AUTH: User record
    AUTH-->>GW: JWT token
    GW-->>FE: JWT token
    FE-->>U: Store in cookies, redirect
```

### AI Chat Message Flow

```mermaid theme={null}
sequenceDiagram
    participant U as User
    participant FE as Frontend (3000)
    participant GW as Gateway (4000)
    participant BOT as Bot Service (4002)
    participant KB as Knowledge Service (4005)
    participant OR as OpenRouter
    participant DB as PostgreSQL

    U->>FE: Send message
    FE->>GW: POST /api/bots/:id/chat
    GW->>BOT: Forward with auth headers
    BOT->>DB: Get agent config + history
    BOT->>KB: Retrieve KB context (vector search + graph)
    KB-->>BOT: Relevant chunks + citations
    BOT->>OR: LangChain invoke (system prompt + KB context + message)
    OR-->>BOT: AI response
    BOT->>DB: Save messages, record billing
    BOT-->>FE: Response + sources
    FE-->>U: Display with citation tooltips
```

### Billing and Credit Accounting Flow

```mermaid theme={null}
sequenceDiagram
    participant SVC as Service
    participant FN as record_external_cost_event()
    participant DB as PostgreSQL

    SVC->>FN: raw_cost_usd, org_id, provider, operation
    FN->>DB: Load billing defaults (margin, credit value)
    FN->>DB: Lock wallet row (FOR UPDATE)
    FN->>DB: Calculate charged_usd and credits_burned
    FN->>DB: Update wallet balance
    FN->>DB: Insert external_cost_events row
    FN->>DB: Insert credit_ledger row
    FN-->>SVC: Billing result (event_id, balance_after)
```

### Knowledge Base RAG Flow

<Steps>
  <Step title="Ingest Content">
    User uploads document or adds URL source. Frontend sends to Knowledge Service via Gateway.
  </Step>

  <Step title="Resolve Connector">
    ConnectorRegistry resolves source to the appropriate ContentConnector (YouTube, Instagram, Twitter, RSS, URL, blog-platform, or document).
  </Step>

  <Step title="Normalize Content">
    Connector fetches and returns `NormalizedContent[]` with dedup keys and content hashes. For social media: one `kb_document` per post with metadata.
  </Step>

  <Step title="Process Media">
    Vision text extraction (OpenRouter) for images/PDFs. Transcription service for audio/video content. Text chunking with metadata preservation.
  </Step>

  <Step title="Generate Embeddings">
    Gemini Embedding 2 generates multimodal embeddings (3072 dimensions) for text, images, video, and audio content.
  </Step>

  <Step title="Store Vectors">
    Vectors stored in ChromaDB (local) or Pinecone (production). Metadata saved to PostgreSQL.
  </Step>

  <Step title="Query and Retrieve">
    User queries KB. Service generates query embedding, retrieves chunks, and Bot Service uses chunks as context for OpenRouter. AI generates response with enriched citations.
  </Step>
</Steps>

### Knowledge Graph Flow

```mermaid theme={null}
graph LR
    A[Document Ingested] --> B[BullMQ Queue]
    B --> C[Entity Extraction<br/>Text + Vision + Audio]
    C --> D[Store in PostgreSQL<br/>kg_entities / kg_relationships]
    D --> E[Community Detection<br/>kg_communities]
    D --> F[Document Summaries<br/>kg_document_registry]

    G[Query Time] --> H[Graph Traversal<br/>Recursive CTEs]
    G --> I[Vector Search +<br/>Graph Context]
    G --> J[KB Map Overview]
    H & I & J --> K[Enriched RAG Response]
```

## Security

<AccordionGroup>
  <Accordion title="Authentication">
    * JWT-based authentication
    * Secure password hashing (bcrypt)
    * Token refresh mechanism
    * Multi-tenant isolation
  </Accordion>

  <Accordion title="Authorization">
    * Two-tier RBAC (platform + organization)
    * Resource-level permissions (groups + direct grants)
    * `checkResourceAccess()` middleware on all resource routes
  </Accordion>

  <Accordion title="Data Protection">
    * AES-256-GCM encryption for sensitive config values
    * Parameterized SQL queries (no ORM)
    * HTTPS in production
    * Token-based file access with expiration
  </Accordion>
</AccordionGroup>

## Deployment

All services run under **Docker Compose**. There are three environments:

| Environment | Branch                  | URL                                                                              | Notes                            |
| ----------- | ----------------------- | -------------------------------------------------------------------------------- | -------------------------------- |
| Development | `main` (auto-deploy)    | [https://bsio2.brainstormer.io](https://bsio2.brainstormer.io)                   | Features land here first         |
| Staging     | `staging` (auto-deploy) | [https://staging-bsio2.noesisinfo.co.in](https://staging-bsio2.noesisinfo.co.in) | Pre-release verification         |
| Production  | — (manual deploy)       | [https://app.brainstormer.io](https://app.brainstormer.io)                       | Live product, manual deploy only |

`main` auto-deploys to the development server. Production is **manual only** — no auto-deploy from any branch.