Skip to main content
Brainstormer supports a wide range of file types for knowledge base ingestion. Every document you upload goes through an intelligent processing pipeline that extracts text, generates embeddings, and makes the content searchable by your agents.
Knowledge base detail view with document statistics

Knowledge base detail page showing document stats and content

​
Supported File Types

​
How to Upload

1

Open your knowledge base

Navigate to Knowledge in the sidebar and select the knowledge base you want to add content to.
2

Upload files

Click Upload Documents and either:
  • Drag and drop files into the upload area
  • Click to browse and select files from your computer
You can upload multiple files at once.
3

Monitor processing

Each uploaded file shows its processing status:
  • Pending — Waiting in the processing queue
  • Processing — Being parsed, chunked, and embedded
  • Completed — Ready for search and retrieval
  • Failed — Processing error (click for details)

​
The Processing Pipeline

When you upload a document, it goes through these stages:

​
1. Document Loading

The system selects the best parser for your file type:
  • LlamaParse — Used for complex PDFs and DOCX files with tables, images, and multi-column layouts. Requires a LlamaParse API key.
  • LangChain loaders — Used for simpler formats (TXT, CSV, basic PDFs). Always available as a fallback.
If LlamaParse is not configured, the system automatically falls back to LangChain loaders. Processing still works, but complex document layouts may not be parsed as accurately.

​
2. Content Extraction

For non-text media, additional extraction steps run:
  • Vision text extraction — Images and PDF pages with embedded images are analyzed by OpenRouter vision models to extract text, descriptions, and context.
  • Transcription — Audio and video content is transcribed to text using Whisper.

​
3. Chunking

Extracted text is split into chunks for embedding:
  • Default chunk size: 1,000 characters
  • Default overlap: 200 characters
Overlap ensures that context at chunk boundaries is preserved. Each chunk retains metadata about its position in the original document.

​
4. Embedding

Each chunk is converted into a 3072-dimensional vector using Gemini Embedding 2:
  • Text chunks — Standard text embedding
  • Image chunks — Multimodal embedding (the image itself is embedded, not just extracted text)
  • Audio/video chunks — Embedded from transcribed text

​
5. Vector Storage

Embeddings are stored in your configured vector database:
  • Pinecone — Used in production for scalable, managed vector search
  • ChromaDB — Used in local development

​
6. Optional: Graph Extraction

If enabled, the system extracts entities (people, organizations, concepts, products) and relationships from your content to build a knowledge graph. See Knowledge Graph for details.

​
Processing Queue

Document processing runs asynchronously through a job queue (BullMQ + Redis). This means:
  • Uploads return immediately — you do not need to wait for processing
  • Up to 5 documents are processed concurrently
  • Failed jobs are retried up to 3 times automatically
  • You can continue adding more documents while others are processing
Large files (especially video and long PDFs) take longer to process. For the fastest results, start with your most important documents and add the rest while you configure your agent.

​
Adding Web Sources

In addition to file uploads, you can add web-based content:
  • URLs — Any publicly accessible web page
  • RSS feeds — Automatically pull in all items from a feed
  • Sitemaps — Discover and add all pages from a sitemap
Web sources go through the same processing pipeline as uploaded files. RSS feeds and sitemaps can be configured for auto-sync, periodically checking for new content and ingesting it automatically.

​
Managing Documents

From your knowledge base’s document list, you can:
  • View status — See processing state and any error details
  • Browse chunks — See how a document was split and what was extracted
  • Delete — Remove a document and its associated chunks and embeddings
  • Re-process — Re-run the processing pipeline on a failed document
Deleting a document removes its chunks and vector embeddings permanently. If the document was being used by linked agents, those agents will no longer be able to cite content from that document.