Skip to main content

​
Overview

This document outlines the migration strategy from base64-encoded file handling to the URL-based cloud storage system. The migration eliminates critical limitations of base64 encoding while ensuring zero downtime and backward compatibility.

​
Migration Overview

Before (Base64)

  • 33% file size overhead
  • 25MB max file size
  • Full file in memory
  • Model-specific compatibility issues

After (URL-based)

  • No encoding overhead
  • 2GB+ file support
  • Streaming access
  • Universal LLM compatibility

​
Migration Phases

​
Phase 1: Infrastructure Preparation

Status: Complete
  • Database schema: message_attachments and file_storage_config tables
  • Local filesystem storage with secure paths
  • Organization-based storage partitioning
  • Access token generation

​
Phase 2: Core Service Implementation

Status: Complete
  • FileUploadService: validation, storage, token-based access, processing queue
  • AttachmentRepository: CRUD operations, processing status, cleanup

​
Phase 3: API Integration

Status: Complete
  • Upload endpoint: POST /api/agents/attachments/upload
  • File access endpoint: GET /api/agents/attachments/:accessToken/file
  • Chat integration: attachments passed as URL references

​
Phase 4: Frontend Integration

Status: Complete
  • Drag-and-drop file upload
  • Progress tracking and error handling
  • File type validation on client
  • Preview generation
  • Message display with download links

​
Phase 5: LLM Provider Integration

Status: Complete All major LLM providers supported with URL-based file access:
Fallback mechanism: If URL access fails, the system automatically falls back to base64 encoding.

​
Performance Benchmarks

​
Risk Mitigation

SHA-256 hash verification for all files. Automatic integrity checks before LLM processing.
64-character cryptographically secure access tokens. Organization-based access control. Regular token rotation and expiration.
Optimized database queries with strategic indexing. Connection pooling. CDN integration planned.
Comprehensive testing with all LLM providers. Automatic fallback to base64 when URLs are not supported.

​
Rollback Strategy

If critical issues are encountered:
1

Immediate Rollback

Disable new file upload endpoint. Route uploads through legacy base64 processing.
2

Gradual Migration Back

Convert recent URL-based attachments to base64 storage. Maintain database records for audit.
3

Re-enablement

Implement feature flags for gradual re-enablement after fixes.

​
Success Metrics

​
Future Enhancements

  • Amazon S3: Full integration with signed URLs
  • Google Cloud Storage: GCS for Google Cloud deployments
  • Azure Blob Storage: Azure ecosystem integration
  • CDN: Global content delivery network
  • Image Processing: Real-time resizing and optimization