Agent Provisioning

Overview

Agent Provisioning turns a creator’s source URL (a YouTube or Instagram channel) into a fully-provisioned, chat-ready agent. A single server-side orchestrator — AgentProvisioningService (services/bot/src/services/agent-provisioning.service.ts) — owns the entire chain: detect the source, scrape recent content, analyze the creator’s voice, create the knowledge base and agent, link and index sources, and (optionally) publish a public slug.

This page documents the internal platform architecture. For the consumer-facing async API, see the Provisioning quickstart and the agents:from-url reference.

Callers

Two entry points run the same provision() method. Behavioural differences are encoded as input flags.

Caller	Entry route	Backend	`waitForIndexing`	Default `mode`
In-app wizard	`POST /bots/provision`	enqueue build, poll `GET /bots/provision/builds/:buildId`	`false`	`full`
Public async API	`POST /v1/agents:from-url` (gateway)	`POST /agent-builds` -> BullMQ worker	`true`	`preview`

Both paths enqueue a BullMQ agent-provisioning job and return 202 with a buildId plus poll/stream URLs; neither blocks the HTTP request. The wizard returns as soon as the agent exists and indexes in the background. The public API waits for the bounded index to reach a terminal state before emitting ready.

Phase State Machine

provision() advances a monotonic phase FSM, emitting one ProvisioningEvent per transition with an interpolated progress weight.

queued -> detecting -> scraping -> analyzing -> provisioning -> indexing -> ready
                                                                          \-> failed

Phase	Progress	What happens
`queued`	0	Job accepted, build row created
`detecting`	5	Platform + handle parsed from URL; plan/credit pre-flight
`scraping`	15	`POST /knowledge/light-scrape` reads recent public content + brand
`analyzing`	35	`CreatorAnalysisService` derives the system prompt, welcome, description
`provisioning`	65	KB created, sources batched, `bots` + RBAC + tools + prompt config + widget config
`indexing`	82	(async only) poll KB sources until terminal or timeout
`ready`	100	Agent chat-ready; result returned/persisted
`failed`	100	Carries a `ProvisioningErrorCode` (`scrape_failed`, `analysis_failed`, `kb_required`, `plan_limit`, `insufficient_credits`, …)

Document-only inputs skip scraping/analyzing and produce a blank agent with a warning. Failures throw ProvisioningServiceError, which carries the error code, an HTTP status, and a retryable flag.

Redis-to-SSE Fan-out

Progress is decoupled from the SSE connection through Redis, so the SSE client can land on a different bot replica than the one running the job. Each event is fanned out three ways by handleEvent in provisioning.queue.ts: persisted to agent_builds, published to Redis provision:progress:{buildId}, and — on terminal events only — delivered to the consumer’s optional HMAC-signed webhook (provision-bridge.service.ts: services/bot/src/services/provision-bridge.service.ts). The SSE route maps phases to SSE event names (ready -> done, failed -> error, else progress), uses the event’s emittedAt as the SSE id: for Last-Event-ID replay, replays the current snapshot on connect, and closes immediately if the build is already terminal.

The `agent_builds` State Machine

Migration 080_agent_provisioning.sql adds agent_builds — the durable state machine behind every build (both callers; application_id is NULL for the wizard path). It powers:

Snapshot / poll — GET /agent-builds/:buildId returns the latest ProvisioningEvent (stored in last_event).
SSE replay — last_event + last_event_id let a reconnecting client resume.
Idempotency — a client Idempotency-Key short-circuits duplicate submits via findByIdempotencyKey.
Handle dedup — findReusableByHandle returns a fresh, published agent for the same (org, platform, normalized_handle), making repeat submits instant and free.
Preview TTL — preview builds get expires_at = now + 7 days.
Full result — migration 081 adds a result JSONB column so the wizard’s step-2 review hydrates from a polled build.

A node-cron worker (workers/provisioning-cleanup.worker.ts, every 6h, env-gated PROVISIONING_CLEANUP_ENABLED=true) soft-deletes expired preview resources (KB links + knowledge base) and stamps cleaned_at.

Service-Account JWT & Cost Attribution

Bot-side writes run in-process via the existing repositories. Knowledge-side work (light-scrape, KB create, sources/batch) and the three AI suggestion endpoints run over internal HTTP, authenticated with a short-lived (600s) service-account JWT minted via signJwt(..., config.JWT_SECRET):

signJwt(
  {
    userId: input.createdBy, // service-account UUID (API) or real user UUID (wizard)
    email: "[email protected]",
    platformRole: "user",
    organizations: [{ id: input.organizationId, role: input.orgRole || "member" }],
  },
  config.JWT_SECRET,
  600,
);

For the public API, every API key resolves to a real service_account_user_id (migration 080, api_applications.service_account_user_id). That UUID becomes created_by/owned_by, the resource_permissions grant target, and the billing user_id — so all provisioning costs attribute to the service account, not a synthetic string. Each provider-cost building block (creator analysis, light-scrape, KB embedding, AI suggestions) sets its own narration at its own boundary; the orchestrator records no un-narrated direct cost.

Knowledge Base Readiness for Auto-Publish

When input.publish is true, the orchestrator verifies that the agent has at least one linked knowledge base with at least one indexed document before publishing the public slug. If the KB is missing or still indexing, the build fails with:

Error code	Meaning
`kb_required`	No linked KB, or no indexed documents yet. The build cannot publish a live slug.

This prevents public agents from going live without grounded context. The in-app wizard path (publish: false) creates the agent as a draft and defers the readiness check to the user-driven Go Live step.

Plan-Limit & Credit Pre-flight

assertWithinLimits() runs server-side before any work, so both callers are covered. It reads billing.getPlanLimitsAndUsage() and throws ProvisioningServiceError(..., httpStatus 402):

Credit balance is always enforced — creditBalance <= 0 throws insufficient_credits even for superadmins.
Resource limits (maxAgents, maxKbs) throw plan_limit, but are skipped when bypassPlanLimits is set (superadmins, mirroring the gateway billing-guard).

The sync wizard route maps the 402 straight back to the client; the async worker converts it into a failed event. Orgs with no billing account are treated as unmetered.

Agent Creation Contract Compliance

The orchestrator honors the full Agent Creation Contract — POST /bots alone is insufficient. In one run it:

Creates the bots row + RBAC grant (createBot, owner = owned_by).
Links the KB via kb_agent_links (upsertKBLinks).
Writes bot_tools including builtin:kb_context_injection (and hitl:escalate_to_human for support intent).
Writes bot_prompt_configs/system (mode: "fixed") so the editor + version history populate.
Writes bots.widget_config.welcome (+ brand + starters) when configured.
Pins the latest prompt versions, then publishes a unique public slug when publish is set.

Bounded Preview Ingestion

preview is a mode flag on the same provision() path. It defaults to maxItems: 8, the only ingestion hint plumbed to connectors via sources/batch (recentFirst / transcriptsOnly are accepted but best-effort). The async submit route caps ingestion.maxItems at 50. The indexing wait uses a 120s deadline for preview (vs 180s for full) and proceeds to ready on timeout, letting background indexing continue.

Key Files

File	Role
`services/bot/src/services/agent-provisioning.service.ts`	The orchestrator (`provision()`)
`services/bot/src/queues/provisioning.queue.ts`	BullMQ worker, 3-way event fan-out, SSRF-guarded webhook
`services/bot/src/services/provision-bridge.service.ts`	Redis pub/sub -> in-process SSE emitter
`services/bot/src/routes/provisioning.routes.ts`	`agent-builds` create / poll / SSE routes
`services/gateway/src/routes/public-api.ts`	`/v1/agents:from-url` surface (scope, rate-limit, Turnstile)
`services/bot/src/workers/provisioning-cleanup.worker.ts`	Preview TTL cleanup cron
`infrastructure/postgres/migrations/080_agent_provisioning.sql`, `081`	`agent_builds` table, service-account user, `result` column

Full internal contract: docs/architecture/AGENT_PROVISIONING_API.md.

Overview

Services

Architecture