Domain Onboarding Flow

Architecture Overview

Domain onboarding now uses Cloudflare Workflows as the orchestration layer:

POST /api/assets (type=website)
    ↓
AssetOnboardWorkflow.create()
    ↓
DomainOnboardWorkflow.create()
    ↓
domain-onboard-consumer (queue execution)
    ↓
backlink-classify-consumer (URL classification)

---

Mermaid Diagram

---

Entry Points

1. POST /api/assets (Website Asset)

File: src/endpoints/assets.js

When a user adds a website asset:

1. Asset record created in assets table
2. Domain record created via ensureDomain()
3. AssetOnboardWorkflow.create() triggered
4. Workflow delegates to DomainOnboardWorkflow
5. Returns workflow_id and status_url

// Response
{
  "success": true,
  "id": "asset_abc123",
  "domain_id": 456,
  "onboarding": {
    "started": true,
    "orchestration": "workflow",
    "workflow_id": "wf_xyz789",
    "status_url": "/api/admin/workflow/asset-onboard/wf_xyz789"
  }
}

2. POST /api/admin/domains/onboard

File: src/endpoints/admin-domains.js

Direct domain onboarding (admin):

1. Validates domain list
2. Creates domain records
3. DomainOnboardWorkflow.create() for each
4. Returns workflow IDs

3. triggerDomainOnboarding()

File: src/lib/domain-onboarding.js

Shared utility for triggering domain onboarding:

• Called by assets.js, competitors.js, etc.
• Workflow-first with queue fallback
• Handles deduplication (skips if already onboarded)

---

Workflow Steps

DomainOnboardWorkflow

async run(event, step) {
  // Step 1: Initialize
  await step.do('initialize', async () => {
    // Set onboard_status = 'in_progress'
    // Store workflow_id in domain record
  });

  // Step 2: Fetch Backlinks
  await step.do('fetch-backlinks', async () => {
    // Call DataForSEO Backlinks API
    // Store in backlinks table
    // Queue URLs for classification
  });

  // Step 3: Fetch Keywords
  await step.do('fetch-keywords', async () => {
    // Call DataForSEO Ranked Keywords API
    // Store in domain_keyword_rankings
  });

  // Step 4: Fetch Summary
  await step.do('fetch-summary', async () => {
    // Call DataForSEO Domain Overview API
    // Store in domain_summaries
  });

  // Step 5: Finalize
  await step.do('finalize', async () => {
    // Set onboard_status = 'complete'
    // Set onboard_completed_at
  });

  return { success: true, domain_id, backlinks_count, keywords_count };
}

---

Fallback Behavior

If workflows are unavailable, the system falls back to queue-based orchestration:

// In triggerDomainOnboarding()
if (env.DOMAIN_ONBOARD_WORKFLOW) {
  // Try workflow first
  const instance = await env.DOMAIN_ONBOARD_WORKFLOW.create({ params });
  return { orchestration: 'workflow', workflow_id: instance.id };
} else if (env.DOMAIN_ONBOARD_QUEUE) {
  // Fallback to queue
  await env.DOMAIN_ONBOARD_QUEUE.sendBatch(jobs);
  return { orchestration: 'queue', jobs_count: jobs.length };
}

---

Cost Breakdown

Operation	API	Cost
Backlinks	DataForSEO Backlinks	$0.04 per 50 URLs
Keywords	DataForSEO Ranked Keywords	$0.03 per 100 keywords
Summary	DataForSEO Domain Overview	$0.02
Properties	DataForSEO Domain Properties	$0.01
URL Classification (Stage 3)	ZenRows	$0.001 per URL
URL Classification (Stage 4)	OpenAI	$0.01 per URL

Total per domain: ~$0.10 + ($0.01 × low-confidence URLs)

---

Monitoring

Check workflow status:

curl /api/admin/workflow/domain-onboard/{workflow_id}

Response:

{
  "id": "wf_abc123",
  "status": "running",
  "steps": [
    { "name": "initialize", "status": "completed" },
    { "name": "fetch-backlinks", "status": "running" },
    { "name": "fetch-keywords", "status": "pending" }
  ]
}