DataForSEO Integration

Complete guide to DataForSEO API integration, rate limits, and usage patterns.

Overview

DataForSEO provides keyword research, SERP tracking, and competitive intelligence data. RankFabric uses several DataForSEO APIs:

Instant Pages - Fetch HTML snapshots of websites
Keyword Suggestions - Related keywords with search volume
Keyword Overview - Full keyword metrics (volume, CPC, intent, history)
Categories for Keywords - Category classification
Keywords for Keywords - Related keyword discovery
SERP API - Search ranking positions
Domain Metrics - Top domains per category

Configuration

Environment Variables

Set in wrangler.toml:

[vars]
DATAFORSEO_LABS_LIMIT = "100"
DATAFORSEO_LABS_MAX_REQUESTS = "1"
DATAFORSEO_DAILY_BUDGET = "50"

Secrets

Store credentials securely:

echo "your-login" | wrangler secret put DATAFORSEO_LOGIN
echo "your-password" | wrangler secret put DATAFORSEO_PASSWORD

Get credentials: Sign up at dataforseo.com

APIs Used

1. Instant Pages

Fetch HTML snapshots of websites for analysis.

Endpoint: POST https://api.dataforseo.com/v3/on_page/instant_pages

Used In: src/lib/dataforseo.js → fetchInstantPage()

Request:

[{
  "target": "example.com",
  "check_spell": false,
  "disable_cookie_popup": true,
  "load_resources": false,
  "enable_javascript": false
}]

Response:

{
  "tasks": [{
    "result": [{
      "items": [{
        "url": "https://example.com",
        "content": "<!DOCTYPE html>...",
        "meta": {...}
      }]
    }],
    "cost": 0.01
  }]
}

Cost: ~$0.01 per page

Rate Limit: 2000 requests/minute

Usage:

Keyword research flow (analyze site HTML)
Content extraction for LLM analysis

2. Keyword Suggestions

Get related keywords with search volume.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/keyword_suggestions/live

Used In: src/lib/dataforseo.js → fetchKeywordSuggestions()

Request:

[{
  "keyword": "seo tools",
  "location_code": 2840,
  "language_code": "en",
  "limit": 100
}]

Response:

{
  "tasks": [{
    "result": [{
      "items": [
        {
          "keyword": "best seo tools",
          "search_volume": 8100,
          "competition": 0.75,
          "cpc": 12.50
        }
      ]
    }]
  }]
}

Cost: ~$0.004 per 100 keywords

Batch Limit: 100 keywords per request

Usage:

Generate related keywords
Expand keyword lists

3. Keyword Overview

Full keyword data including 72-month history.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/keyword_overview/live

Used In: src/endpoints/keyword-overview.js

Request:

[{
  "keywords": ["seo tools", "keyword research"],
  "location_code": 2840,
  "language_code": "en"
}]

Response:

{
  "tasks": [{
    "result": [{
      "items": [
        {
          "keyword": "seo tools",
          "search_volume": 27100,
          "difficulty": 75,
          "cpc": 15.30,
          "competition": 0.85,
          "monthly_searches": [
            {"month": 11, "year": 2025, "search_volume": 27100}
          ],
          "keyword_properties": {
            "intent": "commercial"
          }
        }
      ]
    }]
  }]
}

Cost: ~$0.004 per keyword

Batch Limit: 100 keywords per request

Usage:

Get comprehensive keyword metrics
Historical trend analysis

4. Categories for Keywords

Get DataForSEO category IDs for keywords.

Endpoint: POST https://api.dataforseo.com/v3/dataforseo_labs/google/categories_for_keywords/live

Used In: src/endpoints/categories-for-keywords.js

Request:

[{
  "keywords": ["seo tools", "keyword research"],
  "location_code": 2840,
  "language_code": "en"
}]

Response:

{
  "tasks": [{
    "result": [{
      "items": [
        {
          "keyword": "seo tools",
          "categories": [12045, 12046]
        }
      ]
    }]
  }]
}

Cost: ~$0.004 per 100 keywords

Usage:

Auto-categorize keywords
Map to DataForSEO category taxonomy

5. Keywords for Keywords

Get related keywords ranked by relevance.

Endpoint: POST https://api.dataforseo.com/v3/keywords_data/google_ads/keywords_for_keywords/live

Used In: src/endpoints/keywords-for-keywords.js

Request:

[{
  "keywords": ["seo tools"],
  "location_code": 2840,
  "language_code": "en",
  "sort_by": "relevance"
}]

Response:

{
  "tasks": [{
    "result": [{
      "keywords": [
        {
          "keyword": "best seo tools",
          "competition": "HIGH",
          "search_volume": 8100
        }
      ]
    }]
  }]
}

Cost: ~$0.05 per request

Usage:

Keyword expansion
AI prompt enrichment

6. SERP API

Get search engine ranking positions.

Endpoint: POST https://api.dataforseo.com/v3/serp/google/organic/live/advanced

Used In: src/lib/dataforseo.js → fetchSerpResults()

Request:

[{
  "keyword": "seo tools",
  "location_code": 2840,
  "language_code": "en",
  "device": "desktop",
  "depth": 100
}]

Response:

{
  "tasks": [{
    "result": [{
      "items": [
        {
          "type": "organic",
          "rank_group": 1,
          "rank_absolute": 1,
          "url": "https://example.com/seo-tools",
          "title": "Best SEO Tools",
          "description": "..."
        }
      ]
    }]
  }]
}

Cost: ~$0.003 per keyword

Usage:

SERP tracking
Competitive analysis

Rate Limits & Quotas

Account Limits

Default limits (varies by plan):

Requests per minute: 2000
Requests per day: Unlimited (cost-based)
Concurrent requests: 100

Budget Management

RankFabric tracks daily budget in KV:

Budget Key: budget:daily

Check budget:

const spent = await env.DFS_BUDGETS.get('budget:daily');
const limit = env.DATAFORSEO_DAILY_BUDGET || 50;

Budget enforcement:

if (parseFloat(spent) >= parseFloat(limit)) {
  throw new Error('Daily budget exceeded');
}

Reset: Automatic at midnight UTC

Caching Strategy

KV Cache

DataForSEO responses are cached in KV to reduce costs.

Cache Keys:

dfs:categories:{keyword}
dfs:suggestions:{keyword}:{location}
dfs:serp:{keyword}:{location}:{device}

TTL:

Category data: 30 days
Keyword suggestions: 7 days
SERP data: 1 day

Example:

const cacheKey = `dfs:suggestions:${keyword}:${location}`;
const cached = await env.DFS_RUNS.get(cacheKey);
if (cached) return JSON.parse(cached);

// Fetch from DataForSEO
const result = await fetchFromDataForSEO(...);

// Cache result
await env.DFS_RUNS.put(cacheKey, JSON.stringify(result), {
  expirationTtl: 86400 * 7
});

Category Taxonomy

DataForSEO provides 3,182 categories stored in KV.

KV Key: dataforseo-categories.json

Structure:

{
  "12045": {
    "id": 12045,
    "name": "Internet & Telecom > Web Services > SEO & SEM",
    "path": ["Internet & Telecom", "Web Services", "SEO & SEM"]
  }
}

Usage:

// Get all categories
const categoriesJson = await env.DFS_RUNS.get('dataforseo-categories.json');
const categories = JSON.parse(categoriesJson);

// Map category ID to name
const categoryName = categories[12045]?.name;

Error Handling

Common Errors

401 Unauthorized:

{
  "status_code": 20103,
  "status_message": "Incorrect login or password"
}

Fix: Verify DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD secrets

429 Rate Limit:

{
  "status_code": 40901,
  "status_message": "Rate limit exceeded"
}

Fix: Implement backoff/retry or reduce request rate

402 Insufficient Funds:

{
  "status_code": 40204,
  "status_message": "Insufficient account balance"
}

Fix: Add credits to DataForSEO account

Retry Logic

Automatic retry with exponential backoff:

async function retryWithBackoff(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === maxRetries - 1) throw err;
      const delay = Math.pow(2, i) * 1000;
      await new Promise(res => setTimeout(res, delay));
    }
  }
}

Retry on:

Network errors
5xx server errors
Rate limit errors (with longer delay)

Don't retry on:

401 authentication errors
400 bad request errors
402 insufficient funds

Cost Optimization

Best Practices

Use batch endpoints:

// Good: 100 keywords in 1 request ($0.004)
fetchKeywordOverview(keywords: [...100])

// Bad: 100 requests ($0.40)
for (const kw of keywords) {
  fetchKeywordOverview([kw])
}

Cache aggressively:
- Category data rarely changes → 30 day TTL
- Search volume changes monthly → 7 day TTL
Use cheaper APIs when possible:
- Keyword Suggestions ($0.004) vs Keywords for Keywords ($0.05)
Set daily budgets:
```
[vars]
DATAFORSEO_DAILY_BUDGET = "25"
```

Monitor costs:

-- Track daily spending in ClickHouse
SELECT 
  toDate(created_at) as date,
  SUM(cost) as total_cost
FROM dataforseo_requests
GROUP BY date
ORDER BY date DESC;

Implementation Details

Authentication

HTTP Basic Auth with Base64 encoding:

function getAuthHeader(env) {
  const login = env.DATAFORSEO_LOGIN;
  const password = env.DATAFORSEO_PASSWORD;
  const credentials = `${login}:${password}`;
  const encoded = btoa(unescape(encodeURIComponent(credentials)));
  return `Basic ${encoded}`;
}

Request Format

All DataForSEO endpoints expect POST with JSON array:

const payload = [{
  keyword: "seo tools",
  location_code: 2840,
  language_code: "en"
}];

const response = await fetch(endpoint, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': getAuthHeader(env)
  },
  body: JSON.stringify(payload)
});

Response Parsing

Standard response structure:

const data = await response.json();
const tasks = data.tasks || [];
const results = [];

for (const task of tasks) {
  const taskResults = task.result || [];
  for (const result of taskResults) {
    const items = result.items || [];
    results.push(...items);
  }
}

Cost tracking:

const cost = tasks[0]?.cost || 0;
await trackDataForSEOCost(cost);

Location Codes

Common location codes:

Code	Location
2840	United States
2826	United Kingdom
2124	Canada
2036	Australia
2276	Germany

Full list: DataForSEO Location Codes

Language Codes

Common language codes:

Code	Language
en	English
es	Spanish
fr	French
de	German
ja	Japanese

Full list: DataForSEO Language Codes

File Locations

Main library: src/lib/dataforseo.js
Adapters: src/adapters/dataforseo-instant-pages-adapter.js
Endpoints: src/endpoints/*
Category mapping: src/lib/dataforseo-categories.js

Overview​

Configuration​

Environment Variables​

Secrets​

APIs Used​

1. Instant Pages​

2. Keyword Suggestions​

3. Keyword Overview​

4. Categories for Keywords​

5. Keywords for Keywords​

6. SERP API​

Rate Limits & Quotas​

Account Limits​

Budget Management​

Caching Strategy​

KV Cache​

Category Taxonomy​

Error Handling​

Common Errors​

Retry Logic​

Cost Optimization​

Best Practices​

Implementation Details​

Authentication​

Request Format​

Response Parsing​

Location Codes​

Language Codes​

File Locations​

See Also​

Overview

Configuration

Environment Variables

Secrets

APIs Used

1. Instant Pages

2. Keyword Suggestions

3. Keyword Overview

4. Categories for Keywords

5. Keywords for Keywords

6. SERP API

Rate Limits & Quotas

Account Limits

Budget Management

Caching Strategy

KV Cache

Category Taxonomy

Error Handling

Common Errors

Retry Logic

Cost Optimization

Best Practices

Implementation Details

Authentication

Request Format

Response Parsing

Location Codes

Language Codes

File Locations

See Also