Diagnostics & Troubleshooting

Debug endpoints, health checks, and troubleshooting procedures.

Health Checks
Debug Endpoints
Webhook Troubleshooting
Queue Diagnostics
Common Issues
Logging & Monitoring

Health Checks

ClickHouse Connection Test

Test ClickHouse connectivity and query a sample table.

curl https://your-worker.workers.dev/test/clickhouse

Success Response:

{
  "status": "ok",
  "clickhouse_connected": true,
  "sample_query_result": [...]
}

Failure Response:

{
  "status": "error",
  "error": "Connection timeout",
  "clickhouse_connected": false
}

Initialize ClickHouse Schema

Create ClickHouse tables on first deployment.

curl https://your-worker.workers.dev/init/clickhouse

Response:

{
  "status": "initialized",
  "tables_created": [
    "keyword_snapshots",
    "serp_runs",
    "app_analytics"
  ]
}

Auto-initialization:

If CLICKHOUSE_AUTO_INIT=true in environment, schema initializes automatically on first request.

Debug Endpoints

Debug Room Data

Debug Apple room/grouping data structure.

curl "https://your-worker.workers.dev/api/app-store/room/debug?room_id=apple_games"

Response:

{
  "room_id": "apple_games",
  "categories": [...],
  "hierarchy": {...},
  "raw_data": {...}
}

Debug Catalog

Debug complete catalog structure.

curl https://your-worker.workers.dev/api/app-store/catalog/debug

Response:

{
  "total_categories": 142,
  "by_platform": {
    "apple": 78,
    "google_play": 64
  },
  "hierarchy_depth": 4,
  "orphaned_categories": []
}

Queue Status

Get queue metrics and health.

curl https://your-worker.workers.dev/api/admin/queue-status

Response:

{
  "queues": {
    "rankfabric-tasks": {
      "pending": 15,
      "in_progress": 3,
      "completed_today": 1247
    },
    "clickhouse-ingestion": {
      "pending": 5,
      "in_progress": 1
    }
  }
}

Webhook Troubleshooting

Test Webhook Authentication

The worker sends webhooks to Base44/React when category syncs complete.

Webhook Headers:

X-Webhook-Secret: <NEXUS_WEBHOOK_SECRET>
Content-Type: application/json

Payload Example:

{
  "event": "categories_synced",
  "platform": "apple",
  "device": "iphone",
  "categories_count": 78,
  "timestamp": "2025-11-17T10:00:00Z"
}

Common Webhook Issues

401 Unauthorized

Cause: Secret mismatch between worker and receiver.

Fix:

# Verify worker secret is set
wrangler secret list | grep NEXUS_WEBHOOK_SECRET

# Set matching secret on both sides
echo "your-secret-here" | wrangler secret put NEXUS_WEBHOOK_SECRET

# Set same secret in React app environment
NEXUS_WEBHOOK_SECRET=your-secret-here

404 Not Found

Cause: Webhook URL incorrect or endpoint doesn't exist.

Fix:

# Verify webhook URL is set
wrangler secret list | grep BASE44_WEBHOOK_URL

# Set correct URL
echo "https://base44.app/api/functions/webhookCategories" | \
  wrangler secret put BASE44_WEBHOOK_URL

Connection Timeout

Cause: Receiver is down or not accessible.

Fix:

Check receiver app is running
Verify URL is publicly accessible
Check firewall/network rules
Test with curl:

curl -X POST https://your-webhook-url \
  -H "X-Webhook-Secret: your-secret" \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

Manual Webhook Test

Test webhook delivery manually:

# Set your secret
WEBHOOK_SECRET="your-secret-here"

# Test webhook call
curl -X POST https://your-webhook-url/api/functions/webhookCategories \
  -H "X-Webhook-Secret: $WEBHOOK_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "event": "test",
    "platform": "apple",
    "device": "iphone",
    "categories_count": 1
  }'

Expected: 200 OK with {"success": true}

Webhook Setup Checklist

Generate secret: openssl rand -hex 32
Set secret in worker: wrangler secret put NEXUS_WEBHOOK_SECRET
Set webhook URL: wrangler secret put BASE44_WEBHOOK_URL
Set same secret in React app environment
Deploy worker: wrangler deploy
Test webhook with curl
Trigger actual webhook (rebuild catalog)
Check logs on both sides

Queue Diagnostics

View Queue Logs

Tail worker logs in real-time:

wrangler tail --format pretty

Filter for specific queue jobs:

wrangler tail --format pretty | grep "harvest_keywords"

Dead Letter Queue

Check dead letter queue for failed jobs:

# View DLQ messages (Cloudflare dashboard)
# Navigate to: Queues → rankfabric-dlq → Messages

Retry Failed Jobs

Manually requeue failed jobs:

# Get failed job from DLQ
# Resubmit to main queue via admin endpoint (to be implemented)

Clear Queue

⚠️ Destructive operation

# Stop all queue consumers
wrangler queues consumer stop rankfabric-tasks

# Clear messages (via Cloudflare dashboard)

# Restart consumers
wrangler queues consumer start rankfabric-tasks

Common Issues

ClickHouse Connection Errors

Symptom: Failed to connect to ClickHouse

Solutions:

Check credentials:

wrangler secret list | grep CLICKHOUSE

Verify secrets are set:

echo "your-host" | wrangler secret put CLICKHOUSE_HOST
echo "your-user" | wrangler secret put CLICKHOUSE_USER
echo "your-password" | wrangler secret put CLICKHOUSE_PASSWORD
echo "your-database" | wrangler secret put CLICKHOUSE_DATABASE

Test connection:

curl https://your-worker.workers.dev/test/clickhouse

Reinitialize schema:

curl https://your-worker.workers.dev/init/clickhouse

Base44 Authentication Failures

Symptom: 401 Unauthorized from Base44 API

Solutions:

Verify Base44 secrets:

wrangler secret list | grep BASE44

Check JWT secret:

echo "your-jwt-secret" | wrangler secret put BASE44_JWT_SECRET

Test Base44 connection:

curl https://your-worker.workers.dev/diagnostics/keyword?keyword=test

Queue Stuck

Symptom: Jobs stay in pending status indefinitely

Solutions:

Check queue dashboard:
- Cloudflare Dashboard → Queues → rankfabric-tasks
- Look for error rate spike
Check worker logs:

wrangler tail --format pretty | grep ERROR

View dead letter queue:
- Check if jobs moved to DLQ
- Investigate error messages
Restart queue consumer:

# Redeploy worker to restart consumers
wrangler deploy

DataForSEO Quota Exceeded

Symptom: 429 Too Many Requests from DataForSEO

Solutions:

Check daily budget:

# Budget is tracked in KV: budget:daily
wrangler kv:key get --namespace-id=<your-namespace> "budget:daily"

Increase budget limit:

// In wrangler.toml
[vars]
DATAFORSEO_DAILY_BUDGET = "100"

Wait for reset:
- Budget resets at midnight UTC
- Or wait for account quota reset
Check account balance:
- Log into DataForSEO dashboard
- Verify account credits

App Details Not Found

Symptom: 404 Not Found for app details endpoint

Solutions:

Verify app exists in database:

-- Via wrangler d1 execute
SELECT * FROM apps WHERE id = 'com.example.app';

Check platform parameter:

# Correct
curl "...?platform=google_play"

# Wrong
curl "...?platform=android"

Trigger fresh crawl:

curl -X POST .../api/app-store/crawl/full \
  -d '{"platform":"google_play","categories":["..."]}'

Logging & Monitoring

Enable Debug Logging

Set log level in environment:

# wrangler.toml
[vars]
LOG_LEVEL = "debug"

Structured Logging

All logs include structured fields:

{
  "timestamp": "2025-11-17T10:00:00Z",
  "level": "info",
  "message": "Crawl started",
  "project_id": "proj_123",
  "platform": "google_play"
}

Key Log Patterns

Successful crawl:

[crawl] Starting project: proj_google_play_...
[crawl] Queued 10 categories
[crawl] Completed 10/10 categories
[crawl] Project complete: 150 apps discovered

Failed job:

[queue] Processing job: harvest_keywords
[queue] ERROR: Failed to fetch keywords
[queue] Retrying job (attempt 2/3)

Webhook delivery:

[webhook] Sending to: https://base44.app/...
[webhook] Response: 200 OK

Monitoring Dashboards

Cloudflare Dashboard:

Workers → Analytics → Requests, Errors, CPU time
Queues → Metrics → Queue depth, Processing time
D1 → Metrics → Reads, Writes, Storage

Custom Metrics (ClickHouse):

Query app crawl health:

SELECT 
  platform,
  COUNT(*) as total_apps,
  MAX(updated_at) as last_update
FROM apps
GROUP BY platform;

Query SERP tracking activity:

SELECT 
  toDate(tracked_at) as date,
  COUNT(*) as serp_runs
FROM serp_runs
WHERE tracked_at > now() - INTERVAL 7 DAY
GROUP BY date
ORDER BY date DESC;

Debugging Checklist

When something goes wrong:

Check worker logs:
```
wrangler tail --format pretty
```
Verify secrets are set:
```
wrangler secret list
```
Test external connections:
- ClickHouse: GET /test/clickhouse
- Base44: GET /diagnostics/keyword?keyword=test
Check queue status:
```
curl .../api/admin/queue-status
```
Review recent deployments:
```
wrangler deployments list
```
Check for rate limits:
- DataForSEO quota
- Worker CPU time limits
- D1 read/write limits

Verify data integrity:

wrangler d1 execute rankfabric_db --command "SELECT COUNT(*) FROM apps"

Getting Help

For operational issues:

Check this diagnostics guide
Review Runbook
Check worker logs
Search GitHub issues

For integration questions:

For API questions:

Table of Contents​

Health Checks​

ClickHouse Connection Test​

Initialize ClickHouse Schema​

Debug Endpoints​

Debug Room Data​

Debug Catalog​

Queue Status​

Webhook Troubleshooting​

Test Webhook Authentication​

Common Webhook Issues​

401 Unauthorized​

404 Not Found​

Connection Timeout​

Manual Webhook Test​

Webhook Setup Checklist​

Queue Diagnostics​

View Queue Logs​

Dead Letter Queue​

Retry Failed Jobs​

Clear Queue​

Common Issues​

ClickHouse Connection Errors​

Base44 Authentication Failures​

Queue Stuck​

DataForSEO Quota Exceeded​

App Details Not Found​

Logging & Monitoring​

Enable Debug Logging​

Structured Logging​

Key Log Patterns​

Monitoring Dashboards​

Debugging Checklist​

Getting Help​

Table of Contents

Health Checks

ClickHouse Connection Test

Initialize ClickHouse Schema

Debug Endpoints

Debug Room Data

Debug Catalog

Queue Status

Webhook Troubleshooting

Test Webhook Authentication

Common Webhook Issues

401 Unauthorized

404 Not Found

Connection Timeout

Manual Webhook Test

Webhook Setup Checklist

Queue Diagnostics

View Queue Logs

Dead Letter Queue

Retry Failed Jobs

Clear Queue

Common Issues

ClickHouse Connection Errors

Base44 Authentication Failures

Queue Stuck

DataForSEO Quota Exceeded

App Details Not Found

Logging & Monitoring

Enable Debug Logging

Structured Logging

Key Log Patterns

Monitoring Dashboards

Debugging Checklist

Getting Help