Task Debugging Guide - Stuck Documents

Date: March 17, 2026

Problem: Documents Stuck in "Processing" Status

You have 3 documents stuck in processing status for hours without error messages:

medical-arthritis4(4).jpg - 149 minutes
medical-eyes-results-1.pdf - 632 minutes
medical-eyes-results-1.pdf - 662 minutes

All show:

Status: processing or uploaded
Progress: 0% or 10%
Error message: None (empty)

Why This Happens

Documents get stuck without error messages when:

OCR API Call Hanging
- The OCR provider API is slow or unresponsive
- Network timeout not configured
- API key invalid but not returning error
File Path Issues
- File moved or deleted after upload
- Permissions prevent reading file
- Path constructed incorrectly
Celery Worker Crash
- Worker process died during task
- Memory exhaustion
- Uncaught exception
Configuration Missing
- OCR provider not configured (but you have it configured ✅)
- Model settings invalid

How to Fix

Step 1: Use Task Manager UI

Navigate to: http://localhost:3000/task-monitor

What you'll see:

Table of stuck documents
Age indicator (red "Stalled" badge if >60 min)
Error column (shows "-" if no error logged)
"Retry OCR" button

Action:

Click "Retry OCR" button for each stuck document
Monitor progress bar (should move from 0% → 10% → 100%)
Wait 1-5 minutes for completion

Step 2: Check OCR Provider Status

# Test OCR configuration
curl -X GET http://localhost:8000/api/v1/ai-config/default-for-task/ocr \
  -H "Authorization: Bearer YOUR_TOKEN"

Expected:

{
  "provider": {"name": "OpenAI Production"},
  "model": {"name": "Gemini 3.1 Flash Lite"},
  "max_tokens": 4096
}

✅ You have this configured correctly.

Step 3: Monitor Celery Logs

# Watch Celery task execution
journalctl -f -u celery 2>&1 | grep -i "ocr_document"

# Or check log file
cat /var/log/celery.log | grep -i "ocr_document" | tail -20

Look for:

Task start: Task started
Progress: Progress: ocr_start
Errors: ERROR messages
Completion: Task completed successfully

Step 4: Manual Retry via API

# Login to get token
curl -X POST http://localhost:8000/api/v1/auth/login \
  -d "username=admin@health-assistant.local" -d "password=admin123"

# Retry stuck document
curl -X POST http://localhost:8000/api/v1/task-monitor/documents/retry/DOCUMENT_ID \
  -H "Authorization: Bearer YOUR_TOKEN"

Response:

{"message": "Document OCR will be retried", "document_id": "..."}

Step 5: Check Database Status

# Connect to DB and check status
from app.core.database import AsyncSessionLocal
from app.models.document_model import DocumentModel
from sqlalchemy import select
import asyncio

async def check():
    async with AsyncSessionLocal() as session:
        result = await session.execute(
            select(DocumentModel).where(DocumentModel.status == 'processing')
        )
        docs = result.scalars().all()
        for doc in docs:
            print(f'{doc.filename}: {doc.status}, {doc.progress}%')

asyncio.run(check())

Improved Error Logging

The new TaskLogger utility (in backend/app/workers/task_logger.py) now captures:

Structured JSON logs - Easy to parse
Error categorization - config, file, api, validation, system
Progress tracking - Start, progress, success, error
Timeout detection - Auto-fail after 5 minutes
Sanitized data - No API keys logged

Future Prevention

1. Timeout Configuration

Tasks now auto-fail after 5 minutes to prevent infinite hanging.

2. Progress Monitoring

Task Manager UI shows:

Age in minutes
Stalled badge (>60 min)
Progress bar
Error messages

3. Automatic Retry

"Retry OCR" button resets status and triggers new task.

4. Better Logging

All OCR tasks now log:

Start time
Each stage (config_loaded, path_found, ocr_start, etc.)
Error category
Duration

Quick Fix Commands

# 1. Retry all stuck documents
curl -X POST http://localhost:8000/api/v1/task-monitor/documents/retry/DOC_ID1 \
  -H "Authorization: Bearer TOKEN"
curl -X POST http://localhost:8000/api/v1/task-monitor/documents/retry/DOC_ID2 \
  -H "Authorization: Bearer TOKEN"

# 2. Check status
curl -X GET http://localhost:8000/api/v1/task-monitor/documents/processing \
  -H "Authorization: Bearer TOKEN"

# 3. Monitor Celery
journalctl -f -u celery | grep "ocr_document"

Expected Timeline

After clicking "Retry OCR":

0-30 seconds: Task starts, progress → 10%
30-60 seconds: OCR processing, progress → 50%
60-120 seconds: OCR completes, progress → 100%
Status: Changes to "completed"

If it fails again:

Check Celery logs for error
Verify file exists in uploads directory
Test OCR API directly
Restart Celery worker: systemctl restart celery

Summary

Current State:

3 documents stuck (0.5-11 hours)
No error messages logged
OCR provider configured ✅
Celery running ✅