A common implementation pattern in backend systems is to call an external service directly inside a request handler.
For example:
| @router.post(“/process-document”) async def process_document(file: UploadFile): content = await file.read() response = requests.post( “https://external-service.com/process”, files={“file”: content} ) return response.json() |
This works without issues in development and low-traffic environments.
The problem starts when traffic increases or when the external service has variable response times.
Where It Starts Failing
-
The request remains open too long
If the external service takes 15–20 seconds to respond, your API request is blocked for that entire duration.
Under concurrent load:
- Worker threads remain occupied
- New requests start queuing
- Timeouts increase
- Overall responsiveness drops
Even endpoints unrelated to this flow can start slowing down because resources are shared.
-
Your performance depends on something you don’t control
- When the external dependency slows down, your system slows down.
- When it hits rate limits, your system starts failing.
- When it has a temporary outage, your endpoint immediately reflects that failure.
- This tight coupling makes the API layer fragile.
-
No structured retry or failure handling
If the external call fails due to:
- Network issues
- Temporary service disruption
- Timeout
- The entire request fails.
There is no controlled retry mechanism, no backoff strategy, and no way to isolate failed jobs from successful ones.
For small systems, this may be acceptable. Under production load, it becomes difficult to manage.
A More Stable Pattern
Instead of performing heavy or slow work inside the request lifecycle, separate it.
A simple pattern:
API → Queue → Worker → External Service → Store Result
The API accepts the request, stores the required input, pushes a message to a queue, and immediately returns a job ID.
Processing happens asynchronously.
Example:
| @router.post(“/process-document”) async def process_document(file: UploadFile): job_id = str(uuid.uuid4()) # store file (S3 or database) sqs.send_message( QueueUrl=“YOUR_SQS_URL”, MessageBody=json.dumps({“job_id”: job_id}) ) return { “job_id”: job_id, “status”: “processing” } |
A background worker then consumes the message and calls the external service.
This keeps the API fast and predictable, regardless of how long the external system takes.
Why This Helps
- API requests are no longer blocked by long-running operations
- Concurrency can be controlled at the worker level
- Retries can be configured using queue settings
- Failed jobs can be redirected to a Dead Letter Queue
- Worker scaling can be adjusted independently of API scaling
The system becomes more stable under load because responsibilities are separated.
Where This Commonly Appears
This issue is often seen when integrating with:
- AI APIs
- Payment gateways
- Government or regulatory APIs
- Document conversion services
- Large report generation systems
AI integrations tend to expose the problem quickly because response times are usually longer and rate limits are strict.
However, the architectural issue is not specific to AI. It applies to any slow or unpredictable external dependency.
Direct synchronous calls are still acceptable in low-traffic systems or when the external service responds consistently in under a couple of seconds.
But once stability under load becomes important, handling long-running work asynchronously is usually the safer design choice.