Mini-Batch Backfill Beats the Monolithic Pipeline

Ayush Mania

The Pipeline That Looked Fine

Saral's search runs over an internal corpus first. When a query is narrow enough that the corpus returns fewer than fifteen results, the server backfills from an external talent database. The first version of that backfill was the obvious one:

Call the external API for candidates matching the query.
Enrich each one - a separate API call per profile, ~8 seconds each.
Ingest into our corpus.
Score against the original query.
Append to the result set.
Update the result count for the client.

The recruiter sees the search start, then a spinner. The spinner hangs. Two minutes later the client times out. The backend, oblivious, finishes thirty seconds later and writes results that nobody is polling for.

The Real Cost

The naive monolithic pipeline serializes work that should stream and makes the slowest external call the floor on user-perceived latency.

What Streaming Actually Looks Like

The fix is to process in mini-batches of three to five candidates and let the client pick up new rows on its existing five-second poll.

For each batch of 3-5 candidates:
  enrich → ingest → score → append → bump resultCount

The client does not change. The same /results?searchId=X endpoint that was already polling now sees the count rising and the list growing every five seconds. The first batch lands in roughly 25 to 40 seconds instead of 90 to 120. By the time the recruiter has read the first three candidates, three more have appeared. There is no infrastructure change here. Same workers, same database, same external API. The only difference is where the commit boundary sits.

Three Discovery Tiers, Run In Parallel

Search briefs vary wildly. A precise query against a niche role might return zero candidates from the external API. A broad query returns more than we need. We cannot know in advance which one we got.

Tier	Strategy	Why It's Here
Tier 1	Original query, mapped 1:1 to the API filters	Strongest matches first
Tier 2	Broadened: drop experience and location, expand role family	Catches misses from strict filtering
Tier 3	SERP fallback	Last resort, reliably non-empty

Each tier produces a stream of candidates. Each stream feeds the same mini-batch pipeline. The result set is the union, in arrival order. Tier 1 returns the strongest matches first; Tier 2 ensures the page fills out when Tier 1 came up short; Tier 3 is the floor.

Dedup Is Not an Afterthought

Three tiers running in parallel against three slightly different queries will hit the same person more than once. Two workers pulling jobs from the same queue can race on a freshly inserted row. The naive "check if exists, then insert" pattern has a window where both checks return false and both inserts succeed.

Two Layers of Defense

Pre-call dedup against a known set.
Before any external API call, the worker loads knownUsernames for the search - every external identifier already attempted, regardless of outcome. The query going to the API is filtered against this set. The stub row for the candidate is written before the enrichment call so future workers see them as known. Database-level uniqueness.
The (searchId, candidateId) pair has a unique constraint. Inserts use ON CONFLICT DO NOTHING. If the pre-call dedup misses - and it will, sometimes, under concurrent ingestion - the database catches it without raising. The order matters. Doing the unique-constraint check first and the in-memory check as a hot-path optimization gives correctness regardless of how the workers happen to be scheduled.

What Goes Wrong Without Retry

The external API's discovery endpoint is non-deterministic under load. A query that times out at 60 seconds on the first attempt will succeed on retry, seconds later, on the same parameters. There is no rate-limit signal, no queue notification - it just times out. The original code had no retry. It assumed timeouts meant "no candidates available." About 41% of backfills returned zero stubs, and the failure rate was uncorrelated with query specificity, which was the tell. A retry wrapper with a single retry on timeout or 5xx and a five-second delay raised the hit rate above 95%. No infrastructure change, no additional cost. The retry is bounded - retryLimit: 1, retryDelay: 5s - because the goal is to absorb the API's non-determinism, not to mask a real outage.

Why retry only on 5xx and timeouts?

What about 429s?

Why bound the retry to 1?

Per-Candidate Failures Don't Block the Search

In the monolithic pipeline, one stuck enrichment blocks the entire batch. In the mini-batch pipeline, one stuck enrichment blocks one candidate. The other four in the batch finish, get scored, and get appended. The stuck one fails forward - marked failed_no_email or failed_transient - with retry available from the UI. This is the real reason the architecture matters. Production failures are rarely catastrophic; they are stragglers. A pipeline that depends on the slowest worker in a batch to complete will always be at the mercy of the worst tail latency. A pipeline that commits per-candidate is bounded by the median.

What This Looks Like Operationally

A few patterns fall out of this design:

Append-only result rows. Once a candidate is in the result set, they do not move. Pagination depends on this.
Server-set rank. The Engine assigns rank as candidates flow in. The frontend never re-sorts.
Status machine on candidates. queued → running → succeeded | failed_*. The frontend renders by status; failures are not hidden.
Stale-search recovery. If a worker crashes, the row is marked failed. If the entire search goes silent for ten minutes, an idle reaper marks it stale and lets the user retry.

The pipeline is fundamentally simple. The work is in deciding what is allowed to run concurrently and what is allowed to fail in isolation. Once those answers are right, the architecture mostly writes itself.

When This Is Overkill

If your backfill source returns in under a second and never times out, mini-batching adds complexity for little gain. If you have one tier of discovery and no cross-tier dedup problem, you do not need the knownUsernames precheck. The pattern is shaped by the constraints - flaky external API, multi-tier discovery, recruiter-grade UX expectations - and earns its keep when those constraints are real. The takeaway is not "always mini-batch." It is: locate the slowest external call in your pipeline, draw the commit boundary just before and just after it, and let the client pick up the difference. If you do that and nothing else, you have already paid for most of the win.

Share this post: