How I Learned About Race Conditions the Hard Way

A while ago, we had a bug in scrupp.com that looked simple at first but was caused by a race condition.

We were using Symfony Messenger to process scraping jobs in the background.

The flow was simple:

1. A client submits LinkedIn links.
2. Symfony Messenger sends the jobs to workers.
3. Each worker scrapes the LinkedIn profile.
4. The result is saved to the lead in the database.

The problem happened when two different clients scraped the same LinkedIn profile at almost the same time.

Both jobs matched the same lead in our database.

What Happened

Imagine we already had this lead:

{
  "linkedin": "linkedin.com/in/john",
  "email": null,
  "phone": null
}

Worker A started processing the lead.

At almost the same time, Worker B started processing the same lead.

Both workers loaded the same old data.

Worker A found an email.

{
  "email": "john@example.com",
  "phone": null
}

Worker B found a phone number.

{
  "email": null,
  "phone": "+123456789"
}

Worker A saved first.

Now the lead had the email.

But Worker B was still working with the old copy of the lead.

When Worker B saved its result, it also saved the old email: null.

The final result became:

{
  "email": null,
  "phone": "+123456789"
}

The email found by Worker A was gone.

Why It Happened

The problem was not Symfony Messenger itself.

The problem was that two workers were allowed to update the same lead at the same time.

Both workers read the lead before either one finished saving.

This is called a race condition.

The final result depends on which worker finishes last.

The Lesson

Background workers make applications much faster, but they also create new problems.

When multiple workers can update the same database record, we have to think about concurrency.

Instead of replacing the whole lead with an old copy, updates should only change the fields that the worker actually found.

For example:

UPDATE leads
SET phone = '+123456789'
WHERE id = 123;

This is safer than saving the whole old lead object again.

For more complex cases, transactions, row locking, version checks, or other concurrency controls may also be needed.

The main lesson was simple:

Just because two jobs are correct on their own does not mean they are safe when they run at the same time.