Scraping 40,000 videos a day — and what breaks at scale

We built a platform that pays clippers to post short reels across multiple platforms — YouTube, Instagram, TikTok and Twitter — and tracks their performance. At peak, it was scraping 40,000 videos a day, feeding a system that's now tracked over 2 billion views. Here's what we learned scaling it — some of it the hard way.

The 1% problem becomes the daily problem

At small scale, an edge case is exactly that — an edge. Something rare enough to fix "when we get to it." When you're scraping 40,000 videos a day across four different platforms, a bug that only triggers 1% of the time isn't rare anymore — it's hitting hundreds of videos every single day.

Four platforms means four different sets of quirks: metadata that's occasionally missing, view counts that update on different schedules, formats that change without warning, rate limits that kick in at different thresholds. Individually, each of these is a minor edge case. Multiplied by tens of thousands of videos a day, every one of them becomes a daily occurrence you have to handle gracefully, not an exception you can patch later.

Perfect in dev, a shambles in production

This is the one that catches almost every team out eventually: scraping logic that works perfectly against a handful of test videos, then behaves completely differently at real volume.

The problem isn't sloppy code — it's that a small test sample can't reveal what only shows up at scale. A platform might silently throttle you after your two-thousandth request of the day. A video's data might be structured slightly differently depending on region, account age, or content type. None of that shows up until you're actually running at production volume, against real, messy, inconsistent data from the outside world.

What actually helped:

• Testing against a large, varied sample of real videos rather than a small clean set, to surface the inconsistencies early
• Building in retries, backoffs and graceful failure for every platform individually, since each one breaks in different ways
• Treating monitoring as core infrastructure — with scraping at this volume, you find out about a broken data source from your dashboards, not from a clipper asking why their view count looks wrong

Small inefficiencies get expensive fast

A slightly inefficient request that's fine for 100 videos a day becomes a real infrastructure cost at 40,000 videos a day, across four platforms, run continuously. At this volume, performance isn't a nice-to-have — it directly affects your running costs.

The architecture has to expect growth, not just handle today's volume

A scraping and tracking system built to comfortably handle today's volume tends to become the bottleneck the moment you add another platform, another few thousand clippers, or another few thousand videos a day. Some of the most valuable engineering decisions on this project weren't about handling the current volume — they were about making sure the next stage of growth didn't require a rebuild.

The takeaway

Building a system that works against a handful of test videos is one thing. Building one that keeps working reliably across multiple third-party platforms, at real volume, with data you don't control, is a different discipline entirely. It's less about writing clever scraping logic and more about assuming platforms will change, data will be messy, and designing so you find out before your clippers do.

If you're building something that depends on external platforms and real scale, it's worth having this conversation before you write the first line of code, not after the first outage.