Wednesday morning the support queue is full of people who can't finish checkout, and nobody on the team can reproduce it.
Not much, is the uncomfortable answer. It proved the software behaved correctly under the conditions somebody thought to write down. It says nothing about the conditions nobody thought to write down. And that second pile is where nearly all of your production bugs are sitting.
Which means the fix isn't "write more tests." I've watched teams add a thousand tests after an incident and ship the same class of bug six months later, because the tests were modeling the same clean little world that failed them the first time. The gap isn't test quantity. It's the distance between your test conditions and real use.
A test is a tiny controlled experiment. You establish a known state, do a thing, compare the result to what you expected.
That control is the whole point. It's also the blind spot. In a test environment the data is known, traffic is light, the network is fast, and the third-party services are usually fake. Production gives you none of that. Users show up in unpredictable numbers, type things nobody planned for, run browsers from 2016, and combine features in ways the product team never discussed in any meeting.
There's a useful distinction buried here that gets flattened a lot. Testing checks behavior against expectations. QA asks whether the expectations were ever right, and whether anyone will notice fast enough when a defect slips past.
When someone asks why a thing worked in testing and broke in prod, the answer is usually boring. Staging and production were different, and one of those differences mattered.
The gaps accumulate quietly. Staging runs a newer Postgres than prod. A timeout that's 30 seconds in staging is 5 seconds in prod — or absent entirely, so the app falls back to some default somebody set in 2019 and forgot. Prod sits behind a firewall and a CDN that staging has never touched. A service account with god-mode permissions in staging is locked down in prod.
I like the insurance-company example because it's so unglamorous. Regional carrier in the Midwest, launching claims upload. In QA, documents land in a test bucket in about two seconds. In production, every upload routes through a corporate proxy with a file-size limit. Large PDFs from policyholders fail. No error surfaces. The only trace is in proxy logs the app team doesn't read.
Sandboxes have the same problem. Payment processors, identity providers, shipping APIs — they all hand you one, and none of them behave like the live service. Different rate limits. Different error messages. Sometimes different response shapes.
What actually helps: infrastructure as code, pinned runtime versions in containers, a config diff before every release, and a few read-only checks against prod after you deploy. Most companies can't afford a full copy of production and never will. Fine. What matters is that each difference is a decision somebody made on purpose, not an accident nobody noticed.
People who write test cases know how the software is supposed to work. Which makes it genuinely hard to picture how everyone else will use it.
They don't read the requirements. They double-click Submit because the page felt slow. They open the same order in two tabs and edit both. They hit back halfway through checkout. Someone pastes an emoji into a field that feeds a twenty-year-old billing system. Someone else starts a form on their phone at lunch, comes back at 8pm, and the session's long dead.
Old accounts are the sneaky one. Test accounts are new and clean. Real accounts have history — a subscription canceled and reactivated, a billing address in Puerto Rico that one shipping rule treats differently, a permission role left over from a reorg two years back. Any feature that assumes accounts look like fresh test accounts will break first for your longest-tenured customers. Those are the ones who call.
Devices and accessibility too. A layout that's fine on desktop shoves a required button off-screen on a small Android phone. A modal traps keyboard focus and a screen-reader user can't finish a purchase. Nothing fails. A paying customer is just stuck.
Concurrency is the hardest. Two people booking the last seat at the same instant. A webhook landing before the transaction that created its record has committed. One tester clicking through a workflow alone will basically never hit these.
Testing only the happy path is the classic mistake, and honestly it's an easy one to make under deadline. Better coverage comes from asking a different question for each feature: how would someone interrupt this, repeat it, or misuse it? Exploratory testing catches a lot of this early, and it's the part of QA that's hardest to automate for exactly that reason.
Test data is tidy because somebody made it tidy. Name, valid email, five-digit ZIP. Production data has been dragged through years of real use, migrations, imports, and mistakes.
A field that's required today was optional four years ago, so there are thousands of rows with nulls your new code doesn't handle. Same customer, three times, thanks to a CRM import. Apostrophes and accents and non-Latin characters break validation and CSV exports. O'Brien has been wrecking badly escaped queries since before most of us were born. Orders point at deleted products. Invoices with zero line items. A report that takes two seconds on a thousand test rows takes six minutes on forty million, or just eats all the memory and dies.
Synthetic data only contains the cases its author thought of. Staging copies of production go stale in a couple months and then everyone's quietly testing against 2023.
Masked or anonymized snapshots, refreshed on a schedule, get you most of the way. HIPAA makes that harder, I know. Where it does, generate data that matches production's share of nulls, its character sets, its record ages. Not perfect. Much better than tidy.
And rehearse migrations separately. Run the script against something realistic before release night, while there's still time to care about the table locks and constraint violations it turns up.
An app can be flawless with 50 test users and fall over at 5,000. Functional tests confirm the software gives the right answer. They don't tell you whether it keeps giving the right answer when everything arrives at once.
Fast queries start queuing on locks. A connection pool sized for development runs dry and requests stack up behind it. An external API starts throwing 429s because you blew through your rate limit. A small memory leak kills a server after six hours of steady traffic. Right after deploy, cold caches dump a burst of requests straight onto the database.
Load testing checks response times at expected traffic. Stress testing pushes until something snaps. Soak testing holds steady for hours looking for leaks. Spike testing is the sudden rush. You want all four eventually, and you want to start with whichever one matches how your worst day actually looks.
k6, JMeter, Locust — take your pick, they're all free and they'll all script a user journey. Make the journey realistic, though. Search, log in, buy. In that order, mixed together, the way people actually shop.
Short baseline run on every significant build. Bigger runs before a major launch. Teams that treat performance as everyday work catch the regressions while they're small enough to be boring.
Some failures only exist in production. A memory leak that needs three days to hurt. A certificate expiring on a Saturday. A partner API changing its response format and telling no one. This is the part where QA has to keep going after release, and where a lot of teams quietly stop.
The tooling is good now. Structured logs you can actually search. Sentry grouping exceptions and telling you which release introduced them. Metrics for error rate and latency. Distributed tracing following one request across services. Collectively that's observability — you can tell what the system is doing by reading what it emits.
Tune the alerts. If everything pages somebody, nobody reads the pager. Alert on things a customer would notice. A jump in checkout errors, sure. A single 500 somewhere in a batch job, probably not.
Smoke tests are a handful of automated checks right after deploy. Can people log in. Can a test transaction be created and voided. Do the integrations answer. With a rollback plan the team has actually practiced, those checks turn a four-hour outage into an eight-minute one.
One thing almost everyone skips: before the release, agree on what a healthy deploy looks like, who's watching the dashboards, and what specifically triggers a rollback. Not after. Before. Otherwise the person on call is making that call alone at 11pm with no sleep.
Test with realistic data. Masked snapshots, or modeled sets with nulls, duplicates, legacy formats, and production volume.
For each feature, list the ways someone could interrupt it, repeat it, or misuse it. Write tests for the likeliest handful.
Put a name on the first hour after deploy and have that person compare error rates and latency against the previous version.
Configuration: Variables, secrets, flags, permissions all match production. Drift is what produces "but it worked in staging."
Test count tells you almost nothing. I've seen teams with four thousand green tests ship a release that hurt customers, because all four thousand modeled a cleaner world than the one the software lives in.
Better question than "did the tests pass" is "have we tested the conditions this thing will actually run under." Asking it drags realistic data and traffic and human behavior into release planning, where they belong.
Automation and manual work do different jobs and neither replaces the other. Automation is fast, repeatable, and excellent at regressions. Exploratory testing runs on a person's judgment, including the vague sense that something's off even when every assertion passes — and that sense has caught more bugs in my experience than any dashboard. Performance testing shows you the system under pressure. Monitoring shows you the system once customers are actually in it. A blameless review afterwards turns each incident into a test that didn't exist before.
That loop is the whole thing. Teams doing it well still ship bugs. They just catch more of them early, and the ones that get through don't stay broken for long.
InfineneTech works with businesses on development and QA around exactly this gap. Testing strategy built around real usage. Automated regression suites. Application and performance testing under realistic conditions. Launch prep that includes deployment checks, monitoring, and a rollback plan somebody has actually run. Plus ongoing maintenance so quality doesn't drift once the launch excitement wears off.
Nobody can promise software that never fails. What a disciplined process buys you is fewer surprises reaching customers, and a shorter path from "something's wrong" to "found it, fixed it, rolled it back."
If your releases keep passing testing and then causing trouble, look at InfineneTech's Quality Assurance and software development services, or get in touch and tell them where your process hurts.
