The Hidden Cost of Skipping Automated Testing (And Why Most Developers Do It Anyway)
Manual testing feels faster today — but it's quietly costing businesses thousands down the line. Here's why most developers skip automation anyway, and what it's really costing you.
In 2019, I was working at GHTK, the largest shipping company in Vietnam at the time, with around 30,000 shippers on the platform. I was part of the KPI team, responsible for collecting shipping data, calculating shipper salaries, and generating reports for accountants and directors that were later used to run payroll.
There were more than 40 shipper types, each with its own salary formula, and only four developers to cover all of it — I was one of them. Back then, none of us had any real testing experience, and the project had no automated tests at all. Every pull request was reviewed line by line by the tech lead and technical manager before it got merged, and that review was the only safety net we had.
Because I owned the logic covering around 15 shipper types — roughly 70% of the total — I ended up re-reading my own code over and over before I dared to ship it. The stakes were real: even a tiny rounding error in a single package's budget could cost the company billions of Vietnamese dong at scale. Quality was ultimately caught by the QA team and later audited by accountants, not by the code itself. And yes, bugs still reached production — some from our side, some from other teams like big data.
The one thing every developer, lead, and manager on that team feared was a wrong salary calculation. It meant real revenue loss and weeks of retroactive collection to fix. I was doing well by every metric the team measured, but every time my changes hit production, I couldn't sleep. I kept replaying the logic in my head, wondering if I'd missed an edge case. I watched a colleague spend days untangling a payroll bug that cost the company $200,000. I got lucky — careful, obsessive manual review kept me clear of anything that serious — but "getting lucky" is not a strategy, and I knew it even then.
In 2021, I left GHTK to join Sotatek as a fullstack senior developer. I was confident that what I built would behave exactly as expected — after all, I'd survived years of payroll work without a critical incident. I was wrong.
Sotatek is an outsourcing company where projects come from many different clients, each with its own tech stack, architecture, business logic, and team. New requirements needed to be implemented or extended without breaking existing behavior, and I was often juggling four projects at once. Without the ability to focus on a single domain, I would forget existing logic, and a change that fixed one feature would quietly break another. Every update meant tracing through unrelated code just to be sure nothing else would break.
That's when I realized this wasn't just my problem — it was the team's problem. I was spending roughly 20% of my time building new features and 80% of my time debugging regressions caused by recent changes. So I started researching how to fix that ratio, and automated testing kept coming up as the answer. I started with unit tests.
A unit test verifies that a single function behaves correctly under expected — and unexpected — conditions, and it catches the moment a change to that function silently breaks something that depends on it. Here's a simple example.
An e-commerce platform sells many types of products, including hats, clothes, and shoes. Every product uses the same function to calculate the final price after applying a discount.
// discount.ts
export function calculateDiscount(price: number, discount: number): number {
if (discount < 0 || discount > 100) {
throw new Error("Invalid discount");
}
return price - (price * discount) / 100;
}A few months later, the business launches a promotion: shoes now carry an additional $10 handling fee after the discount is applied. A developer updates the shared function, confident it's safe because they know the checkout flow for shoes uses it.
// discount.ts
export function calculateDiscount(price: number, discount: number): number {
if (discount < 0 || discount > 100) {
throw new Error("Invalid discount");
}
const handlingFee = 10;
return price - (price * discount) / 100 + handlingFee;
}The developer double-checks the shoe checkout, sees the correct total, and ships the change to production. What they didn't realize is that the same function is also used by hats, clothes, and every other product category. Now every customer buying anything on the platform is silently overcharged. Customers notice, some leave for a competitor, and the company loses both revenue and trust.
Adding a unit test up front prevents exactly this. Something as simple as this:
// discount.test.ts
import { calculateDiscount } from "./discount";
describe("calculateDiscount", () => {
it("should apply 20% discount correctly", () => {
expect(calculateDiscount(100, 20)).toBe(80);
});
it("should return original price when discount is 0%", () => {
expect(calculateDiscount(100, 0)).toBe(100);
});
it("should return 0 when discount is 100%", () => {
expect(calculateDiscount(100, 100)).toBe(0);
});
it("should throw an error for invalid discount", () => {
expect(() => calculateDiscount(100, 120)).toThrow(
"Invalid discount"
);
});
});With this test in place, the moment a developer introduces the new handling fee, the test suite fails before the change ever reaches production:
FAIL discount.test.ts
calculateDiscount
✕ should apply 20% discount correctly (3 ms)
✓ should return original price when discount is 0%
✓ should return 0 when discount is 100%
✓ should throw an error for invalid discount
● calculateDiscount › should apply 20% discount correctly
expect(received).toBe(expected) // Object.is equality
Expected: 80
Received: 90
5 |
6 | it("should apply 20% discount correctly", () => {
> 7 | expect(calculateDiscount(100, 20)).toBe(80);
| ^
8 | });
9 |
10 | it("should return original price when discount is 0%", () => {
Test Suites: 1 failed, 1 total
Tests: 1 failed, 3 passed, 4 total
Snapshots: 0 total
Time: 0.356 sI shared what I'd learned about unit testing with the team, and once we adopted it, we could change business logic with confidence, without breaking existing features. The time we spent debugging dropped sharply, and we could focus on building instead of firefighting. But that alone wasn't enough — the next challenge showed up when several features had to work together correctly.
Take the same discount example from before and stretch it further: a customer placing an order doesn't just trigger a discount calculation. The checkout flow also validates inventory, calculates shipping, processes payment, creates the order, updates stock, and sends a confirmation email. Each piece can work perfectly in isolation and still fail when combined with the others.
This is where unit tests reach their limit. They verify individual pieces of logic, but they can't tell you whether multiple components integrate correctly to complete a real business workflow. That gap is what integration tests are for. Here's an example.
When a customer clicks "Place Order", the checkout service performs two important steps:
- Charge the customer's credit card.
- Save the order to the database.
The implementation looks straightforward:
async checkout(order: Order) {
// Charge the customer's credit card.
await paymentService.charge(order);
// Save the order to the database.
await orderRepository.create(order);
}Both components already have their own unit tests:
describe("PaymentService", () => {
it("should charge the customer successfully", async () => {
...
});
});
describe("OrderRepository", () => {
it("should save the order", async () => {
...
});
});Everything passes:
PASS PaymentService
PASS OrderRepository
Test Suites: 2 passed
Tests: 2 passedThen production happens. A customer places an order, the payment gateway successfully charges $150, and milliseconds later the database becomes temporarily unavailable:
❌ Failed to insert order
DatabaseError: Connection timeoutNow there's a serious business problem: the customer was charged, but no order was ever created. From the customer's side, their bank account was debited and nothing arrived — a fast way to lose trust in the platform.
This is exactly what an integration test is designed to catch — but only if the test actually simulates the failure instead of just describing it. Here's what that looks like in practice.
The real fix isn't to charge first and refund afterward if something goes wrong — that still leaves a window where a customer is charged for an order that doesn't exist yet, and a refund is one more thing that can itself fail. A better fix is to change the order of operations: create the order first, inside a database transaction, then charge the payment, and only commit the transaction once both steps succeed. If the payment fails, the transaction rolls back and the order is discarded — the "charged with no order" state becomes structurally impossible instead of something we patch after the fact.
async checkout(order: Order) {
return db.transaction(async (trx) => {
// Create the order first, inside the transaction.
const savedOrder = await orderRepository.create(order, { transaction: trx });
// Only charge the customer once the order is safely queued.
await paymentService.charge(savedOrder);
return savedOrder;
// If we reach here, the transaction commits and the order becomes permanent.
// If anything above throws, the transaction rolls back and the order is discarded.
});
}Now the integration test simulates the order being created successfully, followed by the payment failing, and asserts that the order never ends up persisted:
describe("CheckoutService", () => {
it("rolls back the order when payment fails", async () => {
const createSpy = jest
.spyOn(orderRepository, "create")
.mockResolvedValue({ ...order, id: "order_123" });
const chargeSpy = jest
.spyOn(paymentService, "charge")
.mockRejectedValue(new Error("Card declined"));
await expect(checkoutService.checkout(order)).rejects.toThrow(
"Card declined"
);
expect(createSpy).toHaveBeenCalledWith(order, expect.anything());
expect(chargeSpy).toHaveBeenCalledWith(
expect.objectContaining({ id: "order_123" }),
);
// The order was created inside the transaction, but since payment
// failed, the transaction rolled back — no order should exist.
await expect(orderRepository.findById("order_123")).resolves.toBeNull();
});
});Run this test, and it passes:
PASS checkout.integration.test.ts
CheckoutService
✓ rolls back the order when payment fails (19 ms)
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 totalWhat this test really proves is the thing that matters to the business: when something goes wrong mid-checkout, the customer is never left charged for an order that doesn't exist. The order attempt gets rolled back before payment is ever touched, so there's nothing to refund and nothing for the customer to notice — the failure disappears before it ever reaches them.
This is what makes an integration test worth writing: it doesn't just check that an error is thrown — anyone can write that assertion and still ship a bug. It simulates a realistic failure across two dependencies and verifies the system did the right thing in response — in this case, that the customer's money is never touched unless the order is guaranteed to exist.
I brought this same case to my team, and once we started writing integration tests around every workflow like this one, the change in how we worked was immediate. At first, the time we spent writing code actually went up — integration tests take longer to design than unit tests, since they need realistic scenarios, seeded data, and simulated failures. But the payoff showed up almost right away in how little time we spent chasing bugs afterward.
Our time split went from roughly 20% coding and 80% debugging and fixing production issues, to close to 50% coding, 45% writing automated tests, and just 5% left for debugging and handling genuine business edge cases we hadn't anticipated. We were writing more tests than ever, yet shipping faster overall, because we were no longer burning days tracing a bug back through a workflow that no one fully remembered the shape of.
Unlike unit and integration tests, load and stress testing wasn't something I went looking for as part of a natural progression. I first encountered this challenge while working on a trading platform. One business requirement completely changed how I thought about testing: after release, the product had to handle 50,000 concurrent users on their VPS — a single server, not a cluster — at the same time, with live prices streaming and orders being placed. Unit tests and integration tests could tell me the order logic and price calculations were correct. Neither could tell me whether the server would still be standing once 50,000 people opened it at once. That gap is what sent me into researching load and stress testing.
The distinction matters here. Load testing simulates the exact traffic you're expected to handle — in this case, 50,000 concurrent users — and checks that response times and error rates stay acceptable. Stress testing pushes past that number to find the point where the system actually breaks, so you know your real ceiling instead of hoping 50,000 happens to be safely under it.
To simulate that traffic, I wrote a k6 script that ramps up virtual users gradually, holds them at the target for a few minutes, then ramps back down — much closer to how real users arrive than firing 50,000 requests all at once, you can think the script look like this:
// load-test.js
import http from "k6/http";
import { check, sleep } from "k6";
export const options = {
stages: [
{ duration: "1m", target: 50000 }, // ramp up
{ duration: "3m", target: 50000 }, // hold at target
{ duration: "1m", target: 0 }, // ramp down
],
thresholds: {
http_req_duration: ["p(95)<300"], // 95% of requests under 300ms
http_req_failed: ["rate<0.01"], // less than 1% error rate
},
};
export default function () {
const res = http.get("https://client-vps.example.com/api/market-price");
check(res, {
"status is 200": (r) => r.status === 200,
"response time < 300ms": (r) => r.timings.duration < 300,
});
sleep(1);
}Running it with k6 run load-test.js against the client's actual VPS, the first result was rough:
scenarios: (100.00%) 1 scenario, 50000 max VUs, 5m max duration
✗ status is 200
✗ response time < 300ms
http_req_duration..............: avg=4.8s p(95)=9.6s
http_req_failed................: 42.30% ✗ 21150 out of 50000
vus_max.........................: 50000Nearly half the requests were failing, and the ones that succeeded took seconds instead of milliseconds. Digging into it, I found a handful of very ordinary problems compounding on a single VPS: the Node process wasn't using the server's other CPU cores, every price update was hitting the database directly instead of a cache, and the connection pool was small enough that it saturated almost immediately once thousands of clients connected at once.
None of these were exotic fixes — clustering the process across CPU cores, caching market data reads in Redis, and sizing the connection pool properly for the expected concurrency. Running the same load test after those changes told a very different story:
scenarios: (100.00%) 1 scenario, 50000 max VUs, 5m max duration
✓ status is 200
✓ response time < 300ms
http_req_duration..............: avg=180ms p(95)=290ms
http_req_failed................: 0.04% ✗ 20 out of 50000
vus_max.........................: 50000That satisfied the client's requirement, but it left one question unanswered: how much headroom did we actually have above 50,000? So I reused the same script, just raising the target in each stage to 70,000, and pushed it into stress-test territory to find the real breaking point:
scenarios: (100.00%) 1 scenario, 70000 max VUs, 5m max duration
✗ status is 200
✗ response time < 300ms
http_req_duration..............: avg=2.1s p(95)=5.8s
http_req_failed................: 18.70% ✗ 13090 out of 70000Somewhere between 50,000 and 70,000 concurrent users, that VPS started to degrade. Knowing exactly where that line sat let me tell the client, with actual numbers instead of a guess, how much safety margin they had above their target — and what would need to change first if their user base ever grew past it.
Add these numbers up and the pattern is consistent: without automated tests, quality depends entirely on how carefully — and how anxiously — a developer reviews their own code. That approach can work for a while, especially with disciplined people and careful manual QA. But it doesn't scale, it doesn't transfer to new team members, and it eventually fails, usually at the worst possible time: a wrong salary calculation, a customer charged twice, a checkout page that collapses during the one week of the year it matters most.
The teams I've seen recover from this didn't do it by hiring more careful reviewers or asking developers to "be more careful." They did it by shifting quality left — unit tests to protect logic, integration tests to protect workflows, and load or stress tests to protect the system under real traffic. The upfront cost is real: writing tests takes time that could go toward a new feature. But that's exactly the shift I saw on my own team — from 20% coding and 80% firefighting, to roughly 50% coding, 45% writing automated tests, and only 5% left for genuine edge cases. We spent more time writing tests, and still shipped faster, because almost none of it was lost to debugging.
That's the trade every engineering team eventually has to make: pay a small, predictable cost now, or an unpredictable, much larger one later.
That's all for this sharing. I hope it's useful, whether you're a developer weighing where to invest your testing effort, or someone evaluating how a team builds and ships software.
If you have any suggestions, feedback, or topics you'd like me to explore in future posts, feel free to reach out.
Thank you for reading, and I look forward to sharing more engineering lessons with you in the future.