How I Turned 15 Struggling Pods into 2 Healthy Ones
Our team struggled with a service that kept triggering resource alerts even after scaling to 15 Kubernetes pods. Instead of a full rewrite, I investigated the root causes and reduced it from 15 overloaded pods to just 2 healthy ones.
This story began a few years ago when I came across an issue with the voucher service that had been raised in our tech group. The service was responsible for distributing vouchers to users, and I had never worked on it before. However, the service had become a significant challenge for the team. Despite consuming 15 pods and requiring substantial CPU and memory resources, other developers were struggling to identify the root cause of its performance issues. Numerous technical discussions had taken place, but no clear solution emerged, and a complete service rewrite was starting to be seriously considered.
As someone who enjoys solving challenging problems, I wanted to take on this issue. One of the concerns I raised with my manager was that rewriting the entire service without understanding the root cause would only address the symptoms, not the underlying problem. If we didn't know why the issue occurred, what would prevent the same problem from happening again in the future? In that case, we might find ourselves facing another costly rewrite down the road.
With that in mind, I took full ownership of the service and was given the responsibility of improving its performance and stability.
Once I started investigating the service, I discovered a wide range of issues contributing to its poor performance. These included database overfetching, missing indexes on frequently queried columns, inefficient database operations implemented with Prisma, circular dependencies between components, and several other architectural and code-quality problems that I will explain in detail now:
One challenge I've noticed repeatedly in this project is overfetching—when a query pulls far more data than necessary. This often happens either due to overlooking code quality or reusing existing logic without fully understanding it. Overfetching may seem harmless at first, but it can significantly impact performance and maintainability. Let's dive in and see a concrete example in the code below.
async function getEmployeeLanguage(req, res, next) {
try {
req?.log?.info('[Employee] - get employee language start');
const { params } = req;
const { userId } = params;
const user = await feature.user.getUser({ user_id: userId });
const userLanguage = feature.user.getUserLanguage(user.information.language);
req?.log?.info('[Employee] - get employee language success');
helper.response({
data: {
language: userLanguage,
},
next,
});
} catch (e) {
req?.log?.info('[Employee] - get employee language fail');
helper.respExcept({ e, next });
}
}As the code shows, the logic seems straightforward: it calls getUser to retrieve the user's language. At first glance, this looks simple and elegant. However, getUser actually fetches all of the user's data in our system—over 60 columns—and even triggers additional queries across other tables. The resulting SQL is complex, and the response time is significant: even in our development environment, this single API call to get the user's language takes almost one second.
To address overfetching, the fix is actually quite simple: adjust the query to retrieve only the data that is truly required. However, the more important takeaway is the mindset—always be intentional about what data is being requested. This practice not only improves performance but also enhances security, as we often handle sensitive user information that should not be exposed unnecessarily.
These images below demonstrate the applied fix. As you can see, the generated SQL is much cleaner because the query now retrieves only the field required by the API—in this case, the user's language. By eliminating unnecessary data retrieval and related queries, the API response time improved dramatically, becoming nearly 10 times faster than the original implementation.
This improvement highlights an important engineering principle: always fetch only the data you need. Doing so not only reduces database load and improves application performance, but also minimizes the exposure of sensitive information, resulting in a more secure and maintainable system.
export const getUserLanguageByUserId = async (userId) => {
const userLanguage = await db.user.getDocById(userId, { attributes: ['language'] });
error.isNull(userLanguage, 'message: employee not found', { code: 404 });
return getUserLanguage(userLanguage.language);
};
Another issue I discovered that has a significant impact on system performance and resource consumption is the execution of unnecessary actions within application logic.
A common example is an "Update Voucher Details" screen, where administrators can view and modify voucher information. Suppose an administrator only wants to change the voucher's name. Ideally, the system should update only that specific field. However, in many cases, the update logic processes the entire voucher object instead. So here is the code I found:
async updateVoucher(
id: number,
data: VoucherUpdateData
): Promise<VoucherPayload> {
const existVoucher = await this.prisma.voucher.findFirst({
where: { id },
select: voucherUpdateSelect,
});
if (!existVoucher) {
errorHelper.notFound({
code: INTERNAL_ERROR_CODE.VOUCHER_NOT_FOUND,
message: 'voucher not found',
});
}
const translationsData = buildTranslationsData(data.translations);
const updateTranslationsData = translationsData.map((translation) => {
return {
where: {
language: translation.language,
},
data: translation,
};
});
const thumbnailData = data.detail.thumbnail as string;
const coverData = data.detail.cover as string;
const updateVoucherData: Prisma.VoucherUpdateInput = {
...data.detail,
thumbnail: thumbnailUploadS3Key,
cover: coverUploadS3Key,
partner: {
connect: {
id: data.partner.id,
},
},
voucherDetailTranslation: {
updateMany: updateTranslationsData,
},
};
const updatedVoucher = await this.prisma.voucher.update({
where: { id },
data: updateVoucherData,
select: voucherSelect,
});
}The code had the same overfetching issue we discussed earlier. The developer reused the voucherUpdateSelect configuration without fully understanding how Prisma works, which resulted in many unnecessary SQL queries being executed. As shown in the image below, the application performs numerous unnecessary operations, including additional database queries, validations, business rule checks, and data transformations for fields that have not changed. While each individual operation may seem insignificant, their combined effect can substantially increase response times, database load, and overall resource consumption.
The root cause is often a lack of consideration for what has actually changed. Rather than updating only the modified data, the system blindly reprocesses everything. This not only impacts performance but also increases code complexity and makes the system more difficult to maintain over time. So when I tested this by updating a voucher with nothing changed, the image below shows the SQL generated:
As I counted, around 20 queries were generated for doing nothing—that's crazy right, querying information, updating information, and querying again to return it. Imagine checking into a hotel and asking the receptionist to correct the spelling of your name. Instead of updating a single field, the receptionist calls housekeeping, room service, security, accounting, the restaurant, the spa, and the parking department to verify information that hasn't changed. After 20 phone calls, your name is finally updated.
Fortunately, the fix for this issue is relatively straightforward.
The first approach is to let the front-end send only the fields that have actually changed. This is easy to implement because the front-end already has both the original data and the updated values entered by the admin or user. The back-end can then update only those specific fields. This approach is highly efficient and works well when no additional business validation is required.
The second approach extends the first one for cases where business rules require extra validation against the current database rather than the data sent from the front-end, such as checking the current voucher status in the database, verifying the number of users who have received the voucher, or validating other related data. In this case, the back-end should query only the information required for validation, perform the necessary checks, and then update only the fields that have changed. This is also good for security—for example, if the business rule only allows updating a voucher's status from draft to active, sending only voucher_id and status as active will be prevented if that transition isn't valid. So just a small change can bring a surprising result, as you can see in the two images below.
This was one of the most impactful initiatives I led to improve the performance and maintainability of our Voucher Service. For some context, during a Tech Group meeting, several developers mentioned that they were struggling to work with the service. The main challenge was Prisma. It had been introduced years ago by engineers who had already left the company, and over time the knowledge of how it worked was gradually lost. As a result, developers found it difficult to troubleshoot issues, understand complex queries, and confidently implement new features.
After investigating the service, I discovered that Prisma was being heavily used to generate complex subqueries. While it worked, it made the codebase harder to understand and maintain. I proposed migrating these database operations to TypeORM, which allowed us to use more explicit joins and a structure that was already familiar to most engineers in the company.
The goal was not only to improve query performance, but also to make the Voucher Service easier for future developers to maintain and extend. The scope of the migration was significant:
- 52 APIs required updates
- More than 150 Prisma operations needed to be migrated
- The service did not follow the Repository Pattern, causing database access logic to be scattered throughout the codebase
To reduce risk, I divided the migration into three phases. Each phase targeted approximately 17–18 APIs and was scheduled to be completed within one week. For every phase, I followed the same strategy:
- Move existing Prisma operations into dedicated repositories following the Repository Pattern.
- Implement equivalent TypeORM queries alongside the existing Prisma logic.
- Verify that both implementations returned identical results.
- Introduce a feature flag to control whether the service read data through Prisma or TypeORM.
This feature-flag approach gave us a safe migration path. If the TypeORM implementation behaved as expected, we could gradually increase adoption and eventually remove the old Prisma code. If any issues were discovered, we could instantly switch back to the Prisma implementation without impacting users while continuing to investigate and fix the problem.
Once all three phases were completed and the TypeORM implementation had proven stable in production, the final step was straightforward: remove the feature flags, delete the legacy Prisma code, and fully standardize the service on TypeORM. This approach allowed us to modernize a critical service with minimal risk while simultaneously improving performance, code readability, and long-term maintainability. You can see simple code below for more clarity.
/**
* @description Get list voucher by list voucher ids
* @param ids - List voucher ID want to find
* @param useTypeORM - Optional parameter to determine whether to use TypeORM or Prisma.
*/
public async getVoucherByIds(
ids: number[],
useTypeORM: boolean
): Promise<VoucherEntity[]> {
if (useTypeORM) {
return this.replicaRepo.find({
where: {
id: In(ids),
deleted: null,
},
});
}
// TODO: Will remove later when TypeORM worked good
return prisma.voucher.findMany({
where: {
id: { in: ids },
deleted: null,
},
select: voucherSelect,
}) as unknown as VoucherEntity[];
}Above is just simple code showing how I implemented the migration process. What surprised me is that migrating from Prisma (subqueries for the reference table) to TypeORM (joins for the reference table) not only made things easier to maintain, but also brought a really good response time improvement compared to the previous implementation, as you can see in the image below.
This was the final issue I discovered in the Voucher Service. The previous three improvements significantly increased performance and reduced resource consumption. However, despite those optimizations, the memory usage of the service continued to grow over time. The growth was slow, but it was consistent, which indicated that there was still an underlying problem that had not been addressed.
After monitoring the service and investigating several possible causes, I eventually identified the root issue. The Voucher Service was built using the NestJS framework, where business logic is organized into modules. For example:
- Voucher-related logic lives in the Voucher Module
- Campaign-related logic lives in the Campaign Module
- Queue processing logic lives in the Queue Module
- Referral-related logic lives in the Referral Module
In theory, this architecture makes the codebase easier to maintain because responsibilities are separated by feature. Teams can work on individual modules without needing to understand the entire application. However, this design can become problematic when dependencies between modules are not carefully managed.
Over time, modules began reusing logic from one another:
- The Voucher Module depended on the Campaign Module and Queue Module.
- The Campaign Module depended on the Voucher Module and Queue Module.
- The Queue Module depended on the Voucher Module and Campaign Module.
- The Referral Module depended on all three modules.
As the system evolved, these dependencies formed a complex network of circular references. To make the application compile and run, forwardRef() was used extensively throughout the codebase. While forwardRef() is a valid NestJS feature for resolving circular dependencies, it should generally be treated as an exception rather than a standard architectural pattern.
In the Voucher Service, the situation had become particularly severe. Circular dependencies existed across many modules, and forwardRef() had become the default solution whenever a dependency issue appeared.
From my experience working with NestJS applications, this is often a warning sign of deeper architectural problems. Excessive circular dependencies increase coupling between modules, make the codebase harder to understand, complicate testing, and can contribute to memory management issues as the application grows. Instead of relying on forwardRef(), I believe a better long-term solution is to reorganize the code and clearly separate responsibilities. In many cases, introducing shared services, extracting common business logic, or redesigning module boundaries can completely eliminate the need for circular dependencies.
So now we know that circular dependencies were the problem, but I think the more important question is why they happened.
After carefully reviewing the voucher service code, I found a common coding pattern that can easily lead to circular dependencies. I'll use a simple example to illustrate it. Once we understand this pattern and improve our code design, we won't have to worry about circular dependencies anymore.
The first issue is when core database operation logic lives directly in the service layer. When another service needs to reuse that logic, it has to import the entire service, which increases coupling between modules and can eventually create circular dependencies.
Let's take a look at the example below.
class VoucherService {
private readonly prisma: any; // Database access
private readonly campaignService: CampaignService; // ❌ Creates circular dependency
public async getVoucherByCode(code: string): Promise<any> {
return this.prisma.voucher.findByCode(code);
}
// ❌ VoucherService is responsible for voucher logic but also depends on CampaignService.
// This couples two domains together and makes the dependency graph:
// VoucherService -> CampaignService -> VoucherService
// Circular dependencies increase complexity and can cause DI/container issues.
public async getVoucherDistributedInformation(
code: string
): Promise<{ campaign: any; voucher: any }> {
const campaign: any =
await this.campaignService.getCampaignByDistributedVoucherCode(code);
const voucher: any = await this.getVoucherByCode(code);
return {
campaign,
voucher,
};
}
}
class CampaignService {
private readonly prisma: any; // Database access
private readonly voucherService: VoucherService; // ❌ Creates circular dependency
public async getCampaignByDistributedVoucherCode(
distributedCode: string
): Promise<any> {
return this.prisma.campaign.findByDistributedCode(distributedCode);
}
// ❌ CampaignService depends on VoucherService while VoucherService already depends on CampaignService.
// This creates a dependency cycle:
// VoucherService -> CampaignService -> VoucherService
// Cross-domain orchestration should be moved to a dedicated use-case/facade service
// to keep services independent and easier to maintain/test.
public async distributeVoucherByCampaignId(
campaignId: string
): Promise<void> {
const campaign: any = await this.prisma.campaign.findById(campaignId);
const voucherCodeDistributed: any =
await this.voucherService.getVoucherByCode(
campaign.voucherCode
);
}
}In this example, both VoucherService and CampaignService contain their own database access logic. At first glance, this looks reasonable because each service is responsible for its own domain. However, problems start to appear when one service needs to reuse data access logic from another service.
For example, CampaignService needs to retrieve voucher information, so it injects VoucherService. Later, VoucherService needs campaign information, so it injects CampaignService. This creates the following dependency graph:
VoucherService → CampaignService → VoucherService
This is a circular dependency.
The root cause is that database operations are implemented directly inside the service layer. Since the database queries are tightly coupled with business logic, other services must import the entire service just to reuse a simple query. As the codebase grows, these cross-service dependencies accumulate and eventually form dependency cycles.
The impact is not limited to circular dependency warnings. It also increases coupling between modules, makes services harder to test, complicates dependency injection, and makes future maintenance more difficult.
A better approach is to separate database access from business orchestration. Services should focus on business logic within their own domain, while shared data access or cross-domain workflows should be moved to repositories, data access layers, or dedicated orchestration/facade services. So I used the repository pattern to fix that—let's take a look.
// No contain business logic, only database operation
class VoucherRepository {
constructor(private readonly prisma: any) {}
public async findByCode(code: string): Promise<any> {
return this.prisma.voucher.findByCode(code);
}
}
// No contain business logic, only database operation
class CampaignRepository {
constructor(private readonly prisma: any) {}
public async findById(id: string): Promise<any> {
return this.prisma.campaign.findById(id);
}
public async findByDistributedVoucherCode(
code: string
): Promise<any> {
return this.prisma.campaign.findByDistributedVoucherCode(code);
}
}// Only contain business logic, no database operation handled in this layer, let repository do that
class VoucherService {
constructor(
private readonly voucherRepository: VoucherRepository,
private readonly campaignRepository: CampaignRepository,
) {}
public async getVoucherDistributedInformation(
code: string
): Promise<{ campaign: any; voucher: any }> {
const campaign: any =
await this.campaignRepository.findByDistributedVoucherCode(code);
const voucher: any =
await this.voucherRepository.findByCode(code);
return {
campaign,
voucher,
};
}
}// Only contain business logic, no database operation handled in this layer, let repository do that
class CampaignService {
constructor(
private readonly campaignRepository: CampaignRepository,
private readonly voucherRepository: VoucherRepository,
) {}
public async distributeVoucherByCampaignId(
campaignId: string
): Promise<void> {
const campaign: any =
await this.campaignRepository.findById(campaignId);
const voucher: any =
await this.voucherRepository.findByCode(
campaign.voucherCode,
);
}
}To resolve this issue, we introduced the Repository Pattern and moved database access logic out of the service layer.
Previously, services were responsible for both business logic and database operations. As a result, when one service needed data owned by another domain, it had to inject the entire service just to reuse a query. Over time, these cross-service dependencies created circular dependencies.
With the Repository Pattern, database operations are extracted into dedicated repositories. Services no longer depend on each other to retrieve data. Instead, they depend on repositories that are responsible only for data access.
This changes the dependency flow from:
VoucherService → CampaignService → VoucherService
to:
VoucherService → VoucherRepository
VoucherService → CampaignRepository
CampaignService → CampaignRepository
CampaignService → VoucherRepository
As a result, the circular dependency is removed because services no longer reference each other directly.
This approach also provides several additional benefits:
- Clear separation between business logic and data access.
- Reduced coupling between modules.
- Easier unit testing by mocking repositories.
- Better maintainability as the codebase grows.
- Lower risk of introducing new circular dependencies in the future.
By keeping repositories responsible for data access and services responsible for business logic, each layer has a single responsibility, making the overall architecture easier to understand and maintain.
Now let's look at a second pattern that can just as easily lead to circular dependencies: letting pure functions live in the service layer. A pure function has two properties—first, given the same input, it always returns the same result; second, it has no side effects (no reading/writing a database, no network calls, no file or global variable access, etc.). Here's an example of a pure function.
// ✅ Pure function
//
// - Depends only on its input parameters (a, b)
// - Does not access database, API, cache, file system, or global state
// - Does not modify any external data (no side effects)
// - Same input always produces the same output
function isGreaterThanOrEqual(a: number, b: number): boolean {
if (isNaN(a) || isNaN(b)) return false;
return a >= b;
}
// ✅ Pure function
//
// - Depends only on voucher and currentTimestamp inputs
// - Contains only validation logic
// - Does not access database, services, or external resources
// - Does not modify the voucher object
// - Same inputs always produce the same result (either pass or throw an error)
function validateVoucher(
voucher: Voucher,
currentTimestamp: number,
): void {
if (!voucher.name || !voucher.code || !voucher.status || !voucher.end_at) {
throw new Error('Invalid voucher: missing required fields');
}
if (
typeof voucher.name !== 'string' ||
typeof voucher.code !== 'string' ||
typeof voucher.status !== 'string'
) {
throw new Error(
'Invalid voucher: name, code, and status must be strings',
);
}
if (typeof voucher.end_at !== 'number') {
throw new Error('Invalid voucher: end_at must be a number');
}
if (voucher.end_at < currentTimestamp) {
throw new Error(
'Invalid voucher: end_at must be greater than or equal to the current timestamp',
);
}
}As you can see, pure functions are self-contained units of logic that depend only on their inputs and produce predictable outputs without interacting with external systems such as databases, services, caches, files, or global state. Because they have no dependencies, they do not need to live inside a service.
If pure functions are placed in the service layer, we can run into the same problem described in the first issue: when another service wants to reuse the function, it has to import the entire service that contains it. This unnecessarily increases coupling between services and can eventually contribute to circular dependencies.
Instead, pure functions should be extracted into utility classes, helper modules, or dedicated domain libraries so they can be reused without introducing additional service dependencies. So I spent one more sprint refactoring and reorganizing the whole code structure, and more than 20 circular dependencies were successfully removed. Finally, here is the result metric I got on the voucher service running in production.
As you can see, the voucher service is now running on only two healthy pods with very low resource consumption over the last seven days. Before these issues were fixed, the service required up to 15 pods and was consuming resources at its maximum limits. This optimization significantly reduced infrastructure costs for the company, which was one of my primary goals throughout this project.
During the process, I also increased test coverage from 10% to 70%, added more than three times as many logs to improve observability and debugging, and refactored the codebase to make it easier to maintain, migrate, and extend. The overall structure of the service is now much more organized, allowing other developers to work on it more efficiently.
One important lesson I learned while fixing the voucher service is that software systems rarely become problematic because of a single feature or a single bad decision. Instead, they gradually degrade over time through many small compromises in code quality. Each individual change may seem harmless when it is introduced, and the service may continue to function normally. However, as these small issues accumulate, they eventually interact with one another and create problems that are difficult to predict and even harder to diagnose.
This experience reinforced my belief that code quality is not a luxury—it is an investment. Good design, proper testing, clear logging, and continuous refactoring may seem expensive in the short term, but they prevent much larger operational and maintenance costs in the future.
That's all for this sharing. I hope you found it useful and enjoyed following the journey of identifying and fixing the issues within the voucher service.
If you have any suggestions, feedback, or topics that you'd like me to explore in future posts, feel free to reach out.
Thank you for taking the time to read this article, and I look forward to sharing more engineering lessons and experiences with you in the future.