One approach mirrors the underlying system. The CLI exposes the full object model and API surface area. The other approach structures interaction. It organizes complexity so developers can navigate it with less cognitive effort.

As systems grow in scale, that distinction becomes more important.

Heroku: Structuring the Common Path

I recently heard that Heroku is effectively moving into maintenance mode. That news made me reflect on how much influence it had on developer experience, especially through its CLI.

For many developers, Heroku was the first time infrastructure felt simple.

You ran:

heroku create

git push heroku master

And your application was live.

The effectiveness of this flow was not accidental. Heroku built deployment directly on top of Git. Instead of introducing a new operational model, it aligned infrastructure with an existing developer habit.

You pushed code. The platform handled build, release, routing, and process management.

Heroku did not expose infrastructure primitives like instances, networking rules, or load balancers. Those concerns existed, but they were abstracted behind an opinionated workflow.

The CLI was intentionally limited. It did not expose every possible knob. It constrained the surface area in exchange for cohesion.

Heroku did not mirror infrastructure. It structured access to it around the most common developer workflow.

That design decision reduced both operational and decision complexity.

AWS CLI: Complete System Exposure

The AWS CLI represents a different philosophy.

It exposes more than 300 services and thousands of operations. Nearly every API call is accessible. In some cases, the CLI even provides capabilities not available in the console.

It is comprehensive.

But it does not impose a unified workflow across services. Each service has its own namespace, naming conventions, and flags. Discoverability is limited. Exploration usually requires documentation rather than interaction.

You cannot realistically hold the entire surface area in your head. You must know what you are looking for.

The AWS CLI mirrors the system architecture faithfully. It reflects organizational scale and API breadth.

That fidelity is powerful. But it transfers the burden of structure to the developer.

K9s: Reducing Interaction Cost

Kubernetes is also a large and complex system.

The kubectl CLI exposes the Kubernetes object model directly: pods, deployments, replica sets, stateful sets, services, config maps, ingress objects, and more. Working with it often involves multiple terminals and carefully assembled commands.

At Probo, while running production workloads on Kubernetes, we used K9s extensively.

K9s does not hide Kubernetes complexity. It preserves the object model. What it changes is the interaction cost.

Instead of repeatedly assembling commands, you navigate a live, continuously updating view of cluster state. You can sort by CPU or memory with a keystroke. You can switch namespaces quickly. You can inspect, edit, and drill into objects without constructing new command chains each time.

It uses real-time watch APIs, so state updates are continuous rather than discrete.

For engineers unfamiliar with kubectl, onboarding is faster because the system becomes discoverable. You explore available objects instead of recalling exact syntax.

K9s mirrors Kubernetes. But it organizes access to it in a way that reduces cognitive effort.

Claude CLI: Structuring Interaction With Uncertainty

Claude CLI applies similar structuring principles, but in a different category of system.

Traditional CLIs interact with deterministic systems. You issue a command and receive a predictable result. Even if the system is complex, the behavior is stable.

AI systems are different. They are probabilistic and opaque. Outputs are not guaranteed. Internal reasoning is not directly visible. Without structure, interaction can feel uncertain.

Claude CLI addresses that by structuring the interaction loop.

It provides visible processing feedback instead of blocking silently. It adapts cleanly to terminal resizing. It surfaces contextual shortcuts when relevant rather than requiring memorization. It avoids rigid flows while still offering structured guidance. When proposing code changes, it allows review before acceptance.

These choices reduce uncertainty. They introduce checkpoints where trust matters. They help developers reason inside a probabilistic system rather than treating it like a deterministic one.

Claude CLI does not simplify the model. It structures how developers engage with it.

The Real Shift

The evolution of the CLI is not from commands to interactivity.

It is from exposing complexity to structuring it.

As systems expand in capability — whether cloud infrastructure, container orchestration, or AI models — faithfully mirroring internal APIs becomes insufficient. The surface area grows beyond what can be navigated through recall alone.

The more complex the system, the more responsibility shifts to the interface.

Modern platform teams are no longer just exposing capability. They are shaping how engineers think inside complex systems.

There is a lot happening in the tech scene right now.

New model releases, new coding agents, and more conversations around autonomous coding. Things are moving fast. I don’t know if it’s a bubble around me or if it’s actually everywhere, but it’s very real in my circle. The people I follow, the people I talk to in real life. Everyone seems to be discussing the same shift.

It feels important. Like this is not just another tool upgrade, but a change in direction.

At the same time, I sometimes feel a bit discouraged. Not depressed, but something closer to uncertainty. What we used to do, and how we used to do it, is changing. The way I learned to code, the way I spent years getting better at it, doesn’t map one-to-one anymore.

I’m not resisting this change. I’m actively using these tools. Claude, Codex, multiple agents. I experiment, I adapt, I try to understand what works. Work still gets done. In many cases, it gets done faster.

The joy of coding is still there for me. It just looks different now.

I’m not writing every single line of code anymore. Instead, I’m guiding agents. Shaping direction. Reviewing outcomes. Translating thoughts into something executable through these tools. That process still gives me joy.

Recently, this became very clear while I was building my claude.md.

I spent time writing down my engineering philosophy. Not as theory, but as a summary of what I’ve learned over the years. Things like YAGNI with seams, being pragmatic about DRY, preferring simple code over clever abstractions, designing for deletion, treating security as a first-class concern, and focusing on integration tests over excessive mocking.

Writing it forced me to reflect. On mistakes I’ve made. On systems that became hard to change. On abstractions that looked elegant early on but became liabilities later. On production incidents that shaped how I think about observability, testing, and security.

I felt genuine pride while writing it. Not because it was perfect, but because it felt honest. Like compressing years of experience into something explicit. Something that could guide not just me, but also the agents I work with.

It wasn’t a new realization. More like a confirmation.

The joy of coding was never about typing code.

It was always about the end product.

But the process mattered. It was challenging, sometimes frustrating, and that’s exactly what made it satisfying. The thinking. The tradeoffs. The discipline to decide what not to build. The willingness to delete things that no longer served the system.

Today, that same process still exists. It has just moved up a layer.

Instead of expressing judgment through keystrokes, I express it through constraints, principles, reviews, and guidance. Through documents like claude.md. Through how I steer agents rather than how fast I type.

I don’t know how the future will look. I don’t know how far autonomy will go, or what the role of an engineer will eventually become.

What I do know is that the part I’ve always enjoyed hasn’t disappeared. It has just shifted.

The joy was never in typing code. It was in the process of thinking deeply, making tradeoffs, and turning ideas into systems that work.

If that process still exists in some form, I’m curious to see where this goes.

Anyone who has spent years building real systems knows that this sense of stability is mostly an illusion. Software engineering is not built on certainty. It is built on continuous change.

The Promise of “Final Answers”

Every few years, the industry confidently presents a new answer that is supposed to last.

• Java was once considered the safest long-term choice

• Python became the language of productivity

• JavaScript expanded beyond the browser and powered everything

• React was seen as the solution to UI complexity

• Kubernetes promised to make scaling manageable

• Big Data was expected to reveal insights at scale

• Machine Learning became a differentiator

• Now AI is framed as unavoidable

Each of these waves arrived with confidence and implied permanence. None of them truly delivered it. The tools themselves are not the problem. The problem is the belief that any tool can remove uncertainty from the career.

Two Tracks That Shape Every Career

A software engineering career usually runs on two tracks at the same time.

Knowledge That Compounds

Some knowledge lasts and deepens with experience.

• How systems fail under load

• Why simple designs outlive clever ones

• How performance, cost, and reliability trade off

• How incentives influence architecture more than documentation

• How users behave in unexpected but repeatable ways

This knowledge compounds. It remains useful regardless of trends.

Adaptation That Never Stops

The second track is adaptation.

Languages change. Frameworks evolve. Architectures reset. Tooling gets replaced. These skills are learned, applied, and eventually left behind. They do not compound in the same way.

Careers tend to stall when engineers confuse these tracks. Some chase novelty without building depth. Others rely only on past experience and resist change. Longevity comes from balancing both.

The Hidden Mental Load

Software engineering is endless problem-solving. Not the kind that ends with closure, but the kind where solving one problem quietly introduces several more. You rarely finish work. You move from one set of constraints to another.

Over time, this creates mental pressure. FOMO creeps in slowly. A new framework trends. A peer switches jobs. Salary benchmarks move. Even when things are going well, it can feel like you are falling behind.

Financial uncertainty adds to this. When savings, investments, or expenses are not fully under control, every slowdown feels risky and every market shift feels personal.

Pressure Inside Organizations

Within companies, the pressure continues through reviews, ratings, promotions, and increments. These systems are designed to encourage growth, but they often feel like a race with no clear finish line.

The focus slowly shifts:

• from learning to proving

• from building to performing

• from long-term thinking to short-term validation

The work stops being only about improving systems. It becomes about constantly optimizing yourself.

When Things Break Beyond Your Control

There are also failures engineers cannot influence.

Poor management decisions. Overhiring during good times. Sudden strategy shifts. Layoffs driven by financial models rather than performance. You can do everything right and still lose your job.

Unlike many traditional professions, software engineering offers little institutional protection. There is no government-backed job security and no guarantee that years of contribution will translate into continuity. Stability depends heavily on market conditions and leadership choices.

This reality quietly shapes behavior. People save aggressively, spend cautiously, and stay alert even when things seem stable.

What Experience Eventually Teaches

With time, most engineers learn a difficult but useful truth. This career does not reward certainty. It rewards the ability to adapt without losing direction or burning out.

You stop asking what to learn forever. You start asking what helps you stay relevant now. You separate identity from tools, focus on fundamentals, adapt when necessary, and let go when required.

Closing Thought

Seeing the uncertainty in software engineering does not mean disliking the work. Both can coexist.

Many people stay in this field not because it is stable, but because they genuinely enjoy problem solving. There is a quiet satisfaction in turning vague ideas into working systems, in watching something abstract become real, and in seeing logic take shape on a screen. That sense of creation is difficult to replace.

Titles change. Tools evolve. Guarantees remain elusive. But the joy of solving problems and seeing things work keeps people going through the cycles.

Software engineering is not a ladder. It is a landscape that keeps changing shape. Uncertainty is not a temporary phase. It is part of the job.

Learning to live with it, while still enjoying the craft, is what allows people to last.

Continuation of Cricket Scale, Part 3

We often get so excited about doing “technical work” that we forget first principles and end up deep in yak shaving — optimizing the wrong thing, adding complexity no one asked for, or celebrating fixes that shouldn’t have existed in the first place.

At Cricket Scale, some of the most impactful wins weren’t about clever infra or cutting-edge tools. They came from the not-so-technical grind: listing APIs, questioning every spike, trimming redundant calls, and removing waste. Unglamorous chores, but they saved us more than any scaling trick ever could.

So in this post, I’ll start discussing those chores — the work that rarely makes it into conference talks, but quietly makes systems faster, cheaper, and easier to run.

1. Questioning the Existence

During load testing, we noticed a few APIs with throughput that stood out — far higher than expected. Our tests mimicked real user behavior in the app, mapping common flows to the APIs behind them. That exercise quickly pointed us in the direction of listing every API and looking at the numbers: throughput, p90/p95/p99 latencies, average response times, and error rates.

why are these APIs being called so often?

The first question we asked was simple: why are these APIs being called so often?

• Instant Refresh Assumptions — Frontend logic assumed users always needed the “freshest” state, so certain endpoints were being hit far more frequently than necessary.

• Reloads and Missed Caching — Some APIs reloaded every time a user landed on a key screen (like the home page), even when nothing had changed. A few endpoints also slipped past client-side caching. Each case seemed small in isolation, but multiplied at DAU scale, they created massive unnecessary load.

Take the home page as an example. Every time a user landed there, our feed API was called — which made sense, because the feed had to be fresh. We couldn’t afford to show expired events.

But alongside the feed, our hamburger API was also being triggered on every visit. This endpoint pulled details like user balance, skill score, and help ticket status — things that didn’t need to be refreshed with the same urgency as the feed.

Because the home page has the highest DAU, the hamburger API ended up matching the feed API in throughput. And that wasn’t the end of it — each hamburger call fanned out into other services, hitting the wallet service and skill score service to fetch details.

So what looked like a harmless reload turned into a cascade of unnecessary load, inflating throughput across multiple systems. At scale, this kind of oversight meant we were spending infra and engineering effort just to serve data users didn’t actually need that often.

why does this API exist at all?

Once we answered why they were called so often, the next question was even more fundamental: why does this API exist at all?

• Zombie APIs — Endpoints left behind from old experiments or features long since retired.

• Unfinished Features — Half-built ideas that never made it to production, but still left endpoints active.

Cleaning these up wasn’t glamorous work — it meant auditing flows, talking to product managers about what was still relevant, and carefully deprecating endpoints so nothing broke. But each removal was one less thing to scale, one less source of noise in our metrics, and one more bit of clarity in how the system actually worked.

Does this product feature even matter?

We didn’t stop at APIs. We went a step further and questioned the product features themselves.

We made a list of APIs that were expensive to run and mapped them against their importance to the user experience. Wherever we saw features that weren’t contributing much, or had very low to no usage, we sat with the product team and evaluated whether they were still worth keeping.

In many cases, the honest answer was no. Those features were quietly adding cost, complexity, and operational overhead without giving users any real value. With the product team’s support, we removed them altogether.

At scale, sometimes the best optimization isn’t caching or tuning — it’s simply deleting what no longer matters.

What about expensive but necessary APIs?

Then there were cases where the APIs were genuinely expensive but couldn’t just be deleted. A good example was expired or past events in which users had participated — essentially their long-term portfolio.

For an app that had been live for more than a year, this meant a massive amount of historical data. Fetching and rendering all of it on demand didn’t make sense — the cost was high, the latency was painful, and most users only cared about their recent activity.

Here, the solution wasn’t removal, but redesign. Instead of hitting expensive APIs every time, we thought carefully about what actually needed to be shown up front, and what could be deferred, summarized, or tucked behind pagination.

This was another kind of grind — not heroic infra scaling, but thoughtful product and engineering decisions about what truly mattered to the user experience.

2. The Ops Grind

Not all of the grind was about APIs or product features. A big part of scale was operational — managing observations, tuning infra, and staying ahead of limits.

Observability and Checklists

We built a top-level system view, inspired by Dream11’s observability work, that could instantly tell us which service or dependency in the entire system was having an issue. And with the help of a master dashboard, we had the ability to drill down further — moving from the big picture to the exact service, API, or dependency causing trouble.

👉 If you’d like me to go deeper into how we approached observability — from designing top-level views to drill-down dashboards and fine-tuning alerting — let me know in the comments. That might be the focus of a future post.

Alongside this, we relied heavily on checklists, timely enhancements, and alert fine-tuning. This wasn’t glamorous, but it gave us clarity and helped us avoid firefights turning into outages.

Infra Limits and Quotas

From the infrastructure side, the grind looked different but was just as important. Scaling wasn’t only about adding nodes or tuning autoscalers — it also meant making sure we never hit the practical limits of the cloud.

Cloud providers usually enforce soft limits on resources, and if these are overlooked, they can easily cause downtime during peak events. To stay ahead, we sat with each team to understand their scaling requirements and helped set their scaling configs in place well in advance. Depending on their needs, we also did capacity reservations so resources were guaranteed when traffic spiked.

Beyond quotas, we focused on resilience:

• Creating backups so we could quickly recover in case of failures.

• Preparing SOPs for incidents so everyone knew what to do if something went wrong.

• Warming up redundant dependencies so failover paths were ready and wouldn’t add cold-start latency during a match.

None of this was glamorous, but without it, all the scaling design in the world wouldn’t have mattered — the system would have hit a ceiling we didn’t control.

The Playbook Grind

Scaling wasn’t just about systems — it was about people. For the first day of the largest cricketing event, the IPL, we developed detailed SOPs that laid out exactly how things would run. Nothing was left to chance.

• A game marshal was appointed to oversee coordination.

• We documented the points of contact across customer experience, business, marketing, and other key teams.

• A dedicated Slack channel was created for real-time coordination.

• An on-call rotation was set up so engineers knew exactly who was responsible at any given moment, ensuring quick response without chaos.

• We defined pre-match activities (scaling checks, quota reviews, dry runs), during-match activities (traffic monitoring, alerts, notifications), and post-match activities (user comms, debriefs, reporting).

Each step was written down, and reviewed. It wasn’t glamorous engineering work, but it created alignment. Everyone knew their role, the handoffs were smooth, and there was no confusion when the real traffic hit.

Closing Thoughts

The “not-so-technical grind” rarely makes it into architecture diagrams or scaling war stories, but it’s often where the real wins come from. Listing APIs, questioning why they exist, debating product features, preparing infra checklists, and writing SOPs — none of it is glamorous, but all of it compounds at scale.

Sometimes the most powerful scaling move isn’t adding more infra. It’s asking simple, first-principle questions and having the discipline to trim, prepare, or delete.

Karpenter promised something fundamentally different. Instead of relying on ASGs, it provisions EC2 instances directly, based on rules you define with just two CRDs: Provisioner and AWSNodeTemplate. That means instead of being locked into a fixed pool of machines, Karpenter can pick the best available and cheapest instance at runtime.

That idea was powerful enough for us to give it a try in our lower environments, as it wasn’t production-ready at the time.

Why We Looked at Karpenter

The motivation was simple: our workloads were scaling up and the cracks in Cluster Autoscaler were starting to appear.

• Speed: Cluster Autoscaler could take about 2 minutes for a new node to become ready. With Karpenter (plus a lightweight AMI), it was closer to 40 seconds.

• Flexibility: We wanted more freedom in choosing machines, mixing on-demand with spot instances, and selecting from a wider set of EC2 types.

• Cost: By leaning on spot capacity and flexible provisioning, we could save up to 40% on compute without changing how workloads were deployed.

• Drift Detection: Karpenter can detect when nodes become outdated or misaligned with current requirements and proactively replace them. (Note: This feature was introduced in later versions, not available in early 2023)

This wasn’t about chasing new tech for the sake of it rather it was about building flexibility into our scaling model so we could adapt under unpredictable demand.

The Early Days

When we started using Karpenter in 2023, it was far from stable. The stability journey was gradual:

• Pre-v0.32: Breaking changes were common, sometimes with every minor release

• v0.32: Marked a turning point where APIs started stabilizing

• v1.0.0: Finally delivered the production-ready stability we needed

Some of the specific pain points:

• Cryptic errors: Pods stuck in Pending with vague “capacity not available” messages. Nodes failing silently because of IAM misconfigurations.

• IAM complexity: We had to manage multiple roles (controller, node bootstrap, EC2 instance profile). If trust policies didn't line up, nodes would launch but fail to register. For example, if the node instance profile lacked the necessary EKS worker node policies, pods would stay pending even though EC2 showed healthy instances.

• Breaking changes:

  • v0.27 changed default consolidation logic, breaking scaling policies.

  • v0.32 overhauled scheduling APIs, forcing CRD rewrites.

Despite all this, the potential upside kept us going.

How Karpenter Works (in brief)

Cluster Autoscaler depends on Auto Scaling Groups: you define which instances you want, and the autoscaler requests capacity by resizing those groups.

Karpenter removes ASGs from the equation instead:

1. A pod is pending.

2. Karpenter evaluates your Provisioner and AWSNodeTemplate. {in 2023}

3. It chooses the cheapest + most available EC2 type that satisfies requirements (on-demand or spot).

4. It launches the instance directly and joins it to the cluster.

That direct model is what makes Karpenter faster and more flexible.

What Worked, What Hurt, What Broke

After months of experimenting, tweaking provisioners, and firefighting, we started to see a clearer picture of what Karpenter actually meant for us. It wasn’t just theory or benchmarks anymore, it was lived experience. The results were mixed: genuine wins, persistent frustrations, and a few near-disasters that taught us hard lessons.

Here’s how it really played out.

The Good: When Demand Hit Us Out of Nowhere

The first time Karpenter proved its worth. It was a weekday afternoon, a fairly low-traffic period, when suddenly one of our events got picked up on social media. Within minutes, requests shot up nearly 3×.

With Cluster Autoscaler, this would’ve meant a painful 2–3 minutes of waiting for new nodes. Instead, Karpenter spun up fresh capacity in close to 40 seconds. Pods scheduled almost immediately, the customer impact graph barely wobbled.

The second win was cost. By default, Karpenter mixed spot and on-demand intelligently, shaving ~40% off our compute bill without needing us to micro-manage instance types. That kind of flexibility and no more rigid ASGs felt transformative.

The third win was subnet planning. With Cluster Autoscaler was always a headache. IP exhaustion could quietly block scaling, and fixing it meant manual subnet juggling across availability zones.

Karpenter flipped that model entirely. As long as we tagged subnets with karpenter.sh/discovery=<cluster-name>, it could shift nodes across them automatically. We could add bigger subnets without re-architecting capacity, and scaling bottlenecks from IP exhaustion mostly disappeared.

And by the time v1.0.0 shipped, Karpenter was finally stable enough. It went from “experimental toy” to a critical part of our stack.

The Bad: Living on the Edge of Release Notes

But before v1.0, running Karpenter felt like being part of a beta program we never signed up for. Some minor upgrades came with breaking changes. Provisioners that worked last week suddenly refused to validate this week.

The documentation made things worse. Most of it still assumed EKS clusters used the old aws-auth ConfigMap, even though AWS had deprecated it in favor of EKS access entries back in 2022. We ended up digging through GitHub issues more than reading official docs.

And then there was setup overhead. IAM roles, trust relationships, provisioner tuning. It was never a plug-and-play install. Every new environment meant hours of yak-shaving before workloads could actually run.

These weren’t “cute edge cases.” They were the kind of pain that leaves scars where you’re refreshing Grafana dashboards while combing through opaque controller logs, trying to understand why autoscaling, the thing you promised would be smoother, is actively betraying you.

When to Consider Karpenter

• Choose Karpenter if:

  • Node startup latency is hurting your workloads.

  • You’re under cost pressure and want to use spot effectively.

  • You want flexibility to adapt to unpredictable demand.

• Stick with Cluster Autoscaler if:

  • Stability > flexibility for your environment.

  • Your clusters are small, predictable, or heavily regulated.

  • Your team doesn’t have bandwidth to manage IAM and configuration complexities.

• Recommended starting point: Begin with the latest v1.x stable release. Avoid older versions entirely.

Final Reflection

Adopting Karpenter early wasn't smooth. We endured cryptic errors, broken upgrades, and IAM headaches. But the payoff was real: faster scaling, lower costs, and flexibility that Cluster Autoscaler couldn't match.

The real lesson isn't about Karpenter itself, it's about pragmatic engineering. Sometimes the safe choice (Cluster Autoscaler) is fine. But sometimes, the risky bet (Karpenter in 2023) pays off because it unlocks new ways to scale under pressure.

In our case, it was worth it. If you're hitting the walls of traditional autoscaling, it might be time to consider the trade-offs.

What’s Next

Considering Karpenter for your setup? The implementation has its gotchas—IAM configuration, provisioner/Nodepool tuning, and migration strategies all need careful planning. If there's enough interest, I'll cover the practical setup details and lessons learned in a follow-up post.

When we first brought Redis into the mix, it felt like the silver bullet. Everything got faster — API responses dropped, counters updated instantly, and user experience stayed snappy even during match peaks.

At peak, ~99.7% of requests were served in under 100 ms, with an average response time of just ~15 ms.

Redis gave us the low-latency backbone we needed to keep the platform responsive, even as millions of requests were flowing every minute. It quickly became our default answer for anything that needed to be fast:

• Hot caches for APIs

• Real-time counters for trades, traders, portfolio

• Sorted sets for leaderboards

• Sessions, rate limiting, and throttling

And for a while, Redis felt like the silver bullet.

But Redis also came with its own set of challenges, and those challenges became more noticeable as we scaled:

The Rug for Our Tech Debt

Redis didn’t just make our systems faster — it also made our problems less visible. In hindsight, we were guilty of sweeping tech debt under the Redis rug.

• Slow database queries? Put them behind Redis.

• Unoptimized schema? Hide it under a cache.

• Missing indexes? Redis will cover it up.

• Poorly thought-out architectures? Add Redis in the middle and call it “scalable.”

It worked — until the rug was pulled away. The moment Redis faltered, all the hidden dirt came spilling out: the database choked, the APIs stalled, and the architectural shortcuts stood exposed.

The lesson for us was clear: Redis is excellent for performance, but it’s not a substitute for good database practices or sound architecture.

We learned this the hard way. Sometimes it was a Redis downtime that exposed all the hidden issues at once. Other times, it was during load testing combined with a bit of chaos testing that showed just how fragile the underlying systems really were.

Not all redis data is equal: Ephemeral vs Permanent

Redis shines when it’s used for ephemeral data: sessions with TTLs, rate limiting, short-lived caches. If Redis goes down, the system can repopulate or tolerate the loss.

But things get tricky when Redis is used for permanent data — like leaderboards without TTLs. Suddenly, Redis isn’t a cache anymore; it’s acting as a database, but without durability guarantees. And when it’s not available, the question becomes less technical and more product-driven:

• What should the user see when Redis is down?

• Do you try to calculate the leaderboard on the fly? (expensive and slow)

• Or do you handle it gracefully? For example, showing a message that “leaderboard is being built” or temporarily hiding the feature.

This is where having a product mindset matters. If Redis is critical to the experience, you need a fallback plan that puts the user first — whether that’s hiding the feature, showing stale data, or communicating clearly that it’s temporarily unavailable.

Redis taught us that system design and product design can’t be separated. At scale, resilience isn’t just about infra choices; it’s also about how you want users to experience failure.

Hot & Heavy: The trouble with Keys at scale

One of the toughest Redis lessons came during a high-stakes India cricket match. Traffic spiked, DAUs surged, and suddenly our app started crashing. The events feed — the most visible part of the experience — stopped working.

On debugging, we discovered Redis response times had shot up. The culprit: a hot key.

For simplicity, think of it as one prebuilt event feed cached under a single Redis key. (In reality, we had multiple keys with combinations like user segmentation and language, but the point remains the same.) That feed was updated asynchronously through an algorithm, and all requests hit it.

During normal days, this design held up. But when India played — and tens of millions of users came in at once — that one key became a single point of failure, even though Redis was running in a cluster. All the load funneled into a single shard, overwhelming it, while other shards stayed underutilized.

The first bottleneck we hit wasn’t CPU or memory — it was network throughput. That shard simply couldn’t push responses fast enough. We also realized our key size made the situation worse: the feed payload was large, so every request pushed a heavy response over the network.

In subsequent years, hot keys continued to be a recurring theme. This time, they showed up as high CPU utilization and latency spikes on particular shards.

To solve this there can be multiple solutions:

• Multiple read replicas: adding replicas and enabling reads from them to spread the load

• Distributing load across multiple keys with the same value, to avoid a single point of pressure. This would involve some code changes to your cache get method.

• Client-side caching at the Kubernetes pod level, so not every request had to hit Redis. This reduced load on Redis, but brought its own challenges — especially update propagation and cache invalidation across hundreds of pods.

• Compression for big keys where payload size couldn’t be reduced at the source. This helped ease network throughput issues, but added CPU overhead for compress/decompress trade-offs.

In our case, since this was a feed, the writes (i.e. feed update logic) weren’t very frequent. That meant replication lag wasn’t a concern, which made the Redis master/replica method a clean fit. We scaled Redis simply by enabling multiple reads across replicas. The Redis client we used (ioredis) even exposed this as a simple boolean flag — making the shift straightforward while leveraging higher network throughput–enabled instances.

Costing and Migrations - The price of speed

Our Redis journey wasn’t only about performance — cost and operations played a big role too.

We started with self-hosted Redis, which worked fine in the early days. But as scale grew, the cracks began to show:

• Downtime: our setup couldn’t keep up with rapid traffic spikes, and the feed (again) was the culprit.

• Recovery pain: bringing Redis back after failures was slow and messy, especially under load.

• Management overhead: handling failovers, patching, and scaling manually took too much engineering time.

The first big shift came when we migrated to AWS ElastiCache. That move gave us reliability and faster operations, but it also introduced new challenges:

• To meet higher memory requirements, we had to pick larger instances.

• That meant running 8-core or 16-core machines even though Redis itself is single-threaded — so most of those extra cores just sat idle. We were paying for CPU we couldn’t use, simply to get more memory.

• This led to inefficient resource utilization, and since managed offerings from AWS aren’t cheap at scale, the cost quickly became significant.

Eventually, we moved again — this time to RedisLabs (now Redis Enterprise). Surprisingly, this turned out to be a cheaper alternative while maintaining the same scale. Our guess is that Redis Enterprise optimizes resource utilization differently — possibly by running multiple Redis processes on the same underlying machine — which reduced waste compared to ElastiCache. Whatever the exact reason, it gave us a better balance between cost efficiency and operational reliability.

The Obvious Good Practices (That Are Easy to Forget)

A lot of Redis pain can be avoided by sticking to the basics. These aren’t fancy tricks — just good hygiene that makes a big difference at scale:

• Monitor slowlogs: keep an eye on commands that take longer than expected. They’re often early warnings of a bigger issue.

• Choose data types thoughtfully: don’t just reach for strings. Sorted sets, hashes, and bitmaps all have trade-offs in memory and performance.

• Set TTLs by default: ephemeral data should expire on its own; otherwise, you risk memory bloat.

• Avoid big keys: break payloads into smaller chunks to reduce network and serialization overhead.

• Watch replication lag: especially if you’re reading from replicas under heavy load.

• Plan for failover: assume Redis will go down at some point and decide what users should see when it does.

At cricket scale, these fundamentals turn from “nice to have” into “must have.”

Final Thoughts

Redis was one of the most valuable tools in our stack. It helped us achieve p99 latencies under 100 ms at scale and kept the platform responsive during peak cricket moments.

At the same time, it highlighted the importance of using the right tool for the right job. Redis worked very well for certain use cases, but it also showed its limits when we stretched it too far.

Integrating Google Calendar as the “Control Plane”

The first building block of Scaler was surprisingly non-technical: a shared Google Calendar. We realized that our traffic spikes weren’t random — they were event-driven. And who in the company knew about events ahead of time? The marketing team. Instead of building an elaborate custom interface, we decided to piggyback on something everyone already used and trusted.

Step 1: Set up a shared calendar

We created a dedicated Google Calendar called Scaling Events. Access was shared with engineering and marketing. Each person could add or edit events based on their role. Engineers got the calendar ID, which let us fetch the data programmatically, while non-technical teammates just saw “another calendar” show up in their Google Calendar UI.

Step 2: Encode scaling intent as events

Each event in the calendar represented a scaling profile. The title was simple and human-readable—high, medium, night, low—and always tied to a cricket match, campaign, or product launch. These were created after consulting the marketing/business team, so engineers didn’t have to guess when to scale.

Step 3: Describe what to scale

The event description field became our schema. Inside it, we listed which resources needed scaling—comma-separated:

HPA, ASG, RDS, REDIS

That way, a marketing manager could say “this IPL playoff match is high traffic, scale Redis and RDS accordingly”without touching infrastructure code.

Step 4: Process the inputs

Once these events were in place, the next step was automation. We wrote an Airflow DAG to poll the Google Calendar API every 10 minutes. Any scheduler would have worked, but at Probo we leaned heavily on Airflow for orchestrating time-based tasks, so it fit right in.

Airflow DAG: Calendar → Scaling Actions

The DAG worked in two layers:

1. Template Definition

   A generic scaling_template described how each resource could scale across different traffic levels. It covered everything from RDS instance types to Redis dataset size, broken down by profiles (default, medium, high, night).

   Example (simplified for RDS):

{
    "RDS": {
        "<ENVIRONMENT>": [
            {
                "name": "<Name of the RDS Cluster>",
                "scaling_enabled": "<true/false>",
                "instance_ids": ["<RDS Instance Names>"],
                "traffic_config": {
                    "default": {
                        "instance_type": "<db.instance.type>",
                        "min_capacity": "<number>",
                        "max_capacity": "<number>"
                    },
                    "medium": {
                        "instance_type": "<db.instance.type>",
                        "min_capacity": "<number>",
                        "max_capacity": "<number>"
                    },
                    "high": {
                        "instance_type": "<db.instance.type>",
                        "min_capacity": "<number>",
                        "max_capacity": "<number>"
                    },
                    "night": {
                        "instance_type": "<db.instance.type>",
                        "min_capacity": "<number>",
                        "max_capacity": "<number>"
                    }
                }
            }
        ]
    },
    "calendar_id": "<YOUR_CALENDAR_ID>@group.calendar.google.com"
}

1. Configuration in Git

   To keep things auditable, environment-specific configs were stored in Git. Every change (say, adding a new RDS cluster to scale) was reviewed like any other code change.

   Example (simplified):

{
    "RDS": {
        "PROD": [
            {
                "name": "main-db",
                "scaling_enabled": true,
                "instance_ids": ["db-instance-1", "db-instance-2"],
                "traffic_config": {
                    "default": {
                        "instance_type": "db.t4g.medium",
                        "min_capacity": 1,
                        "max_capacity": 1
                    },
                    "medium": {
                        "instance_type": "db.r6g.large",
                        "min_capacity": 3,
                        "max_capacity": 3
                    },
                    "high": {
                        "instance_type": "db.r6g.xlarge",
                        "min_capacity": 10,
                        "max_capacity": 50
                    }
                }
            }
        ]
    },
    "calendar_id": "<YOUR_CALENDAR_ID>@group.calendar.google.com"
}

This two-step structure—templates for patterns, Git for actual config—kept the system both flexible and reviewable. Marketing could decide when to scale, engineers defined how scaling worked, and Airflow acted as the glue in between.

Closing thoughts

What we built with Google Calendar + Airflow may look deceptively simple, but it was the foundation of Scaler. Instead of chasing fancy abstractions, we gave the business team a language they already understood—calendar events—and wired it directly into infrastructure logic.

“This meant engineers weren’t firefighting during every cricket match, and marketers could directly influence capacity without opening a single AWS console.”

That said, this is just the first layer. There’s a lot more beneath the surface—like:

• The processing logic: how calendar events were parsed, mapped to Git configs, and converted into scaling actions on AWS resources (ASGs, RDS, Redis, HPA).

• Safety nets & guardrails: how we ensured that a bad calendar entry or misconfigured template didn’t accidentally scale down production during a high-stakes event.

• Real-world outcomes: cost savings, reliability gains, and some hard lessons learned about prediction vs. reaction.

If you’d like me to go deeper into these areas, let me know—I can cover them in the next post.

During our journey at Probo, we experienced this first-hand. The start of an IPL match would bring sharp surges in traffic — far beyond what daily active user metrics or linear growth curves could prepare us for. A single ball could trigger millions of requests at once.

This is commonly referred to as cricket scale.

It was different from normal scale — spiky, sudden, and emotionally driven. Handling it required rethinking not only our infrastructure, but also our processes and how our teams operated.

This series is an attempt to document the lessons we learned along the way.

What to Expect

• Part 1 → EC2 vs Kubernetes: picking the right scaling foundation

• Part 2 → Redis: A Silver Bullet With a Cost

• Part 3 → The Not-So-Technical Grind

While RMG applications may no longer operate in India, the challenges of spiky traffic patterns remain. Concert ticketing, IPO subscriptions, elections, or live streaming platforms all face their own versions of the cricket scale.

Through this series, I’ll share what worked for us, what didn’t, and how we built systems (and teams) that could survive the pressure of cricket scale.

In the second year of Probo, even regular India cricket matches (not just IPL) pushed our systems to the edge. Traffic patterns during the game were sudden, steep, and unpredictable.

Our EC2-based setup, which relied on autoscaling groups, was reliable but slow to react. Scaling new instances could take minutes, while traffic spikes arrived in seconds. We realized this approach might not hold up as Probo grew. We needed something quicker.

At the same time, we were tinkering with Kubernetes. It promised more configurable scaling — the ability to tune how workloads scaled up and down, to use horizontal and vertical pod autoscalers, and to react to more than just CPU or memory.

EC2 — The Baseline

When we started, running Node.js applications on AWS EC2 instances was straightforward:

• Simple model: launch an instance, deploy the app, attach it to a load balancer.

• Autoscaling groups: scale in/out based on CPU and memory

• Predictable performance: Node.js processes on EC2 gave us stable latency under load.

We used pm2 as the process manager, running multiple processes per instance — usually n-1, where n is the number of cores — to maximize CPU utilization while leaving some headroom.

This setup worked fine for steady traffic and moderate growth. But at cricket scale, the coarse granularity of EC2 autoscaling meant we were always a step behind the spikes.

Kubernetes — The Next Step

Kubernetes looked like the natural answer:

• Configurable scaling: horizontal and vertical pod autoscalers, custom metrics, and more control over how workloads scale up and down.

• Bin packing: schedule multiple pods with different resource needs onto the same EC2 node, making better use of CPU and memory.

• Faster churn: pods can come up quicker than booting whole VMs.

On paper, it solved the problems we were facing. But our early results showed the opposite — applications that ran smoothly on EC2 performed worse when moved to Kubernetes.

The reasons became clear: networking overhead, noisy neighbors, CoreDNS bottlenecks, and the operational burden of tuning clusters under sudden bursts. I’ve written about some of these challenges in more detail here: Production-Grade Pain: Lessons From Scaling Kubernetes on EKS.

The Trade-offs

In theory, Kubernetes gave us more knobs to tune. In practice, we had a small team and IPL was around the corner. There were already higher-priority tasks on the table, and investing deep time into tuning Kubernetes clusters wasn’t realistic.

So we took the pragmatic route:

• For high-scale events like IPL matches, we resorted to EC2 instances

• Instead of pouring resources into scaling Kubernetes, we invested time into building a switcher — a mechanism that shifted traffic to EC2 instances when we expected spikes.

This simple switcher eventually evolved into a full-fledged scaler solution, giving us the reliability of EC2 when we needed it most, while still allowing us to experiment with Kubernetes in parallel.

The trade-off was clear: predictability now vs flexibility later. And at cricket scale, predictability was what kept us alive.

Scaler — Inhouse Scaling Solution

That switcher eventually evolved into a full-fledged scaler solution.

It started simple: a predictive scaling strategy. The marketing team would share expected traffic numbers before big matches, we looked at DAU, and from there we planned the number of EC2 instances we’d need.

The scaler worked off calendar events, each tagged with a scale profile:

• High-scale for IPL and international cricket.

• Medium-scale for smaller matches or anticipated spikes.

• Low-scale for regular days.

• And to optimize cost, we also introduced a night-time-scale, dialing resources down when traffic naturally dropped.

This hybrid model gave us predictability when it mattered most, and cost efficiency when it didn’t — while still leaving room to experiment with Kubernetes in the background.

👉 If you’d like me to go deeper into how this inhouse scaler worked, let me know — I can cover it in a dedicated post.

Closing thoughts

Cricket scale forced us to move faster than we planned. It showed us that scaling infrastructure isn’t about chasing the latest tool — it’s about making trade-offs that match your workload and stage of growth.

“Thinking in systems” is the same principle, applied to the systems we build.

What System Thinking Means to Me

For me, system thinking means building with the whole picture in mind, not just the task in front of me.

I don’t just aim to “complete” something; I aim to create a system — one with a solid, reliable core that doesn’t break under pressure. Around that core, it should be extensible so it can grow, configurable so it can adapt, and resilient enough to handle edge cases without special hacks.

It’s about designing in a way where today’s solution doesn’t limit tomorrow’s possibilities. A well-thought-out system makes scaling, maintenance, and evolution natural — not painful.

An Example from My Work

In my early years at Probo, I was tasked with catching unfair usage patterns — for example, users running multiple cloned apps to place trades. Some of these behaviours qualified as fraud, and my job was to prevent them and block those users.

On paper, it was a simple task: detect the pattern → block the user. But Probo is building something unique, and the fraud patterns were still evolving. If I had only solved for the few cases we knew about, the system would have failed the moment new patterns appeared.

So my approach was to build a system — one that was extensible, configurable, and observable, without directly depending on me for every new case. I implemented a rule engine combined with our in-house tagging system.

The workflow was simple:

1. A user journey event — like signup or login — was sent to the fraud rule engine.

2. The user, along with their metadata, was evaluated against a set of rules.

3. If a violation was found, the user was tagged under a specific fraud category.

4. Each fraud tag carried its own set of restrictions, which could be applied instantly.

This meant that as fraud behaviours changed — and they did — we could respond quickly without rewriting the entire detection logic. The system became a foundation we could keep improving, instead of a one-time patch.

The Cost of Task-First Thinking

I’ve seen this countless times — a feature gets shipped, works fine for a while, and then has to be completely re-written when new use cases emerge.

The reason is usually the same: we focused on getting the task done, not on how it fits into the bigger system. The design was fine for today, but brittle for tomorrow.

This doesn’t just cause rewrites — it also takes a hit on scalability. A solution that seems lightweight at low usage can turn into a bottleneck as traffic grows.

In fact, when we first built the fraud rule engine at Probo, daily active users were relatively low. At that time, it could have been tempting to run it as a synchronous system in the user flow. But had we done that, today’s scale would have meant massive slowdowns in normal operations. By designing it as an event-driven, decoupled system, we avoided a scalability wall that would have crippled us later.

The task-first approach delivers speed in the moment — but it often leaves behind hidden costs that surface only when scale and complexity catch up.

System Thinking ≠ Over-Engineering

Thinking in systems doesn’t mean building something that will last untouched for the next 10 years. In a fast-paced startup, velocity is critical. You can’t freeze progress in the name of perfect architecture.

What it does mean is judging wisely the level of configuration and flexibility a solution really needs. Sometimes spending 2–3 extra days on a task today can save you weeks of painful rewrites a few months later.

That’s an underappreciated skill — knowing when to invest that extra effort, and when to deliberately choose the quick path with eyes wide open.

A Time I Over-Engineered

I learned this lesson the hard way.

Once, we were experimenting with ways to create delight for users when they won a trade. For example, if someone won more than a certain amount, we wanted to show a celebratory meme. The idea was simple: a small surprise that might encourage virality.

If I had built this in the most direct way, it could have been done in 2–3 days. Instead, I spent almost two weeks building a templatization engine that could handle multiple scenarios — based on winning amount, number of trades, leaderboard rank, and more.

The reality? The experiment didn’t pan out. We closed it without ever needing those additional templates. The two weeks of careful system-building had little practical value.

That experience reminded me: system thinking is valuable, but over-engineering is still a risk. The real skill is not just in designing flexible systems, but in knowing when to keep it simple and wait for validation.

Shortcuts Aren’t Evil — Blind Shortcuts Are

Just like self-aware teams can take a shortcut consciously when the trade-off is worth it, systems thinking accepts that sometimes you must optimise for the immediate need.

The difference is:

• Without systems thinking: “This works for now — ship it.”

• With systems thinking: “This works for now, but here’s the cost, and here’s how we’ll address it later.”

That awareness is what keeps future you from cursing past you.

Why It Matters

Software issues often don’t come from a single bad decision. More commonly, they build up over time as a series of small, local choices that don’t work well together.

Thinking in systems helps reduce this risk. It encourages designing with future growth and change in mind, so today’s solution doesn’t become tomorrow’s bottleneck.

By approaching problems this way, you’re not just addressing the immediate need — you’re making it easier and more efficient to handle what comes next.

You join a standup and everyone is clear on what’s next.

They know what’s blocked, what’s in progress, and why it matters.

Work moves steadily without needing constant direction.

This is what a self-aware team looks like: a group that doesn’t just work on assigned tasks, but understands the context, priorities, and impact behind them. They can adapt, make decisions, and keep moving even when a manager isn’t present.

Why Self-Awareness Matters in Teams

In most engineering teams, people are good at completing the work in front of them. But if the “why” is missing, two things often happen:

• Effort goes into optimising for the wrong outcomes.

• Progress slows when dependencies shift or leaders aren’t available.

Without the bigger picture, developers can make short-term decisions that create technical debt or move the system further away from the ideal state. These choices often seem harmless in isolation, but over time they compound — leading to a situation that’s expensive, risky, or slow to reverse.

Self-awareness means building a shared understanding of:

• The problem we’re solving.

• The outcome we’re aiming for.

• The trade-offs we’re willing to make along the way — and their impact on the long-term vision.

When developers hold this understanding, they:

• Know what to do next to move closer to the ideal state.

• Are self-sufficient in identifying and creating their own work.

• Can set their own pace without waiting for constant approvals.

• Feel a stronger sense of autonomy and ownership over the outcome.

From a manager’s perspective, this translates to:

• Fewer delays and bottlenecks.

• Less day-to-day handholding.

• Smoother progress even when priorities shift.

A self-aware team is not just more effective — it’s easier to lead.

How to Build Self-Awareness in Teams

Self-awareness doesn’t happen automatically — it’s cultivated. And one of the most important ingredients is ownership.

Ownership comes when people are aligned to the outcome, not just assigned tasks. When team members understand the “why,” the ideal state, and the trade-offs involved, they naturally start making decisions that move the work forward. They don’t wait to be told what to do next — they create their own next steps within that shared context.

The following practices help build that alignment and ownership from the start:

1. Start With Motivation, Not Tasks

Every project begins with a short, clear statement of why it matters:

• The problem: What issue or gap are we addressing?

• The impact: What improves if we succeed?

• The cost of inaction: Why this work is important now. This helps with prioritisation

This context helps the team see the work as part of a bigger picture.

2. Define the Ideal State

In real-world engineering, we often take shortcuts to ship faster. That’s fine — as long as the team knows what the end goal looks like.

The ideal state answers:

• “If there were no constraints, what would the best version of this be?”

• “In six months, what result would we be proud to share?”

This keeps short-term compromises from becoming long-term defaults.

3. Build Acceptance Criteria Together

Self-awareness grows when the team co-creates the definition of done.

Involve the team directly:

• “What must be true for us to call this complete?”

• “What would make us confident this solves the problem?”

These discussions often uncover assumptions or blind spots early. When the team helps define success, they are more likely to maintain quality and consistency without reminders.

4. Keep the Document Alive

The initial project document — motivation, ideal state, and acceptance criteria — works best when it stays relevant throughout the project:

• Update it when priorities or scope change.

• Refer to it in standups and retros.

• Mark shortcuts explicitly so they can be revisited later.

In practice, this can be difficult to maintain alongside day-to-day work. Treat it as a good to have — even partial updates can help keep the team aligned and avoid losing sight of the bigger picture.

5. Connect Daily Work Back to the Big Picture

A self-aware team keeps the “why” in mind during execution:

• Every task links back to the project motivation.

• Code reviews ask: “Does this move us closer to the ideal state?”

• Shortcuts are taken consciously, documented, and followed up.

Challenges and Navigating Resistance

Not everyone in a team starts with the same level of ownership or clarity. Some naturally take initiative, while others wait for direction. Building self-awareness across the board means recognising these differences and addressing the barriers that hold people back.

Common barriers include:

• Ego — “I already know what’s best.”

• Fear — concern that ownership means more responsibility without adequate support.

• Low clarity — not knowing how to apply self-awareness in day-to-day work.

A big part of overcoming these challenges is the manager’s role. Managers need to give confidence to team members by providing freedom and autonomy, allowing them to fail safely, and taking time to understand their perspectives. This builds both competence and trust over time.

Practical ways to reduce resistance:

• 1-on-1 conversations to understand motivation and hesitation.

• Appreciation and recognition when initiative is shown.

• A free hand where possible, instead of micromanaging.

• Probing questions to guide thinking, rather than direct answers.

• Modelling the behaviour — openly sharing context, acknowledging trade-offs, and showing how to adapt when things change.

When managers create an environment where autonomy is safe and valued, resistance fades, confidence grows, and team members begin anticipating the next move instead of waiting for instructions.

The Payoff

When teams share the same context and are aligned to the outcome, they can decide their own next steps, maintain momentum, and adapt without friction.

For developers, that’s a sense of autonomy and ownership.

For managers, it’s confidence that progress will continue without constant intervention.

A self-aware team doesn’t just complete tasks — it consistently moves the work closer to the ideal state, even when the path changes.

Do you think your team is self-aware?

I’d love to hear how you’ve built (or struggled to build) it — drop a comment or DM me your thoughts.

Lately, platform teams have gained a lot of attention, often compared to DevOps or SRE teams. But having worked in platform teams across multiple orgs, I’ve seen that while the promise of a platform team is great, the reality often gets messy.

Here are a few challenges I’ve faced—and lessons learned along the way.

1. Platform Work Has a Long Feedback Loop

Platform projects are usually long-term. Add slow adoption (especially in large orgs), and it can take weeks or even months to know if a release made an impact.

This is very different from product teams, which get feedback quickly from users, metrics, or A/B tests. For platform work, success is often subtle and delayed—and harder to measure.

Over time, that delay can drain motivation.

What helped:

• Celebrate small wins. Even a working POC, adoption by one team, or shaving minutes off CI time is a win.

• Define platform-specific metrics.

  • Improving observability? Track MTTD (mean time to detect) or MTTR (mean time to repair).

  • Building CI/CD tooling? Track deploy frequency, failure rate, or downtime hours.

• Demo internally. Show your work. Build trust.

2. Don’t Become the Bottleneck

In fast-moving orgs, platform teams often become a blocker—usually unintentionally.

Product teams depend on the platform for infra access, deploy automation, observability setup, or network configs. But when automation lags or processes are unclear, product teams either wait—or find hacks.

This puts platform teams in a tough spot:

Ship fast and fix it later? Or slow things down to “do it right”?

What helped:

• Convention over configuration. Standardize defaults, configs, naming patterns—anywhere a dev has to make a decision, help them make the right one faster.

• Invest in documentation-as-a-product. If someone’s stuck, docs should be the unblocker—not Slack.

• Pre-empt needs. Stay one step ahead by tracking repeat asks and turning them into products.

3. Avoid Becoming the Dumping Ground

This one’s common: anything that doesn’t fit cleanly into a product team’s scope ends up on a platform ticket. From one-off S3 buckets to VPN questions to “hey can you help with this test suite,” the list grows quickly.

Most of these aren’t complex. The issue isn’t difficulty—it’s unclear ownership.

Platform teams then spend increasing time on support work, often needing separate sprints or even dedicated sub-teams to handle tickets.

What helped:

• Clarify ownership boundaries. If a task doesn’t need deep platform knowledge, it shouldn’t come to the platform team.

• Create playbooks and self-serve tools. Most support tickets can be avoided with good internal guides and small CLI tools.

• Enable, don’t gate. Help product teams help themselves.

Final Thoughts

Platform teams can be incredibly impactful—but only when they stay focused.

The goal isn’t to be everything to everyone. It’s to build leverage, not load.

That means saying no, simplifying where it matters, and making space to celebrate the progress you’re making behind the scenes.

🚀 Let’s Chat

What’s your experience been like working with or inside a platform team?

Have you run into any of these challenges—or solved them differently?

Drop a comment or reach out—I’d love to learn from your story.

You can also follow Thoughtful Engineering for more lessons from real systems and real teams.

Before diving into developer experience (DevX), it’s important to understand the role of a platform team.

What Is a Platform Team?

Gergely Orosz described it well in his blog The Pragmatic Engineer: platform teams build the internal building blocks that product teams use to ship customer-facing features. A good platform helps teams move faster, more safely, and with less friction.

Platform teams can work at many layers—providing infrastructure, central tooling, service creation flows, CI/CD, config management, observability, experimentation, and more.

Developer Experience ≠ Just Developer Tools

When your users are developers, it’s easy to assume they’ll figure things out. So we ship command-line tools with verbose flags, dashboards that need digging, or runbooks buried in wikis—because “they’ll manage.”

But this mindset leads to:

• Over-complicated setup processes

• Layers of configuration

• Weak documentation and poor onboarding

• Confusion around what’s owned, automated, or expected

This is very different from how we treat non-developer end-users, where we aim for delight and simplicity.

Just because our users are developers doesn’t mean they deserve less.

Why DevX Matters More Than Ever

In 2025, developer velocity is a product moat. The easier it is for engineers to create, test, and ship code, the faster a company moves.

Good DevX:

• Reduces cognitive load

• Speeds up onboarding

• Prevents avoidable mistakes

• Encourages consistency without enforcing rigidity

For platform teams, DevX isn’t a nice-to-have. It’s the core product.

A Simple Example

Let’s walk through a simplified, ideal developer experience.

Context: Nilesh from the Growth team needs to launch a rewards microservice.

What the flow looks like in a DevX-focused org:

1. Nilesh logs into the internal developer dashboard.

2. He clicks “Create new app”, selects a language (say, Node.js), and chooses dependencies like Postgres and Redis.

3. He’s given a GitHub repo with:

   • Boilerplate folder structure

   • Linting and testing setup

   • Docker Compose with service dependencies

   • CI/CD pipeline configured with build, test, and deploy stages

4. He writes code. CI/CD deploys it to staging, then prod, without needing help from infra or DevOps.

What Nilesh skipped:

• Filing infra requests

• Setting up GitHub permissions

• Picking or aligning framework versions

• Writing config files for lint, test, and deploy from scratch

• Coordinating with DevOps for deploy pipelines

This isn’t about 100% automation—it’s about removing unnecessary steps so engineers can focus on solving business problems.

What DevX Really Means

Developer experience isn’t about flashy portals or adding more tools. It’s about thoughtful defaults, reducing friction, and enabling autonomy without chaos.

At its core, good DevX means:

• Better documentation and discoverability

• Simple integration and testing paths

• Clear ownership models

• Fast feedback loops

• Decentralised decisions with guardrails

Closing Thoughts

A high-growth engineering org is like a well-oiled machine. The platform team is the layer that keeps the rest running smoothly. When DevX is neglected, platform efforts become bottlenecks. But when DevX is intentional, it becomes a multiplier.

If you’re building or evolving a platform team, start with developer experience.

It’s not a feature—it’s the foundation.

If this resonated with you, I’d love to hear how your team approaches DevX.

You can also subscribe to Thoughtful Engineering for more posts like this.

But after some time, we noticed a few problems:

• These people were getting tired and burned out. They were always on call and always needed.

• The rest of the team was slower, not because they didn’t care, but because they waited for these few to take the lead.

• Everyone said reliability was important, but only a few were actually working on it. It looked like a shared responsibility, but in reality, it wasn’t.

No one did this on purpose—it just happened over time. People assumed, “They’ll handle it.”

But what if one of them was on leave? Or decided to leave the company?

We realized that for our team to grow in a healthy way, ownership couldn’t stay with just a few people. It needed to be shared, clear, and part of how we worked every day.

Looking back, the problem wasn’t just that a few people were handling all the alerts. The real question was:

Why did everyone else stay away?

It wasn’t because people didn’t care. It was more about things like:

• “I don’t know if this concerns me“

• “I don’t know enough to help.”

• “It’s not my area—someone else owns it.”

• “This is how it’s always worked here.”

No one was told to stay out, but the way things were set up made them feel that way. Even if they wanted to help, they didn’t feel confident or responsible.

Problem 1: “I don’t know if this concerns me”

This was the most common reason people stayed away from alerts. When something broke or an alert fired, many didn’t step in—not because they didn’t care, but because they weren’t sure if it was their responsibility.

A big part of the issue was how our alerting was set up:

• Alerts were too generic and not tied to specific services or teams.

• Many alerts were triggered on shared infrastructure, so it was unclear what part of the system was actually affected.

• Most alerts landed in a shared Slack channel, without context or clear ownership.

So even when someone saw an alert, their first thought was usually:

“Is this mine? Should I look into it? Or is someone else already on it?”

That hesitation added delay. And more often than not, the same people who always responded… responded again.

We learned that without clear signals, most people will assume it’s someone else’s job. Especially in high-stakes situations, people avoid stepping in if they’re unsure.

This was the first thing we had to fix—make ownership visible and obvious.

Problem 2: “I don’t know enough to help”

Even when people knew the alert was related to their area, many still didn’t take action. The reason? They didn’t feel confident enough.

We heard things like:

• “I don’t know where to start.”

• “What if I try something and make it worse?”

• “Someone else will probably fix it faster than me.”

In many cases, the required knowledge lived only in a few people’s heads. There were no clear runbooks, no shared dashboards, and no easy way to understand what was going on unless you had been around long enough or had seen it before.

This made reliability feel like a “specialist’s job,” not something everyone could contribute to.

We realized that without easy access to context or tooling, even willing team members won’t step up. Not because they don’t want to—but because they feel stuck.

To fix this, we had to invest in documentation, observability, and simple debugging tools—so anyone on the team could at least start investigating, even if they weren’t the expert.

Problem 3: “It’s not my area—someone else owns it.”

Even when someone understood the alert and knew how to help, they often stayed away because they believed it wasn’t their job.

We’d hear things like:

• “This isn’t my service.”

• “That team usually handles this.”

• “I don’t want to step on anyone’s toes.”

This mindset became more common as we started splitting our systems and teams. Silos started to form—each team had their own services, their own dashboards, their own processes. While separation helped with focus, it also created invisible walls. People became hesitant to step outside their boundaries, even when they had something useful to add.

The end result? A narrow view of ownership. Teams only focused on their piece, even if issues were affecting others too.

We realised that clear ownership is important, but collaboration across teams is just as critical. Especially in complex systems, problems rarely sit neatly within one team’s boundaries.

A simple step that helped break silos was starting a rotating on-call program, which gave more people system-wide context and shared responsibility.

Problem 4: “This is how it’s always worked”

Sometimes the biggest blocker is not technical—it’s habit.

Many people stayed away from alerts or reliability work simply because that’s how things had always been. A few people had always handled it, so the rest of the team assumed that’s how it’s supposed to be.

There was no discussion, no decision—it just became the norm over time.

This mindset is hard to spot, and even harder to change. It leads to passive behaviour, where people wait instead of act. Even when ownership shifts or teams grow, these old patterns often stay in place.

To break this pattern, we realized that change doesn’t always start with a policy. It starts with conversations.

• In 1-1s, we talked openly about reliability work and whether people felt confident owning it.

• In career conversations, we highlighted how taking initiative on such problems is a sign of growth.

• During RCAs and postmortems, we made space for people to ask questions and understand areas outside their team.

Over time, these regular conversations helped shift mindsets. People started seeing ownership not as extra work, but as part of being a strong engineer.

Closing Thoughts

We didn’t fix ownership gaps overnight. It took a mix of changes—some technical, like routing alerts better, and some cultural, like encouraging open conversations during 1-1s and RCAs.

But the biggest shift came when we stopped relying on a few “go-to” people and made reliability a shared goal.

We’re still evolving, but one thing is clear: ownership doesn’t have to come from authority. It grows when people feel safe, supported, and informed enough to step up.

And that’s what we’ve been working toward—a culture where ownership is not assigned, it’s shared.

Seen similar patterns in your team? I’d love to hear how you’ve approached this.

If this resonated with you, subscribe to get future posts—straight from the production floor.

You might have read dozens of articles around Monolith vs Microservices. And most of them suggest to keep your distance with the microservices until you have solid reasons to do so.

But micro-services are tempting, you can get these solid reasons well early into your development cycle or as late as never. There are numerous articles on how to split monolith to microservices. But very few articles talk about how to do your monolith right. This is an attempt at the latter.

Context

At Probo, we are a small team of about 30 engineers. One basic principle we try to adopt is to keep things simple. As a result, we are running a monolithic version of our backend from day 1 and we are well into our 4th year. We have a seen a journey of 0 trades to 250Mn+ trades in a day.

Things that helped us in this journey

Code structure

• Our code structure is inspired by Domain Driven Design. From the start itself we had three major bounded contexts that we recognised and developed the code around it.

• It was as simple as — User (profile, referral, etc), Trading (order-management, trade-management etc) and Payments (user balance, transaction history etc) ecosystem.

We tried to keep interaction between these contexts loosely coupled. Most of the database joins were in their own bounded contexts.

• We had a very basic layered code architecture — Presentation → Domain logic → Data management layer.

• Concept of services in a monolith, with a focus on preventing domain leakage.

• If required each individual service used to communicate with other service through service<>service methods and not by data layer.

Splitting into Server & Worker modes

• We encountered performance issues well early into our journey when everything was handled with a single process. To solve, we first identified the background tasks and separated them out of server process.

• As a result, both server process and various worker processes could now independently scale without affecting one another.

The bottleneck is always the database

We use AWS managed database offering — MySQL RDS

• As mentioned by Kailash Nadh, “97.42%* of all scaling bottlenecks stem from databases”. And we can confirm that it is true!

• Focus on fundamentals. “No Indexes” or “incorrect Indexing” is the most popular answer I get during my interviews when I ask about downtimes.

• Leveraging performance insights of RDS helped us pin point multiple issues. We were able to solve long running queries, reduce expensive queries, understand transactions.

• Investing time in learning about database parameters helped us scale our database better and avoid downtimes. This is required even when you are using a managed database like AWS RDS.

• One thing we did actively was to post slow queries on slack — this kept us on toes!

• Use of database proxy helped scale our application horizontally, without worrying of connection limitations.

• On a team level, everyone had access to database monitoring dashboards, that really helped to understand and learn about the issues together. I have seen multiple organisations which limit database access to a certain team, namely devops or dba team which I feel limits the developers to understand and write optimised code.

Caching Layer

• Initially, we started investing in our caching layer to ease of pressure on our database and improve API response times.

• The setup quickly expanded to solve other use cases like Leaderboard, Authentication, Realtime, Feed, Graphs etc.

Independent endpoint scaling

• Segregating our high throughput endpoints, helped us in scaling horizontally and help isolate issues faster.

• Example — when a notification is sent to our users, we update the read/view status on our end. This is a burst traffic that impacts other APIs performance. Identifying these Api’s and redirecting to their own set of dedicated instances helped us prevent slowness.

Language Specific Tweaking

• Whatever language you pick, there are always some basic level of configuration tweaking that is required to scale efficiently.

• We use NodeJS majorly, and since it is single threaded we run it in cluster mode using PM2 to efficiently utilize our compute resources.

• Some other configurations like--max-old-space-size=SIZE (sets the max memory size of V8's old memory section), --prof (for profiling) were used

Observability & Alerting

• We spent a considerable amount of time in setting up our observability right. I wouldn’t say it is perfect, but it gets the job done.

• Our logging infrastructure is self-hosted, a simple Elasticsearch cluster with Kibana. Though it looks simple, but scaling Elasticsearch has been a challenge.

• Initially, the scale on our infrastructure was a self-inflicted scale, since a lot of logs were not useful. To tackle this problem, we created a simple library — https://engineering.probo.in/from-chaos-to-clarity-the-evolution-of-logging-from-unstructured-to-structured-part-1-78b059bed362

• For metrics, we utilised last9. It helped us to spend time on creating effective dashboards instead of spending time on installation/maintenance & scaling.

• Last9’s alert manager helped us in democratising alerts among our teams. Despite a monolith, each team owned their metrics and alerts.

Not everything is good in Wonderland.

Deployment time

There is a significant increase in deployment time due to

• CI Pipeline tests and checks

• Rolling deployment, hence the time taken to complete the deployment is directly proportional to the number of instances

Fear of change

There is always a fear that a change might take complete service down. This has resulted in apprehensions in the frequency of deployments or during high traffic times, etc.

Multiple dependencies & Resource Sizing

• Dependencies in terms of databases, cache etc keep on growing with use cases. Thus increasing your surface area for downtimes.

• Horizontal scaling also puts pressure on your resources, like Database connections, which leads to unoptimized sizing. Proxy & Pooling to the rescue!

In conclusion, it’s quite possible to scale a monolithic architecture to handle a high volume of transactions, given the right practices and considerations.

We’ve focused on keeping our codebase simple and structured, while there are challenges but we’ve found solutions that work for us. The journey has been rewarding and educational, and we hope our experience can provide useful insights for others facing similar scaling challenges. At Probo, with increasing scale, we get regular opportunities to re-evaluate our architecture. Stay tuned for more updates!

If you enjoyed reading this, you’re one of us — someone who loves solving deep technical challenges. We’re building high-performance systems and tackling complex engineering problems every day.

Follow - https://engineering.probo.in/

Subscribe now