In a previous post, we discussed the rise of the developer platform and how developer productivity is one of the main reasons why many organizations are either looking for or building an internal developer platform (IDP). According to a recent global survey done by Stripe, on a scale of 0 – 100%, developers responded that only 68.4% of their time is productive, which means that developers could be nearly 50% more productive than today:
(100% — 68.4%) / 68.4% = 46%
Realizing the potential for improvement, it’s easy to find teams implementing a microservices-based architecture, where the goal is to improve the developer productivity while achieving faster development, better scalability, smaller independent teams, independent deployment, etc. But, while a microservices-based architecture certainly has its many advantages, it can drastically increase complexity and bring teams back to square one to address developer productivity.
Now seeing the new challenges, many organizations set the goal of building their internal developer platform (IDP) to address developer productivity. While this is undoubtedly an essential and valid initiative, we see many of these teams abandon the idea of building their own IDP after being into it for a year or two.
Before we dig through some of the main reasons why we see teams abandoning this idea, another stat from the Stripe survey is that building custom technology is one of the top initiatives developers believe is hindering their productivity in their company:
Keeping that in mind, let’s explore some of the main reasons why teams are abandoning the idea or initiative of building their custom IDP:
Reason 1: Teams underestimate the complexity of building an IDP
I have worked with many teams that are very bullish on building their own IDP. For them, a custom IDP can be the silver bullet solution for all their developer productivity challenges, but they quite often underestimate the complexity of building an IDP.
Handling the first few requirements, users, and developers often give teams the confidence that scaling won’t be difficult. Still, as they start onboarding dozens, hundreds, or even thousands of developers, applications, services, and clusters, they quickly realize that custom scripts and automation won’t be enough to handle that scale. They start then getting push back from developers on the overall experience and developer productivity, application deployment gets impacted, the visibility and management of applications post-deployment become even more complex than before, it becomes hard to implement governance and controls across different applications, teams, requirements, and more.
Reason 2: No workflow to address different governance requirements
Different teams and different applications often require different levels of control and governance.
Although the statement above is quite simplistic, applying different levels of control and governance across different teams and services can become quite difficult, and DevOps and Platform Engineering teams now more than ever are tasked with implementing controls across security, RBAC, network policies, resource consumption, and more. As reason 1, applying those at a small scale can many times be handled through custom scripts and seem easy enough, but as you scale, the complexity of applying and maintaining these levels of controls for your different services and teams will grow exponentially.
Reason 3: Lack of observability and post-deployment management
It is interesting to see how people spend so much time figuring out Continuous Delivery (let’s not even get started on Continous Delivery vs. Continuous Deployment) and overlook the operational aspect of applications post-deployment.
Being able to continuously operate your applications post-deployment with deep observability and control of the application, its dependencies, and governance controls is essential to enable developers to fulfill the promise of speed and agility.
As teams abandon their IDPs, it is not uncommon to see the lack of strong capabilities around instrumenting, measuring, observing, and managing applications post-deployment as one of the main reasons. An IDP without a strong continuous operational model won’t give developers a proper link between Continuous Integration, Delivery, and Operation. In the end, a developer’s job does not end with the code just being deployed.
Reason 4: Lack of integration into an existing stack
It’s incredible to see the speed at which development teams adopt new tools today across monitoring, alerting, incident management, and others. At the same time, DevOps and Platform Engineering teams need to adapt infrastructure to ever-changing requirements from both business and applications.
As these requirements change and evolve, they realize that building an IDP should not only be able to address current requirements but also integrate them into new tools that are adopted by development teams, extended to address additional requirements, and be able to leverage existing and new infrastructure components that are adopted at the organizational level.
An IDP that can only address today’s requirements will decrease developer adoption over time. This has also been one reason why teams fear when hearing the term “Platform as a Service” because they fear being stuck into a platform that cannot grow and evolve together with their internal requirements. That may be a topic for the next discussion!
Building an IDP that can address this can be incredibly challenging if not planned ahead of time. It will certainly require dedicated time and resources to properly do it, which teams also underestimate many times.
Reason 5: Lack of clear definition across roles and responsibilities
With the adoption of microservices-based architecture and technologies such as Kubernetes, it’s quite common to see infrastructure-related tasks being introduced as part of the application delivery and management process.
While DevOps and Platform teams have spent a good amount of time learning about Kubernetes, you often see developers pushing back, saying it introduces unnecessary complexities that were not part of the development process before and being now responsible for understanding, creating, and maintaining components such as deployments, ingress rules, charts and more does not help address the developer productivity concern, which is a fair point.
Microservices and Kubernetes introduced infrastructure-related components not only at the application deployment time but also in the application operational aspect, where understanding, supporting, and troubleshooting applications’ object dependencies have added even more complexity for developers, which in turn, creates an additional burden on the DevOps and Platform engineering team, blurring the responsibilities between the different teams.
Reason 6: Its a product, not just custom scripts
As teams start putting together their IDP, it is not uncommon to see it being treated as a point-in-time initiative driven primarily by the DevOps or Platform Engineering team.
May teams realize too late that building an IDP will require them to drive it as an actual product, require them many times to:
- Perform periodic user/developer interviews, so they deliver what will address the requirement of their end-users rather than their own thoughts and objectives.
- Establish a roadmap and delivery cadence so developers are aware of what’s coming next and when
- Perform developer advocacy inside their own organization to evangelize and train developers on the benefits and how to move to this new platform
- Establish a support team
These are just a few of the tasks tied to building and driving their own product. When you don’t treat your IDP as an actual product but rather as a group of scripts to address a specific scenario you had in your mind, the chances are that your developers will never successfully adopt your IDP. Your investment of time and resources won’t be worth it.
Reason 7: Cost
We are a large organization, and cost is not a concern. We want to invest in building something that will solve our problem, and we are unique
Does the quote above sound familiar?
While this is a great statement, after a year into it, many teams realizing that building a custom IDP is:
- Not core to their business
- Takes at least 25% of the time of DevOps or Platform Engineers to code custom IDP code
- Does not add business value
- They are not as unique as they initially thought when it comes to developer productivity and requirements.
I’m a fan of how Jeff Lawson defines Build vs. Die (yep, not Buy but Die!). In his Ask your Developer book, he describes that everything that your end customer or users directly access should be built by your team because that is what brings uniqueness to your business, but how you deliver these functionalities to your end-users does not matter to them, and I couldn’t agree more.
If you think of your bank, when you open your online bank app, you don’t care if the team behind it used microservices, Kubernetes, or even mainframes. All you care about is the experience and reliability of the application.
Conclusion
An Internal Developer Platform (IDP) now is top of mind for many teams when trying to address the developer productivity issue within their organizations, but building something that can address today’s requirements, allow your developers to move fast, and that can evolve together with your requirements can be more challenging than expected. If not planned well initially, it can turn into a cost-consuming initiative that won’t drive the desired outcomes.
I believe this brings us right back to the result from Stripe’s survey on initiatives hindering developer productivity across different companies:
In the next post, we will discuss how we see different teams successfully addressing the points above and how they leverage their IDPs to drive developer productivity at its best.
