Internal developer portals: a springboard to platform engineering

Many engineering leaders look for a centralized deployment platform where everything operates seamlessly. This involves simplifying complicated processes, allowing developers to focus on creating quality software. To achieve this goal, a central hub for monitoring and managing the entire software development lifecycle is crucial.

Understanding the difficulties of platform engineering reveals successful techniques, practical procedures, and the appropriate technology for your needs. Internal developer portals are key to modern corporate performance because they provide streamlined access to vital information.

Recently, there's been a significant surge in adopting internal developer portals (IDPs) to enhance the developer experience and expand existing knowledge. It is a natural next step in the platform strategy evolution to introduce an internal developer portal and is also a common pilot project for many organizations.

However, aligning different dimensions, such as processes, infrastructure, and UX, while creating the right contract between the platform and the portal can be challenging. A well-executed portal acts as a gateway, encouraging widespread platform adoption among intended users.

Why organizations invest in developer portals

We are aware that workplaces that heavily rely on digital technology benefit greatly from simplifying developers' tasks. For example, you can reduce time to production from two months to a few days and reduce ongoing infrastructure maintenance by 80%. 

The goal is to streamline tasks through code reuse, adherence to technical standards, and design guidelines. However, developers often face challenges in integrating these elements, which leads to missed objectives and trade-offs in their workflows. Supplying the right interfaces for developers resolves these issues.

Enabling engineers in their day-to-day work

Internal developer portals move the focus away from top-down directives and toward developer-centric approaches. These gateways function as the "connective tissue," supporting a variety of functions. This includes the following.

• Compliance and governance: Ensuring adherence to organizational standards, such as coordinating code freezes from a central point.
• Streamlining feature deployment and service scaffolding: Providing flexibility to create experiences tailored to your organization’s needs.
• Simplifying cloud and infrastructure as code: Facilitating collaboration and eliminating duplication through an inner source approach.
• Improving incident management and on-call processes: Clarifying system interactions and ownership to enhance incident triage and escalation.
• Managing permissions: Enabling self-service management of access to external or internal systems, with approval flows or just-in-time access.
• Supporting self-service capabilities: Integrating with CI/CD systems, triggering custom actions, or managing feature toggles.
• Enhancing visibility: Offering insights into service catalogs, API versioning, team topologies, and costs, and enabling productivity tracking with scorecards.
• Centralized documentation: Consolidating information to prevent developers from searching across multiple sources.

The challenges before internal developer portals

Prior to the introduction of IDPs, organizations used custom web portals to handle routine tasks like onboarding, access control, and provisioning new cloud accounts. 

Instead of empowering developers, each technological department frequently produced its own version of a web portal, focusing mainly on control. This strategy led to a complex combination of different systems and manual labor, poor user experiences, and slow performance. Finding a balance is difficult because developers often state a preference for code-based interfaces over complex portals. 

Legacy shortcomings

The primary shortcomings of the legacy approach included:

• Lack of an API-first mindset: Custom portals often struggled to streamline processes due to varying or non-existent API formats.
• Poorly designed interfaces: Infrastructure engineers, not being front-end experts, created web portals with non-intuitive interfaces.
• Limited extensibility: Adding features and integrations was difficult and time-consuming, with high coupling and technical constraints hindering progress.
• Duplication of effort: Multiple teams building their portals in slightly different ways led to redundant efforts and inefficiencies.

Modern IDP solutions have addressed these issues by offering intuitive GUIs alongside code-based interfaces such as SDKs, direct API access, and CLI tools, enhancing overall automation efforts.

The shift to modern internal developer portals

The transition to current IDP systems such as Backstage or Port is a natural progression in response to increased complexity, integration needs, and efficiency expectations. These systems collect data and optimize operations, lowering developer friction and eliminating context switching across tools. To fully realize these benefits, it is critical to establish a robust agreement between internal developer portals and platforms.

Understanding developer portal architecture

Internal developer portals are designed to support a wide range of capabilities tailored to your operating model, tech stack, common practices, and team structures. 

Let’s take a look at the most notable areas of IDP architecture at a high level.

The core design principle of internal developer portals

The central element of IDP architecture is self-service, allowing developers to access various internal and external capabilities. They can access the core via a web interface, direct API call, or command line interface (CLI) tooling. It integrates seamlessly with your internal platform, facilitating smooth operations and interactions.

Key components of IDP solutions

Let’s take a glimpse at the components of internal developer portals.

1. Software catalog: A centralized repository storing relevant information about software inventory, team structures, API schemas, and versioning.
2. Blueprints: Templates that serve as starting points for scaffolding applications or infrastructure components, promoting consistency and speed in development.
3. Approval flows: Mechanisms for tracking and approving access to cloud resources or external systems, ensuring transparency and compliance within the organization.
4. Custom plugins: Tools that allow users to extend the portal's capabilities through integrations or custom workflows. These often can be developed using low/no-code approaches or dedicated programming frameworks.
5. Insights and scorecards: Real-time data dashboards providing metrics on service states, compliance status, and other key performance indicators, helping teams stay informed and aligned.

Platform or portal first?

When considering whether to implement an internal developer portal or platform first, it’s essential to understand the distinct roles each system plays and the implications for your platform strategy.

Introducing an internal developer portal first

Starting with an internal developer portal focuses on building the abstraction layer, setting the stage for self-service and automation. This approach provides flexibility in modeling interactions and mitigating future issues related to updates in underlying solutions. 

However, it doesn't address existing problems with unstable managed services or ticket-driven processes. Essentially, while a portal enhances accessibility and ease of use, it doesn’t solve foundational issues within your infrastructure.

Introducing a platform first

Conversely, implementing a new platform requires developers to adapt to a new operating model and migrate their services to fully leverage the platform. Developers can selectively access the processes they need, promoting a smoother transition and better alignment with operational goals.

Comparison between platform and portal first

Let's compare what each approach offers.

Strategic recommendation

In most cases, building a stable, scalable, and efficient platform infrastructure should be the starting point. This approach unlocks essential capabilities that you can later make accessible through an internal developer portal. 

Starting with the platform enables cooperation with developers to identify and remove process bottlenecks, standardize procedures, and build a solid base. Subsequently, you can build a portal on top of this knowledge, reducing the need for constant adjustments and providing a more cost-effective solution.

However, in rare cases where teams share a clear vision and demonstrate strong collaboration, it may be possible to develop both the platform and portal in parallel. This requires careful coordination and a unified approach to ensure both initiatives converge seamlessly.

This nuanced decision warrants input from others closely involved in platform engineering. Consulting with industry peers can provide additional perspectives and help tailor the strategy to your organization's specific needs and circumstances.

Integrating your platform with developer portals

Integrating internal developer portals with platforms aims to address developers' challenges by combining the accessibility of portals with the robustness of platforms. Here are key considerations and common integration patterns to guide this process.

Key considerations for integration

Looked at holistically, all efforts to build platforms or portals aim to solve developers' problems. Let’s go through some aspects of integration.

1. Level of Coupling

• Ensure the portal and platform have a balanced level of coupling. This enhances maintainability, scalability, and the ability to develop and test each component in isolation. A well-coupled system allows seamless interaction while maintaining the independence necessary for updates and scalability.

2. Level of Ownership

• Define clear ownership of services, automation, and interfaces to prevent confusion.  It is imperative that all teams are aware of their roles in order to promote efficiency and responsibility. This transparency facilitates efficient management and troubleshooting.

3. Level of Abstraction

• Determine the appropriate level of abstraction based on your organization's maturity. Mature organizations might benefit from open standards in technology, while less mature or less engineering-focused organizations might need a higher level of abstraction to simplify processes and enhance usability.

Common portal-platform integration patterns

Now let's look at common portal-platform integration patterns from a more technical perspective.

1. Declarative configuration in Git

• This is the most common pattern for custom-built IDP solutions. It is relatively easy to implement and often integrated with CLI tooling for additional validation and automation. This approach is ideal for developers who prefer code integrations over web interfaces, promoting consistency and ease of use.

2. Agent-based integration

• Commonly used in advanced IDP solutions and SaaS products, this pattern provides real-time data access and integrates with multiple third-party systems. Agents can monitor and manage various services, offering a more dynamic and responsive integration approach.

3. Direct API integration

• The traditional method of system integration, this approach is highly flexible but requires more development effort. Direct API integration allows precise control over interactions between the portal and platform, accommodating specific use cases and custom requirements.

Choosing the right pattern

The choice of integration pattern depends on the specific use case and organizational needs, where each pattern offers distinct advantages.

• Declarative configuration in Git: Best for environments where code-based configurations are preferred, ensuring version control and automation.
• Agent-based integration: Suitable for dynamic environments requiring real-time data and extensive third-party integrations.
• Direct API integration: Ideal for custom integrations needing precise control and flexibility.

Holistic integration strategy

Viewing platforms and portals as complementary systems helps in building a cohesive strategy. Portals provide the gateway, while platforms offer the foundation. Successful integration requires careful consideration of coupling, ownership, and abstraction levels, along with the appropriate integration pattern. This combined approach brings significant value, enhancing developer productivity and operational efficiency.

Common pitfalls in building internal developer portals

Creating successful internal developer portals (IDPs) can be challenging. Here are some common pitfalls to avoid:

1. Vertical Approach: Developing multiple portals with limited capabilities often results in inconsistency and hampers end-to-end process automation. Aim for a single portal that offers a comprehensive view, integrating service catalogs, costs, resource management, and blueprints.
2. Static Views: Relying on static data instead of live, automatically collected data across software and resource catalogs limits the portal's usefulness. Ensure the portal allows actions like spawning new microservices and creating new environments to maintain relevance and reduce administrative overhead.
3. Lack of Developer Buy-In: Developing an IDP without input from the developer community can lead to a mismatch of use cases and actual needs. Involve developers early and continuously to ensure the portal addresses their requirements.
4. Limited Contribution: Treating the portal solely as an infrastructure project caps innovation and growth. Encourage shared contributions from various teams to foster continuous improvement and adaptability.
5. Poor Design: Building the portal solely by infrastructure teams without design expertise can result in a non-user-friendly interface. Invest in design skills to create an intuitive and appealing portal.
6. Abandonware: Failing to actively develop and maintain the portal after the initial launch can lead to obsolescence. Ensure a dedicated team is responsible for ongoing development and support.

Conclusion: Aligning platform and portal initiatives

Internal developer portals serve as a springboard for attaining platform engineering objectives, but their implementation is complicated. Organizations frequently experience integration issues between portal and platform activities, resulting in different solutions with overlapping features. Proper alignment and mutual support among these projects are critical to increasing developer autonomy and productivity.

Recognizing the synergy between portals and platforms is essential for realizing their full potential. By avoiding common pitfalls and fostering collaboration, organizations can create robust, user-friendly IDPs that drive platform engineering success. Meeting your organization’s demands and prioritizing features, especially integrations, poses a significant challenge. A dedicated integrations team helps deliver value quickly, but internal custom-built portals often struggle to keep up with evolving developer needs.