Transforming IT: From a solution-driven model to a capability-driven model

[tl;dr Moving from a solution-oriented to a capability-oriented model for software development is necessary to enable enterprises to achieve agility, but has substantial impacts on how enterprises organise themselves to support this transition.]

Most organisations which manage software change as part of their overall change portfolio take a project-oriented approach to delivery: the project goals are set up front, and a solution architecture and delivery plan are created in order to achieve the project goals.

Most organisations also fix project portfolios on a yearly basis, and deviating from this plan can often very difficult for organisations to cope with – at least partly because such plans are intrinsically tied into financial planning and cost-saving techniques such as capitalisation of expenses, etc, which reduce bottom-line cost to the firm of the investment (even if it says nothing about the value added).

As the portfolio of change projects rise every year, due to many extraneous factors (business opportunities, revenue protection, regulatory demand, maintenance, exploration, digital initiatives,  etc), cross-project dependency management becomes increasingly difficult. It becomes even more complex to manage solution architecture dependencies within that overall dependency framework.

What results is a massive set of compromises that ends up with building solutions that are sub-optimal for pretty much every project, and an investment in technology that is so enterprise-specific, that no other organisation could possibly derive any significant value from it.

While it is possible that even that sub-optimal technology can yield significant value to the organisation as a whole, this benefit may be short lived, as the cost-effective ability to change the architecture must inevitably decrease over time, reducing agility and therefore the ability to compete.

So a balance needs to be struck, between delivering enterprise value (even at the expense of individual projects) while maintaining relative technical and business agility. By relative I mean relative to peers in the same competitive sector…sectors which are themselves being disrupted by innovative technology firms which are very specialist and agile within their domain.

The concept of ‘capabilities’ realised through technology ‘products’, in addition to the traditional project/program management approach, is key to this. In particular, it recognises the following key trends:

  • Infrastructure- and platform-as-a-service
  • Increasingly tech-savvy work-force
  • Increasing controls on IT by regulators, auditors, etc
  • Closer integration of business functions led by ‘digital’ initiatives
  • The replacement of the desktop by mobile & IoT (Internet of Things)
  • The tension between innovation and standards in large organisations

Enterprises are adapting to all the above by recognising that the IT function cannot be responsible for both technical delivery and ensuring that all technology-dependent initiatives realise the value they were intended to realise.

As a result, many aspects of IT project and programme management are no longer driven out of the ‘core’ IT function, but by domain-specific change management functions. IT itself must consolidate its activities to focus on those activities that can only be performed by highly qualified and expert technologists.

The inevitable consequence of this transformation is that IT becomes more product driven, where a given product may support many projects. As such, IT needs to be clear on how to govern change for that product, to lead it in a direction that is most appropriate for the enterprise as a whole, and not just for any particular project or business line.

A product must provide capabilities to the stakeholders or users of that product. In the past, those capabilities were entirely decided by whatever IT built and delivered: if IT delivered something that in practice wasn’t entirely fit for purpose, then business functions had no alternative but to find ways to work around the system deficiencies – usually creating more complexity (through end-user-developed applications in tools like Excel etc) and more expense (through having to hire more people).

By taking a capability-based approach to product development, however, IT can give business functions more options and ways to work around inevitable IT shortfalls without compromising controls or data integrity – e.g., through controlled APIs and services, etc.

So, while solutions may explode in number and complexity, the number of products can be controlled – with individual businesses being more directly accountable for the complexity they create, rather than ‘IT’.

This approach requires a step-change in how traditional IT organisations manage change. Techniques from enterprise architecture, scaled agile, and DevOps are all key enablers for this new model of structuring the IT organisation.

In particular, except for product-strategy (where IT must be the leader), IT must get out of the business of deciding the relative value/importance of individual product changes requested by projects, which historically IT has been required to do. By imposing a governance structure to control the ‘epics’ and ‘stories’ that drive product evolution, projects and stakeholders have some transparency into when the work they need will be done, and demand can be balanced fairly across stakeholders in accordance with their ability to pay.

If changes implemented by IT do not end up delivering value, it should not be because IT delivered the wrong thing, but rather the right thing was delivered for the wrong reason. As long as IT maintains its product roadmap and vision, such mis-steps can be tolerated. But they cannot be tolerated if every change weakens the ability of the product platform to change.

Firms which successfully balance between the project and product view of their technology landscape will find that productivity increases, complexity is reduced and agility increases massively. This model also lends itself nicely to bounded domain development, microservices, use of container technologies and automated build/deployment – all of which will likely feature strongly in the enterprise technology platform of the future.

The changes required to support this are significant..in terms of financial governance, delivery oversight, team collaborations, and the roles of senior managers and leaders. But organisations must be prepared to do this transition, as historical approaches to enterprise IT software development are clearly unsustainable.

Transforming IT: From a solution-driven model to a capability-driven model

The role of microservices in Enterprise Architectural planning

Microservices as an architectural construct is gaining a lot of popularity, and the term is slowly but surely finding its place among the lexicon of developers and software architects. The precise definition of ‘microservices’ varies (it is not a formal concept in computer science, as to my knowledge it is not grounded in any formal mathematical abstraction) but the basic concept of a microservice being a reusable lightweight architectural component doing one thing and doing it extremely well is a reasonable baseline.

What do ‘Microservices’ mean to organisations used to understanding their world in terms of Applications? What about all the money some organisations have already spent on Service Oriented Architecture (SOA)? Have all the Services now been supplanted by Microservices? And does all this stuff need to be managed in a traditional command-and-control way, or is it possible they could self-manage?

The answers to these questions are far from reaching a consensus, but here folllows an approach that may help begin to help architects and planners rationalise all these concepts, taken from a pragmatic enterprise perspective.

Note that the definitions below do not focus on technologies so much as scenarios.

Applications

Applications are designed to meet the needs of a specific set of users for a specific set of related tasks or activities. Applications should not be shared in an enterprise sense. This means applications must not expose any re-usable services.

However, application functionality may depend on services exposed by many domains, and applications should use enterprise services to the fullest extent possible. To that extent, applications must focus on orchestrating rather than implementing services and microservices, even though it is generally through the act of building applications that services and microservices are identified and created.

Within a given domain, an Application should make extensive use of the microservices available in that domain.

Applications require a high degree of governance: i.e., for a given user base, the intent to build and maintain an application for a given purpose needs to be explicitly funded – and the domain that that application belongs to must be unambiguous. (But noting that the functionality provided by an application may leverage the capabilities of multiple domains.)

Services

Services represent the exposed interfaces (in a technical sense) of organizational units or functional domains. Services represent how that domain wants information or services to be consumed and/or published with respect to other domains.

Services represent a contract between that domain and other domains, and reflect the commitment the domain has to the efficient operation of the enterprise as a whole. Services may orchestrate other services (cross-domain) or microservices (intra-domain). The relationship between applications and services is asymmetric: if a service has a dependency on an application, then the architecture needs reviewing.

Services require a high degree of governance: i.e., the commitments/obligations that a domain has on other domains (and vice-versa) needs to be formalised and not left to individual projects to work out. It means some form of Service Design methodology, including data governance, is required – most likely in parallel with application development.

Microservices

Within a domain, many microservices may be created to support how that domain builds or deploys functionality. Microservices are optimised for the domain’s performance, and not necessarily for the enterprise, and do not depend upon microservices in other domains.

Applications that are local to a domain may use local microservices directly, but microservices should be invisible to applications or services in other domains.

However, some microservices may be exposed as Services by the domain where there is seen to be value in doing so. This imposes an external obligation on a domain, which are enforced through planning and resourcing.

Microservices require a low degree of governance: the architectural strategy for a domain will define the extent to which microservices form part of the solution architecture for a domain. Microservices developers choose to make their services available and discoverable in ways appropriate to their solution architecture.

In fact, evidence of a successful microservices based approach to application development is an indicator of a healthy technical strategy. Rather than require microservices to be formally planned, the existence of microservices in a given domain should be a hint as to what services could or should be formally exposed by the domain to support cross-divisional goals.

It is quite unlikely that meaningful (resilient, robust, scaleable) Services can be identified and deployed without an underlying microservices strategy. Therefore, the role of a domain architect should be to enable a culture of microservice development, knowing that such a culture will directly support strategic enterprise goals.

Domains

A domain consists of a collection of applications, services and microservices, underpinning a business capability. It is an architectural partition of the organisation, which can be reflected in the firm’s organisational structure. It encapsulates processes, data and technology.

In particular, domains represent business capabilities and are clusters of business and technology – or, equivalently, business function and technology.

The organisational structure as it relates to domains must reflect the strategy of the firm; without this, any attempt at developing a technology strategy will likely fail to achieve the intended outcome.

So, the organisational structure is key, as without appropriate accountability and a forward-looking vision, the above concepts will not usefully survive ad-hoc organisational changes.

While organisational hierarchies may change, what a business actually does will not change very often (except as part of major transformation efforts). And while different individuals may take different responsibilities for different domains (or sets of domains), if an organisation change materially impacts the architecture, then architectural boundaries and roadmaps need to change accordingly so that they re-align with the new organisational goals.

Summary

Partitioning an organisation’s capabilities is a necessary prerequisite to developing a technology strategy. A technology strategy can be expressed in terms of domains, applications, services and (the presence of) microservices.

The key is to not be overly prescriptive, and to allow innovation and creativity within and across domains, but providing sufficient constraints to encourage innovation while simultaneously keeping a lid on complexity. The presence of microservices and a viable microservice strategy within a domain can be used as a good proxy for enterprise technology strategy alignment.

The role of microservices in Enterprise Architectural planning