Microservice Principles and Enterprise IT Architecture

The idea that the benefits embodied in a microservices approach to solution architecture is relevant to enterprise architecture is a solid one.

In particular, it allows bottom-up, demand-driven solution architectures to evolve, while providing a useful benchmark to assess if those architectures are moving in a way that increases the organization’s ability to manage overall complexity (and hence business agility).

Microservices cannot be mandated top-down the same way Services were intended to be. But fostering a culture and developing an IT strategy that encourages the bottom-up development of microservices will have a significant sustainable positive impact on a business’s competitiveness in a highly digital environment.

I fully endorse all the points Gene has made in his blog post.

follows

Julia Set Fractal

Ruth Malan is fond of noting that “design is fractal”. In a comment on her post “We Just Stopped Talking About Design”, she observed:

We need to get beyond thinking of design as just a do once, up-front sort of thing. If we re-orient to design as something we do at different levels (strategic, system-in-context, system, elements and mechanisms, algorithms, …), at different times (including early), and iteratively and throughout the development and evolution of systems, then we open up the option that we (can and should) design in different media.

This fractal nature is illustrated by the fact that software systems and systems of systems belonging to an organization exist within an ecosystem dominated by that organization which is itself a system of systems of the social kind operating within a larger ecosystem (i.e. the enterprise). Just as structure follows strategy then becomes a constraint on strategy…

View original post 670 more words

Microservice Principles and Enterprise IT Architecture

The role of microservices in Enterprise Architectural planning

Microservices as an architectural construct is gaining a lot of popularity, and the term is slowly but surely finding its place among the lexicon of developers and software architects. The precise definition of ‘microservices’ varies (it is not a formal concept in computer science, as to my knowledge it is not grounded in any formal mathematical abstraction) but the basic concept of a microservice being a reusable lightweight architectural component doing one thing and doing it extremely well is a reasonable baseline.

What do ‘Microservices’ mean to organisations used to understanding their world in terms of Applications? What about all the money some organisations have already spent on Service Oriented Architecture (SOA)? Have all the Services now been supplanted by Microservices? And does all this stuff need to be managed in a traditional command-and-control way, or is it possible they could self-manage?

The answers to these questions are far from reaching a consensus, but here folllows an approach that may help begin to help architects and planners rationalise all these concepts, taken from a pragmatic enterprise perspective.

Note that the definitions below do not focus on technologies so much as scenarios.

Applications

Applications are designed to meet the needs of a specific set of users for a specific set of related tasks or activities. Applications should not be shared in an enterprise sense. This means applications must not expose any re-usable services.

However, application functionality may depend on services exposed by many domains, and applications should use enterprise services to the fullest extent possible. To that extent, applications must focus on orchestrating rather than implementing services and microservices, even though it is generally through the act of building applications that services and microservices are identified and created.

Within a given domain, an Application should make extensive use of the microservices available in that domain.

Applications require a high degree of governance: i.e., for a given user base, the intent to build and maintain an application for a given purpose needs to be explicitly funded – and the domain that that application belongs to must be unambiguous. (But noting that the functionality provided by an application may leverage the capabilities of multiple domains.)

Services

Services represent the exposed interfaces (in a technical sense) of organizational units or functional domains. Services represent how that domain wants information or services to be consumed and/or published with respect to other domains.

Services represent a contract between that domain and other domains, and reflect the commitment the domain has to the efficient operation of the enterprise as a whole. Services may orchestrate other services (cross-domain) or microservices (intra-domain). The relationship between applications and services is asymmetric: if a service has a dependency on an application, then the architecture needs reviewing.

Services require a high degree of governance: i.e., the commitments/obligations that a domain has on other domains (and vice-versa) needs to be formalised and not left to individual projects to work out. It means some form of Service Design methodology, including data governance, is required – most likely in parallel with application development.

Microservices

Within a domain, many microservices may be created to support how that domain builds or deploys functionality. Microservices are optimised for the domain’s performance, and not necessarily for the enterprise, and do not depend upon microservices in other domains.

Applications that are local to a domain may use local microservices directly, but microservices should be invisible to applications or services in other domains.

However, some microservices may be exposed as Services by the domain where there is seen to be value in doing so. This imposes an external obligation on a domain, which are enforced through planning and resourcing.

Microservices require a low degree of governance: the architectural strategy for a domain will define the extent to which microservices form part of the solution architecture for a domain. Microservices developers choose to make their services available and discoverable in ways appropriate to their solution architecture.

In fact, evidence of a successful microservices based approach to application development is an indicator of a healthy technical strategy. Rather than require microservices to be formally planned, the existence of microservices in a given domain should be a hint as to what services could or should be formally exposed by the domain to support cross-divisional goals.

It is quite unlikely that meaningful (resilient, robust, scaleable) Services can be identified and deployed without an underlying microservices strategy. Therefore, the role of a domain architect should be to enable a culture of microservice development, knowing that such a culture will directly support strategic enterprise goals.

Domains

A domain consists of a collection of applications, services and microservices, underpinning a business capability. It is an architectural partition of the organisation, which can be reflected in the firm’s organisational structure. It encapsulates processes, data and technology.

In particular, domains represent business capabilities and are clusters of business and technology – or, equivalently, business function and technology.

The organisational structure as it relates to domains must reflect the strategy of the firm; without this, any attempt at developing a technology strategy will likely fail to achieve the intended outcome.

So, the organisational structure is key, as without appropriate accountability and a forward-looking vision, the above concepts will not usefully survive ad-hoc organisational changes.

While organisational hierarchies may change, what a business actually does will not change very often (except as part of major transformation efforts). And while different individuals may take different responsibilities for different domains (or sets of domains), if an organisation change materially impacts the architecture, then architectural boundaries and roadmaps need to change accordingly so that they re-align with the new organisational goals.

Summary

Partitioning an organisation’s capabilities is a necessary prerequisite to developing a technology strategy. A technology strategy can be expressed in terms of domains, applications, services and (the presence of) microservices.

The key is to not be overly prescriptive, and to allow innovation and creativity within and across domains, but providing sufficient constraints to encourage innovation while simultaneously keeping a lid on complexity. The presence of microservices and a viable microservice strategy within a domain can be used as a good proxy for enterprise technology strategy alignment.

The role of microservices in Enterprise Architectural planning

Mining the gold in the legacy application hills

One recurring challenge that CIOs & CTOs have is talent retention: in the face of continuing and accelerating innovation in technology, and continued lowering of barriers to entry for those who want to get involved in technology, finding and retaining people who are professionally satisfied with looking after so-called ‘legacy’ technology is increasingly difficult.

The most visible impact is inexorable rising cost of change, which CIOs usually deal with in the traditional way: by outsourcing or offshoring the work to locations where similar talent can be found, but cheaper.

But this tends not to solve the problem: rather, the ‘hard’ bits of the legacy applications are ignored in favour of writing new code that the current maintainers can understand. Fear, uncertainty and doubt then creeps in when any of the ‘legacy’ software needs to change. The end result is increased complexity and no sustainable reduction in cost of change.

Only a few years ago, mainframes were considered to be legacy applications. Today, applications written 7+ years ago are the new legacy applications in most organisations, but ostensibly any application whose original architects and maintainers have moved on are now ‘legacy’.

With all the focus on new, exciting technologies, what technical strategy should be adopted to maximise valuable existing functionality, retain and engage development talent, and allow legacy architectures to fit seamlessly and sustainably into new, innovative (cloud/mobile-based) architectures?

Legacy applications have a significant advantage over new applications: they reflect what is effective and useful from a business perspective (even if their implementations do not reflect what is efficient). With the benefit of hindsight, it is possible to look at these applications through a modular lens, and define architectural boundaries of change that can be enforced through technology.

Such an approach can then lead to potential opportunities for decomposing complex monolithic applications, optimising infrastructure usage, deploying components onto anti-fragile infrastructure (elastic fabrics), and finally identifying potentially reusable business services and components that can form building blocks for new applications using innovative technologies.

In particular, it allows complexity to be managed, useful working software to be maximised, and talent to be retained, through providing a route to working with new, in-demand technologies.

In defining a technical strategy supporting a enterprise-wide digital strategy, this is a feasible, cost manageable way to evolve the large portfolio of legacy applications that many large organisations have, both from the bottom up, as well as the top down.

The broad approach is well captured in this presentation here from a global financial services firm, and the overall architectural approach is well described in this book, from 2010.

In the last few years, there have been significant advances in certain enabling technologies (around modularity, APIs, IaaS/PaaS, containers, HTML, etc) and increased focus on enabling architectural disciplines (like business, data, and solution architecture) that are making such a strategy feasible as part of a medium/long-term plan to address IT complexity.

In short, if firms do not have some strategy for dealing with their legacy technology, they will be unable to compete in the increasingly digital economy that is impacting almost every industry out there.

Mining the gold in the legacy application hills

The Reactive Development Movement

A new movement, called ‘Reactive Development’ is taking shape:

http://readwrite.com/2014/09/19/reactive-programming-jonas-boner-typesafe

The rationale behind why creating a new ‘movement’, with its own manifesto, principles and related tools, vendors and evangelists, seems quite reasonable. It leans heavily on the principles of Agile, but extends them into the domain of Architecture.

The core principles of reactive development are around building systems that are:

  • Responsive
  • Resilient
  • Elastic
  • Message Driven

This list seems like a good starting point..it may grow,but these are a solid starting point.

What this movement seems to have (correctly) realised is that building these systems aren’t easy and don’t come for free..so most folks tend to ignore these principles *unless* these are specific requirements of the application to be built. Usually, however, the above characteristics are what you wish your system had when you see your original ‘coherent’ architecture slowly falling apart while you continually accommodate other systems that have a dependency on it (resulting in ‘stovepipe’ architectures, etc’).

IMO, this is the right time for the Reactive Movement. Most mature organisations are already feeling the pain of maintaining and evolving their stovepipe architectures, and many do not have a coherent internal technical strategy to fix this situation (or, if they do, it is often not matched with a corresponding business strategy or enabling organisational design – a subject for another post). And the rapidly growing number of SaaS platforms out there means that these principles are a concern for even the smallest startups, participating as part of a connected set of services delivering value to many businesses.

Most efforts around modularity (or ‘service oriented architectures’) in the end try to deliver on the above principles, but do not bring together all the myriad of skills or capabilities needed to make it successful.

So, I think the Reactive Development effort has the opportunity bring together many architecture skills – namely process/functional (business) architects, data architects, integration architects, application architects and engineers. Today, while the individual disciplines themselves are improving rapidly, the touch-points between them are still, in my view, very weak indeed – and often falls apart completely when it gets to the developers on the ground (especially when they are geographically removed from the architects).

I will be watching this movement with great interest.

The Reactive Development Movement

Why modularity?

Modularity isn’t new: like a number of techniques developed in the 60’s and 70’s, modularity is seeing a renaissance with the advent of cloud computing (i.e., software-defined infrastructure).

Every developer knows the value of modularity – whether it is implemented through objects, packages, services or APIs. But only the most disciplined development teams actually maintain modular integrity over time. This is because it is not obvious initially what modules are needed: usually code evolves, and only when recurring patterns are identified is the need for modules identified. And then usually some refactoring is required, which takes time and costs money.

For project teams intent on meeting a project deadline, refactoring is often seen as a luxury, and so stovepipe development continues, and continues. And, of course, for outsourced bespoke solutions, it’s not necessarily in the interest of the service provider to take the architectural initiative.

In the ‘old-school’ way of enterprise software development, this results in an obvious scenario: a ‘rats nest’ of system integrations and stove-pipe architectures, with a high cost of change, and correspondingly high total cost of ownership – and all the while with diminishing business value (related to the shrinking ability of businesses to adapt quickly to changing market needs).

Enter cloud-computing (and social, big-data, etc, etc). How does this effect the importance of modularity?

Cloud-computing encourages horizontal scaling of technology and infrastructure. Stove-pipe architectures and large numbers of point-to-point integrations do not lend themselves to scalable architectures, as not every component of a large monolithic application can be or should be scaled horizontally, and maintenance costs are high as each component has to be maintained/tested in lock-step with other changes in the application (since the components weren’t designed specifically to work together).

In addition, the demands/requirements of vertical integration encourage monolithic application development – which usually implies architectures which are not horizontally scalable and which are therefore expensive to share in adaptable, agile ways. This gets progressively worse as businesses mature and the business desire for vertical integration conflicts with the horizontal needs of business processes/services which are shared by many business units.

 

The only way, IMHO, to resolve this apparent conflict in needs is via modular architectures – modularity in this sense representing the modular view of the enterprise, which can then be used to define boundaries around which appropriate strategies for resolving the vertical/horizontal conflicts can be resolved.

Modules defined at the enterprise level which map to cloud infrastructures provide the maximum flexibility: the ability to compose modules in a multitude of ways vertically, and to scale them out horizontally as demand/utilisation increases.

Current approaches to enterprise architecture create such maps (often called ‘business component models’), but to date it has been notoriously difficult to translate this into a coherent technical strategy – previous attempts such as SOA provide mixed results, especially where it matters: at enterprise scale.

There are many disciplines that must come together to solve this vertical/horizontal integration paradox, starting with enterprise architecture, but leaning heavily on data architecture, business architecture, and solution architecture – not to mention project management, business analysis, software development, testing, etc.

I don’t (yet!) have the answers to resolving this paradox, but technologies like infrastructure-as-a-service, OSGi, API and API management, big data and the impact mobile has on how IT is commissioned and delivered all have a part to play in how this will play out in the enterprise.

Enterprises that realise the opportunity will have a huge advantage: right now, the small players have the one thing they don’t have: agility. But the small players don’t have what the big players have: lots and lots of data. The commercial survivors will be the big players that have figured out how they can leverage their data while rediscovering agility and innovation. And modularity will be the way to do it.

 

Why modularity?

The vexing question of the value of (micro)services

In an enterprise context, the benefit and value of developing distributed systems vs monolithic systems is, I believe, key to whether large organisations can thrive/survive in the disruptive world of nimble startups nipping at their heels.

The observed architecture of most mature enterprises is a rats nest of integrated systems and platforms, resulting in high total cost of ownership, reduced ability to respond to (business) change, and significant inability to leverage the single most valuable assets of most modern firms: their data.

In this context, what exactly is a ‘rats nest’? In essence, it’s where dependencies between systems are largely unknown and/or ungoverned, the impact of changes between one system and another is unpredictable, and where the cost of rigorously testing such impacts outweigh the economic benefit of the business need requiring the change in the first place.

Are microservices and distributed architectures the answer to addressing this?

Martin Fowler recently published a very insightful view of the challenges of building applications using a microservices-based architecture (essentially where more components are out-of-process than are in-process):

http://martinfowler.com/articles/distributed-objects-microservices.html

The led to a discussion on LinkedIn prompted by this post, questioning the value/benefit of serviced-based architectures:

Fears for Tiers – Do You Need a Service Layer?

The upshot is, there is no agreed solution pattern that addresses the challenges of how to manage the complexity of the inherently distributed nature of enterprise systems, in a cost effective, scalable way. And therein lies the opportunities for enterprise architects.

It seems evident that the existing point-to-point, use-case based approach to developing enterprise systems cannot continue. But what is the alternative? 

These blog posts will delve into this topic a lot more over the coming weeks, with hopefully some useful findings that enterprises can apply with some success.. 

 

The vexing question of the value of (micro)services