THE FUTURE OF ESBs (AND SOA)

There are some interesting changes happening in technology, which will likely fundamentally change how IT approaches technology like Enterprise Service Buses (ESBs) and concepts like Service Oriented Architecture (SOA).

Specifically, those changes are:

  • An increased focus on data governance, and
  • Microservice technology

Let’s take each in turn, and conclude by suggesting how this will impact how ESBs and SOA will likely evolve.

Data Governance

Historically, IT has an inconsistent record with respect to data governance. For sure, each application often had dedicated data modellers or designers, but its data architecture tended to be very inward focused. Integration initiatives tended to focus on specific projects with specific requirements, and data was governed only to the extent it enabled inidividual project objectives to be achieved.

Sporadic attempts at creating standard message structures and dictionaries crumbled in the face of meeting tight deadlines for critical business deliverables.

ESBs, except in the most stable, controlled environments, failed to deliver the anticipated business benefits because heavy-weight ESBs turned out to be at least as un-agile as the applications they intended to integrate, and since the requirements on the bus evolve continually, application teams tended to favour reliable (or at least predictable) point-to-point solutions over enterprise solutions.

But there are three new drivers for improving data governance across the enterprise, and not just at the application level. These are:

  • Security/Privacy
  • Digital Transformation
  • Regulatory Control

The security/privacy agenda is the most visible, as organisations are extremely exposed to reputational risk if there are security breaches. An organisation needs to know what data it has where, and who has access to it, in order to ensure it can protect it.

Digital transformation means that every process is a digital-first process (or ‘straight-through-processing’ in the parlance of financial services). Human intervention should only be required to handle exceptions. And it means that the capabilities of the entire enterprise need to be brought to bear in order to provide a consistent connected customer experience.

For regulated industries, government regulators are now insisting that firms govern their data throughout that data’s entire lifecycle, not only from a security/privacy compliance perspective, but also from the perspective of being able to properly aggregate and report on regulated data sets.

The same governance principles, policies, processes and standards within an enterprise should underpin all three drivers – hence the increasing focus on establishing the role of ‘chief data officeer’ within organisations, and resourcing that role to materially improve how firms govern their data.

Microservice Technology

Microservice technology is an evolution of modularity in monolithic application design that started with procedures, and evolved through to object-oriented programming, and then to packages/modules (JARs and DLLs etc).

Along the way were attempts to extend the metaphor to distributed systems – e.g., RPC, CORBA, SOA/SOAP, and most recently RESTful APIs – in addition to completely different ‘message-driven’ approachs such as that advocated by the Reactive Development community.

Unfortunately, until fairly recently, most applications behind distributed end-points were architecturally monolithic – i.e., complex applications that needed to go through significant build-test-deploy processes for even minor changes, making it very difficult to adapt these applications in a timely manner to external change factors, such as integrations.

The microservices movement is a reaction to this, where the goal is to be able to deploy microservices as often as needed, without the risk of breaking the entire application (or of having a simple rollback process if it does break). In addition, microservice architectures are inherently amenable to horizontal scaling, a key factor behind its use within internet-scale technology companies.

So, microservices are an architectural style that favours agile, distributed deployment.

As such, one benefit behind the use of microservices is that it allows teams, or individuals within teams, to take responsibility for all aspects of the microservice over its lifetime. In particular, where microservices are exposed to external teams, there is an implied commitment from the team to continue to support those external teams throughout the life of the microservice.

A key aspect of microservices is that they are fairly lightweight: the developer is in control. There is no need for specific heavyweight infrastructure – in fact, microservices favor anti-fragile architectures, with abundant low-cost infrastructure.

Open standards such as OSGi and abstractions such as Resource Oriented Computing allow microservices to participate in a governed, developer-driven context. And in the default (simplest) case, microservices can be exposed using plain-old RESTful standards, which every web application developer is at least somewhat familiar with.

Data Governance + Microservices = Enterprise Building Blocks

Combining the benefits of both data governance and microservices means that firms for the first time can start buiding up a real catalog of enterprise-re-usable building blocks – but without the need for a traditional ESB, or traditional ESB governance. Microservices are developed in response to developer needs (perhaps influenced by Data Governance standards), and Data Standards can be used to describe, in an enterprise context, what those (exposed) microservices do.

Because microservices technologies allow ‘smart endpoints’ to be easily created and integrated into an application architecture, the need for a central ‘bus’ is eliminated. Developers can create many endpoints with limited complexity overhead, and over time can converge these into a small number of common services.

With respect to the Service Registry function provided by ESBs, the new breed of API Management tools may be sufficient to provide any lookup/resolution capabilities required (above and beyond those provided by the microservice architecture itself). API Management tools also keep complexity out of API development by taking care of monitoring, analytics, authentication, protocol conversion and basic throttling capabilities – for those APIs that require those capabilities.

Culturally, however, microservices requires a collaborative approach to software development and evolution, with minimum top-down command-and-control intervention. Data governance, on the other hand, is necessarily driven top-down. So there is a risk of a cultural conflict between top-down data governance and bottom-up microservice delivery: both sides need to be sensitive to the needs of the other side, and be prepared to make compromises occasionally.

In conclusion, the ESB is dead…but long live (m)SOA.

THE FUTURE OF ESBs (AND SOA)

The role of microservices in Enterprise Architectural planning

Microservices as an architectural construct is gaining a lot of popularity, and the term is slowly but surely finding its place among the lexicon of developers and software architects. The precise definition of ‘microservices’ varies (it is not a formal concept in computer science, as to my knowledge it is not grounded in any formal mathematical abstraction) but the basic concept of a microservice being a reusable lightweight architectural component doing one thing and doing it extremely well is a reasonable baseline.

What do ‘Microservices’ mean to organisations used to understanding their world in terms of Applications? What about all the money some organisations have already spent on Service Oriented Architecture (SOA)? Have all the Services now been supplanted by Microservices? And does all this stuff need to be managed in a traditional command-and-control way, or is it possible they could self-manage?

The answers to these questions are far from reaching a consensus, but here folllows an approach that may help begin to help architects and planners rationalise all these concepts, taken from a pragmatic enterprise perspective.

Note that the definitions below do not focus on technologies so much as scenarios.

Applications

Applications are designed to meet the needs of a specific set of users for a specific set of related tasks or activities. Applications should not be shared in an enterprise sense. This means applications must not expose any re-usable services.

However, application functionality may depend on services exposed by many domains, and applications should use enterprise services to the fullest extent possible. To that extent, applications must focus on orchestrating rather than implementing services and microservices, even though it is generally through the act of building applications that services and microservices are identified and created.

Within a given domain, an Application should make extensive use of the microservices available in that domain.

Applications require a high degree of governance: i.e., for a given user base, the intent to build and maintain an application for a given purpose needs to be explicitly funded – and the domain that that application belongs to must be unambiguous. (But noting that the functionality provided by an application may leverage the capabilities of multiple domains.)

Services

Services represent the exposed interfaces (in a technical sense) of organizational units or functional domains. Services represent how that domain wants information or services to be consumed and/or published with respect to other domains.

Services represent a contract between that domain and other domains, and reflect the commitment the domain has to the efficient operation of the enterprise as a whole. Services may orchestrate other services (cross-domain) or microservices (intra-domain). The relationship between applications and services is asymmetric: if a service has a dependency on an application, then the architecture needs reviewing.

Services require a high degree of governance: i.e., the commitments/obligations that a domain has on other domains (and vice-versa) needs to be formalised and not left to individual projects to work out. It means some form of Service Design methodology, including data governance, is required – most likely in parallel with application development.

Microservices

Within a domain, many microservices may be created to support how that domain builds or deploys functionality. Microservices are optimised for the domain’s performance, and not necessarily for the enterprise, and do not depend upon microservices in other domains.

Applications that are local to a domain may use local microservices directly, but microservices should be invisible to applications or services in other domains.

However, some microservices may be exposed as Services by the domain where there is seen to be value in doing so. This imposes an external obligation on a domain, which are enforced through planning and resourcing.

Microservices require a low degree of governance: the architectural strategy for a domain will define the extent to which microservices form part of the solution architecture for a domain. Microservices developers choose to make their services available and discoverable in ways appropriate to their solution architecture.

In fact, evidence of a successful microservices based approach to application development is an indicator of a healthy technical strategy. Rather than require microservices to be formally planned, the existence of microservices in a given domain should be a hint as to what services could or should be formally exposed by the domain to support cross-divisional goals.

It is quite unlikely that meaningful (resilient, robust, scaleable) Services can be identified and deployed without an underlying microservices strategy. Therefore, the role of a domain architect should be to enable a culture of microservice development, knowing that such a culture will directly support strategic enterprise goals.

Domains

A domain consists of a collection of applications, services and microservices, underpinning a business capability. It is an architectural partition of the organisation, which can be reflected in the firm’s organisational structure. It encapsulates processes, data and technology.

In particular, domains represent business capabilities and are clusters of business and technology – or, equivalently, business function and technology.

The organisational structure as it relates to domains must reflect the strategy of the firm; without this, any attempt at developing a technology strategy will likely fail to achieve the intended outcome.

So, the organisational structure is key, as without appropriate accountability and a forward-looking vision, the above concepts will not usefully survive ad-hoc organisational changes.

While organisational hierarchies may change, what a business actually does will not change very often (except as part of major transformation efforts). And while different individuals may take different responsibilities for different domains (or sets of domains), if an organisation change materially impacts the architecture, then architectural boundaries and roadmaps need to change accordingly so that they re-align with the new organisational goals.

Summary

Partitioning an organisation’s capabilities is a necessary prerequisite to developing a technology strategy. A technology strategy can be expressed in terms of domains, applications, services and (the presence of) microservices.

The key is to not be overly prescriptive, and to allow innovation and creativity within and across domains, but providing sufficient constraints to encourage innovation while simultaneously keeping a lid on complexity. The presence of microservices and a viable microservice strategy within a domain can be used as a good proxy for enterprise technology strategy alignment.

The role of microservices in Enterprise Architectural planning

APIs, more APIs, and .. crypto-currencies

I recently attended #APIconUK, a conference on APIs organised by the folks at Programmable Web. This event was aimed at developers (bottom-up API development) rather than more traditional enterprise customers (top-down SOA-types), and as such was very poorly represented by enterprise-minded folks. Which, IMO, is a bit short-sighted.

The conference was good – a nice mix of hands on, technical and strategic presentations with plenty of time to chat to people between sessions.

Sadly I couldn’t make all the sessions, but key themes I picked up were:

  • Emphasis on treating APIs as a product
  • Computational APIs using Wolfram Alpha’s Mathematica
  • The rise of Linked Data
  • API Security
  • Payment APIs (Stripe)
  • Monetisation of APIs using crypto-currencies
  • Mobile developers & APIs

Adding all these together, what does it really mean?

  • APIs must be treated as first-class citizens with respect to technology management..inventoried/marketed, governed and supported in much the same way applications have been in the past.
  • It is possible to rapidly build useful services using sophisticated but usually not enterprise-standard tools like Mathematica, but to expose the end result as a RESTful API, which the enterprise IT teams can then re-implement in standard technologies. This allows consumers of the API to gain early benefit, and allow enterprise IT time to implement it in a sustainable,medium-term cost effective way.
  • Linked Data – the idea of attaching semantics to API information – is key for Machine Learning. APIs have the opportunity to become dynamic and self-forming when coupled with semantic information via Linked Data standards. Machine Learning is, in turn, key to automate or significantly improve processes that have relied on traditional BI reporting methods.
  • API security is a huge concern, but lots of vendors and solutions are beginning to appear in this space. In essence, it is quick and easy for anyone to create an API (just build a RESTful interface), but API management tools can help control who accesses that API, etc. It also imposes a level of governance to avoid proliferation of dubiously useful APIs.
  • Payment APIs are the next big thing. Stripe demonstrated a live build-it-from-scratch example of how to use their payment APIs, and it is remarkably simple. It doesn’t even need a client side SDK, as you can invoke the API via tools like CURL etc. An excellent example of how a simple API can hide the complexity of the back-end processing that must go on to make it appear simple..
  • Monetisation of APIs could be huge to incentivise the development of reusable services in direct response to demand. This is a significant topic, but essentially the current API model relies on many (internet-based) API providers ultimately making money from advertising, and not from the API itself. Introducing an API currency which is exchanged whenever external APIs are used, or external services use your API, is an intriguing concept, which may be useful even in a ‘traditional’ corporate environment, and would allow service providers to monetise their services directly without having to rely on ‘eyeball’ oriented revenue.
  • Mobile developers today still tend to build the back-end in a way that is highly optimised for the mobile application – in other words, any APIs built (RESTful or otherwise) tend to be designed for the specific needs of the mobile app, and not provided as a general service. Mobile developers have the opportunity to take the lead in ensuring that any mobile app results in the publishing of one or more enterprise-relevant services that other (mobile) app developers could use. Clearly this has an impact on how systems are designed and built, which is likely why the focus on mobile hasn’t produced good enterprise-relevant APIs.

Where does this leave the enterprise CTO? Here’s my take:

1) Mobile apps is where a lot of attention is these days..make sure every mobile project catalogs its data and makes an effort to publish reusable APIs. Do not let every mobile app have its own back-end server implementation, unless it is essentially just an orchestration layer.

2) Encourage developers to build and catalog APIs..even creating a simple, human-discoverable internal catalog like the one Programmable Web has would be a good start.

3) Understand your data from a business perspective, and document it using semantic web standards. This not only helps with IT planning, but is also directly usable in API and machine-learning initiatives, through standards like RDF and JSON-LD. Data architects are the best folks to tackle this.

4) Use API Management tools to help manage and control the creation and use of APIs.

5) Adopt principles like those proposed by the Reactive Development Movement to ensure that architectures behind APIs meet basic architectural standards, so they have some hope of coping with unplanned growth and utilisation.

6) Any app that does Payments should use a well-defined Payments API. It’s beyond the right time to develop internal Payment API standards for apps that have payment processing requirements..the increasing use of alternative currencies will expedite the need for this, as many traditional or legacy systems will not be able to cope with these currencies, so the API should hide this. Plus. it will help manage the transition from current state to target application landscape for payments-related applications. A model-driven approach to payments is key to this, especially as international payments standards take time to evolve.

7) APIs can enable innovative (read: non-standard) use of IT solutions, as long as the innovation can be encapsulated behind standard APIs. This gives the enterprise IT folks time to plan how to support the API, but still allow (limited) business adoption of the API for immediate business value.

With respect to how this all relates to SOA-type enterprise integration strategies, I’m still forming a view. Specifically, the internet itself does not have the concept of a ‘bus’. Someday it possibly will. But this is one space where the innovation still needs to come from enterprises, which still struggle with implementing successful SOA architectures that extend much beyond departmental boundaries.

Finally, the relationship between APIs and crypto-currencies is potentially so huge, that I will cover that another day..

APIs, more APIs, and .. crypto-currencies