Why IoT is changing Enterprise Architecture

[tl;dr The discipline of enterprise architecture as an aid to business strategy execution has failed for most organizations, but is finding a new lease of life in the Internet of Things.]

The strategic benefits of having an enterprise-architecture based approach to organizational change – at least in terms of business models and shared capabilities needed to support those models – have been the subject of much discussion in recent years.

However, enterprise architecture as a practice (as espoused by The Open Group and others) has never managed to break beyond it’s role as an IT-focused endeavor.

In the meantime, less technology-minded folks are beginning to discuss business strategy using terms like ‘modularity’, which is a first step towards bridging the gap between the business folks and the technology folks. And technology-minded folks are looking at disruptive business strategy through the lens of the ‘Internet of Things‘.

Business Model Capability Decomposition

Just like manufacturing-based industries decomposed their supply-chains over the past 30+ years (driving an increasingly modular view of manufacturing), knowledge-based industries are going through a similar transformation.

Fundamentally, knowledge based industries are based on the transfer and management of human knowledge or understanding. So, for example, you pay for something, there is an understanding on both sides that that payment has happened. Technology allows such information to be captured and managed at scale.

But the ‘units’ of knowledge have only slowly been standardized, and knowledge-based organizations are incentivized to ensure they are the only ones to be able to act on the information they have gathered – to often disastrous social and economic consequences (e.g., the financial crisis of 2008).

Hence, regulators are stepping into to ensure that at least some of this ‘knowledge’ is available in a form that allows governments to ensure such situations do not arise again.

In the FinTech world, every service provided by big banks is being attacked by nimble competitors able to take advantage of new, more meaningful technology-enabled means of engaging with customers, and who are willing to make at least some of this information more accessible so that they can participate in a more flexible, dynamic ecosystem.

For these upstart FinTech firms, they often have a stark choice to make in order to succeed. Assuming they have cleared the first hurdle of actually having a product people want, at some point, they must decide whether they are competing directly with the big banks, or if they are providing a key part of the electronic financial knowledge ecosystem that big banks must (eventually) be part of.

In the end, what matters is their approach to data: how they capture it (i.e., ‘UX’), what they do with it, how they manage it, and how it is leveraged for competitive and commercial advantage (without falling foul of privacy laws etc). Much of the rest is noise from businesses trying to get attention in an increasingly crowded space.

Historically, many ‘enterprise architecture’ or strategy departments fail to have impact because firms do not treat data (or information, or knowledge) as an asset, but rather as something to be freely and easily created and shunted around, leaving a trail of complexity and lost opportunity cost wherever it goes. So this attitude must change before ‘enterprise architecture’ as a concept will have a role in boardroom discussions, and firms change how they position IT in their business strategy. (Regulators are certainly driving this for certain sectors like finance and health.)

Internet of Things

Why does the Internet Of Things (IoT) matter, and where does IoT fit into all this?

At one level, IoT presents a large opportunity for firms which see the potential implied by the technologies underpinning IoT; the technology can provide a significant level of convenience and safety to many aspects of a modern, digitally enabled life.

But fundamentally, IoT is about having a large number of autonomous actors collaborating in some way to deliver a particular service, which is of value to some set of interested stakeholders.

But this sounds a lot like what a ‘company’ is. So IoT is, in effect, a company where the actors are technology actors rather than human actors. They need some level of orchestration. They need a common language for communication. And they need laws/protocols that govern what’s permitted and what is not.

If enterprise architecture is all about establishing the functional, data and protocol boundaries between discrete capabilities within an organization, then EA for IoT is the same thing but for technical components, such as sensors or driverless cars, etc.

So IoT seems a much more natural fit for EA thinking than traditional organizations, especially as, unlike departments in traditional ‘human’ companies, technical components like standards: they like fixed protocols, fixed functional boundaries and well-defined data sets. And while the ‘things’ themselves may not be organic, their behavior in such an environment could exhibit ‘organic’ characteristics.

So, IoT and and the benefits of an enterprise architecture-oriented approach to business strategy do seem like a match made in heaven.

The Converged Enterprise

For information-based industries in particular, there appears to be an inevitable convergence: as IoT and the standards, protocols and governance underpinning it mature, so too will the ‘modular’ aspects of existing firms operating models, and the eco-system of technology-enabled platforms will mature along with it. Firms will be challenged to deliver value by picking the most capable components in the eco-system around which to deliver unique service propositions – and the most successful of those solutions will themselves become the basis for future eco-systems (a Darwinian view of software evolution, if you will).

The converged enterprise will consist of a combination of human and technical capabilities collaborating in well-defined ways. Some capabilities will be highly human, others highly technical, some will be in-house, some will be part of a wider platform eco-system.

In such an organization, enterprise architects will find a natural home. In the meantime, enterprise architects must choose their starting point, behavioral or structural: focusing first on decomposed business capabilities and finishing with IoT (behavioral->structural), or focusing first on IoT and finishing with business capabilities (structural->behavioral).

Technical Footnote

I am somewhat intrigued at how the OSGi Alliance has over the years shifted its focus from basic Java applications, to discrete embedded systems, to enterprise systems and now to IoT. OSGi, (disappointingly, IMO), has had a patchy record changing how firms build enterprise software – much of this is to do with a culture of undisciplined dependency management in the software industry which is very, very hard to break.

IoT raises the bar on dependency management: you simply cannot comprehensively test software updates to potentially hundreds of thousands or millions of components running that software. The ability to reliably change modules without forcing a test of all dependent instantiated components is a necessity. As enterprises get more complex and digitally interdependent, standards such as OSGi will become more critical to the plumbing of enabling technologies. But as evidenced above, for folks who have tried and failed to change how enterprises treat their technology-enabled business strategy, it’s a case of FIETIOT – Failed in Enterprise, Trying IoT. And this, indeed, seems a far more rational use of an enterprise architect’s time, as things currently stand.

 

 

 

 

 

 

Advertisements
Why IoT is changing Enterprise Architecture

Strategic Theme #2/5: Enterprise Modularity

[tl;dr Enterprise modularity is the state-of-the-art technologies and techniques which enable agile, cost effective enterprise-scale integration and reuse of capabilities and services. Standards for these are slowly gaining traction in the enterprise.]

Enterprise Modularity is, in essence, Service Oriented Architecture but ‘all the way down’; it has evolved a lot over the last 15-20 years since SOA was first promoted as a concept.

Specifically, SOA historically has been focused on ‘services’ – or, more explicitly, end-points, the contracts they expose, and service discovery. It has been less concerned with the architectures of the systems behind the end-points, which seems logically correct but has practical implications which has limited the widespread adoption of SOA solutions.

SOA is a great idea that has often suffered from epic failures when implemented as part of major transformation initiatives (although less ambitious implementations may have met with more success, but limited to specific domains).

The challenge has been the disconnect between what developers build, what users/customers need, and what the enterprise desires –  not just for the duration of a programme, but over time.

Enterprise modularity needs several factors to be in place before it can succeed at scale. Namely:

  • An enterprise architecture (business context) linked to strategy
  • A focus on data governance
  • Agile cross-disciplinary/cross-team planning and delivery
  • Integration technologies and tools consistent with development practices

Modularity standards and abstractions like OSGi, Resource Oriented Computing and RESTful computing enable the development of loosely coupled, cohesive modules potentially deployable as stand-alone services and which can be orchestrated in innovative ways using   many technologies – especially when linked with canonical enterprise data standards.

The upshot of all this is that traditional ESBs are transforming..technologies like Red Hat Fuse ESB and open-source solutions like Apache ServiceMix provide powerful building blocks that allow ESBs to become first-class applications rather than simply integration hubs. (MuleSoft has not, to my knowledge, adopted an open modularity standard such as OSGi, so I believe it is not appropriate to use as the basis for a general application architecture.)

This approach allows for the eventual construction of ‘domain applications’ which, more than being an integration hub, actually hosts the applications (or, more accurately, application components) in ways that make dependencies explicit.

Vendors such as CA Layer7, Apigee and AxWay are prioritising API management over more traditional, heavy-weight ESB functions – essentially, anyone can build an API internal to their application architecture, but once you want to expose it beyond the application itself, then an API management tool will be required. These can be implemented top-down once the bottom-up API architecture has been developed and validated. Enterprise or integration architecture teams should be driving the adoption and implementation of API management tools, as these (done well) can be non-intrusive with respect to how application development is done, provided application developers take a micro-services led approach to application architecture and design. The concept of ‘dumb pipes’ and ‘smart endpoints are key here, to avoid putting inappropriate complexity in the API management layer or in the (traditional) ESB.

Microservices is a complex topic, as this post describes. But modules (along the lines defined by OSGi) are a great stepping stone, as they embrace microservice thinking without necessarily introducing distributed systems complexity. Innovative technologies like Paremus Service Fabric  building on OSGi standards, help make the transition from monolithic modular architectures to a (distributed) microservice architecture as painless as can be reasonably expected, and can be a very effective way to manage evolution from monolithic to distributed (agile, microservice-based) architectures.

Other solutions, such as Cloud Foundry, offer a means of taking much of the complexity out of building green-field microservices solutions by standardising many of the platform components – however, Cloud Foundry, unlike OSGi, is not a formal open standard, and so carries risks. (It may become a de-facto standard, however, if enough PaaS providers use it as the basis for their offerings.)

The decisions as to which ‘PaaS’ solutions to build/adopt enterprise-wide will be a key consideration for enterprise CTOs in the coming years. It is vital that such decisions are made such that the chosen PaaS solution(s) will directly support enterprise modularity (integration) goals linked to development standards and practices.

For 2015, it will be interesting to see how nascent enterprise standards such as described above evolve in tandem with IaaS and PaaS innovations. Any strategic choices made in these domains must consider the cost/impact of changing those choices: defining architectures that are too heavily dependent on a specific PaaS or IaaS solution limits optionality and is contrary to the principles of (enterprise) modularity.

This suggests that Enterprise modularity standards which are agnostic to specific platform implementation choices must be put in place, and different parts of the enterprise can then choose implementations appropriate for their needs. Hence open standards and abstractions must dominate the conversation rather than specific products or solutions.

Strategic Theme #2/5: Enterprise Modularity

THE FUTURE OF ESBs (AND SOA)

There are some interesting changes happening in technology, which will likely fundamentally change how IT approaches technology like Enterprise Service Buses (ESBs) and concepts like Service Oriented Architecture (SOA).

Specifically, those changes are:

  • An increased focus on data governance, and
  • Microservice technology

Let’s take each in turn, and conclude by suggesting how this will impact how ESBs and SOA will likely evolve.

Data Governance

Historically, IT has an inconsistent record with respect to data governance. For sure, each application often had dedicated data modellers or designers, but its data architecture tended to be very inward focused. Integration initiatives tended to focus on specific projects with specific requirements, and data was governed only to the extent it enabled inidividual project objectives to be achieved.

Sporadic attempts at creating standard message structures and dictionaries crumbled in the face of meeting tight deadlines for critical business deliverables.

ESBs, except in the most stable, controlled environments, failed to deliver the anticipated business benefits because heavy-weight ESBs turned out to be at least as un-agile as the applications they intended to integrate, and since the requirements on the bus evolve continually, application teams tended to favour reliable (or at least predictable) point-to-point solutions over enterprise solutions.

But there are three new drivers for improving data governance across the enterprise, and not just at the application level. These are:

  • Security/Privacy
  • Digital Transformation
  • Regulatory Control

The security/privacy agenda is the most visible, as organisations are extremely exposed to reputational risk if there are security breaches. An organisation needs to know what data it has where, and who has access to it, in order to ensure it can protect it.

Digital transformation means that every process is a digital-first process (or ‘straight-through-processing’ in the parlance of financial services). Human intervention should only be required to handle exceptions. And it means that the capabilities of the entire enterprise need to be brought to bear in order to provide a consistent connected customer experience.

For regulated industries, government regulators are now insisting that firms govern their data throughout that data’s entire lifecycle, not only from a security/privacy compliance perspective, but also from the perspective of being able to properly aggregate and report on regulated data sets.

The same governance principles, policies, processes and standards within an enterprise should underpin all three drivers – hence the increasing focus on establishing the role of ‘chief data officeer’ within organisations, and resourcing that role to materially improve how firms govern their data.

Microservice Technology

Microservice technology is an evolution of modularity in monolithic application design that started with procedures, and evolved through to object-oriented programming, and then to packages/modules (JARs and DLLs etc).

Along the way were attempts to extend the metaphor to distributed systems – e.g., RPC, CORBA, SOA/SOAP, and most recently RESTful APIs – in addition to completely different ‘message-driven’ approachs such as that advocated by the Reactive Development community.

Unfortunately, until fairly recently, most applications behind distributed end-points were architecturally monolithic – i.e., complex applications that needed to go through significant build-test-deploy processes for even minor changes, making it very difficult to adapt these applications in a timely manner to external change factors, such as integrations.

The microservices movement is a reaction to this, where the goal is to be able to deploy microservices as often as needed, without the risk of breaking the entire application (or of having a simple rollback process if it does break). In addition, microservice architectures are inherently amenable to horizontal scaling, a key factor behind its use within internet-scale technology companies.

So, microservices are an architectural style that favours agile, distributed deployment.

As such, one benefit behind the use of microservices is that it allows teams, or individuals within teams, to take responsibility for all aspects of the microservice over its lifetime. In particular, where microservices are exposed to external teams, there is an implied commitment from the team to continue to support those external teams throughout the life of the microservice.

A key aspect of microservices is that they are fairly lightweight: the developer is in control. There is no need for specific heavyweight infrastructure – in fact, microservices favor anti-fragile architectures, with abundant low-cost infrastructure.

Open standards such as OSGi and abstractions such as Resource Oriented Computing allow microservices to participate in a governed, developer-driven context. And in the default (simplest) case, microservices can be exposed using plain-old RESTful standards, which every web application developer is at least somewhat familiar with.

Data Governance + Microservices = Enterprise Building Blocks

Combining the benefits of both data governance and microservices means that firms for the first time can start buiding up a real catalog of enterprise-re-usable building blocks – but without the need for a traditional ESB, or traditional ESB governance. Microservices are developed in response to developer needs (perhaps influenced by Data Governance standards), and Data Standards can be used to describe, in an enterprise context, what those (exposed) microservices do.

Because microservices technologies allow ‘smart endpoints’ to be easily created and integrated into an application architecture, the need for a central ‘bus’ is eliminated. Developers can create many endpoints with limited complexity overhead, and over time can converge these into a small number of common services.

With respect to the Service Registry function provided by ESBs, the new breed of API Management tools may be sufficient to provide any lookup/resolution capabilities required (above and beyond those provided by the microservice architecture itself). API Management tools also keep complexity out of API development by taking care of monitoring, analytics, authentication, protocol conversion and basic throttling capabilities – for those APIs that require those capabilities.

Culturally, however, microservices requires a collaborative approach to software development and evolution, with minimum top-down command-and-control intervention. Data governance, on the other hand, is necessarily driven top-down. So there is a risk of a cultural conflict between top-down data governance and bottom-up microservice delivery: both sides need to be sensitive to the needs of the other side, and be prepared to make compromises occasionally.

In conclusion, the ESB is dead…but long live (m)SOA.

THE FUTURE OF ESBs (AND SOA)

The long journey for legacy application evolution

The CTO of Sungard recently authored this article on Sungard’s technical strategy for its large and diverse portfolio of financial applications:

http://www.bankingtech.com/247292/how-to-build-a-better-sungard/

The technical strategy is shockingly simple on the surface:

  • All products are to provide hosted solutions, and
  • All new products must use HTML5 technology

This (one assumes) reflects the changing nature of how Sungard would like to support its clients – i.e., moving from bespoke shrink-wrapped applications deployed and run in client environments, to providing hosted services that are more accessible and cheaper to deploy and operate.

Executing this strategy will have, I believe, some interesting side effects, as it is quite difficult to meet those to primary objectives without addressing a number of others along the way. For example, the development of key capabilities such as:

  • Elasticity
  • Dev-Ops
  • Mobile
  • Shared Services
  • Messaging/Integration

Elasticity

Simply put, hosted services require elasticity if cost effectiveness is to be achieved. At the very least, the capability to quickly add server capacity to a hosted service is needed, but – especially when multiple hosted services are offered – the ability to reallocate capacity to different services at different times is especially key.

For legacy applications, this can be a challenge to do: most legacy applications are designed to work with a known (ahead of time) configuration of servers, and significant configuration and/or coding work is needed when this changes. There is also a reliance on underlying platforms (e.g., application servers) to themselves be able to move to a truly elastic environment. In the end, some product architectures may be able to do no more than allow products to host a replica of the client’s installation on their data centres, shifting the operational cost from the client to the provider. I suspect that it won’t be long before these products become commercially unviable.

Dev-Ops

Dev-ops is a must-have in a hosted environment – i.e., effective, efficient processes that support robust change from programmer check-in to operational deployment. All SaaS-based firms understand the value of this, and while it’s been talked about a lot in ‘traditional’ organisations, the cultural and organisational changes needed to make it happen are often lagging.

For traditional software vendors in particular, this is a new skill: why worry about dev-ops when all you need to do is publish a new version of your software for your client to configure, test and deploy – probably no more than on a quarterly basis, or even much longer. So a whole new set of capabilities need to be built-up to support a strategy like this.

Mobile

HTML5 is ready-made for mobile. By adopting HTML5 as a standard, application technology stacks will be mobile-ready, across the range of devices, enabling the creation of many products optimised for specific clients or client user groups.

In general, it should not be necessary to develop applications specifically for a particular mobile platform: the kinds of functionality that need to be provided will (in the case of financial services) likely be operational in nature – i.e., workflows or dashboards, etc.

Applications specifically designed to support trading activities may need to be custom-built, but even these should be easier to built on a technology stack that supports HTML5 vs more traditional 2-tier fat client & database architectures.

Traditional 3-tier architectures are obviously easier to migrate to HTML5, but in a large portfolio of (legacy) applications, these may be in a relative minority.

Shared Services

In order to build many more innovative products, new products need to build upon the capabilities of older or existing products – but without creating ‘drag’ (i.e., software complexity) that can slow down and eventually kill the ability to innovate.

The way to do this is with shared services, which all teams recognise and evolve, and who build their products on top of their shared capabilities. Shared services mean better data governance and architectural oversight, to ensure the most appropriate services are identified, and to ensure the design of good APIs and reusable software. Shared services pave the way for applications to consolidate functionality at a portfolio level.

Messaging/Integration

On the face of it, the Sungard strategy doesn’t say anything about messaging or integration: after all, HTML5 is a client GUI technology, and hosted services are (in principle) primarily about infrastructure.

And one could certainly meet those stated objectives without addressing messaging/integration. However, once you start to have hosted services, you can start to interact with clients from a business development perspective on a ‘whole capability’ basis – i.e., offer solutions based on the range of hosted services offered. This will require a degree of interoperability between services predicated primarily on message passing of some sort.

On top of that, with hosted services, connection points into client environments need to be more formally managed: it is untenable (from a cost/efficiency perspective) to have custom interface points for every client, so some standards will be needed.

Messaging/Integration is closely related to shared services (they both are concerned with APIs and message content etc), but are more loosely coupled, as the client should not (to the fullest extent possible) know or care about the technical implementation of the hosted services, and instead focus on message characteristics and behaviour as it relates to their internal platforms.

Conclusion

The two objectives set by Sungard are a reflection of what is both meaningful and achievable for a firm with a portfolio of its size and complexity. Products that choose to adopt the directives to the letter will eventually find they have no strategic future.

Product owners, on the other hand, that understand where the CTO is really trying to get to, will thrive and survive in an environment where legacy financial applications – feature-rich as they are – are at risk of being outed by new internet-based firms that are able to leverage the best of the latest technologies and practices to scale and take market share very quickly from traditional vendors.

Portfolio owners on the buy-side (i.e., financial service firms) should take note. A similar simply stated strategy may be all that is needed to get the ball rolling.

The long journey for legacy application evolution