Why IoT is changing Enterprise Architecture

[tl;dr The discipline of enterprise architecture as an aid to business strategy execution has failed for most organizations, but is finding a new lease of life in the Internet of Things.]

The strategic benefits of having an enterprise-architecture based approach to organizational change – at least in terms of business models and shared capabilities needed to support those models – have been the subject of much discussion in recent years.

However, enterprise architecture as a practice (as espoused by The Open Group and others) has never managed to break beyond it’s role as an IT-focused endeavor.

In the meantime, less technology-minded folks are beginning to discuss business strategy using terms like ‘modularity’, which is a first step towards bridging the gap between the business folks and the technology folks. And technology-minded folks are looking at disruptive business strategy through the lens of the ‘Internet of Things‘.

Business Model Capability Decomposition

Just like manufacturing-based industries decomposed their supply-chains over the past 30+ years (driving an increasingly modular view of manufacturing), knowledge-based industries are going through a similar transformation.

Fundamentally, knowledge based industries are based on the transfer and management of human knowledge or understanding. So, for example, you pay for something, there is an understanding on both sides that that payment has happened. Technology allows such information to be captured and managed at scale.

But the ‘units’ of knowledge have only slowly been standardized, and knowledge-based organizations are incentivized to ensure they are the only ones to be able to act on the information they have gathered – to often disastrous social and economic consequences (e.g., the financial crisis of 2008).

Hence, regulators are stepping into to ensure that at least some of this ‘knowledge’ is available in a form that allows governments to ensure such situations do not arise again.

In the FinTech world, every service provided by big banks is being attacked by nimble competitors able to take advantage of new, more meaningful technology-enabled means of engaging with customers, and who are willing to make at least some of this information more accessible so that they can participate in a more flexible, dynamic ecosystem.

For these upstart FinTech firms, they often have a stark choice to make in order to succeed. Assuming they have cleared the first hurdle of actually having a product people want, at some point, they must decide whether they are competing directly with the big banks, or if they are providing a key part of the electronic financial knowledge ecosystem that big banks must (eventually) be part of.

In the end, what matters is their approach to data: how they capture it (i.e., ‘UX’), what they do with it, how they manage it, and how it is leveraged for competitive and commercial advantage (without falling foul of privacy laws etc). Much of the rest is noise from businesses trying to get attention in an increasingly crowded space.

Historically, many ‘enterprise architecture’ or strategy departments fail to have impact because firms do not treat data (or information, or knowledge) as an asset, but rather as something to be freely and easily created and shunted around, leaving a trail of complexity and lost opportunity cost wherever it goes. So this attitude must change before ‘enterprise architecture’ as a concept will have a role in boardroom discussions, and firms change how they position IT in their business strategy. (Regulators are certainly driving this for certain sectors like finance and health.)

Internet of Things

Why does the Internet Of Things (IoT) matter, and where does IoT fit into all this?

At one level, IoT presents a large opportunity for firms which see the potential implied by the technologies underpinning IoT; the technology can provide a significant level of convenience and safety to many aspects of a modern, digitally enabled life.

But fundamentally, IoT is about having a large number of autonomous actors collaborating in some way to deliver a particular service, which is of value to some set of interested stakeholders.

But this sounds a lot like what a ‘company’ is. So IoT is, in effect, a company where the actors are technology actors rather than human actors. They need some level of orchestration. They need a common language for communication. And they need laws/protocols that govern what’s permitted and what is not.

If enterprise architecture is all about establishing the functional, data and protocol boundaries between discrete capabilities within an organization, then EA for IoT is the same thing but for technical components, such as sensors or driverless cars, etc.

So IoT seems a much more natural fit for EA thinking than traditional organizations, especially as, unlike departments in traditional ‘human’ companies, technical components like standards: they like fixed protocols, fixed functional boundaries and well-defined data sets. And while the ‘things’ themselves may not be organic, their behavior in such an environment could exhibit ‘organic’ characteristics.

So, IoT and and the benefits of an enterprise architecture-oriented approach to business strategy do seem like a match made in heaven.

The Converged Enterprise

For information-based industries in particular, there appears to be an inevitable convergence: as IoT and the standards, protocols and governance underpinning it mature, so too will the ‘modular’ aspects of existing firms operating models, and the eco-system of technology-enabled platforms will mature along with it. Firms will be challenged to deliver value by picking the most capable components in the eco-system around which to deliver unique service propositions – and the most successful of those solutions will themselves become the basis for future eco-systems (a Darwinian view of software evolution, if you will).

The converged enterprise will consist of a combination of human and technical capabilities collaborating in well-defined ways. Some capabilities will be highly human, others highly technical, some will be in-house, some will be part of a wider platform eco-system.

In such an organization, enterprise architects will find a natural home. In the meantime, enterprise architects must choose their starting point, behavioral or structural: focusing first on decomposed business capabilities and finishing with IoT (behavioral->structural), or focusing first on IoT and finishing with business capabilities (structural->behavioral).

Technical Footnote

I am somewhat intrigued at how the OSGi Alliance has over the years shifted its focus from basic Java applications, to discrete embedded systems, to enterprise systems and now to IoT. OSGi, (disappointingly, IMO), has had a patchy record changing how firms build enterprise software – much of this is to do with a culture of undisciplined dependency management in the software industry which is very, very hard to break.

IoT raises the bar on dependency management: you simply cannot comprehensively test software updates to potentially hundreds of thousands or millions of components running that software. The ability to reliably change modules without forcing a test of all dependent instantiated components is a necessity. As enterprises get more complex and digitally interdependent, standards such as OSGi will become more critical to the plumbing of enabling technologies. But as evidenced above, for folks who have tried and failed to change how enterprises treat their technology-enabled business strategy, it’s a case of FIETIOT – Failed in Enterprise, Trying IoT. And this, indeed, seems a far more rational use of an enterprise architect’s time, as things currently stand.

 

 

 

 

 

 

Why IoT is changing Enterprise Architecture

Know What You Got: Just what, exactly, should be inventoried?

[tl;dr Application inventories linked to team structure, coupled with increased use of meta-data in the software-development process that is linked to architectural standards such as functional, data and business ontologies, is key to achieving long term agility and complexity management.]

Some of the biggest challenges with managing a complex technology platform is knowing when to safely retire or remove redundant components, or when there is a viable reusable component (instantiated or not) that can be used in a solution. (In this sense, a ‘component’ can be a script, a library, or a complete application – or a multitude of variations in-between.)

There are multiple ways of looking at the inventory of components that are deployed. In particular, components can view from the technology perspective, or from the business or usage perspective.

The technology perspective is usually referred to as configuration management, as defined by ITIL. There are many tools (known as ‘CMDB’s) which, using ‘fingerprints’ of known software components, can automatically inventorise which components are deployed where, and their relationships to each other. This is a relatively well-known problem domain – although years of poor deployment practice means that random components can be found running on infrastructure months or even years after the people who created it have moved on. Although the ITIL approach is imminently sensible in principle, in practice it is always playing catch-up because deployment practices are only improving slowly in legacy environments.

Current cloud-aware deployment practices encapsulated in dev-ops are the antithesis of this approach: i.e., all aspects of deployment are encapsulated in script and treated as source code in and of itself. The full benefits of the ITIL approach to configuration management will therefore only be realised when the transition to this new approach to deployment is fully completed (alongside the virtualisation of data centres).

The business or usage perspective is much harder: typically this is approximated by establishing an application inventory, and linking various operational accountabilities to that inventory.

But such inventories suffer from key failings..specifically:

  • The definition of an application is very subjective and generally determined by IT process needs rather than what makes sense from a business perspective.
  • Application inventories are usually closely tied to the allocation of physical hardware and/or software (such as operating systems or databases).
  • Applications tend to evolve and many components could be governed by the same ‘application’ – some of which may be supporting multiple distinct business functions or products.
  • Application inventories tend to be associated with running instances rather than (additionally) instantiable instances.
  • Application inventories may capture interface dependencies, but often do not capture component dependencies – especially when applications may consist of components that are also considered to be a part of other applications.
  • Application and component versioning are often not linked to release and deployment processes.

The consequence is that the application inventory is very difficult to align with technology investment and change, so it is not obvious from a business perspective which components could or should be involved in any given solution, and whether there are potential conflicts which could lead to excessive investment in redundant components, and/or consequent under-investment in potentially reusable components.

Related to this, businesses typically wish to control the changes to ‘their’ application: the thought of the same application being shared with other businesses is usually something only agreed as a last resort and under extreme pressure from central management: the default approach is for each business to be fully in control of the change management process for their applications.

So IT rationally interprets this bias as a license to create a new, duplicate application rather than put in place a structured way to share reusable components, such that technical dependencies can be managed without visibly compromising business-line change independence. This is rational because safe, scalable ways to reuse components is still a very immature capability most organisations do not yet have.

So what makes sense to inventory at the application level? Can it be automated and made discoverable?

In practice, processes which rely on manual maintenance of inventory information that is isolated from the application development process are not likely to succeed – principally because the cost of ensuring data quality will make it prohibitive.

Packaging & build standards (such as Maven, Ant, Gradle, etc) and modularity standards (such as OSGi for Java, Gems for Ruby, ECMAsript for JS, etc) describe software components, their purpose, dependencies and relationships. In particular, disciplined use of software modules may allow applications to self-declare their composition at run-time.

Non-reusable components, such as business-specific (or context-specific) applications, can in principle be packaged and deployed similarly to reusable modules, also with searchable meta-data.

Databases are a special case: these can generally be viewed as reusable, instantiated components – i.e., they may be a component of a number of business applications. The contents of databases should likely best be described through open standards such as RDF etc. However, other components (such as API components serving a defined business purpose) could also be described using these standards, linked to discoverable API inventories.

So, what precisely needs to be manually inventoried? If all technical components are inventoried by the software development process, the only components that remain to be inventoried must be outside the development process.

This article proposes that what exists outside the development process is team structure. Teams are usually formed and broken up in alignment with business needs and priorities. Some teams are in place for many years, some may only last a few months. Regardless, every application component must be associated with at least one team, and teams must be responsible for maintaining/updating the meta-data (in version control) for every component used by that team. Where teams share multiple components, a ‘principle’ team owner must be appointed for each component to ensure consistency of component meta-data, to handle pull-requests etc for shared components, and to oversee releases of those components. Teams also have relevance for operational support processes (e.g., L3 escalation, etc).

The frequency of component updates will be a reflection of development activity: projects which propose to update infrequently changing components can expect to have higher risk than projects whose components frequently change, as infrequently changing components may indicate a lack of current knowledge/expertise in the component.

The ability to discover and reason about existing software (and infrastructure) components is key to managing complexity and maintaining agility. But relying on armies of people to capture data and maintain quality is impractical. Traditional roadmaps (while useful as a communication tool) can deviate from reality in practice, so keeping them current (except for communication of intent) may not be a productive use of resources.

In summary, application inventories linked to team structure, and Increased use of meta-data in the software-development process that is linked to broader architectural standards (such as functional, data and business ontologies) are key to achieving agility and long term complexity management.

Know What You Got: Just what, exactly, should be inventoried?

Strategic Theme #2/5: Enterprise Modularity

[tl;dr Enterprise modularity is the state-of-the-art technologies and techniques which enable agile, cost effective enterprise-scale integration and reuse of capabilities and services. Standards for these are slowly gaining traction in the enterprise.]

Enterprise Modularity is, in essence, Service Oriented Architecture but ‘all the way down’; it has evolved a lot over the last 15-20 years since SOA was first promoted as a concept.

Specifically, SOA historically has been focused on ‘services’ – or, more explicitly, end-points, the contracts they expose, and service discovery. It has been less concerned with the architectures of the systems behind the end-points, which seems logically correct but has practical implications which has limited the widespread adoption of SOA solutions.

SOA is a great idea that has often suffered from epic failures when implemented as part of major transformation initiatives (although less ambitious implementations may have met with more success, but limited to specific domains).

The challenge has been the disconnect between what developers build, what users/customers need, and what the enterprise desires –  not just for the duration of a programme, but over time.

Enterprise modularity needs several factors to be in place before it can succeed at scale. Namely:

  • An enterprise architecture (business context) linked to strategy
  • A focus on data governance
  • Agile cross-disciplinary/cross-team planning and delivery
  • Integration technologies and tools consistent with development practices

Modularity standards and abstractions like OSGi, Resource Oriented Computing and RESTful computing enable the development of loosely coupled, cohesive modules potentially deployable as stand-alone services and which can be orchestrated in innovative ways using   many technologies – especially when linked with canonical enterprise data standards.

The upshot of all this is that traditional ESBs are transforming..technologies like Red Hat Fuse ESB and open-source solutions like Apache ServiceMix provide powerful building blocks that allow ESBs to become first-class applications rather than simply integration hubs. (MuleSoft has not, to my knowledge, adopted an open modularity standard such as OSGi, so I believe it is not appropriate to use as the basis for a general application architecture.)

This approach allows for the eventual construction of ‘domain applications’ which, more than being an integration hub, actually hosts the applications (or, more accurately, application components) in ways that make dependencies explicit.

Vendors such as CA Layer7, Apigee and AxWay are prioritising API management over more traditional, heavy-weight ESB functions – essentially, anyone can build an API internal to their application architecture, but once you want to expose it beyond the application itself, then an API management tool will be required. These can be implemented top-down once the bottom-up API architecture has been developed and validated. Enterprise or integration architecture teams should be driving the adoption and implementation of API management tools, as these (done well) can be non-intrusive with respect to how application development is done, provided application developers take a micro-services led approach to application architecture and design. The concept of ‘dumb pipes’ and ‘smart endpoints are key here, to avoid putting inappropriate complexity in the API management layer or in the (traditional) ESB.

Microservices is a complex topic, as this post describes. But modules (along the lines defined by OSGi) are a great stepping stone, as they embrace microservice thinking without necessarily introducing distributed systems complexity. Innovative technologies like Paremus Service Fabric  building on OSGi standards, help make the transition from monolithic modular architectures to a (distributed) microservice architecture as painless as can be reasonably expected, and can be a very effective way to manage evolution from monolithic to distributed (agile, microservice-based) architectures.

Other solutions, such as Cloud Foundry, offer a means of taking much of the complexity out of building green-field microservices solutions by standardising many of the platform components – however, Cloud Foundry, unlike OSGi, is not a formal open standard, and so carries risks. (It may become a de-facto standard, however, if enough PaaS providers use it as the basis for their offerings.)

The decisions as to which ‘PaaS’ solutions to build/adopt enterprise-wide will be a key consideration for enterprise CTOs in the coming years. It is vital that such decisions are made such that the chosen PaaS solution(s) will directly support enterprise modularity (integration) goals linked to development standards and practices.

For 2015, it will be interesting to see how nascent enterprise standards such as described above evolve in tandem with IaaS and PaaS innovations. Any strategic choices made in these domains must consider the cost/impact of changing those choices: defining architectures that are too heavily dependent on a specific PaaS or IaaS solution limits optionality and is contrary to the principles of (enterprise) modularity.

This suggests that Enterprise modularity standards which are agnostic to specific platform implementation choices must be put in place, and different parts of the enterprise can then choose implementations appropriate for their needs. Hence open standards and abstractions must dominate the conversation rather than specific products or solutions.

Strategic Theme #2/5: Enterprise Modularity

Reactive Development vs Legacy Applications

To follow up my earlier post on the Reactive Development movement, I went to a meet-up explaining the rationale behind the concepts and hopefully to explain why it deserves its own ‘manifesto’ – particularly when compared with the architectures of many legacy applications owned by large/mature organisations, which did not specifically apply such principles.

To recap, Reactive Development is focused around designing and building applications conforming with the following 4 architectural principles:

  • Responsive
  • Resilient
  • Message driven
  • Elastic

When a new phrase is coined in technology, there are usually commercial interests behind it, and ‘reactive development’ is no different: the hosts of this particular event (Typesafe) are front-and-centre of the so-called Reactive-software movement and the Reactive Manifesto.

But, leaving that aside, is there anything new here? Why should architects care, haven’t they been dealing with these challenges all along?

Below is my assessment of how ‘legacy’ applications stack up against the Reactive Manifesto; this gives a hint as to the challenge (and opportunity) facing organisations faced with replacing their legacy architectures with more modern architectures. (In this case, ‘legacy’ means applications built over the past 10-15 years, not the mainframe dinosaurs..).

Responsive

Historically, responsiveness was primarily determined from a user perspective: since many applications were built to be delivered via HTTP (HTML, Javascript, etc), this mostly concerned application server architectures (and, indirectly, database responsiveness). Application server architectures are optimised for user-led use cases – i.e., to build responsive GUIs supporting many concurrent users. The case for using complex application servers (i.e., middleware such as WebLogic, WebSphere, TomCat, etc) has historically been harder to make for non-GUI-oriented application development (i.e., where user session management is not a concern). In particular, these architectures are generally assuming that the ‘users’ are human, and not other systems, which has a significant impact on how such systems would be architected.

In summary, application responsiveness – excluding databases – has been driven primarily by GUI requirements, resulting in application architectures optimised for user use-cases which only need to meet ‘good enough’ performance criteria. This can be expensive to fix when ‘good enough’ is no longer good enough.

Resilient

In ‘traditional’ architectures, resilience has been provided via techniques such as clustering, replication or redundant fail-over capacity. Such approaches require static infrastructure configurations, and the middleware software underpinning applications must be explicitly designed to support such configurations; this usually means proprietary enterprise-strength vendor solutions. Such solutions in turn have a heavy bias towards expensive infrastructure with a low MTBF (mean-time-between-failure), as recovering from failure is still an expensive and labour-intensive process – even if the application continues to operate at a degraded level through the failure. Conversely, an approach focused on achieving low mean-time-to-recovery (MTTR) would acknowledge infrastructure failure as likely to happen, and seek to minimise the time taken to get failed processes up and running again.

In summary, application resilience has historically relied on low MTBF rather than on low MTTR, and consequently resilience comes at high infrastructure and support cost. 

Message Driven

Message-driven, in the context of Reactive Development, means non-blocking and location-transparent. Publish/subscribe messaging architectures conform to that definition, and certainly these exist today. But most architectures are not message driven: the use of RESTful APIs, for example, does not by default conform to this principle, as REST is synchronous and can be location-specific. However, it is certainly possible to build message-driven architectures on top of REST.

The real point of the message-driven principle is to enable the other reactive principles of resilience, elasticity and  responsiveness. It means that even if you have point-to-point messages, as long as they conform to Reactive Principles, you will get the Reactive Manifesto benefits of “load management, elasticity, and flow control” – benefits that are not easy to obtain with legacy static point-to-point feed-driven architectures.

In most enterprise environments, good pub/sub solutions have existed for some time; however, without requiring application designers to conform to the ‘message-driven’ principle, when you want to get data from System X to System Y, why use the overhead of a (relatively expensive) enterprise messaging system when all that is needed is a simple file extract?

In summary, the lack of an historical message-driven focus for system design has resulted in a proliferation of complex and expensive-to-maintain point-to-point system dependencies, despite the existence of enabling technologies.

Elastic

Traditional architectures are not elastic. By elastic, we mean the ability to scale down as well as up. Legacy architectures are static: it can often take months to introduce new infrastructure into a production environment for a ‘legacy’ application.

The best ‘traditional’ architectures use load balancing and allow the easy addition of new servers into a farm (such as a web server farm or compute grid). This enables scaling up, but doesn’t particularly support scaling down: i.e., taking servers out of the farm and making them available for other applications is still a very manual, labour intensive exercise – and is definitely not something that can be done many times during the day, as demand and load varies.

In summary: legacy architectures may at best be able to scale up, but they cannot easily scale down, resulting in wasted capacity and unnecessarily high infrastructure costs.

Conclusion

My main take-away from the Reactive Development  meet-up is really about how distributed computing has advanced over the last 20 years, or, more specifically, how distributed computing concepts are becoming key skills that your ‘average’ developer needs to be able to grasp to cope with ‘modern’ application development.

Distributed computing in the enterprise is hard: you just have to look at CORBA, SOA and ESBs to see how history has treated ambitious attempts at distributed computing. Distributed technologies such as

  • Distributed shared mutable state,
  • Serialisable distributed transactions,
  • Synchronous RPC, and
  • Distributed Objects

have all failed to gain mass traction because they require specialist knowledge, which usually implies vendor lock-in when deployed at enterprise scale – which equates to long-term expense.

By applying Reactive principles, and using technologies that can specifically enable them, it should be possible to build enterprise-scale applications at significantly lower long-term cost and with increasing rather than decreasing business value.

Note: I have no view as yet as to whether vendors rallying behind the Reactive Manifesto are on the right track with their solutions.  In my opinion, there is still too much emphasis on the run-time aspects of Reactive Development, and not enough on change planning and system design, which is not something any vendor can solve for an organisation, but which the application of Reactive Principles can have a very positive effect on.

Reactive Development vs Legacy Applications

Why modularity?

Modularity isn’t new: like a number of techniques developed in the 60’s and 70’s, modularity is seeing a renaissance with the advent of cloud computing (i.e., software-defined infrastructure).

Every developer knows the value of modularity – whether it is implemented through objects, packages, services or APIs. But only the most disciplined development teams actually maintain modular integrity over time. This is because it is not obvious initially what modules are needed: usually code evolves, and only when recurring patterns are identified is the need for modules identified. And then usually some refactoring is required, which takes time and costs money.

For project teams intent on meeting a project deadline, refactoring is often seen as a luxury, and so stovepipe development continues, and continues. And, of course, for outsourced bespoke solutions, it’s not necessarily in the interest of the service provider to take the architectural initiative.

In the ‘old-school’ way of enterprise software development, this results in an obvious scenario: a ‘rats nest’ of system integrations and stove-pipe architectures, with a high cost of change, and correspondingly high total cost of ownership – and all the while with diminishing business value (related to the shrinking ability of businesses to adapt quickly to changing market needs).

Enter cloud-computing (and social, big-data, etc, etc). How does this effect the importance of modularity?

Cloud-computing encourages horizontal scaling of technology and infrastructure. Stove-pipe architectures and large numbers of point-to-point integrations do not lend themselves to scalable architectures, as not every component of a large monolithic application can be or should be scaled horizontally, and maintenance costs are high as each component has to be maintained/tested in lock-step with other changes in the application (since the components weren’t designed specifically to work together).

In addition, the demands/requirements of vertical integration encourage monolithic application development – which usually implies architectures which are not horizontally scalable and which are therefore expensive to share in adaptable, agile ways. This gets progressively worse as businesses mature and the business desire for vertical integration conflicts with the horizontal needs of business processes/services which are shared by many business units.

 

The only way, IMHO, to resolve this apparent conflict in needs is via modular architectures – modularity in this sense representing the modular view of the enterprise, which can then be used to define boundaries around which appropriate strategies for resolving the vertical/horizontal conflicts can be resolved.

Modules defined at the enterprise level which map to cloud infrastructures provide the maximum flexibility: the ability to compose modules in a multitude of ways vertically, and to scale them out horizontally as demand/utilisation increases.

Current approaches to enterprise architecture create such maps (often called ‘business component models’), but to date it has been notoriously difficult to translate this into a coherent technical strategy – previous attempts such as SOA provide mixed results, especially where it matters: at enterprise scale.

There are many disciplines that must come together to solve this vertical/horizontal integration paradox, starting with enterprise architecture, but leaning heavily on data architecture, business architecture, and solution architecture – not to mention project management, business analysis, software development, testing, etc.

I don’t (yet!) have the answers to resolving this paradox, but technologies like infrastructure-as-a-service, OSGi, API and API management, big data and the impact mobile has on how IT is commissioned and delivered all have a part to play in how this will play out in the enterprise.

Enterprises that realise the opportunity will have a huge advantage: right now, the small players have the one thing they don’t have: agility. But the small players don’t have what the big players have: lots and lots of data. The commercial survivors will be the big players that have figured out how they can leverage their data while rediscovering agility and innovation. And modularity will be the way to do it.

 

Why modularity?