The Importance of Principles in Managing Complexity

[tl;dr The first step to managing complexity is to define what is important. Principles are a great way to reach consensus on what is important, and establishing governance around adherence to those principles are the first step to getting on top of change-related complexity.]

Managing complexity is not easy, whether you are in a large organisation, or in a relatively green-field environment where growth is rapid and complexity is a far-off problem you hope to have some day…

The fact is, if your organisation suffers from a complexity problem (in technology, and implicitly, processes and data), and you are still in (a profitable) business, then it is that complexity which, for better or for worse, has been the instrument of that organisation’s success. As a consequence, it is quite hard to convince people to implement complexity management (read ‘enterprise architecture’) before complexity actually becomes a problem.

Sometimes, however, unmanaged complexity brings an organisation’s growth to a grinding halt, in the worst cases leading to eventual shutdown. (In this context, an organisation can be an independent business or a business unit within a larger company.)

Historically, investment in IT incurred large capital costs, so usually there was a large focus on realising value from that investment – i.e. to exploit it to the maximum. Exploiting technology usually means adding on more and more functionality to existing systems in ways that meet business imperatives.  But business imperatives tend to be fairly short-term in nature. Without guiding principles, the act of exploiting technology leads to ever more expensive change-related costs.

For any self-respecting IT organisation, a core set of principles, actively governed, is an absolute necessity. Principles transcend solution architectures and technologies, and enable project autonomy as decisions can be made locally without requiring ‘central’ approval, provided principles are being adhered to.

Such principles, call them ‘implementation principles’ should guide how IT organisations do their work and manage change.

But many principles that stay only within the IT organisation may break in the face of over-whelming business imperatives: on the basis that the (immediate) business needs comes first, IT often takes it on the chin and accumulates technical debt.

So it is important to agree principles that business stakeholders agree to and supports, and for which they are willing to have an open discussion about in the context of delaying projects, increasing costs, prioritising change, or driving organisational change.

Let’s call these principles ‘value principles’ – they are focused on how the business can realise value from its investments, particularly in IT.

In order to understand what this means in practice, it may be worth returning to the ‘Zero CapEx IT’ concept introduced in my last post. What if all IT was sourced through (virtual or actual) 3rd parties, so that all of IT was delivered as software-as-a-service?

In this case, the ‘implementation principles’ would be whatever guidelines or standards were used by the SaaS provider to ensure they could deliver change reliably, and maintain service according to agreed levels. The SaaS provider has a vested interest in the client getting value out of their service, but the only levers they have to ensure that is their ability to react in a timely manner to change requests.

From the business’ perspective, adopting an IT (SaaS) service requires structural change to ensure the business can take advantage of the technology. This may mean assigning roles and responsibilities, defining new activities, maybe creating whole new departments, etc. The business may be paying for the service, but unless it pro-actively maximises what the service can offer in terms of what the business needs, it may not get the value. The SaaS (i.e., IT) can do little to address this situation, and may end up losing the client if the client does not take action.

On the other hand, if a client is pro-actively engaged, then they will likely want regular changes so they can continually optimise the value of the service to their operations. This puts the onus on the provider to remain agile.

In a corporate environment, where the accountability between value and implementation is often blurred, there is usually little optionality with respect to who your provider is – or, from a provider perspective, who your client is. In fact, the client/provider perspective in a corporate environment can be damaging, as it can reinforce an us-vs-them culture, leading to mistrust and an ‘order-taking’ mentality developing between the client and the provider.

In this environment, complexity will thrive because, after all, the client is tied and therefore has to keep paying for their service. There is little incentive to be agile..if the client wants more quicker, they have to pay for it.

If, on the other hand, the client retains an active, supportive interest in how their IT is delivered, as well as what is delivered, then they will get the IT they want.

If the client is not willing to agree certain base-line principles with their (IT) provider, and to put in place structures to enforce them (including a formal waiver process), then one can expect little to improve.

On the other hand, if the client or business partner will willingly participate in establishing principles and related governance forums, this will set the tone for how the business/IT relationship will evolve going forward.

At the very least, ‘value principles’ (such as, for example, “The business will commit to implementing the necessary structural changes to maximise IT investment”) should be agreed, and the consequence/implications of these to project planning should be understood.

Note that value principles are increasingly including concepts around data governance, which historically have been an ‘implementation’ principle – for example, the principle that ‘information is an asset’.

‘Implementation principles’  are principles which determine how (IT) change is decided and implemented, and how complexity is managed, by the provider. Where the business and technology are inter-twined (i.e., the business value proposition *is* the technology), these would simply form part of the business ‘value principles’. But where the business is agnostic as to the technology, and are only focused on value, then the ‘implementation principles’ are solely for the provider to define and enforce. An example of an ‘implementation’ principle may be ‘Control of technical diversity’, with consequent implications for solution architectures.

Adopting principles means that an organisation has some sense of what it wants to be, even if only aspirationally. It allows individuals or teams to feel comfortable making decisions on their own without feeling they have to get approval, which is key for agile cultures. Establishing formal business+IT forums to review principle waivers helps provide transparency with respect to project risk and platform risk. And it allows Technical Debt to be managed, as every waiver is a recognition of technical debt incurred.

In conclusion, managing complexity is hard. But Principles can make it manageable.

The Importance of Principles in Managing Complexity

The CDO: Enterprise Architect’s best friend

[tl;dr The CDO’s agenda may be the best way to bring the Enterprise Architecture agenda to the business end of the C-suite table.]

Enterprise Architecture as a concept has for some time claimed intellectual ownership of a holistic, integrated view of architecture within an organisation. In that lofty position, enterprise architects have put a stake in the ground with respect to key architecture activities, and formalised them all within frameworks such as TOGAF.

In TOGAF, Data Architecture has been buried within the Information Systems Architecture phase of the Architecture Development Method, along with Application Architecture.

But the real world often gets in the way of (arguably) conceptually clean thinking: I note the rise of the CDO, or Chief Data Officer, usually as part of the COO function of an organisation. (This CDO is not to be confused with the Chief Digital Officer, who’s primary focus could critically be seen as to build cool apps…)

So, why has Data gotten all this attention, while Enterprise Architecture still, for the most part, languishes within IT?

Well, as with most things, its happening for all the wrong reasons: mostly its because regulators are holding businesses accountable for how they manage regulated data, and businesses need, in turn, to hold people to account for doing that. Hence the need for a CDO.

But in the act of trying to understand what data they are liable for managing, and to what extent, CDO’s necessarily have to do Data Architecture and Data Governance. And at this point the CDO’s activities starts dipping into new areas such as business process management, and business & IT strategy – and EA.

If CDOs choose to take an expansive view of their role (and I believe they should), then it would most definitely include all the domains of Enterprise Architecture – especially if one views data complexity and technology complexity as two sides of the same enterprise complexity coin: one as a consequence of business decisions, the other as a consequence of IT decisions.

The good news is that this would at long last put EAs into a role more aligned with business goals than with the goals of the IT organisation. This is not to say that the goals of the IT organisation are in some way wrong, but rather than because the business had abdicated its responsibility for many decisions that IT needs in order to do “the right thing”, IT has had to fill the gaps itself – and in ways which said abdicating businesses could understand/appreciate.

For anyone in the position of CTO, this should help a lot: without prescribing which technologies should be used, businesses then can give real context to their demand which the CTO can interpret strategically, and, in collaboration with his CDO colleague, agree a technology strategy that can deliver on those needs.

In this way, the CDO/CTO have a very symbiotic relationship: the boundaries should be fuzzy, although the responsibilities should be clear: the CDO provides the context, the CTO delivers the technology.

So collaboration is key: while in principle one should be able to describe what a business needs strictly in terms of process and data flows etc, the reality is that business needs and practices change in *response* to technology. In other words, it is not a one-way street. What technology is *capable* of doing can shape what the business *should* be doing. (In many cases, for example, technology may eliminate the need for entire processes or activities.)

Folks on the edge of this CDO/CTO collaboration have a choice: do I become predominantly business facing (CDO), or do I become predominantly technology facing (CTO)? Folks may choose to specialise, or some, like myself, may choose to focus on different perspectives according to the needs of the organisation.

One interesting impact from this approach may  address one of the biggest questions out there at the moment: how to absorb the services and capabilities offered by innovative, efficient, and nimble external businesses into the architecture of the enterprise, while retaining control, compliance and agility of the data and processes? But more on this later.

So, where does all this sit with respect to the ‘3 pillars of a digital strategy’? The points raised there are still valid. With the right collaboration and priorities, it is possible to have different folks fill the roles of ‘client centricity’, ‘chief data officer’ and ‘head of enterprise complexity’. But for the most part, all these roles begin and end with data. A proper, disciplined and thoughtful approach to how and why data is captured, processed, stored, retrieved and transmitted should properly address firm-wide themes such as improving client experience, integrating innovative services, keeping a lid on complexity and retaining an appropriate level of enterprise agility.

Some folks may be wondering where the CIO fits in all this: the CIO maintains responsibility for the management and operation of all IT processes (as defined in The Open Group’s IT4IT model). This is a big job. But it is not the job of the CTO, whose focus is to ensure the right technology is brought to bear to solve the right problem.

The CDO: Enterprise Architect’s best friend

Strategic Theme # 4/5: Systems Theory & Systems Thinking

[tl;dr Systems thinking is the ability to look at the whole as well as the parts of any dynamic system, and understand the consequences/impacts of decisions to the whole or those parts. Understanding and applying systems thinking in software-enabled businesses is in everyone’s interest.]

First thing’s first: I am not an expert in systems theory or systems thinking. But I am learning. Many of the ideas resonate with my personal experience and work practices, so I guess I am a Systems Thinker by nature. The corollary of this is that, it seems, not everyone is a Systems Thinker by nature.

Enterprise Architecture is, in essence, a discipline within the general field of systems theory and systems thinking. It attempts to explain or communicate a system from many perspectives, explicitly looking at the whole as well as the parts.

As every enterprise architect knows, it is impossible to communicate all perspectives and viewpoints in a single view…the concept of a ‘thing’ in enterprise architecture can be very amorphous, a bit like Shrödingers Cat, only tangible when you look at it. So meta-models such as ArchiMate and the Zachman Framework have been devised to help architects manage this complexity.

But this post isn’t about enterprise architecture: the fact is, many complex problems faced by businesses today can only be solved by systems thinking across the board. It cannot be left to a solitary group of ‘enterprise architects’ to try to bring everything together. Rather, anybody involved in business change, continuous improvement (kaizen) or strategy setting must be versed in systems thinking. At the very least, it is this understanding that will allow people to know when and why to engage with enterprise architects.

An excellent example of where systems thinking would be particularly useful is in financial services regulatory reform: regulations impact the entire lifecycle, front-to-back, of every financial product. The sheer volume of regulatory change makes it a practical impossibility to address all regulations all the time, while still operating a profitable business in a competitive environment. The temptation to solve known, specific requirements one-by-one is high,  but without systems thinking complexity will creep in and sustainable change becomes a real challenge.

Although I have a lot to learn on systems thinking, I have found the book ‘Thinking in Systems’ by Donella H Meadows to be a very accessible introduction to the subject (recommended by Adrian Cockcroft and as such it is also a strong influence within the DevOps community). I am also seeing some excellent views on the subject coming out of established architecture practitioners such as Avancier and Tetradian – where challenges and frustrations around the conventional IT-centric approach to enterprise architecture are vented and addressed in very informative ways.

The point of this post is to emphasise that systems thinking (which breaks through traditional organisational silos and boundaries) applied across all professions and disciplines is what will help drive success in the future: culturally, it complements the goals of the ‘lean enterprise’ (strategic theme #1).

Many businesses  struggling to maintain profitability in the face of rapid technology innovation, changing markets, increased competition and ever-rising regulatory obligations would, in my view, do well to embark on a program of education on aspects of this topic across their entire workforce. Those that do can reasonably expect to see a significantly greater return on their (technology) investments in the longer term.

Strategic Theme # 4/5: Systems Theory & Systems Thinking

Strategic Theme #1/5: The Lean Enterprise

[tl;dr Agile and lean concepts are core to any enterprise cultural transformational efforts, but need to extend beyond the realm of the IT function to ensure the desired strategic outcomes from a business perspective are achieved.]

Cultural and process transformation is high on the agenda for many large businesses, as this is (correctly) seen to be necessary to innovate and maintain agility.

However, such transformation is not limited to IT – it needs to extend to financial management, and risk, governance and compliance in order for IT to realise it’s full potential.

A lot of the thinking here is very well captured in the book ‘The Lean Enterprise: How High Performance Organisations Innovate at Scale’ by Humble, Moleskin & O’Reilly.

It covers key ideas such as:

  • extending agile/systems thinking and kaizen (continuous improvement) beyond the realms of IT into financial management, risk, governance and compliance;
  • differentiating exploration from exploitation from a project portfolio perspective (applying real-option theory to portfolio selection and project execution);
  • recognising and transforming the dominant corporate culture (pathological, bureaucratic or generative);
  • starting small and tolerating failure.

This material dovetails nicely with other thought-leading publications such as ‘The Phoenix Project’ (Kim et al) and ‘Continuous Delivery’ by Humble & Farley.

Other ‘lean’ best practices, such as Scaled Agile or LeSS (Large-scale scrum) also address these themes, although these do not attempt to bring the thinking into the non-IT domains of financial management and governance, etc, which (in my view) ignores a key driver for why IT processes in many large organisations are as broken as they are. Never-the-less, they all bring useful insights to the table and any one of them should be part of any transformational agenda.

Addressing complexity is something implicit in this topic: on its own, doing lots of experiments does nothing but create more complexity (even if they do temporarily add value). Systems thinking requires understanding what is no longer needed as much as what is needed (or, alternatively, what is strategic and what is commodity), and is key to preventing complexity from impeding innovation.

For now, I’m not convinced the topic of complexity management has been adequately addressed, but more on this as I dive deeper into this area.

The topic of ‘Lean Enterprise’ is more directly relevant to change management than it is to architecture. However, since architectures invariably follow Conway’s Law (which states that system designs follow the communication structures of organisations), how projects are run, and the level of collaboration established at the outset, can materially impact architectural outcomes, and this must be the starting point for major architectural change,

Strategic Theme #1/5: The Lean Enterprise

The Learning CTO’s Strategic themes for 2015

Like most folks, I cannot predict the future. I can only relate the themes that most interest me. On that basis, here is what is occupying my mind as we go into 2015.

The Lean Enterprise The cultural and process transformations necessary to innovate and maintain agility in large enterprises – in technology, financial management, and risk, governance and compliance. Includes using real-option theory to manage risk.
Enterprise Modularity The state-of-the-art technologies and techniques which enable agile, cost effective enterprise-scale integration and reuse of capabilities and services. Or SOA Mark II.
Continuous Delivery The state-of-the-art technologies and techniques which brings together agile software delivery with operational considerations. Or DevOps Mark I.
Systems Theory & Systems Thinking The ability to look at the whole as well as the parts of any dynamic system, and understand the consequences/impacts of decisions to the whole or those parts.
Machine Learning Using business intelligence and advanced data semantics to dynamically improve automated or automatable processes.

[Blockchain technologies are another key strategic theme which are covered in a separate blog, dappsinfintech.com.]

Why these particular topics?

Specifically, over the next few years, large organisations need to be able to

  1. Out-innovate smaller firms in their field of expertise, and
  2. Embrace and leverage external innovative solutions and services that serve to enhance a firm’s core product offerings.

And firms need to be able to do this while keeping a lid on overall costs, which means managing complexity (or ‘cleaning up’ as you go along, also known as keeping technical debt under control).

It is also interesting to note that for many startups, these themes are generally not an explicit concern, as they tend to be ‘obvious’ and therefore do not need additional management action to address them. However, small firms may fail to scale successfully if they do not explicitly recognise where they are doing these already, and so ensure they maintain focus on these capabilities as they grow.

Also, for many startups, their success may be predicated on larger organisations getting on top of these themes, as otherwise the startups may struggle to get large firms to adopt their innovations on a sustainable basis.

The next few blog posts will explain these themes in a bit more detail, with relevant references.

The Learning CTO’s Strategic themes for 2015

DevOps & the role of Architecture in Achieving Business Success

[TL;DR] Creating a positive re-inforcing effect between enterprise architecture and DevOps will provide firms with a future digital competitive advantage.

I finished reading an interesting book recently:

The Phoenix Project: A Novel About IT, DevOps and Helping Your Business Win (Gene Kim, Kevin Behr, & George Spafford)

Apart from achieving the improbable outcome of turning a story about day-to-day IT into a page-turner thriller (relatively speaking..), it identified that many challenges in IT are of its own making – irrespective of the underlying technology or architecture. Successfully applying lean techniques, such as Kanban, can often dramatically improve IT throughput and productivity, and should be the first course of action to address a perceived lack of resources/capacity in the IT organisation, ahead of looking at tools or architecture.

This approach is likely to be particularly effective in organisations in which IT teams seem to be constantly reacting to unplanned change-related events.

As Kanban techniques are applied, constraints are better understood, and the benefit of specific enhancements to tools and architecture should become more evident to everyone. This forms part of the feedback loop into the software design/development process.

This whole process can be supported by improving the process by which demand into IT (and specific teams within IT) is identified and prioritised: the feedback loop from DevOps continuous improvement activities form part of the product/architecture roadmap which is shared with the business stakeholders.

The application of these techniques dovetails very nicely with advancements in how technology can be commissioned today: specifically, infrastructure-as-a-service (virtualised servers); elastic anti-fragile cloud environments, the development of specialised DevOps tools such as Puppet, Ansible, Chef, etc; the rise of lightweight VMs/containers such as Docker and Mesos, microservices as a unit of deployment, automated testing tools, open modular architecture standards such as OSGi, orchestration technologies such as Resource Oriented Computing, sophisticated monitoring tools using big data and machine learning techniques such as Splunk, etc, etc.

This means there are many paths to improving DevOps productivity. This presents its own technical challenges, but is mostly down to finding the right talent and deploying them productively.

An equal challenge relates to enteprise architecture and scaled agile planning and delivery techniques, which serves to ensure that the right demand gets to the right teams, and that the scope of each team’s work does not grow beyond an architecturally sustainable boundary – as well as ensuring that teams are conscious of and directly support domains that touch on their boundary.

This implies more sophisticated planning, and a conscious effort to continually be prepared to refactor teams and architectures in order to maintain an architecturally agile environment over time.

The identification and definition of appropriate boundaries must start at the business level (for delivering IT value), then to the programme/project level (which are the agents of strategic change), and finally to the enterprise level (for managing complexity). There is an art to identifying the right boundaries (appropriate for a given context), but the Roger Sessions approach to managing complexity (as described in this book) describes a sensible approach.

The establishment of boundaries (domains) allows information architecture standards and processes to be readily mapped  to systems and processes, and to have these formalised through flows exposed by those domains. This in the end makes data standards compliance much easier, as the (exposed) implementation architecture reflects the desired information architecture.

How does this help the DevOps agenda? In general, Agile and DevOps methods work best when the business context is understood, and the scope of the team(s) work is understood in the context of the whole business, other projects/programs, and for the organisation as a whole. This allows the team to innovate freely within that context, and to know where to go if there is any doubt.

DevOps is principally about people collaborating, and (enterprise) architecture is in the end the same. Success in DevOps can only scale to provide firm-wide benefits with a practical approach to enterprise architecture that is sensitive to people’s needs, and which acts as a positive reinforcement of DevOps activities.

In the end, complexity management through enterprise architecture and change agility through dev-ops go hand-in-hand. However, most folks haven’t figured out exactly how this works in practice, just yet…for now, these disciplines remain unconnected islands of activity, but firms that connect them successfully stand to benefit enormously.

DevOps & the role of Architecture in Achieving Business Success

How ‘uncertainty’ can be used to create a strategic advantage

[TL;DR This post outlines a strategy for dealing with uncertainty in enterprise architecture planning, with specific reference to regulatory change in financial services.]

One of the biggest challenges anyone involved in technology has is in balancing the need to address the immediate requirement with the need to prepare for future change at the risk of over-engineering a solution.

The wrong balance over time results in complex, expensive legacy technology that ends up being an inhibitor to change, rather than an enabler.

It should not be unreasonable to expect that, over time, and with appropriate investment, any firm should have significant IT capability that can be brought to bear for a multitude of challenges or opportunities – even those not thought of at the time.

Unfortunately, most legacy systems are so optimised to solve the specific problem they were commissioned to solve, they often cannot be easily or cheaply adapted for new scenarios or problem domains.

In other words, as more functionality is added to a system, the ability to change it diminishes rapidly:

Agility vs FunctionalityThe net result is that the technology platform already in place is optimised to cope with existing business models and practices, but generally incapable of (cost effectively) adapting to new business models or practices.

To address this needs some forward thinking: specifically, what capabilities need to be developed to support where the business needs to be, given the large number of known unknowns? (accepting that everybody is in the same boat when it comes to dealing with unknown unknowns..).

These capabilities are generally determined by external factors – trends in the specific sector, technology, society, economics, etc, coupled with internal forward-looking strategies.

An excellent example of where a lack of focus on capabilities has caused structural challenges is the financial industry. A recent conference at the Bank for International Settlements (BIS) has highlighted the following capability gaps in how banks do their IT – at least as it relates to regulator’s expectations:

  • Data governance and data architecture need to be optimised in order to enhance the quality, accuracy and integrity of data.
  • Analytical and reporting processes need to facilitate faster decision-making and direct availability of the relevant information.
  • Processes and databases for the areas of finance, control and risk need to be harmonised.
  • Increased automation of the data exchange processes with the supervisory authorities is required.
  • Fast and flexible implementation of supervisory requirements by business units and IT necessitates a modular and flexible architecture and appropriate project management methods.

The interesting aspect about the above capabilities is that they span multiple businesses, products and functional domains. Yet for the most part they do not fall into the traditional remit of typical IT organisations.

The current state of technology today is capable of delivering these requirements from a purely technical perspective: these are challenging problems, but for the most part they have already been solved, or are being solved, in other industries or sectors – sometimes in a larger scale even than banks have to deal with. However, finding talent is, and remains, an issue.

The big challenge, rather is in ‘business-technology’. That amorphous space that is not quite business but not quite (traditional) IT either. This is the capability that banks need to develop: the ability to interpret what outcomes a business requires, and map that not only to projects, but also to capabilities – both business capabilities and IT capabilities.

So, what core capabilities are being called out by the BIS? Here’s a rough initial pass (by no means complete, but hopefully indicative):

Data Governance Increased focus on Data Ontologies, Semantic Modelling, Linked/Open Data (RDF), Machine Learning, Self-Describing Systems, Integration
Analytics & Reporting Application of Big Data techniques for scaling timely analysis of large data sets, not only for reporting but also as part of feedback loops into automated processes. Data science approach to analytics.
Processes & Databases Use of meta-data in exposing capabilities that can be orchestrated by many business-aligned IT teams to support specific end-to-end business processes. Databases only exposed via prescribed services; model-driven product development; business architecture.
Automation of data exchange  Automation of all report generation, approval, publishing and distribution (i.e., throwing people at the problem won’t fix this)
Fast and flexible implementation Adoption of modular-friendly practices such as portfolio planning, domain-driven design, enterprise architecture, agile project management, & microservice (distributed, cloud-ready, reusable, modular) architectures

It should be obvious looking at this list that it will not be possible or feasible to outsource these capabilities. Individual capabilities are not developed isolation: they complement and support each other. Therefore they need to be developed and maintained in-house – although vendors will certainly have a role in successfully delivering these capabilities. And these skills are quite different from skills existing business & IT folks have (although some are evolutionary).

Nobody can accurately predict what systems need to be built to meet the demands of any business in the next 6 months, let alone 3 years from now. But the capabilities that separates the winners for the losers in given sectors are easier to identify. Banks in particular are under massive pressure, with regulatory pressure, major shifts in market dynamics, competition from smaller, more nimble alternative financial service providers, and rapidly diminishing technology infrastructure costs levelling the playing field for new contenders.

Regulators have, in fact, given banks a lifeline: those that heed the regulators and take appropriate action will actually be in a strong position to deal competitively with significant structural change to the financial services industry over the next 10+ years.

The changes (client-centricity, digital transformation, regulatory compliance) that all knowledge-based industries (especially finance) will go through will depend heavily on all of the above capabilities. So this is an opportunity for financial institutions to get 3 for the price of 1 in terms of strategic business-IT investment.

How ‘uncertainty’ can be used to create a strategic advantage