Category: Uncategorized Page 9 of 11

I have been trying subtly shift people from saying “big data” to saying “complex events” for some time now. But perhaps I was being too subtle.

Big Data, like master data management (MDM) and customer relationship management (CRM) before it, has a problem. The problem is that's it's a true innovation – so nobody understands it.

Also, it is an innovation that is technology-driven in the sense that it is something that wasn't possible without advances in technology. However, again like MDM and CRM this distinction between being technology-driven and being a true innovation is hard to manage. For these types of innovations, eventually (or rather almost instantly in the case of big data) the phrase is adopted by vendors and then becomes just “a set of technologies” rather than the innovation that those technologies have enabled.

The simplest solution to this problem is to two names. One name for the problem and one name for the solution. In this case the problem is “complex events” and the solution is “big data”. More specifically, the problem is complex events over mixed granularity and time, where some data is a proxy for other data.

Firstly, let's clarify what we mean by a simple event. Take a look at that following row of data:

This data represents a simple event. The way we understand this data as an event is also simple. We just have to put some clear labels on the data, and resolve any codes used, and we can quickly see that the event was the purchase of 2 blue ballpoint pens.

As this is a simple event it lacks the features of a complex event. This purchase occurred at a single moment – in this case some time on the 12th of May 2012. We can debate how accurately we can see the time of the event but this is the type of event that is described entirely at one particular moment.

The second significant feature that the simple event lacks is mixed granularity. Note that the price shown is the price of a single pen. There is also no complexity beyond simple multiplication in determining the total price involved in the event. There is no reference to things like “the total value of pens that the particular person has bought in their lifetime” that is fundamental to understanding the event.

Thirdly, all of the information provided to describe the event is directly related to the transaction. Even the numerical code used to identify that the product was a pen is just a code to simplify some uses of the information. The event wasn't a transaction to acquire part of the means to produce a letter to the purchase's wife's mother. While this might be true it is not part of the event.

These characteristics are what make the event simple. Complex events have the opposite characteristics. Managing these events takes data of mixed granularity, mixed time periods, and includes some data that is a proxy for other data.

If the purchase of the pen itself is a simple event then the decision to buy a pen is a complex event. You might sketch this decision as follows:

In fact, you could quickly add even more complexity to this sketch. This is a genuinely complex event. It crosses time periods because if you are going to be home in 20 minutes and you know you have pens there you don't have to buy one now.

If you are trying to encourage this person to buy a pen you will need to mix granularity and use proxies for the information in the buyer's head. You may know that pens displayed at the front counter sell better than pens at the back of the counter – and you can use this aggregated information – which is a proxy and at a different level of granularity to the rest of the event – when managing the event.

Big Data then is the technology that you can use if you want to understand, influence the outcomes of, or otherwise manage complex events. When big data is presented as Volume, Variety, Variability (etc.) it could be argued that these are features of the data required to support complex events rather than features of big data.

You will often hear “that's not a real big data problem!” which is true but equally you could say “that's not a complex event!”. Importantly, while you could apply the technology of big data to problems that are not real big data problems a complex event is a complex event.

Some may suggest that this distinction between simple and complex events has always been true – and they would be right. The concepts are not new. However, it shouldn't be suggested that we had the ability to manage complex events before the technology allowed us to do so.

It is true that we have been able to create computerised models for some time which have allowed us to, for example, generated a list of customers likely to churn, or pre-process a reasonably accurate next best offer for a customers. However, there is a difference between being able to provide business intelligence and analytical insights across domains where complex events exist, compared to actually being able to work with the complex events themselves, in real-time, across broad domains, cost effectively, and with minimum engineering investment.

So, it's the “complex event” revolution we should be exalting not the “big data” revolution.

Posted with Blogsy

And so I present, the collective nouns for many of those groups I’m well acquainted with after 20 years at this game:

• the “burden” of architects – who blessed with superior knowledge of something – it doesn’t really matter what – use it to protect themselves from everything else
• the “career” of project managers – who seem to forget that they are there for the project and that the project isn’t there as a means to display their many talents
• the “negation” of business process analysts – who will always define themselves by what they are not – they promo-tweet “it’s not about the technology” without any clear sense of irony
• the “anticipation” of change managers – who are by now so convinced that change is continuous and never ending that they must be ever poised for the next change, no matter what it is, without regard to the vast market of ‘everybody else’ who can collectively adapt to ’good’ change well before the precious change-resistant change manager can understand the details of ’bad’ change
• the “deck” of strategists – who make a clear distinction between ’strategy’ and ’implementation’ not because they have a clear view of the rich dimensions of strategy but rather because their two-dimensional view of the world divides real life into “things I’m aware of that I call strategy” and “things I find confusing and complicated that I call implementation”
• the “haggle” of general managers who are struggling just to preserve the pretence of a ‘strong management team’ and thus are happy if they can just keep up and appear to agree with the team, or strategically act contrarian, during an increasingly overlapping portfolio of meetings with people just like them only too scared to say

But I jest. And I’m well aware that the quality is in the exceptions.

My fine and learned colleague Alan Duncan presents a good summary of the data warehouse and business analytics “reboot” on the Information Action blog:

http://informationaction.blogspot.com/2012/08/revolutionising-data-warehouse-business.html

The key takeaway for me is a concept Alan and I are passionate about. The idea that you shouldn't manage your data warehouse as such, you should instead unbundle the data warehouse (and any supporting capabilities) into a set of information services.

Thus, it is these information services that you manage, not the technology components of the data warehouse itself.

Meanwhile, I've started getting syndicated on the SMS blog along with Alan: http://www.smsmt.com/Social/Blog/Matthew-De-George

(everything will be cross-posted here at this stage)

Posted with Blogsy

It's a well worn and cynical observation – that I use often – but it's worth repeating. A “process” is something you'd like other people to follow in situations where you yourself would just use your own judgment. A “framework” is what you think other people need to understand in order for you to do your job correctly.

The way we manage our organisations is filled with these sorts of half solutions. In particular, it's filled with ways of managing complexity that put obligations on others. Putting it another way, the application of the method differs depending on what side you're on – us or them. “They should follow a better process”. “They don't understand our framework”.

Enterprise Architecture is a good example of this. It's locked in a battle with the upper echelons of the corporate world. The premise of enterprise architecture is that you can make systematic decisions about strategy and resource allocation based on building an integrated model of the business, information, applications, and infrastructure (or any number of different layerings and groupings).

Enterprise Architecture (EA) competes with general management and governance principles which align decision making rights with responsibility – regardless of the availability of enterprise architecture models or frameworks. In this context EA could be accused of wanting to eat the cake (of strategic thinking) but not wanting to accept responsibility for the consequences (the fat of being the decision maker).

The problem of creating a systematic approach to strategic resource allocation is a real and important. It deserves to be solved for your organisation. However, it's not a modeling problem, it's a collaboration problem.

I like to talk in terms of “Collaboration Architectures” – this is a sort of “applied architecture” discipline which simply defines the architecture as a “delimited shared understanding”. It's descriptive of the shared understanding of the components, layers, and relationships that the group collaborating all agree on. Most importantly, that delineated shared understand only changes organically with the understanding of the group. You find the collaboration architecture before you change it.

These collaboration architectures are the bridge between soft skills, general management principles and governance, and the desire for systematic approaches to management. It's only in relation to this type of architecture that decisions can be make (including allocation of tasks, resources, performance management, etc).

Collaboration architectures can be mentored by a third party – just like individual meetings can be facilitated or chaired – so there is no conflict of interest between who owns decision rights and who is mentoring the collaboration architecture.

If Enterprise Architecture is to be saved from obscurity – or at least from being constantly misunderstood as it fails to define itself consistently – it needs to focus on the how it facilitates collaboration. It also needs to be clear on what decisions its collaboration architecture supports.

A collaboration architecture for executives coming together to make resource allocation and strategy decisions is powerful. When enterprise architecture plays this role it to is a powerful strategic tool. But it needs to loose a lot of baggage lest it be replaced by other collaboration architectures more suitable to the task.

But that's for another day…

Posted with Blogsy

You likely have an understanding of what a cartel is, and know some historic and contemporary examples – collusion on price in the international shipping industry, gold cartels, and the diamond cartel – all alleged, of course.

Cartels simply agree when they should be competing. Those who support “cartel busting” do so because they believe agreeing when you should be competing breaks the competitive processes that make capitalism good for consumers. It reduces choice and keeps prices artificially high.

While the benefits of capitalism may not be universally accepted, those arguments only make the following analogy more interesting:

What is a cartel really? It’s a group of people that sit around, don’t like to write anything down, and agree on the value of something. Also, they agree on the value of things based on their own interests rather than letting market forces decide.

Is this not precisely what your management team is there to do? Isn’t your management team in that case rather like a cartel?

This isn’t a problem in itself. You need a management team (right?). Otherwise you are admitting your organisation’s aggregation of services, directed by your management team, has no chance of delivering value beyond what your customers could source themselves via a number of coordinated transactions on the market.

The problem isn’t that your management team is like a cartel. The problem is not acknowledging the market-like processes within your organisation that are bypassed when a management team simply “agree amongst themselves” on the value of something. The problem is seeing your organisation only as a governance structure and not seeing the internal market.

The more a management team struggle to “get along” or “be a strong team” the more they loose touch with clients, markets, and employees. How many times have you heard about the need for a “strong management team”? In contrast, how rarely does your management team get pushed internal “market indicators” that they can’t control, or that haven’t been filtered for their consumption?

The problem isn’t the management team agreeing, the problem is the management team agreeing too readily, without sufficient evidence, or without sufficient engagement with the organisation as a whole. In other words, without being linked with the internal market of value within the organisation.

Wherever side of the economic arguments you might sit on – the Austrian economists, for example, don’t have an issue with cartels – the idea of busting cartels is exciting. If you want to bust your management team cartel, the tool of choice is information.

Real cartels bypass the market, bypass the price system. The problem is that organisations don’t even have a price system. They have financial accounting systems, but they only cover a very small proportion of the internal market and are often too readily manipulated and “gamed” by the management team.

In fact, your management team exists in lieu of a price system. That’s almost the very nature of a firm. But the nature of firms is changing. Social media technologies, direct communication channels, and alternative organisational structures such as communities of practice, broadly open the information channels within and across organisations. When this happens the flow and valuation of information become a sort of price system.

While I might once have suggested organisations could benefit from introducing a price system, I believe the tide has turned and it’s now just as important to ensure the management process is re-engaged with this information-based price system that already exists. Not to be driven by it, but to ensure it both informs decision-making and supports the implementation of decisions.

All of our information systems – when implemented with this explicit purpose – can provide a view of this internal market. They can all be viewed as part of this price system. Every activity brokered through an information system can be viewed as a transaction in a market that contributes to one or more market indicators.

Busting your management team cartel is reconnecting with this internal information market. What steps might you take to ensure that your management team isn’t acting like a cartel?

• ensure the management team is presented with external information, that they didn’t generate themselves or that wasn’t directly generated for their consumption.
• ensure the information used to make decisions is shared across the organisation when possible. This uses the internal market to value the information used in decision making.
• Openly compete within the management team. Any management team, particularly one structured across functional lines, has a fundamental conflict of interest regarding which function gets the most resources. If the management team isn’t arguing about resource allocation they aren’t trying to optimise their respective functions.
• Don’t compete. The specific structure of the management team is important. If the organisation’s strengths and weaknesses are understood and agreed within the management team then some roles within the management team are more difficult than others. There are allowed to be “losers”. Some regions are more competitive, some functions are first to struggle when external conditions change. A rich model of the organisation’s operating model should inform the management team’s decisions.

So go cartel busting. If your management team is used to making decisions by simply agreeing amongst yourselves – with little reference to information not explicitly produced for consumption by your team – you may be acting like a cartel, and your consumers, the whole organisation that depends on your services, may be suffering.

Posted with Blogsy

In his bestselling “Thinking, Fast and Slow”, Nobel Prize winner in Economics Daniel Kahneman proposes a model for human thought based on two systems of thinking. “System 1” is defined as fast, instinctive and emotional. Whereas “System 2” is slow, deliberative, and more logical.

Most importantly, “System 2” is lazy and doesn’t even bother showing up to the thinking process unless it has too. In reference to study after study, Kahneman builds a case that people tend to make intuitive decisions, based only on the information available. They use “System 1” because “System 2” is hard – when you use “System 2” you can feel the strain of thinking.

What’s more, when “System 1” is used to make decisions we are confident of the answer even when a lack of evidence exists. They are typically internally consistent decisions based on a number of heuristics that make thinking easier – i.e. less mental strain. Never does “System 1” seek out additional information.

Kahneman proposes that “System 1” operates in What You See Is All There Is (WYSIATI) mode. When asked to answer a question people tend not to incorporate information that isn’t already right in front of them. They rarely think about the sort of information they would require to make an effective decision; but rather think of what decision can be made with the information they have.

As information management consultants, this “System 1” / “System 2” model of thinking about how individuals made decisions can be applied to organisations as a whole.

“System 1” is the way decisions are made with no explicit investment in information management. “System 2” takes more effort, because it must begin with an investment in understanding the information assets you have, the decisions you need to make, and whether there is sufficient available evidence to make an effective decision.

General management devotees make a great fuss about the importance of being decisive – in making the best decision with the available information. And there is certainly value in this approach where no alternative exists. But an alternative does exist.

Creating a systemic approach to delivering the right information, to the right people, to make the critical decisions at the heart of your operating model, is making an investment to build a “System 2” thought process for your organisation. It might be hard work – you’ll feel the strain of thinking – but it’s worthwhile.

“If you design a system where the people who do the work have control of the work, the people change. They take responsibility. It is as Hertzberg taught: if you want people to do a good job, design a good job to do.”

From: http://www.systemsthinking.co.uk/6-seddon-jan10.asp

I think we need to think about capabilities more. Not just use the word but really think it through.

Eg. From here

• trying to come up with “business justification” for technology decisions
• arguing about whether something is a “real data warehouse”
• discussing the difference between an “issue” and a “risk” rather than working through the issue and risk register
• saying “I’m not a technical person” when you really want to say “I’m too important for this discussion”
• saying “you’re not seeing the big picture” when there clearly isn’t one

Sometimes it’s not really much more complicated than this:

Governance is politics in the same room

If you find yourself whispering in the corner “it’s so political around here” you should all get in the same room.