Month: September 2012

My ever-charming wife is currently in Tasmania “doing zoology like a boss” to help understand the impacts on behavior of the facial tumor disease that has decimated the Tasmanian Devil population over the last 15 years or so.

Amanda traps the devils, tags them, and ensures that a bunch of very relevant tests are performed. Just as importantly, she also helps name each of the devils trapped after a prominent “gay” actor: Portia, Rupert, Tom, etc.

In the week she has been away so far she has trapped the allegedly bi-sexual Warren twice. The first time he seemed quite afraid when they released him – uncertain of what might happen.

Interestingly, the second time they trapped him he wasn't scared at all when he was released. He seemed to know he was safe. He looked around a bit and just sort-of wandered off.

This is “pricing it”. It's the process of gaining clarity of the costs and risks. Once something is priced decisions can be made. The increased clarity of the price doesn't always mean incidents of it will go down – it doesn't always create a disincentive – it just creates a price.

The price for Warren eating the free lamb that the traps are deliciously bated with is that he has to spend a day or so in a plastic tube and get a couple of harmless tests done.

As the price is now known it can be used in decision-making. As the price is low, it will be paid more often. The reasonably priced lamb will be consumed more often than unexpected lamb, that looks suspiciously like a trap, with an unknown price.

This is of course just like the famous after-school child care example. In this case the child-care facility decided to charge people for being late to collect their children. This was because they got sick of waiting for late parents.

The only problem was that after they put a price on the late collection of their children the number of late pick-ups actually increased! Because they had priced the after hours care people could decide to use it with no guilt.

The second problem was that the when they stopped the trial and changed back to a zero-penalty policy for late pick up, the number of late pick-ups increased yet again! The parents treated the “new” no-fee policy like after-school care was now on sale! 100% off after hours child care!

(see original paper here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.37.1417&rep=rep1&type=pdf)

So, do you want to “price it”? Really?

Posted with Blogsy

I have been trying subtly shift people from saying “big data” to saying “complex events” for some time now. But perhaps I was being too subtle.

Big Data, like master data management (MDM) and customer relationship management (CRM) before it, has a problem. The problem is that's it's a true innovation – so nobody understands it.

Also, it is an innovation that is technology-driven in the sense that it is something that wasn't possible without advances in technology. However, again like MDM and CRM this distinction between being technology-driven and being a true innovation is hard to manage. For these types of innovations, eventually (or rather almost instantly in the case of big data) the phrase is adopted by vendors and then becomes just “a set of technologies” rather than the innovation that those technologies have enabled.

The simplest solution to this problem is to two names. One name for the problem and one name for the solution. In this case the problem is “complex events” and the solution is “big data”. More specifically, the problem is complex events over mixed granularity and time, where some data is a proxy for other data.

Firstly, let's clarify what we mean by a simple event. Take a look at that following row of data:

This data represents a simple event. The way we understand this data as an event is also simple. We just have to put some clear labels on the data, and resolve any codes used, and we can quickly see that the event was the purchase of 2 blue ballpoint pens.

As this is a simple event it lacks the features of a complex event. This purchase occurred at a single moment – in this case some time on the 12th of May 2012. We can debate how accurately we can see the time of the event but this is the type of event that is described entirely at one particular moment.

The second significant feature that the simple event lacks is mixed granularity. Note that the price shown is the price of a single pen. There is also no complexity beyond simple multiplication in determining the total price involved in the event. There is no reference to things like “the total value of pens that the particular person has bought in their lifetime” that is fundamental to understanding the event.

Thirdly, all of the information provided to describe the event is directly related to the transaction. Even the numerical code used to identify that the product was a pen is just a code to simplify some uses of the information. The event wasn't a transaction to acquire part of the means to produce a letter to the purchase's wife's mother. While this might be true it is not part of the event.

These characteristics are what make the event simple. Complex events have the opposite characteristics. Managing these events takes data of mixed granularity, mixed time periods, and includes some data that is a proxy for other data.

If the purchase of the pen itself is a simple event then the decision to buy a pen is a complex event. You might sketch this decision as follows:

In fact, you could quickly add even more complexity to this sketch. This is a genuinely complex event. It crosses time periods because if you are going to be home in 20 minutes and you know you have pens there you don't have to buy one now.

If you are trying to encourage this person to buy a pen you will need to mix granularity and use proxies for the information in the buyer's head. You may know that pens displayed at the front counter sell better than pens at the back of the counter – and you can use this aggregated information – which is a proxy and at a different level of granularity to the rest of the event – when managing the event.

Big Data then is the technology that you can use if you want to understand, influence the outcomes of, or otherwise manage complex events. When big data is presented as Volume, Variety, Variability (etc.) it could be argued that these are features of the data required to support complex events rather than features of big data.

You will often hear “that's not a real big data problem!” which is true but equally you could say “that's not a complex event!”. Importantly, while you could apply the technology of big data to problems that are not real big data problems a complex event is a complex event.

Some may suggest that this distinction between simple and complex events has always been true – and they would be right. The concepts are not new. However, it shouldn't be suggested that we had the ability to manage complex events before the technology allowed us to do so.

It is true that we have been able to create computerised models for some time which have allowed us to, for example, generated a list of customers likely to churn, or pre-process a reasonably accurate next best offer for a customers. However, there is a difference between being able to provide business intelligence and analytical insights across domains where complex events exist, compared to actually being able to work with the complex events themselves, in real-time, across broad domains, cost effectively, and with minimum engineering investment.

So, it's the “complex event” revolution we should be exalting not the “big data” revolution.

Posted with Blogsy

And so I present, the collective nouns for many of those groups I’m well acquainted with after 20 years at this game:

• the “burden” of architects – who blessed with superior knowledge of something – it doesn’t really matter what – use it to protect themselves from everything else
• the “career” of project managers – who seem to forget that they are there for the project and that the project isn’t there as a means to display their many talents
• the “negation” of business process analysts – who will always define themselves by what they are not – they promo-tweet “it’s not about the technology” without any clear sense of irony
• the “anticipation” of change managers – who are by now so convinced that change is continuous and never ending that they must be ever poised for the next change, no matter what it is, without regard to the vast market of ‘everybody else’ who can collectively adapt to ’good’ change well before the precious change-resistant change manager can understand the details of ’bad’ change
• the “deck” of strategists – who make a clear distinction between ’strategy’ and ’implementation’ not because they have a clear view of the rich dimensions of strategy but rather because their two-dimensional view of the world divides real life into “things I’m aware of that I call strategy” and “things I find confusing and complicated that I call implementation”
• the “haggle” of general managers who are struggling just to preserve the pretence of a ‘strong management team’ and thus are happy if they can just keep up and appear to agree with the team, or strategically act contrarian, during an increasingly overlapping portfolio of meetings with people just like them only too scared to say

But I jest. And I’m well aware that the quality is in the exceptions.

My fine and learned colleague Alan Duncan presents a good summary of the data warehouse and business analytics “reboot” on the Information Action blog:

http://informationaction.blogspot.com/2012/08/revolutionising-data-warehouse-business.html

The key takeaway for me is a concept Alan and I are passionate about. The idea that you shouldn't manage your data warehouse as such, you should instead unbundle the data warehouse (and any supporting capabilities) into a set of information services.

Thus, it is these information services that you manage, not the technology components of the data warehouse itself.

Meanwhile, I've started getting syndicated on the SMS blog along with Alan: http://www.smsmt.com/Social/Blog/Matthew-De-George

(everything will be cross-posted here at this stage)

Posted with Blogsy