Data as a Service for Social Change

They say that hindsight is always 20:20. It's true. I'd been given 5 minutes in front of some of the most influential people and Europe to make my case, and 30 minutes after my nerve wrecking speech, I sat down and thought about everything I couldn't cram into the 5 minutes I had been given.

It's been a while since I shared the press release from the European Commission which highlighted the talk I gave and the work my team did as part of the EIT Young Leaders program. I owe you all an explanation of what I was working on, why I feel so passionate about it, what I said during the speech and what I wished I had the time to cover.

It was nerve wrecking as I stood up to join the other speakers on the stage, Kenneth Cukier, Economist editor and author of the, Alfred Spector, Vice-President of Research and Special Initiatives at Google, and finally Gavin Starks, CEO of the Open Data Institute. I was nervous, these are big names and the audience was just as impressive, consisting of CEOs, CTOs, VPs, representatives from industry, EU Policy makers and European Commissioners.

You can watch a fantastic version of me talk, here on YouTube.

I had been given 5 minutes to talk about how Big Data can be used as a catalyst for social change. Months of work, research and internal team battles turned into just 5 minutes. It is an eternity when you’re standing in front of an audience, but in reality 5 Minutes is just less than three paragraphs of text. That's not much to covey everything the 6 of us, on our team wanted to say.

The Unrealised Opportunity

When you speak to advocates of big data they all preach about how the computing power of today is cheap, digital storage is cheap and we've lots and lots of information about everything we all do every day. All we need to do, they say, is to study the data to see the great advances we can make. The problem is, when you study Big Data, to see lots of advocates, but not as many success stories. Why? What's wrong with the Big Data industry? Why aren't these fantastic opportunities being realised and what can we do about it? - This was the question which led the team and I on a fantastic journey.

Investigating the Obstacles

Data as a service for social change-Slide1
Image by sunsurfr:
Used under Creative Commons license:

Let’s depict a vision of the future; a vision we want to create. Picture a future beach, with this little boy. A little boy who has less chance of suffering from Cancer than anyone today. A little boy who has less chance of suffering from Diabetes than anyone today.

In the future we've taken 25 years of shopping habits for over 1 million people and combined it with health records to reveal lifestyle and dietary choices which increase the risks of diabetes and cancer. We've shared this information with this little boy and his parents and allowed them to change their lifestyle to avoid many of these risks.

This little boy also gets more time with his parents than anyone today. Because his parents spend less time in traffic than we do. They have a smartphone application which provides them with a forecast of road traffic before they leave the house.

Now we have the vision, my team tried to create it. Let’s start with the traffic prediction application, how could we build that?

Building the Future

Data as a service for social change-Slide2
Image by woodleywonderworks:
Used under Creative Commons license:

To build this application we need access to energy consumption data. The application combines electricity consumption data with traffic. As we all wake in the morning we turn on TVs, Radios, Kettles and showers. This causes a spike in the amount of energy that we use. This spike occurs between 1 hour and 30 minutes before we leave our house. This can then be used as an indication of when we will leave home and therefore what future traffic will be like. This electricity data exists, we know this - because the energy companies capture it to bill us, and the energy distributor’s capture it to ensure they supply the right amount of power to the right places at the right time. But it's not freely available - in fact it is locked away.

There are initiatives which try to unlock this data. Schemes like Green Button in the US or MIDATA in the UK which allow individual users to download their data from energy suppliers. These initiatives are focused on providing the data back to the individual. Using these initiatives to power our application we would need to contact each house hold individually and ask for their energy consumption data. For a city like London this could mean asking 3 million households for their data.

As an entrepreneur I've a lot of ideas and very little cash, and asking 3 million people is expensive, and risky. There is a good chance that not everyone will respond - I won't have all the data I need, and contacting 3,000,000 households is very expensive.

What if we could just ask the companies which already own this data? - The energy supply companies and the energy generators. It turns out that there are several issues making them reluctant to release this data:

  • Competitive advantage; what if we shared their data with a competitor. They would know who their biggest customer is?
  • Privacy issues; the energy company customers have provided the data only for billing and not for anything else, data should only be used for the purposes disclosed when it was originally captured.

Currently Open Data Initiatives focus on releasing data for free in a hope that this will spur innovation, and kick start ecosystems. But we haven't seen a huge uptake in this area. There are a number of factors preventing this:

  • Data reliability: The data is supplied for free. Usually this is a best effort delivery. That sounds ok, until you realise that in order for this data to be used commercially it is important to have confidence in it. Essentially the open data community is asking developers to take a leap of faith, to trust their livelihoods, homes, and families will all be safe and secure based on the income generated by a best effort data release. That is a big ask.
  • Timely data: Open Data Initiatives often partner with data producers and manually scrub data of information which may contain individual data, or other commercial sensitive information. This process takes time and as a result a number of the open data Initiatives provide "canned data" from historical data sets. This limits the applications to which this data can be used. We couldn't create a real-time traffic prediction application on this type of historical data alone.

What if we were to invert the question?

Privacy is also a concern; we don't want nor need to pry into an individual’s details, in fact for our app, seeing an individual’s energy consumption data is next to useless. It is like looking at a grain of sand when what we want to see is the beach.

Big Data is by definition big, getting a copy of this data is expensive and slow, and we don't need it. While computing is getting cheaper, and cloud computing is even more efficient, it is still not free and big data requires lots of it.

What if, rather than providing a copy of the data could simply get an opportunity to do some statistical analysis of the data. The data wouldn’t move it would stay within the owning organisation. We could create a software infrastructure which ensured that access to the data was safe and privacy compliant. It would reduce the cost of access for a small start-up, and address the privacy issues and mitigate against the risk associated in disclosing data seen as a competitive advantage.

Data as a service for social change-Slide5
Image by
Used under Creative Commons license:

The data would be safe, but the statistical value of the data would be set free. We would liberate the value of the data.

Free as in Beer (which you have to pay for)

What I'm going to propose sounds, at first a little nuts. I think free data isn't the right thing to base an ecosystem on. In fact I think it discourages the ecosystem from taking root. As mentioned above there are a number of problems with free data which prevent developers from taking dependencies on it. One of the best ways to address is this to pay for the data. Once money swaps hands, SLA (Service Level Agreements) can be put in place and if there is an error in the data, or it is provided late - then there is a rout of recourse and the developer can chase the data provider for recompense. This shared risk and charging model allows the ecosystem to grow. It encourages new data providers to enter the market and allows developers to more confidently base their family’s future on the data they provide.

Incoming Disruption

Data as a service for social change-Slide3
The electricity value chain

This charging model also has the opportunity to disrupt existing market places as it provides additional revenue and a new way for businesses in existing ecosystems to generate revenue. We considered the energy market place in Europe. Drawing the value chain for this market place we see the following:

Power generation companies which sell their energy on via power transmission companies to a Power Distribution Operator (DSO) the DSO in turn supplies the end user via the retail companies we all subscribe to.

For the purposes of our traffic prediction application it is the DSO which possesses all of the real time information we need. The DSO needs to load balance its network to ensure that the right customers get the right amount of power at the right time. To do this the DSO has real-time, live information about power consumption. However the DSO never gets to communicate this information to the end user.

Getting the DSO to release this information, even via the statistical analysis method described above would not be trivial. There are costs associated with it. To deal with this our team proposed the creation of a data broker. The broker could amortise the costs of the technology across a number of data sources. It would provide the marketplace for data services from a range of different industries and it would provide the data providers with additional revenue.

Data as a service for social change-Slide4
How the electricity and DaaS value chains work together

Adding this additional revenue stream into the originally presented value chain we get this: A situation where the DSO is generating income from two different sources. Of course similar additional revenue streams can be obtained by all of the energy companies in the original value chain. This could change the relationships between each of the companies and disrupt the current status quo.

As the broker expands into new industries it will create a market place of data producers which compete for data consumers. This competition should help ensure a low enough price point for the data. Keeping a low price point is important in order to encourage entrepreneurs to really become involved in the market place.

The key to obtaining this future – the future on the beach, is about allowing companies or individuals to come up with ideas, test them, fail quickly, or succeed in a big way. In order to do this we need to create a working ecosystem in which they can experiment. This will provide society with insights and benefits beyond what I or anyone else can outline, and will really make the vision of the boy on the beach a possibility.

Twitter Music – No Thanks

The real beauty of Twitter is the ability to discover cool people, with shared interests and likes. It's like finding that cool group of people in a crowded party. Meeting someone new and really getting to know an area they are interested in and sharing their passion.

Now a music application which lets you do that, that would be cool. It is what Twitter's new music application could have been.

Instead we've an application which provides a bland pastiche version of a generic radio station - one that plays the over hyped, almost factory farmed tunes. - Its like being back at the party but not being able to talk to anyone because of the booming music played by the host.

Now if only there was a cool way to share great new music with those cool guys in the corner ... maybe sound cloud has the answer?


18 Months

It has been a mad 18 months. Well actually I thought it had been relatively sedate, that was until I checked my phone's history.

My mobile phone contract is due for renewal this month, so it was an opportune moment to study my phone with a look to backing up the data prior to getting a new device.

It is funny how the mind works. I can easily recall recent events, but anything beyond three months has been a bit of blur. I thought my travelling had slowed over the last 18 months. I hadn't actually done that much recently, of course checking my phone revealed a different story.

The new Photo application on the phone captures geographic data inside every photo you take and my phone has revealed a rather interesting series of trips across three continents:

Looking forward to a sit down...
Looking forward to a sit down...

Nuts. It's crazy, my phone as a better memory than me! Looking at the photos brought some great events back to life:

  • I became a father
  • I changed job
  • My son's first birthday
  • My wife's birthday
  • My birthday
  • Trips with my family to see relatives in New York and England
  • Giving my sister away!
  • We are now expecting another baby
  • Trip to Tokyo to see one of my best friends

After all that action I think I need a sit down...

Following @documentally ‘s example with beggars, but with a different outcome

Just before Christmas I listened to a fantastic post from @documentally about his trip to Sweeden, where upon a cold winters night he did a good deed for a local beggar and was rewarded with gratitude. I tried the same tonight on a cold trip home in London but was met with a remarkably different response.

I listened with earnest to @documentally's trip to Sweeden on Audioboo, he recalled how upon entering the local kebab shop he had passed a beggar. As @documentally ordered a double portion of his late night chips and donated them to the beggar who had been sitting out side the shop, who gratefully received them.

On the cold walk back to the Hotel I'm staying at in London tonight I popped into a local Tesco Express (late night convenience store). A women exited the store and I heard the homless guy say "... please I'm hungry... ". The woman didn't stop - she just kept walking.

I had only popped in to pick up a toothbrush I had forgotten to pack, but when purchasing the toothbrush I also purchased a prepacked sandwich, and when a left the store the I heard the same refrain from the homeless guy "... please I'm hungry...". So I stopped. I opened my shopping bag and handed the homeless guy the sandwich,.

"I heard you say you were hungry, I've no change but I hope this helps"

"well err..... erm... but... err... it doesn't help with... er.. my medication...."

..I guess beggars in London are a bit too posh for free food... I have the sneaking suspicion that this beggar was earning more money than me by begging on the street!





Passing on “The Toy” – The Story of Doggie and Me

"So what happened to Doggie?" - it was one of Mum's first questions during our phone call. I had been recalling how my one year old son had just managed to walk across our bed and pull down all of the cuddly toys myself and my wife keep on our headboard. Doggie had been sitting on the headboard when he was attacked.


I can't answer my mother's question until I tell you all about Doggie and the adventures we had together. It also goes someway to explaining why, up until his rain of toy terror on our bed my son hadn't got his hands on Doggie.

Doggie has always been here. I can't remember when I got him, in fact now that I think about it, it is really hard to recall a time without Doggie. I think all kids have "the toy", the single toy which gets taken everywhere and does everything, "the toy" that needs to be in bed before they get in bed and with them before they wake up. As a child "the toy" for me was Doggie.

Today of course I can leave Doggie behind, he doesn't need to go everywhere with me and I can share with you that Doggie has never made it inside any of the offices in which I have worked.

Doggie now sits on my headboard keeping watch across the bedroom. However when I pick him up and peer into his battered eyes, feeling his de-fluffed fur I am at an instant transported through a whole range of memories.

Doggie was with me during my time alone in a bedsit in Manchester; he witnessed some truly amazingly bad cooking and never complained. It was a bedsit with a brothel two doors down on the left, a bookies and a pub 3 doors up on the right. I only visited the pub.

He was with me in university in Belfast; sitting in the corner of my bedroom Doggie witnessed some amazing hangovers, a lot of study (I did some, honest, you can ask Doggie), and some kisses with the woman who became my wife.

Doggie was with me while I was at school in England, and before that in South Africa. I don't recall receiving him, but I do know who gave him to me. It was someone who I didn't get to spend enough time with.

My parents, my sister and I travelled out to South Africa when my sister and I were about 5 years old. My Aunty and her family had already moved out there before us.

Of my childhood memories about South Africa hanging out and being babysat by my two "big cousins", Kim and Carolann certainly sticks out. Myself and my sister spent a lot of time with them in South Africa. The big cousins had monopolised the market for babysitting - every time we were babysat, it was Kim and Carolann. The two of them also held, what felt like black belts in the art of tickling children. They were both past masters at it. While I struggle to recall all the details I assume Doggie was there with me too. I'm sure he was flung across the room in fits of tickle induced laughter.

When faced with two babysitters who are black belts in tickling there are two simple options available to any child:

Option 1 : tease the babysitter and get a great reaction, run hide and try to avoid the tickles.

Option 2 : play games with the baby sitter.

Myself and my sister had a general rule: Employ option 1 with Carolann, and employ option 2 with Kim.

Carolann was so easy to tease, and especially so when she had a boyfriend over when her reaction to teasing would be an order of magnitude greater than when she was on her own.

I remember playing a lot of games with Kim and my sister. I remember Kim being creative, doing a lot of drawing and coming up with a really cool homemade spy set. The spy set consisted largely of tin-foil covered boxes marked "camera" and "walky-talky" etc. Maybe this was just prep in case she brought over a boyfriend, I don't know, but I do know it was a lot of fun hanging out with her. My sister and I also learnt that even with this more cooperative babysitting tactic - we still got tickled.

Kim was also the one who gave me Doggie. In the years that followed my family returned to the UK leaving behind my cousins and their family. The baby sitters in England were not as much fun to tease, and were definitely not black belts in tickling. Let’s just say that the competition in England wasn't as good.

The two families kept in touch and I learnt that Kim had started a career in computers. We also heard the news, that Kim had got leukaemia. I remember the phone calls my parents had with my Uncle and Auntie, the searches for experts in both the UK and South Africa. I also recall the family meeting where we heard the very sad news that Kim had passed away. Of course doggie was there through all of that too.

But now I think it is time for Doggie to have fun, for him to return to being "the Toy" and the glee of being thrown across the room in fits of tickle induced laughter. I have to pass Doggie on and let my little boy have him.

I know my son will pick one of his toys to be "the toy", he may pick Doggie or he might pick something else. Whatever he selects I'm going to enjoy playing with my son and discovering which toys he likes. Doggie of course will be there to watch and play too.

When my son is old enough I'll tell him to story of my Doggie, the adventures I've shared with him, and most importantly the stories about the incredibly creative person who gave Doggie to me. I hope my son adds his own adventures and stories to Doggie or whatever toy becomes his "the Toy" and enjoys his toy as much as I enjoyed mine.

In the mean time Merry Christmas to all.

Mince Pie-Tastic

After tasting the shop bought versions I decided that I could do better and I would like to share with tou my results.

Here is my attempt - some lovely home made mince pies. They have been made with additional whiskey. I have been informed by my most wonderful wife that there is a little too much whiskey in them and I'm prohibited from sharing these with my son.

I can't claim all the credit, my brilliant wife helped out loads and rolled out the pastry ensuring a thin but tasty crust. My 'spiked' mince-meat provides a full bodied experience. They really are lovely. I think I got the balance just right - but then I did make them.

So I've brought them into work for an unbiased review... let's see what happens.

My home made mince pies
My home made mince pies

An Amazing An Post Experience

I've got my son, Ben in my arms while in trying and failing to run a bath for him. Ben has other ideas and he's trying to break into my aftershave bottle. The door bell rings and brings someone unexpected, who through a random act of kindness will help us have a good Christmas.

My first thought when the door bell rung was one of frustration - I was so close to getting the little man in the bath. But luckily I hadn't quite managed to get Ben undressed yet.

Like most of the population Emma and I had done a lot of our shopping on line. We had received reminder notes that some of our parcels where ready for collection but being flat out in the run up to Christmas we'd not managed to arrange to collect them.

It is nearly 7pm at night and when I open the door I'm greeted by a guy dressed in a postman's uniform. There is no post van behind him, just a little silver car with the engine still running.

Hi, I'm the guy who normally delivers your post. I noticed that there's were a few parcels waiting for collection, some had been sitting there so long that we had been told by our manager that we have to start sending them back. They look like Amazon boxes and I guessed that they were Christmas presents. I didn't want you guys to miss them - it would make for a terrible Christmas. I'm off work now, on my way home and I volunteered to drop off your parcels. I've got a bunch in the back of the car.

Wow - now these guys are crazy busy this time of the year, I'm sure that this posty was knackered but he'd taken some of his personal time out to drop off our amazon orders.

It was a small random act of kindness which I'm really very grateful for.

Tonight Emma dropped off a big box of Roses chocolates to the post office sorting station as a way of a thank you.

An post - you have some fantastic employees! And a very merry Christmas to you all.

Mr Kipling v Aldi’s Best

So tonight I got the chance to try some mince pies from Aldi. They had some exceedingly stiff competition from Mr Kipling.

As you can see the decorations on the top of the Aldi mince pies were definitely more appealing than that on the Mr Kipling's. These Aldi specimens held so much promise. Also tipping in Aldi's favour was that all of the mince pies in their pack were undamaged! A feat that Mr Kipling couldn't achieve. The Aldi pies shipping with additional packaging which protected the pies.

Filling In The Details
The filling in the Aldi mince pies was delightful, it had a light orange tang to it which I personally really liked. However the filling was very shallow - if only there was more meat! Mr Kipling's on the other hand have a a deep feeling. While not quite as tasty as Aldi's they don't skimp.

Verdict - draw. If Mr Kipling had good packaging - they'd have got this one in the bag!

Mince Pie Month Begins

I love mince pies. So much so that I limit myself to only eating them in December. Today is December 1st so it was with great excitement that I broke my mince pie fast and opened a packed of Mr Kipling's best and was instantly disappointed.

Two of the pies had been smashed on their packet 🙁

Poor job Mr Kipling! poor job.

On the bright side I noted a marked increase in mincemeat volume compared to Mr Kipling's effort last year.

Only one thing for it - I need to make my own!