Big Data on keithrozario.com

MDeC Private Meeting with ODI

Sun, 05 Apr 2015 20:00:17 +0000

Earlier this week I attended a MDeC organized private meeting with Richard Stirling from the Open Data Institute (ODI).The ODI is an institution that hopes to promote the ‘open data’ culture, and founded by a giant of the Tech world, Sir Tim Berners-Lee, which you might remember for inventing a small little thing we call the world wide web.

The meeting was attended by just a handful of folks, some of whom I recognized from a previous Seatti conference I attended, with the audience and topic focus on Open Data (and Big Data) in Malaysia.

The conversation was really good, and broadly speaking touched on 3 key topics. Most of this post is a re-hash from my failing and aged memory, but there's a clearer version of the minutes here from the amazing people of Sinar Malaysia if you're interested in the specifics.

MyProcurement: All government tenders in one Excel file

Tue, 16 Sep 2014 22:54:40 +0000

I've updated this post on 31-Mar-2015, to incorporate the latest changes, and to provide more up to data info on the procurement database. Left everything else in tact.

Happy birthday Malaysia!! Just how awesome is our country, that we celebrate an Independence Day AND a Malaysia Day, not to mention 2 New years day, (or 3 if you count Awal Muharram).

So on that note, I decided to use my IT skills for the good of the country.

To be honest, my IT skills have never been up to par, my day job is more managing/planning/documenting than actual execution of ‘real’ IT work. But it was good for me to dust of the ol’ programming fingers and learn Python to grab some publicly available information and make it more accessible to the less IT centric members of society.

Since I had limited time, and sub-par skills, I decided to set my sights low, and aim to extract all the data from the Malaysian MyProcurement portal, which houses all the results of government tenders (and even direct negotiations) in one single website for easy access. The issue I had with the portal though, was that it only displayed 10 records at a time–from it’s 10,000+ record archive, so there was no way to develop insights into the data from the portal directly, you had to extract it out, but the portal provider did not provide a raw data dump to do this.

So I wrote a simple Python script to extract all the data, and prettified the data in Excel offline. The result is a rather mixed one.

I was happy that I could at least see which Ministeries or Government departments gave out the most contracts, and what the values of those contracts were. All in all, the excel spreadsheet has more than 10,000 tenders with a cumulative value of RM35 billion worth of contracts going back to 2009. The data allowed me to figure out which Ministry gave out the most contracts, the contracts with the highest and lowest value (including one for Rm0.00, and one for just Rm96.00). All in all it was quite informative.

The root cause of crime

Wed, 10 Jul 2013 13:04:19 +0000

Crime has become a hot-button topic these days, and while a lot of fingerpointing and blame-shifting has been going on in political circles, I think it’s wise we took a step back and try to address the root problem rather than its symptoms.

A brilliant piece by Evgeny Morozov from the Slate, points out the following:

[box icon=“chat”]

Forget terrorism for a moment. Take more mundane crime. Why does crime happen? Well, you might say that it’s because youths don’t have jobs. Or you might say that’s because the doors of our buildings are not fortified enough. Given some limited funds to spend, you can either create yet another national employment program or you can equip houses with even better cameras, sensors, and locks. What should you do?

Data guys versus Lawyers and Politicians

Thu, 22 Nov 2012 08:00:59 +0000

Nate Silver is currently the internet darling of the big data folks, not only did he accurately predict the correct outcome in all 49 states for the US presidential election, he correctly pointed out that Florida would be a toss up before eventually leaning towards Obama. That’s like predicting a coin-toss would end on it’s side. While all of that may seem remarkable, this isn’t a story of a boy-genius but rather the dawn of a new age–an age driven by data.

Nate isn’t alone on this, The Slate reports that 2 different pundits got the entire analysis spot on as well, not to mention a 3 man team in North Carolina armed only with robo-callers who also made a spot-on prediction. This isn’t some savant ability that Nate has, this is just pure hard core science at work, and the people that use the science are the ones making the accurate predictions, while the people that ignore it–are left behind.

And just who got left behind? The usual opinion writers, like Ann Coulter who predicted Romney would win by a 273-265 margin, Newt Gingrinch who predicted an ever igger margin for Romney and of course Jim Cramer who predicted such a insane number that it probably isn’t even worth typing here–but I’ll type it anyway. Good ol’ Jim predicted Obama would win by a whoooping 440-98 margin, off by more than a 100 point margin…but at least he got the winner right, and I’m sure predicting the stock market isn’t anything like predicting an election, and 100 points means nothing in the stock market.

Answering the tough questions: Watson vs. Humans

Thu, 30 Aug 2012 08:05:19 +0000

IBM have always been on the cutting edge of innovation, they’ve moved more becoming merely a computer company to what is probably the first truly all encompassing technology company, they don’t just make fancy gadgets or shiny tinga-ma-jigs, they make actual solutions for real-world problems.

In 1996, IBM introduced the world to Deep Blue. Kasparov met Deep Blue and wasn’t impressed, he had no reason to be, he defeated Deep Blue 4-2, and walked away comfortably.

However, in 1997, IBM re-introduced the world to the 2nd version of Deep Blue (unofficially named Deeper Blue), and this time Kasparov was beaten –but not by much. Kasparov is the Tiger Woods, Pele and Michael Jordan of the Chess world, and he was beaten by a super computer with 11.38 GFLOPs of power.

In turns out though, we had nothing to be afraid off, Chess is after all a pretty simple game when you break it down, the number of possible moves are finite, together with the number of possible scenarios to play out. It’s not an easy game to master, but as it turns out playing chess is infinitely easier than just plain talking.

In fact, of all the talking games, Jeopardy seems the most difficult. At the end of this post, I will make an argument to show that Jeopardy – a simple talking game – is about 6,500 times more difficult than Chess (a game we often associate with genius). Turns out Kasparov has to bow to Ken Jennings.

Is MAS updating it's own Wikipedia page?

Thu, 19 Jul 2012 04:26:32 +0000

Continuing my series on bigdata and Google bigquery, I’ve decided to share a rather interesting snippet of information regarding our very own Malaysian Airlines and their wikipedia page.

First, just to illustrate how important Wikipedia is in general, the Malaysian Airlines Wikipedia page gets roughly 30,000 hits per month. That’s just one page of Wikipedia getting more hits than my entire website, I can’t tell you how frustrated that makes me.

Having a negative sounding Wikipedia page is pretty bad for business, particularly if 30,000 potential customers view it every month. That’s a web page that needs some serious attention if you’re the marketing manager of Malaysian Airlines.

Unfortunately for MAS (and every business organization there is), Wikipedia has a policy about updating your own Wikipedia page–you’re not allowed to do it. Wikipedia has to keep to it’s original intention of being an online repository of information that is fair, balanced and neutral. Having marketing gurus or corporate big wigs updating their own Wikipedia entry isn’t exactly in the best intentions of anyone, however Wikipedia doesn’t strictly enforce the policy and leave it up to the crowd.

Fortunately, the crowd have responded, sites like WikiScanner allow users to see which IP addresses updated which Wikipedia articles. Some have gone to the extent of correlating those IP addresses to the owners and determining if companies are updating their own Wikipedia pages against the general guidelines. Let’s see if Malaysian Airlines can join that group of companies who’ve been slapped on the wrist for changing the Wikipedia pages of their organizations.

Wikipedia from a Malaysian perspective

Wed, 18 Jul 2012 04:00:34 +0000

Wikipedia is quite possibly the greatest repository of information mankind has ever seen. It’s built around an amazing concept of allowing anyone the ability to create, document and moderate information in real-time, and so far the concept has proven successful–some may even argue that it’s too successful.

For the past two days, I’ve been writing about Bigquery and Big Data in general, and for the most part I’ve been using the freely available wikipedia dataset in Bigquery to perform some queries and analysis. The results were so interesting, that they warrant a post on their own–and this is that post!

For instance, I was curious who Aiman Abmajid was. For those who aren’t following the blog, Aiman is the undisputed King of Wikipedia in Malaysia. Aiman has single-handedly helped update Malaysian articles on Wikipedia a mind-blowing 13 THOUSAND times–and that’s just the English articles. Almost 6 times more than his closest Malaysian rival.

I was intrigued as to who he was and why was he updating so many Wikipedia entries (some more than 900 times per article), and more I dug the more intriguing it got.

A quick Google search, brought me his Wikipedia which led me to the following:

What is big data

Mon, 16 Jul 2012 07:00:13 +0000

It’s obvious that people have gotten bigger these past few decades, what’s less obvious is how data has grown bigger in the past few years. In fact, 90% of the digital data we have today, was created in the last 2 years. Put another way, in 2010 we had just 10% of the digital data we have today.

In 2011, an estimated 1.2 TRILLION Gigabytes of data was created. That’s roughly 200GB for every man women and child in the world–In just one year. That’s every person in the world watching almost 300 feature length films every day, and this is the average.

The reason is simple, we now keep digital records of our transactions (e-banking and credit cards), our running patterns, our spending habits and even our wedding photos–and that’s just commercial end user applications.

What about corporations who track thousands of data points per second for their manufacturing plants and supermarkets tracking the purchases of customers. We’re creating and gobbling far more data than before, and the trend doesn’t look to be stopping. Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone.