I came across a really cool youtube video from the amazing numberphile series that detailed how companies are already analyzing sports and farming huge amounts of data from sport events like football matches. In fact, these guys are basically farming data from events like the world cup, and then hoping that among that mountain of data lies some insights that could prove useful to football clubs–or football bookies :).

However, this little directive provoked my thoughts, because I’ve always been intrigued by the ‘weeding’ effect of censorship. The ‘weeding’ effect is a simple analogy I came up with while I was –you guessed it– weeding my garden. You see I’ve got a small garden in my home, and every now and then I put a pair of pink rubber gloves and go weeding around by hand, it’s a tough job, but someone has to do it. Now for those of you who’ve weeded anything before you know those nasty little weeds tend to grow in between the grass, and it’s really difficult to pick them up without plucking a fair bit of non-weeds with them. In fact, if you’ve got a lawn like mine–it’s almost impossible to get rid of the weeds without getting rid of the lawn grass as well. You most definitely want to avoid plucking out that expensive lawn grass you laid down.

The same goes with censorship, every time you try to censor something like the word ‘Breast’, you may inadvertently censor out something entirely innocent and useful–like Breast Milk, or Breast cancer, or Breast feeding. So while I really doubt the keyword Breast would lead to anything other than porn for the first 10,000 entries on Google, censoring the word Breast is really an ineffective solution because it could censor out a lot of really useful and relevant information.

A couple of days back, I decided to buy a Christmas present for my wife. In my usual lackadaisical procrastinating style however–I only decided to buy on the 21st of December, which is barely 4 days from Christmas.

In my defense,  I already had a great Christmas present in mind–it was something called the fitbit, a full fledge pedometer that not just records your steps everyday, but also your sleep time and tracks your daily activity. The best part is that the fitbit stores your data on the cloud, where you can analyze it to your hearts content. Not a bad present for a wife whose far more physically active than me :). (does blogging while standing count as a physical activity??)

There was however one problem–the fitbit isn’t available in Malaysia. I tried and I tried, but I couldn’t even find a lowyat seller who had it in stock. Now we really did have a problem.

Fortunately, I found one on Amazon–and Amazon did ship to Malaysia. So after a couple of credit card details were punched in (and a full USD30 for shipping) I manage to place my order on Amazon. The Amazon website promised 2-4 day delivery time-frame but stipulated in no uncertain terms that the order would not reach me before Christmas.

As a tech blog in Malaysia, I thought it’d be interesting to see the latest Auditor-General’s report faired in terms of IT spend from the government. IT spend is a tricky thing, and most don’t understand just how tricky it is, particularly around big IT spend by governments–they often fail. In fact, one of my favorite blogs is dedicated solely to IT failures, aptly titled–IT Project failures.

However, even the Synopsis report of the AG report is a harrowing 87 pages long. It’s not just the length that puts of me off, but rather the sheer dry-ness of the language that is used. Interestingly, not a single diagram exist in the documentation filled with enough monotone text to put even the most ardent auditor to sleep, and I’m no auditor so I nearly dozed off after the 2nd page. I had to take a different approach if I was to get a synopsis of the synopsis, fortunately I work in IT (not auditing or law), and I know of function in Adobe Acrobat that let’s you quickly search a document–it’s called the FIND function, and I was a deadly ninja in the art of the FIND.

So, armed with the FIND function on Adobe Reader, I combed through the document looking for the word ‘system’ and where it tied with an actual IT system too see just how well our government was in delivering IT systems in 2011. Below are just a few paragraphs pertaining to the AG’s report and below are 2 prime examples of the the magnitude of IT failures from Putrajaya.

The story isn’t a typical one, but one that exist in a nuance variety even in Malaysia. Amanda was tricked into exposing herself in front of a webcam by an unknown person. Soon she was blackmailed and finally, photos or her were circulated to her entire school. What followed next was every bit as predictable as it is sad, she was ostracized by her friends and tormented by bullies, she even tells of how she switch schools–multiple times–even moving to a school in a different city!!

Yet, the bullies and torments followed here (aided and enabled by social networks), and Amanda must have reached her limit and at some point she eventually chose to take her own life.

Youtube has taken down the videos, but I felt Amanda’s story should be left for the world to see, as a stark reminder to all of us to look after our children, and I just hope you get to watch the embedded video before even this gets removed. I believe out of respect for Amanda–we should listen to the story she so desperately wanted to tell.

IBM have always been on the cutting edge of innovation, they’ve moved more becoming merely a computer company to  what is probably the first truly all encompassing technology company, they don’t just make fancy gadgets or shiny tinga-ma-jigs, they make actual solutions for real-world problems.

In 1996, IBM introduced the world to Deep Blue. Kasparov met Deep Blue and wasn’t impressed, he had no reason to be, he defeated Deep Blue 4-2, and walked away comfortably.

However, in 1997, IBM re-introduced the world to the 2nd version of Deep Blue (unofficially named Deeper Blue), and this time Kasparov was beaten –but not by much. Kasparov is the Tiger Woods, Pele and Michael Jordan of the Chess world, and he was beaten by a super computer with 11.38 GFLOPs of power.

In turns out though, we had nothing to be afraid off, Chess is after all a pretty simple game when you break it down, the number of possible moves are finite, together with the number of possible scenarios to play out. It’s not an easy game to master, but as it turns out playing chess is infinitely easier than just plain talking.

In fact, of all the talking games, Jeopardy seems the most difficult. At the end of this post, I will make an argument to show that Jeopardy – a simple talking game – is about 6,500 times more difficult than Chess (a game we often associate with genius). Turns out Kasparov has to bow to Ken Jennings.

Just like youtube, slideshare is a free service. However for larger presentations (with Hi-Def Photos) or even videos, you may need to buy the Pro Version which enables up to 100MB uploads per presentation.

I was looking for some detail on Maxis Fibre to Home service until I came across this while trying to to access the Maxis Customer Forum online:

In the early days of the internet, all the data flowing through was done in plaintext, this meant that everything flowing on the internet was fair-game for anyone to hijack and view. It was akin to sending postcards all around, all the post-men and intermediaries could view the entire contents of your messages because it was out there in the open, no need to open sealed envelopes. So everything from your letters to your uncle Bob or your resume for a new job or even your most intimate personal letters could only be sent via postcard–anyone could read it.

There was a strong requirement however to design a mechanism to encrypt data flowing through the internet, because unless you could encrypt data, personal and credit information couldn’t (or rather shouldn’t) have been trasmitted across the internet. So it was important that someone somewhere figure out how data on the internet could be encrypted to enable things like online shopping, social networking, even simple email. So sometime in the mid-90s Netscape (the default browser at the time was Netscape Navigator), took up the gauntlet and invented SSL.

At this point, I’m also reminiscing the days when browsers were actually pay-ware rather than freeware. Remember when Netscape Navigator Gold used to cost money?

There are other more popular tools for big data, but today we’ll focus on Google BigQuery for a very good reason. It’s the only one I know how to use.

Google BigQuery is a full fledge big data tool developed by google and stored on the cloud. There’s a lot more information you can glean from their presentation here. The short story is that Google created this tool online where you can analyze your bigdata for a per use fee, similar to other cloud offerings. Google currently charges $0.035 per GB of data processed or $35 per TB of data. That seems like a small fee, but it adds up pretty quickly, so for the moment bigdata and bigquery aren’t exactly end-user offerings.

I’m just going to quickly jump into a worked example of Google BigQuery before making some remarks. To use BigQuery, you’re first going to have to create an API project in Google and then go to https://bigquery.cloud.google.com

It’s obvious that people have gotten bigger these past few decades, what’s less obvious is how data has grown bigger in the past few years. In fact, 90% of the digital data we have today, was created in the last 2 years. Put another way, in 2010 we had just 10% of the digital data we have today.

In 2011, an estimated 1.2 TRILLION Gigabytes of data was created. That’s roughly 200GB for every man women and child in the world–In just one year. That’s every person in the world watching almost 300 feature length films every day, and this is the average.

The reason is simple, we now keep digital records of our transactions (e-banking and credit cards), our running patterns, our spending habits and even our wedding photos–and that’s just commercial end user applications.

What about corporations who track thousands of data points per second for their manufacturing plants and supermarkets tracking the purchases of customers. We’re creating and gobbling far more data than before, and the trend doesn’t look to be stopping. Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone.