Just because you have webhook, doesn't mean you need a webserver.

With serverless AWS Lambdas you've got a free (as in beer) and always on ability to receive webhooks callbacks without the need for pesky servers. In this post, I'll setup a serverless solution to accept incoming POST from a GitHub webhook.

This post is a very quick brain-dump  of stuff I did over the weekend, in the hopes that I don't forget it :). Will post more in-depth material if time permits over the weekend.

govScan.info, a site I created as a side hobby project to track TLS implementation across .gov.my websites -- now tracks DNS records as well. For now, I'm only tracking MX, NS, SOA and TXT records (mostly to check for dmarc) but I may put more record types to query.

DNS Records are queried daily at 9.05pm Malaysia Time (might be a minute or two later, depending on the domain name) and will be stored indefinitely. Historical records can be queried via the API, and documentation has been updated.

The security community has been abuzz with an absolutely shocker of story from Bloomberg. The piece reports that the Chinese Government had subverted the hardware supply chain of companies like Apple and Amazon, and installed a 'tiny chip' on motherboards manufactured by a company called Supermicro. What the chip did -- or how it did 'it' was left mostly to the readers imagination.

Supermicro's stock price is down a whooping 50%, which goes to show just how credible Bloomberg is as a news organization. But besides the Bloomberg story and the sources (all of which are un-named), no one else has come forward with any evidence to corroborate the piece. Instead, both Apple and Amazon have vehemently denied nearly every aspect of the story -- leaving us all bewildered.

But Bloomberg are sticking to their guns, and they do have credibility -- so let's wait and see. For now, let's put this in the bucket called definitely could happen, but probably didn't happen.

I can only imagine how hard it must be to secure a modern hardware supply chain, but the reason for this post is to share my experience in some supply chain conundrums that occurred to a recent project of mine.

I operate (for fun) a website called GovScan.info,  a python based application that scans various gov.my websites for TLS implementation (or lack thereof). Every aspect of the architecture is written in Python 3.6, including a scanning script, and multiple lambda functions that are exposed via an API, with the entirety of the code available on github.

And thank God for GitHub, because in early August I got a notification from GitHub alerting me to a vulnerability in my code. But it wasn't a vulnerability in anything I wrote -- instead it was in a 3rd-party package my code depended on. 

From my previous post, you can see that I hosted a slide show on a subdomain on hitbgsec.keithrozario.com. The site is just a keynote presentation exported to html format, which I then hosted on an S3 bucket.

The challenge I struggled with, was how to point the domain which I hosted on Cloudflare to the domain hosting the static content.

The recommended way is to just create a simple CNAME entry and point it to the S3 bucket, but that didn't work because the 'crypto' settings on Cloudflare apply to the entire domain -- and not individual subdomains.

And since my website at www.keithrozario.com had a crypto setting of 'Full', the regular CNAME entry kept failing. I could have downgraded to 'Flexible' but that would mean my blog would be downgraded as well -- which wasn't ideal.

Why downgrade my main blog to accommodate a relatively unimportant sub-domain.

Instead found that the solution is to overlay a CloudFront Distribution in front of S3 Bucket -- and then point a CNAME entry to the Distribution.

The solution looks something like this:

I haven’t blogged in a long while – but I have a good(ish!) excuse.

I spent most of August prepping for the #HITBGSEC conference in Singapore. It was my first time presenting at a security conference, and I had an absolute blast. The output of the countless hours I spent is in the embedded youtube video below, and the presentation material can downloaded here[.key] and a html version here (it’s very sluggish, as it’s hosted behind cloudflare + cloudfront + s3 – and it’s 100MB in size)

On the 20th of July, Singaporean authorities announced a data breach affecting SingHealth, the country largest healthcare group. The breach impacted 1.5 million people who had used SingHealth services over the last 3 years.

Oh boy, another data breach with 1.5 million records … yawn.

But Singapore has less than 6 million people, so it’s a BIG deal to this island I currently call home. Here’s what happened.

The lowdown

4-Jul : IHiS’ database administrators detected unusual activity on one of SingHealth’s IT databases 10-Jul : Investigations confirmed the data breach, and all relevant authorities were informed 12-Jul : A Police Report is made 20-Jul : A public announcement is made

The point of entry was ascertained to be “that the cyber attackers accessed the SingHealth IT system through an initial breach on a particular front-end workstation. They subsequently managed to obtain privileged account credentials to gain privileged access to the database”

And finally that “SingHealth will be progressively contacting all patients…to notify them if their data had been illegally exfiltrated. All the patients, whether or not their data were compromised, will receive an SMS notification over the next five days”

Recently, there was a poorly written article in The New Straits Times, that suggested the Malaysian Police would know if you were watching porn online.

Let me cut to the chase, the article is shit.

The software in question, aptly named Internet Crime Against Children Child Online Protective Services (ICACCOPS) is used to detect Child Pornography, and Child Pornography only – as the name clearly implies. It is a collaborative effort by Law Enforcement agencies, and is shared with PDRM, probably as a gesture of good will, and also a collaborative effort.

Gov-TLS-Audit got a brand new domain today. No longer is it sharing a crummy domain with sayakenahack (which is still blocked in Malaysia!), it now has a place to call it’s own.

The domain cost me a whooping $18.00/yr on AWS, and involved a couple hours of registration and migration.

So I felt that while migrating domains, I might as well implement proper security headers as well. Security Headers are HTTP Headers that instruct the browser to deny or allow certain things, the idea being the more information the site tells the browser about itself, the less susceptible it is to attack.

I was shocked to find out that Gov-TLS-Audit had no security headers at all! I assumed AWS (specifically CloudFront) would take care of ‘some’ http headers for me – I was mistaken. Cloudfront takes care of the TLS implementation, but does not implement any security header for you, not even strict-transport-security which is TLS related.

So unsurprisingly, a newly created cloudfront distribution, using the reference AWS implementation, fails miserably when it comes to security headers.

I guess the reason is that HTTP headers are very site-dependant. Had Cloudfront done it automatically, it might have broken a majority of sites And implementing headers is one thing, but fixing the underlying problem is another – totally bigger problem.

But what security headers to implement?

As Malaysians woke up today, to a brand new cabinet of Ministers, many have already begun expressing their dissatisfaction on the lineup. I know better than to wade into these politically charged discussions – but I will point out that my people have long been overlooked for Ministerial positions.

Who are ‘my people’ you ask…

Hackers.

Or if you prefer a less negative word – Geeks. But for the rest of this post, I’ll use the more accurate term of hacker to refer to technically savvy folks who subscribe to the hacker ethic.

Yes, we in the hacker community have long been overlooked for ministerial positions, and I for one, choose to speak out against this travesty. But before I delve into why I think we’ve not played a bigger part in politics, let me first make the case for why we need hackers in parliament.

Why we need hackers in parliament

But it’s not enough that policy makers merely understand technology, they also need to subscribe to the hacker ethic , and bring that ethic into the decisions they make.

What is the hacker ethic? Well I’m glad you asked.

The ethic has no hard definition, but it incorporates things like Sharing, Openness, Decentralization and Free access to computers, etc. The ethic further includes attitudes, like pure meritocracy, the idea that hackers should be judged for their hacking (and nothing else), not age, gender, degrees or even position in a hierarchy. So anytime you see some poor sod who claims to be a hacker, but puts CISSP, PMP, CEH at the end of their LinkedIn profile – you know they’re not really hackers.

You can see ethic played out at hacker conferences throughout the world, hackers are ever willing to share what they’ve built with anyone who’ll listen, and they’re accepting of anyone willing to learn, at any age bracket, without any education or formal training.

The Hacker perspective is an interesting one, and like all perspectives, may not always be right or appropriate, but it’s important for it to be present at the decision making process, if nothing more than to add to the diversity of thought.

So why aren’t there more hackers in decision making levels? Well let’s see what it takes to reach the decision making level in the first place.

[caption id=“attachment_6436” align=“aligncenter” width=“550”]

Boy, that’s a lot of RED![/caption]

Last week, MyNic suffered a massive outage taking out any website that had a .my domain, including local banks like maybank2u.com.my and even government websites hosted on .gov.my.

Here’s a great report on what happened from IANIX. I’m no DNSSEC expert, but here’s my laymen reading of what happened:

1. .my uses DNSSEC
2. Up to 11-Jun,.my used a DNSKEY with key tag:25992
3. For some reason, this key went missing on the 15-Jun, and was replaced with DNSKEY key tag:63366. Which is still a valid SEP for .my
4. Unfortunately, the DS record on root, was still pointing to key tag:25992
5. So DNSSEC starting failing
6. 15 hours later, instead of correcting the error, someone tried to switch off DNSSEC removing all the signatures (RRSIG)
7. But this didn't work, as the parent zone still had a DS entry that pointed to key tag:25992 and hence was still expecting DNSSEC to be turned on.
8. 5 hours after that, they added back the missing DNSKEY key tag:25992 (oh we found it!), but added invalid Signatures for all entries -- still failing.
9. Only 4 hours after that did they fix it, with the proper DS entry on root for DNSKEY key tag:63366and valid signatures.
10. That's a 24 hour outage on all .my domains.

I defend my fellow IT practitioners a lot on this blog, but this is a difficult one. Clearly this was the work of someone who didn’t know what they were doing, and refused to ask for help, instead tried one failed fix after another which made things worse. As my good friend Mark Twain would say – it’s like a Mouse trying to fix a pumpkin.

I don’t fully understand DNSSEC (it’s complicated), but I’m not in charge of a TLD. It’s unacceptable that someone could screw up this badly – and for that screw up to impact so many people, and all we got was a lousy press release.

The point is, it shouldn’t take 24 hours to resolve a DNSSEC issue, especially when it’s such a critical piece of infrastructure. I’ve gone through reports of similar DNSSEC failures, and in most cases recovery takes 1-5 hours. The .nasa.gov TLD had a similar issue, that was resolved in an hour, very rarely do we see a 24 hour outage, so what gives?

I look forward to an official report from MyNIC to our spanking new communications ministry, and for that to be shared to the public.