After reading the awesome DynamoDBBook from Alex DeBrie, I was prompted to fix a long running design issue with Klayers (a separate project I maintain).

Like everybody else that dives into DynamoDB headfirst, I made the mistake of using multiple tables, one for each data entity. After all, a single database consists of multiple tables -- so DynamoDB would logically involve multiple DynamoDB tables as well right?

Wrong!

It turns out, a DynamoDB table is equivalent to a database, and having multiple tables is like having multiple databases, The 'correct' approach, is to load all data into a single DynamoDB table which would allow us to "join" multiple data entities into a single query.

The word "join" is in quotations, as there is no concept of joining data in DynamoDB, all data has to be pre-joined in some way to achieve the performance that DynamoDB promises (sub 10ms response times for tables of any size). If you split your data across multiple tables, you lose the ability to pre-join this data.

So last month I decided to bite the bullet and began re-designing my application to use one table instead of two, and boy did it do my head-in, and wanted to write this post to capture my thoughts on the whole process.

First here's some background of the application.

Let's look at AWS Access Keys inside a Lambda function, from how they are populated into the function's execution context, how long they last, how to exfiltrate them out and use them, and how we might detect an compromised access keys.

But before that, let's go through some basics. Lambda functions run on Firecracker, a microVM technology developed by Amazon. MicroVMs are like docker containers, but provide VM level isolation between instances. But because we're not going to cover container breakouts here, for the purpose of this post we'll use the term container to refer to these microVMs.

Anyway...

Lambda constantly spins up containers to respond to events, such as http calls via API Gateway, a file landing in an S3 bucket, or even an invoke command executed from your aws-cli.

These containers interact with AWS services in the same exact way as any code in EC2, Fargate or even your local machine -- i.e. they use a version of the AWS SDK (e.g. boto3) and authenticate with IAM access keys. There isn't any magic here, it's just with serverless we can remain blissfully ignorant of the underlying mechanism.

But occasionally it's a good idea to dig deep and try to understand what goes on under the hood, and that's what this post seeks to do.

So where in the container are the access keys stored? Well, we know that AWS SDKs reference credentials in 3 places:

• Environment Variables
• The ~/.aws/credentials file
• The Instance Metadata Service (IMDS)

If we check, we'll find that our IAM access keys for lambda functions are stored in the environment variables of the execution context, namely:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

You can easily verify this, by printing out those environment variables in your runtime (e.g. $AWS_ACCESS_KEY_ID) and see for yourself.

OK, now we know where the access stored keys are stored, but how did they end up here and what kind of access keys are they? For that, we need to look at the life-cycle of a Lambda function...

I thought I'd write down my thoughts on contact tracing apps, especially since a recent BFM suggested 53% of Malaysians wouldn't download a contact tracing app due to privacy concerns. It's important for us to address this, as I firmly believe, that contact tracing is an important weapon in our arsenal against COVID-19, and having 54% of Malaysians dismiss outright is concerning.

But first, let's understand what Privacy is.

Privacy is Contextual

Privacy isn't secrecy. Secrecy is not telling anyone, but privacy is about having control over who you tell and in what context.

For example, if you met someone for the first time, at a friends birthday party, it would be completely rude and unacceptable to ask questions like:

• What's your weight?
• What's your last drawn salary?
• What's your age?

In that context you're unlikely to find someone who will answer these questions truthfully.

But...

Age and weight, are perfectly acceptable questions for a Doctor to ask you at a medical appointment, and your last drawn salary is something any company looking to hire you will ask. We've come to accept these questions as OK -- under these contexts.

You might still not want to answer them, which might mean you don't get the job, or the best healthcare -- but you certainly can't be concerned by them. Far more people will answer these same questions truthfully if you change the context from random stranger at a party to doctors appointment.

So privacy is contextual, to justify concerns we have to evaluate both the context and the question before coming to a conclusion.

So let's look at both, starting with the context:

Potassium40 was a project I started to see how fast Lambda could really go. The project attempts to download the robots.txt files from 1 million websites as fast as it can. I chose robots file because -- well it's supposed to be downloaded by robots anyway, so this was both great fun, but also completely ethical as I wasn't scraping people's websites.

This post covers how to perform logging within AWS Lambda. Lambda has built-in integration to Cloudwatch logs, making it a default choice for logs, but the way a distributed system like lambda performs logging, is quite different from how you'd do in a monolithic app.

For the brave folks still reading this -- let's dive in.

Lambda Logging: 101

When building lambda functions, it's always a good idea to sprinkle logging statements throughout your code -- particularly in areas where something important happens. This is just generally good advice regardless of whether you're using Lambda.

A lambda function is a like a little island, surrounded by network. Unlike Fargate containers, of EC2 instances, they do not have EFS, EBS or some other fast storage support. Everything that goes into a lambda, goes in via the network interface (and network only).

Last week I took the AWS Certified Security - Specialty exam -- and I passed with a score of 930 (Woohoo!!)

In this post I cover why I took it, what I did to pass, my overall exam experience, and some tips I learnt along the way.

So let's go.

Why?

Why would anybody pay good money, subject themselves to hours of studying, only to end up sitting in a cold exam room for hours answering many multiple choice questions!

And the reward for that work is an unsigned PDF file claiming you're 'certified', and 'privilege' access to buy AWS branded notebooks and water bottles!! Unless those water bottles come with a reserved instance for Microsoft SQL server in Bahrain, I'm not interested.

But, jokes asides, I did this for fun and profit, and fortunately I really did enjoy the preparing for this exam. It exposed me to AWS services that I barely knew -- and forced me to level-up my skills even on those that I knew.

The exam has a massive focus on VPC, KMS, IAM, S3, EC2, Cloudtrail and Cloudwatch. While lightly touching Guardduty, Macie, Config, Inspector, Lambda, Cloudfront, WAF, System Manager and AWS Shield.

You need to catch you breath just reading through that list!

But for those diligently keeping count -- you'd notice that the majority of those services are serverless -- meaning the exam combined my two technological love-affairs ... security and serverless!

I wasn't lying when I said it was fun. So what about the profit.

I'm not sure how good this would be for my career (I literally got the cert last week), but for $300, it's is relatively cheap, with a tonne of practical value. So trying to get an ROI on this, isn't going to be hard.

For comparison, the CCSP certification cost nearly twice as much, is highly theoretical and requires professional experience.

The results also help me validate my past years of working on serverless projects, proving I wasn't just some rando posting useless hobby projects on GitHub. Instead, I'm now a certified AWS professional, posting useless hobby projects on GitHub (it's all about how you market it!)

So now that we've covered the why, let's move onto how.

GitHub actions is the new kid on the workflow block.

It allows users to orchestrate workflows using familiar git commands like push & pull requests, and un-familiar GitHub events like gollum, issue creation and milestone closures.

In this post, we'll use GitHub actions to orchestrate a build pipeline that will deploy lambda functions using the Serverless framework. There's a lot of tutorials that cover the basics of doing this, but we'll dive deeper and cover off framework plugins and deploying to different environments using feature branches.

In my honest (and truly humble) opinion, VPCs don't make much sense in a serverless architecture -- it's not that they don't add value, it's that the value the add isn't worth the complexity you incur.

After all, you can't log into a lambda function, there are no inward connections allowed. And it isn't a persistent environment, some functions may timeout after just 2-3 seconds. Sure, network level security is still worthy pursuit, but for serverless, tightly managing IAM roles and looking after your software supply chain for vulnerabilities would be better value for your money.

But if you've got a fleet of EC2s already deployed in a VPC, and your Lambda function needs access them. Then you have no choice but to deploy that function in a VPC as well. Or, if your org requires full network logging of all your workloads, then you'll also need VPC (and their flow logs) to comply with such requests.

Don't get me wrong, there is value in having your functions in a VPC, just probably not as much as you think.

Put that aside though, let's dive into the wonderful world of Lambda functions and VPCs

Working Example

First, imagine we deploy a simple VPC with 4 subnets.

1. A Public Subnet with a Nat Gateway inside it.
2. A Private Subnet which routes all traffic through that NAT Gateway
3. A Private Subnet without internet (only local routing)
4. A Private Subnet without internet but with a SSM VPCe inside it

Let's label these subnets (1), (2) ,(3) and (4) for simplicity.

Now we write some Lambda functions, and deploy each of them to each subnet. The functions have an attached security group that allows all outgoing connections, and similarly each subnet has very liberal NACLs that allow incoming and outgoing connections.

Then we create a gateway S3 VPC-endpoint (VPCe), and route subnet (4) to it.

Finally, we enable private DNS on the entire VPC. And then outside the subnet we create a bucket and an System Manager Parameter Store Parameter (AWS really need better terms for these things).

The final network looks like this: