AWS Access Keys Leak in GitHub Repository and Some Improvements in Amazon Reaction
AWS access keys leak via public code repository is a quite known security problem. So common, that popular version control systems offer for free a dedicated service, which looks for hardcoded secrets. Specifically, I refer here to GitHub secret scanning service.
Without a doubt, it’s awesome that such a service was released, however, in this blog post I want to answer the following questions:
- how quickly attackers will detect the key leak?
- how many unique attempts to use the disclosed keys will occur?
- and finally… will Amazon automatically disable my leaked keys?
The experiment
I’ve created an IAM user with programmatic access named github_deployer and attached to it a very limited policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::rabbit-hole-entrance",
"arn:aws:s3:::rabbit-hole-entrance/*"
]
}
]
}On 4th November at 22:05 CET I pushed a commit with hardcoded github_deployer keys:
The Amazon reaction
In less than 60 seconds I got an email from Amazon notifying me about the compromised keys:
When I recall various reports about such keys leaks from 2019 and earlier (for example this story, or this one), the notification email was the ONLY action taken by Amazon. However, nowadays Amazon not only notifies you, but also automatically attaches to compromised user the AWSCompromisedKeyQuarantine policy, which basically explicitly denies some potentially dangerous iam, ec2, organizations, lambda and lightsail actions:
Furthermore, there was automatically created a support ticket for this case. 2 days after I got even a call from Amazon support, what really (positively) surprised me. The support consultant even analysed the logs from my AWS account and notify me about recently launched 3 EC2 instances - what is true, but those instances were spun up by a different IAM user, before the leak so there was no correlation.
Nevertheless, according to this resource I should see the actions coming from support.amazonaws.com in Event History, but I don’t. Fortunately, I can see their actions in CloudTrail, using the following commands:
#unpack the logs
$ find . -type f -exec gunzip {} \;#print all actions taken by the Amazon support
find . -type f -iname "*.json" -exec cat {} \; | jq -cr '.Records[]|[.eventTime, .eventName, .sourceIPAddress] |@tsv' | sort | grep support.amazonaws.com | uniq#some of the actions made by Amazon support team
2020-11-06T06:45:07Z AssumeRole support.amazonaws.com
2020-11-06T06:45:07Z DescribeSnapshots support.amazonaws.com
2020-11-06T06:45:09Z DescribeSpotInstanceRequests support.amazonaws.com
2020-11-06T06:45:15Z DescribeImages support.amazonaws.com
2020-11-06T06:45:18Z ListDomains support.amazonaws.com
2020-11-06T06:45:18Z ListOperations support.amazonaws.com
2020-11-06T06:45:19Z DescribeVolumes support.amazonaws.com
2020-11-06T06:45:23Z DescribeSpotFleetRequests support.amazonaws.com
2020-11-06T06:45:23Z ListUsers support.amazonaws.com
2020-11-06T06:45:24Z ListAccessKeys support.amazonaws.com
2020-11-06T06:50:22Z DescribeInstanceStatus support.amazonaws.com
2020-11-06T06:50:22Z DescribeInstances support.amazonaws.com
2020-11-06T06:50:23Z DescribeAddresses support.amazonaws.com
2020-11-06T06:50:24Z DescribeAccountAttributes support.amazonaws.com
2020-11-06T06:50:24Z DescribePlacementGroups support.amazonaws.com
2020-11-06T06:50:24Z DescribeSecurityGroups support.amazonaws.com
2020-11-06T06:50:24Z DescribeVpcs support.amazonaws.com
The events related with attaching the AWSCompromisedKeyQuarantine policy come from "sourceIPAddress": "AWS Internal".
The attackers’ reaction
My repository was publicly available for less than 2 days and TBH I expected dozens of bots trying to use the leaked keys. Surprisingly I logged “only” 3 such attempts (please note that the CloudTrail logs use GMT time so you should add an hour to convert it to CET timezone):
2020-11-04T21:11:25Z GetCallerIdentity 77.83.1.126
2020-11-04T21:11:26Z ListBuckets 77.83.1.118
2020-11-04T21:57:12Z GetSendQuota 3.8.143.95So, it took attackers ~6 minutes to find my keys on GitHub repository. 6 minutes seem to be pretty long for a bot, huh? This is because the GitHub public events are delayed by 5 minutes (thanks @Code_That_Rocks for pointing this!), what gives the owner and AWS extra time to react.
Maybe giving more permissions to the github_deployer user would encourage attackers to dig more? Maybe using a fresh and experiment-dedicated GitHub account was discouraging for attackers? If you did a similar experiment, but got different results, then please let me know in comments.
TBH I should also confess that I did the same experiment in GitLab repository, by committing same code, but this time exposing keys, which belong to gitlab_deployer IAM user. I wanted to see in action the GitLab Secret Detection service. Firstly, I was surprised that I haven’t received any notification email and then I was even shocked that NOBODY used the gitlab_deployer keys for a week! I’ve realised that the GitLab Secret Detection service requires GitLab runners and some configuration— still, it’s a valuable lesson for me that I should RTFM instead of assuming the service should work like the others.
My colleague did the similar experiment, and by coincidence in a very similar time. So, to give you another sample to compare the results, check out this thread:
Final thoughts
I definitely see a progress in a way how Amazon is handling the access keys leaks. Nevertheless, I think thatAWSCompromisedKeyQuarantine policy is far away from perfect protection, thus the bots will still be hunting for the keys leaks. Furthermore, the sources other than GitHub’s repositories are probably more attractive targets for attackers. Please let me know on my Twitter or LinkedIn if you want to make me similar experiment on other version control systems.
