If you've been working in AWS for long enough, you will know that nothing good comes from configuring resources in the console aka ClickOps. That being said, having a hard and fast rule that everybody should only have ReadOnly access in the console is also not great. I wanted something that would trigger when people are taking manual actions in the console and alert the team to investigate why this was done and what needs to be done to get our IaC deployment in sync with these changes.
For this reason, I've created ClickOops, a simple Lambda that monitors your CloudTrail log files to find manual actions taken in your accounts.
Filtering which events should be classified as a manual action is more involved than you might think. This post by Arkadiy Tetelman goes into detail about what to look for in CloudTrail events, and a sample Python implementation can be found in Towards Data Science's blog.
Deploying ClickOops
Everything you need to deploy this can be found on my GitHub page.
We use AWS ControlTower (CT) to manage our multi-account environment and guardrails. CT sets up a Log Archiving account where all CloudTrail and Config logs are forwarded to (for all accounts and regions in your organisation). This gives you a central place to process these logs without having to worry about deploying (and protecting) these types of solutions across many accounts.
If you are not using CT, you can deploy this for a single account and point it to your CloudTrail logging bucket in the account. Ensure that the key of the files in S3 follows this pattern:
*/{{account_id}}/CloudTrail/*/*.json.gzYou will also need a Slack App configured with Incoming Webhooks. I prefer to configure incoming webhooks to channels that will only be used for one specific purpose to prevent messages from getting lost in busy channels.
After deploying the solution you need to update the SSM parameter containing the Slack Webhook URL. Ironically this is a ClickOps action and you will be notified about this.
ClickOops messages will contain a summary of the IAM action and AWS identity (which should always be a human) which triggered the event. The full CloudTrail event will also be included in the message.
Future work
As AWS releases new services we will need to update the list of CloudTrail events that are allowed to be taken in the console and filter out false positives as I find them. It might also be handy to group accounts and send messages related to them to a separate Slack channel.
I'd love your feedback if you are using something like this in your environments to monitor how users are interacting with AWS, or if you found this helpful.
Updates
[2022-03-15] We've created a standalone Terraform module for this that is easier to consume and has a few bonus features 🤩
We help teams keep their infrastructure-as-code the single source of truth — guardrails, drift detection, and the FinOps to back it up.
Talk to our cloud team →