CloudTrail Partitioner
#Network & Cloud#Cloud Security
A comprehensive cloud security platform that combines vulnerability management, compliance monitoring, and automated remediation capabilities through an agentless architecture to protect cloud infrastructure and applications.
This project establishes partitioned Athena tables for CloudTrail logs
This project establishes partitioned Athena tables specifically for your CloudTrail logs and ensures that the partitions are updated on a nightly basis.
It is based on work by Alex Smolen: Partitioning CloudTrail Logs in Athena
This content is based on the work of Alex Smolen, specifically his post titled 'Partitioning CloudTrail Logs in Athena.'
You can deploy the CDK app right away, but I recommend running it manually first. This approach allows you to confirm that everything is properly configured. Additionally, running it manually will create 90 days of partitions by default. In contrast, the nightly CDK process does not start until 0600 UTC and will only generate partitions for the current day and the next day.
For each account, tables are created with the naming format cloudtrail_000000000000, and a view is also established that combines all these tables.
Related projects: This project was based on the work of Alex Smolen.
This project is effective for many users, but
This project is effective for many users; however, when dealing with larger scales—approximately 100GB of Cloudtrail logs—the way Athena is utilized in this project encounters difficulties.
For this and other reasons, Alex has introduced a new project
For this and other reasons, Alex has introduced a new project called cloudtrail-parquet-glue. This project is detailed in his post titled Use AWS Glue to create CloudTrail Parquet partitions. Additionally, it addresses issues #13 and #14 related to this project.
