6 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
This article details Yelp's approach to handling S3 server access logs at scale. It discusses the challenges of logging, the benefits of object-level logging for debugging and security, and the architecture used to optimize log storage by converting them into a more efficient format.
If you do, here's more
Yelp uses Amazon S3 to store a wide range of data, making it essential to manage access and security effectively. One significant challenge is gaining visibility into how data is accessed. Historically, enabling S3 server access logging was complicated and costly. The introduction of date-based partitioning in November 2023 improved the situation, allowing more efficient querying of logs. Object-level logging is now crucial for troubleshooting, identifying unused data, and demonstrating responsible data management to third parties.
S3 server access logs record API operations for each bucket and its objects. While logs are delivered on a best-effort basis, they can occasionally be delayed or missed. Despite this, they provide valuable insights for debugging access issues, attributing costs to specific services, and responding to security incidents. For example, using SQL queries, teams can analyze logs to find out which IAM roles are generating the most API calls or to investigate suspicious access patterns related to potential compromises.
To manage the high volume of logs generated daily, Yelp has adopted parquet format for log storage. This columnar format significantly compresses data, reducing storage size by 85% and the number of objects by 99.99%. Compaction processes help manage the sheer volume of logs while improving query performance. While AWS offers alternatives, such as CloudTrail Data Events, those come with much higher costs, making the self-managed approach more appealing. The architecture developed leverages Glue tables and aggregation techniques to efficiently identify and manage unused objects in their S3 buckets.
Questions about this article
No questions yet.