Yelp developed a Back-Testing Engine to optimize its Ad Budget Allocation process, which involves distributing campaign budgets between on-platform inventory (like their website and app) and off-platform inventory (the Yelp Ad Network). The system needs to adapt dynamically to the performance of ad campaigns, as even minor adjustments can create significant ripple effects. The Back-Testing Engine allows for simulation of proposed changes using historical campaign data without risking real budgets, helping the team evaluate the potential impact of algorithm changes before implementing them. The engine consists of eight components that work together for simulations. It starts with a parameter search space defined in a YAML file, where users specify the parameters to tune, such as algorithm choices and thresholds. The optimizer, built on the Scikit-Optimize library, selects promising candidates for testing. It iteratively proposes different combinations of parameters, learning from previous simulations to enhance performance metrics like cost per lead. This structured approach enables Yelp to safely innovate and refine its ad allocation strategies while maintaining advertiser trust and improving overall revenue. Each day, Yelp’s system evaluates campaign performance based on past results, ultimately impacting future budget decisions. The Back-Testing Engine mirrors this daily process, allowing the team to forecast outcomes and explore new ideas more confidently. This setup highlights how Yelp is leveraging technology to make data-driven decisions in a complex advertising environment, ensuring they meet advertiser goals while maximizing their own revenue streams.