Owning a data center can be both a practical and strategic move for companies that rely heavily on computing resources. Comma.ai has spent approximately $5 million on their facility, which has saved them an estimated $20 million compared to cloud services. The blog emphasizes that using the cloud can lead to high costs and a lack of control over operations, prompting Comma.ai to take the self-reliant route. They run all their model training and data processing in-house, allowing for more efficient engineering and better performance. The data center setup includes 600 GPUs housed in 75 custom-built TinyBox Pro machines, along with several Dell servers providing about 4PB of SSD storage. The cooling system utilizes outside air instead of traditional power-hungry CRAC systems, which helps keep energy costs lower. Power consumption peaks at 450 kW, and the company faces high electricity rates in San Diego. The infrastructure incorporates tools like Slurm for workload management and a custom storage system called minikeyvalue for high-speed data access, enabling direct training on raw data without caching. For distributed computing, Comma.ai employs PyTorch for model training and a lightweight task scheduler named Miniray for various compute tasks. This flexibility allows them to run everything from tests to model inference efficiently across their machines. Their software architecture is designed for simplicity and speed, with a monorepo system for code management that ensures consistency across the board. The combination of these elements illustrates how a self-operated data center can deliver significant advantages in terms of cost, performance, and control.