Datology is tackling the challenge of managing petabytes of image data through innovative distributed pipelines. With the limitations of existing pre-training datasets, the company believes there’s still potential to enhance AI model training by improving data selection and manipulation. Their research has demonstrated that effective data curation can yield state-of-the-art results without new architectural changes. Their recent achievements include training faster, smaller CLIP models using a 600TB dataset known as DataComp XL, which contains 12.8 billion text-image pairs. The engineering challenges are significant. Datology’s team must design pipelines that support complex data operations like deduplication and filtering while enabling researchers to build customizable workflows. The scale of their datasets means that traditional methods of data manipulation fall short. For instance, the naive approach of comparing every image against every other image would result in impractical computational demands, given the scale of 11 billion images. Instead, they’ve devised more efficient methods to deduplicate data, particularly identifying common problematic entries like "Image Not Found." Another layer of complexity involves adapting their systems to work across various customer environments, which often lack access to cutting-edge hardware like NVIDIA's H100 GPUs. Datology's solutions need to perform well on both high-end and standard hardware. They utilize frameworks such as Spark and Ray to make their operations efficient, ensuring that their pipelines are not only robust but also economically viable for real-world applications. This dual focus on research enablement and customer deployment underpins their engineering efforts.