Announcing the Databricks storage ecosystem: Governing the enterprise data estate, wherever it lives

Open original

Databricks is launching a Software-Defined Storage ecosystem that uses the open-source OpenSharing protocol to link on-premises, edge, and private-cloud systems directly into its Data Intelligence Platform. This zero-copy approach lets teams run serverless compute and train models on local datasets under Unity Catalog governance without migrating any data.

  • Databricks now lets you query on-prem/edge/private-cloud data (via MinIO, Everpure, Qumulo, VAST Data) directly through Unity Catalog with zero-copy access and no egress fees, using the open-source OpenSharing protocol
  • MinIO's AIStor integration is already GA, letting users run live queries on on-prem Iceberg and Delta tables; Everpure and Qumulo are in private preview, VAST Data joins in August
  • Targeted at regulated/data-sovereignty-heavy industries (banks, healthcare, semiconductors, trading firms) facing GDPR/HIPAA/NIS2 rules and untenable cloud egress costs at exabyte scale
  • Connections must pass Databricks's Partner Well-Architected Framework for security/certification before going live, preserving full lineage, access controls, and audit logs across hybrid environments

Enterprises no longer have to haul every byte to the public cloud to tap modern AI and governance. Databricks just rolled out its Software-Defined Storage Ecosystem, linking on-premises, private-cloud, and edge storage directly into its Data Intelligence Platform via the open-source OpenSharing protocol. Big names like MinIO (now GA), Everpure, Qumulo and VAST Data are onboard in various preview stages, exposing petabyte-scale Apache Iceberg™ and Delta tables under Unity Catalog controls without any data copy or egress fees. Regulated industries drove this shift. Semiconductor firms guard classified engineering data. Banks and healthcare providers face GDPR, HIPAA, NIS2 and other residency rules. Trading houses drown in historic tick records that cost too much to move. At exabyte scale, cloud storage and egress fees spiral out of control. And edge workloads—from retail point-of-sale to telecom network telemetry—need millisecond response times you can’t get if data lives off-premises. Under the new model, storage vendors stand up an OpenSharing endpoint. You hook it into Unity Catalog and instantly gain zero-copy, governed access to on-site data from Databricks Serverless Compute, Genie, AgentBricks and built-in LLMs. No complex pipelines, no duplicate datasets, no compliance headaches. The integration follows Databricks’s Partner Well-Architected Framework, ensuring each connection meets security and certification standards before going live. Customers are already testing these links in production. MinIO’s AIStor bridge is GA, enabling live queries on on-prem Iceberg and Delta tables. Everpure and Qumulo roll out private previews, with VAST Data joining in August. Each integration brings the same Unity Catalog view across hybrid estates, so data scientists and engineers treat on-prem data just like cloud data—complete with lineage, access controls and audit logs.

Ask about this article