Apache Fluss · Streaming Storage for the Real-Time Lakehouse

Open original

Apache Fluss released version 1.0 with a new REST Gateway component that lets AI agents access real-time data via HTTP, plus storage improvements for managing context and feature data at scale. The release adds 450,000 lines of code and brings the project from incubation to a mature top-level Apache project.

  • Gateway component written in Rust provides HTTP/REST access to Fluss for agents and applications without requiring native SDKs, with plans to add Model Context Protocol and PostgreSQL wire protocol support.
  • New storage features include row-level TTL for primary key tables, predicate pushdown to skip unnecessary reads at the storage layer, and bucket rescaling to adjust partition capacity as workloads change.
  • Expanded engine support adds Flink bitmap functions, Hudi integration for lakehouse access, and Spark incremental reads, plus production-grade features like coordinator high availability and remote orphan file cleanup.

Apache Fluss 1.0 is a major release of a real-time data system designed to feed AI agents and applications with current information. The project graduated from Apache Incubator status and shipped with over 1,000 commits and 450,000 new lines of code—roughly triple previous releases. The core problem it solves is straightforward: AI systems need fast access to user profiles, recent events, and business state for inference, while training pipelines need both historical data and fresh events. Fluss 1.0 addresses this by expanding how different tools can reach the data, from simple HTTP calls for agents to native Python integration to continuous processing with Flink and Spark. The headline feature is a new Gateway component written in Rust that translates REST requests into Fluss operations. Agents can now list tables, inspect schemas, create tables, and write results back without needing a native SDK—just standard HTTP POST requests. The Gateway runs without a JVM, stays stateless so multiple instances can sit behind a load balancer, and can batch records into a separate Lakehouse system within minutes for deeper historical access. Beyond HTTP, Fluss added native Python client support for looking up context and features, alongside existing Rust and C++ clients. The team also plans to add Model Context Protocol (MCP) support so agents can use Fluss through standard tool interfaces, with PostgreSQL wire protocol and gRPC coming later. On the storage side, Fluss 1.0 introduced fine-grained control over data retention. Different use cases need different windows—agents might want only a few hours of hot data, recovery pipelines need days of logs, and user profiles follow business rules. The system now supports row-level TTL on primary key tables, independent local log TTL (so older logs move to remote storage while staying readable), and dynamic updates to these settings. A new feature called predicate pushdown lets the storage layer skip entire batches of data that can't match a query filter, using column statistics like min/max values to avoid unnecessary transfers and deserialization. This matters for feature computation and training data prep, where you often need only a slice of events—say, temperature readings above 30 from a specific warehouse.

Ask about this article