Redpanda, the real-time data platform for the agentic enterprise, today announced general availability of Apache Iceberg Topics, which enables organizations to query streaming data as Iceberg tables without having to create ETL/ELT pipelines. Redpanda is the first company to offer this functionality in a “Bring Your Own Cloud” (BYOC) environment and for all three major cloud providers – AWS, GCP, and Azure. Apache Iceberg Topics is a critical component of Redpanda’s recently announced agentic AI service for the enterprise as it allows businesses to seamlessly blend real-time operational insights with historical data, all within their existing data lakehouse environments.
“Iceberg has become a groundbreaking technology for organizations to unify their analytical and operational data substrate,” said Alex Gallego, founder and CEO of Redpanda. “Redpanda has been a pioneer in giving enterprise data teams access to the best-in-class tools that automate away complexity and give the best data experiences. Iceberg will be a key technology in the future of agentic AI, serving as the backbone to bring streaming data to power next-generation, industry-defining real-time and AI applications.”
Apache Iceberg has emerged as a critical technology for modern data architectures, addressing inherent limitations within traditional data lakes. By standardizing the metadata that defines groups of files as tables, Iceberg gives data lakes the same benefits as a data warehouse including transactional integrity, portability, and performance, while allowing companies to use multiple query engines and data engineering tools of their choice.
Redpanda’s Iceberg Topics allows users to create Iceberg tables from Redpanda with a simple topic property setting. These tables are then automatically registered with any of the major Iceberg REST catalogs such as Snowflake Open Catalog, AWS Glue, Nessie, and others. Redpanda also provides custom partitioning to improve query performance and built-in dead letter queues that filter invalid data, in addition to automatic schema translation and full fidelity schema evolution for data formats such as Protobuf and Avro. Seamless integration with Iceberg-compatible tools eliminates the need for complex data pipelines and reduces the time to real-time insight by making all streams immediately and transparently available to analysts familiar with existing SQL tooling and environments.
“As LiveRamp continues to build and scale solutions that fuel innovation for our clients, Redpanda has been a key partner in strengthening our streaming data infrastructure and operations,” said Mohsin Hussain, CTO at LiveRamp. “With the new Iceberg integration, we expect to drive faster analytics, simplify data governance, and enhance interoperability, laying the groundwork for more intelligent, AI-driven customer experiences and delivering even greater customer value.”
Key Benefits of Redpanda’s Iceberg Topics:
-
Simplified Data Access: Create Iceberg tables directly from Redpanda topics, enabling seamless access to streaming data for offline analytics.
-
Elimination of Data Movement: Access data from analytics platforms without costly and complex data migrations.
-
Enhanced Efficiency: Bypass custom data engineering jobs and Kafka Connect, reducing time to insight and infrastructure costs.
-
Schema Evolution and Discoverability: Leverage Iceberg’s robust schema management and secure, centralized REST data catalogs for improved data governance.
-
Real-Time to Offline Integration: Blending real-time and historical data for richer analytics insights.
-
Flexible Schema Support: Automatically derive Iceberg schemas from Redpanda schema registry or utilize a default schema for semi-structured views of data.
-
Custom Partitioning: Define custom partitioning scheme to improve query performance.
-
Dead Letter Queue: Built-in dead letter queues automatically set aside invalid data for targeted re-processing.
-
SQL Accessibility: Enable analysts to use standard SQL queries to analyze streaming data within their data lakehouse.
The beta version of Iceberg Topics was launched in December 2024. This capability is now available to all Redpanda Enterprise Edition customers and is available as public beta for Redpanda BYOC customers. For more information, visit the website and register for the Tech Talk.
Redpanda will be exhibiting as a gold sponsor at the Iceberg Summit on Tuesday, April 8 at Hyatt Regency SF Downtown. Attendees are invited to the Redpanda booth to learn more and attend the Redpanda session.
The post Redpanda Announces the General Availability of Apache Iceberg Topics for the Enterprise appeared first on SD Times.
Source: Read MoreÂ
