Apache Iceberg introduces FILE type to integrate unstructured data into lakehouses
The new FILE data type in Apache Iceberg v4 allows unstructured media to be managed with the same transactional rigor as structured data.
Summary
Google has announced the introduction of a new FILE data type for the Apache Iceberg specification, slated for version 4. This development aims to bridge the gap between structured data tables and the unstructured media—such as images, audio, and documents—that typically resides in external object storage.
By moving away from the practice of using simple string URIs to reference files, the FILE type enables the storage of essential metadata directly within the table schema. This change allows multimodal Large Language Models (LLMs) and query engines to access file properties like size, content type, and checksums without making expensive, repetitive calls to external storage buckets.
Technical specifications of the Iceberg FILE type
- Structured Metadata: The FILE type is a specialized nested type that bundles attributes including uri, offset, size, content_type, checksum, and an inline payload field.
- Integrity and Consistency: By storing typed digests or ETags as checksums, the format allows readers to detect if the underlying bytes of a file have changed even if the URI remains identical, preventing silent asset drift.
- Optimized Data Retrieval: The inclusion of an inline payload field allows small binary micro-assets to be stored directly within the table, enabling engines to stream them during columnar scans without additional object storage requests.
- Standardized Multimodal Input: The content_type attribute provides explicit MIME metadata, which eliminates the need for query engines or LLM inference APIs to guess file types based on fragile file extensions.
Why it matters
This update significantly improves the efficiency of multimodal AI workflows by unifying structured business data and raw media under a single transactional governance model. It reduces operational costs for data engineers by minimizing metadata fetch overhead and provides a standardized contract for different query engines to interpret unstructured assets reliably.
Original reference: opensource.googleblog.com →