FOSSWire
Search/
Databases & Storage

MariaDB introduces HyperLogLog via new pluggable aggregate function API

Brief written by the FOSSWire AI editor from MariaDB Official Server How we write

1 min read

Key Takeaway

The new type_hll plugin demonstrates how MariaDB can extend core analytic capabilities like HyperLogLog sketches using a new pluggable aggregate function mechanism.

Summary

MariaDB developer Roman Nozdrin has demonstrated the capability of the database's plugin API to host complete analytic features through the introduction of the type_hll plugin. This plugin implements HyperLogLog (HLL) sketches, which allow for efficient estimation of distinct counts in large datasets without requiring full scans of raw event data.

The implementation relies on three distinct extension mechanisms: Pluggable Data Types (PDT), function plugins, and a newly introduced Pluggable Aggregate Function (PAF) mechanism. While the PDT and function plugins are established features, the PAF mechanism is a significant architectural addition that will officially arrive with the MariaDB 13.3 release, enabling the server to support complex, mergeable aggregate functions outside the core engine.

Technical implementation of the type_hll plugin

  • Pluggable Aggregate Functions (PAF): A new mechanism based on Item_sum_plugin that allows the server to handle building and merging sketches via external shared libraries.
  • Hybrid Extension Model: The plugin utilizes Pluggable Data Types (PDT) for the HLL column type, MariaDB_FUNCTION_PLUGIN for scalar operations, and the new PAF for aggregate logic.
  • HyperLogLog Precision: The implementation provides exact counts for up to 256 distinct values, transitioning to a classic HLL algorithm with 16,384 registers and an expected standard error of approximately 0.8125% for larger sets.
  • SQL Syntax Integration: The new PAF mechanism includes parser and server-side changes to support the FILTER (WHERE ...) syntax and allows individual aggregates to decide if they support the DISTINCT clause.

Why it matters

This development provides a blueprint for database engineers to add specialized analytical types—such as those used in network security monitoring or advertising reach analytics—without modifying the core database engine. It significantly improves performance for big data workloads by allowing users to store and merge compact sketches rather than re-scanning massive tables for distinct counts.

Original reference: mariadb.org →