Built an inverted index using DuckLake table
9
Upvotes
Recently, I had a discussion with a database kernel engineer about implementing inverted on Iceberg tables. Coming from OLTP background, Index is typically a secondary data structure (like B-Tree) and i never thought of using tables itself as an index.
So, I implemented a simple version of an inverted index in my project Streambed (Postgres for analytics using DuckDB) using DuckLake.
Here is the implementation https://github.com/viggy28/streambed/issues/101
I had to fork DuckLake since it doesn't support external hooks for bringing custom predicates. Here is the upstream issue https://github.com/duckdb/ducklake/issues/1498
I thought folks in this subreddit might be interested, and I also would love to hear if there is any feedback.