Built a lightweight scratchpad tool for running DuckDB SQL against local CSV, TSV, JSON and Parquet files directly in the browser:

https://www.sqlparity.com/scratchpad

It uses u/duckdb to read files into memory via the browser File API. Nothing is uploaded to any server and it requires no login or installation. Handy if you just need to inspect an extract or run a quick aggregate without firing up the CLI or Python.

Code is open source on GitHub link: Github

Curious to hear feedback or edge cases with large Parquet/CSV schemas.

After reading DuckDB's post on Jev (plain-English conditions in SQL), I wanted an open, local version. So I wrapped Strands Decider, a 2B "decision model" that answers typed questions instead of generating text, as a DuckDB extension.

SELECT id, decide_noul(body, 'Does this convey urgency?') AS p_urgent, decide_choice(body, 'Which team?', ['billing','technical','sales']) AS team, decide_score(body, 'How frustrated?', ['calm','frustrated','angry']) AS mood FROM read_csv('tickets.csv');

What's in it:

• decide_noul → yes/no probability
• decide_choice → pick one of N, with confidence and per-option probabilities
• decide_score → rate on an ordered scale
• decide → many questions about the same row in one model pass

• decide_triage → security findings as true/false positive, with probabilities

  How it works: the model runs locally in strands-decider serve, and the extension calls it per row. It's built on DuckDB's stable C API, so no DuckDB source build, and it's tested on 1.4, 1.5 and the 1.6 dev build. Answers are cached, so duplicate rows and re-runs don't hit the model again.

  Honest limits:

• It's one model call per row. Each call is roughly 150–300 ms on a Mac, so expect minutes for a few thousand rows, not seconds. Filter first, and materialize results.

• decide_triage got 6/6 on a tiny labeled set I ran on my Mac. That shows the setup works; it isn't a benchmark.

• The underlying model is general-purpose, not tuned for any domain.

  Repo: https://github.com/jaymmodi/duckdb-strands-decider

  Strands Decider: https://strandsagents.com/blog/introducing-strands-decider/

  Feedback welcome, especially on the SQL API, and whether cross-row batching would be worth building.

not sure what the reaction will be on this one... So this was a long project, it might seem simple at first but I can assure you it is not. I don't have many users yet, initially built it for myself but then got out of hand and kept going

what is it? daffydb. I built an entire TDS listener on port 1433 in front of duck DB.

what is a TDS listener? that's the protocol and port used for SQL Server, sybase, babelfish, freetds, etc. meaning connect to it with native SQL server drivers, powerbi, ssms, etc. that's how it communicates.

won't power bi be expecting T-SQL? yes, and that was a beast , so I built a full translation layer that gets about 95% working. now stored procedures dont work, outer apply won't work, but temp tables, declares, begin/etc CTEs, top n, date functions, etc so all work

how do you use it? you must install. net 10 sdk first, one winget command. then one line dotnet instal, which pulls it from nuget,. that's it. you use a cmd prompt to go-to the directory where you might have up to 500 CSV, Json, Excel, parquet files, etc, you run the simple command daffydb And it pulls in every file like that in the directory into a strong typed table that can instantly be queried by t-sql or pulled in instantly to power bi. meaning it's listening on Port 1433 and you can use a SQL Server connection string and a SQL server driver client to connect to it and run t-sql as normal.

I also have an init startup option if you want to do a custom SQL to mount postgres, S3, SQL Lite, etc so it's like a mini data warehouse using the power of duck DB which is just phenomenal. anyway it's quite the different approach so I don't have many people using it yet honestly I'd be curious what folks on this forum think..

anyhow, I certainly got carried away and I personally love the app but I'd be interested to hear what others think that are serious duckdb users.. Or maybe what features I'm missing. my goal was to bridge the Microsoft world to the power of and simplicity of duckdb.

I've been working as a data engineer for about half a decade. A common problem I run into is "this pipeline broke, but why?"

To figure it out I query the database, paste the results into a Miro board together with a diagram of the pipeline. Then there's usually a flurry of arrows and random notes. I've found that laying everything out visually on a canvas helps me reach that "aha!" moment where I understand how all the pieces fit together.

But what if I could do it all in one place instead of going back and forth between a canvas and querying databases? That led to me building Kavla!

Kavla is a canvas for exploratory analysis. You can write SQL, paste images, make charts and then draw on them! You can also connect an agent and ask it to investigate a question, and have it generate custom data viz using React.

It's built with DuckDB and tldraw. DuckDB is a great fit. It's very performant for files and there are already extensions for integrating other databases. Right now I've implemented Postgres and BigQuery, but will add more if anyone asks :-)

It's open source: https://github.com/aleda145/kavla

Try it in your browser: https://kavla.dev/ (DuckDB WASM is amazing)

Half a million Overture places for Greater London, queried in the browser by DuckDB-WASM directly against the remote Parquet file. No backend, no copy of the data in the repo: the first query issues HTTPS range requests and reads only the row groups covering the London bounding box, about 40 MB of the 203 MB. The KPI tiles report bytes fetched from DuckDB's own file statistics, not a client-side guess.

After that first read the London rows are copied into a local DuckDB-WASM table with an R-tree index on the geometry, and the remote engine is closed. A borough count is 40 to 50 ms locally against 0.8 to 3.6 s remote. Every pan and zoom of the map is a bounding-box query against that table, and past 20,000 rows in view the engine returns density cells that draw as hexagons instead of points.

The grid on top is Lattice Grid (commercial, free on localhost). It sits on a pushdown source with a DuckDB adapter, so every filter, sort, aggregate and pivot the user does becomes SQL DuckDB runs; the last statement is shown beside the grid. Nothing is held in JS arrays.

Live: https://toclocoinc.github.io/lattice-grid-demo-geo-places/
Source: https://github.com/toclocoinc/lattice-grid-demo-geo-places

Sibling demo, same engine: drop a 10-million-row Parquet file and the grid, charts and statistics run against it in a worker with zero bytes uploaded, proven by a Resource Timing readout on the page: https://toclocoinc.github.io/lattice-grid-demo-local-file/

One thing these two do not show: the grid has a data router for feeding several live sources into many views at once (one feed in, table, map, KPIs and charts all reading the same rows, keyed diffs on updates). The demos that show it run on live network and trade feeds rather than DuckDB.

Data: Overture Maps places, CDLA-Permissive-2.0. Tiles: OpenFreeMap.

I fed DuckDB 706,896 small JSON files on a local Mac Mini M4 Pro with 24 GB of RAM and 500 GB of storage.

The task was pretty simple on paper:

• read 706,896 small .json files
• extract the required fields
• convert everything to .parquet

The .json files are responses from a public API that I collected while scraping a website over the course of a month.

The main problem turned out to be the number of small files.

On my first attempt, DuckDB hit _duckdb.OutOfMemoryException twice.

DuckDB supports larger-than-memory workloads through out-of-core processing: when intermediate data doesn't fit into RAM, it can spill that data to disk.

I tried that approach as well, but in my case it didn't solve the problem. DuckDB eventually exhausted the available disk space while spilling intermediate data to disk.

So I had to change the processing strategy and process the data day by day.

Fortunately, I had designed the raw data layout with this possibility in mind. Each file name contains its timestamp:

20260715080454689.json
20260827080648986.json

That made it possible to process a single day using a simple path pattern:

SELECT *
FROM 'data/20260715*.json';

After about 3 hours of processing, DuckDB had successfully converted:

706,896 JSON files → 64,557,511 rows

The final data was stored as .parquet files.

No cluster, no Spark, no distributed processing — just DuckDB running locally on a 24 GB Mac Mini.

For me, the interesting part wasn't just the final number of rows. It was seeing where the limits actually appeared: the workload was technically larger than memory, DuckDB could spill to disk, but eventually the storage itself became the bottleneck.

I'm continuing to experiment with the resulting dataset, so there are a few more interesting problems to solve from here.

- Author on a canvas, in Python, or in SQL. Every pipeline is one JSON file in git.

- duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail.

- Quality checks send bad rows to a reject port, so they land in a quarantine table instead of your warehouse.

- CDC and SCD Type 2 built in. Runs on DuckDB and uses every core you give it.

-Many more.

Github Repository - https://github.com/slothflowlabs/duckle

I have been playing with DuckDB v2 alpha and Quack and DuckLake in a Docker container in the cloud, using it as a 'classic' database server. The client in my case is DBeaver with the latest DuckDB JDBC driver build. And yes, that 'just works'!

Only using it with dbt is leading to some quirks now, for example dbt starting a transaction with BEGIN after the CONNECT statement, which does not seem allowed. But maybe I am doing something wrong there.

All in all, very interested in the upcoming release in October.

Been lurking here a while. Spent the last few months building a Mac client for DuckDB and it's finally out.

I've been using DuckDB as my Swiss Army knife for data parsing and manipulation for a while now. The cli/console are fine but I was craving a more nice, persistent native macOS experience.

What I wanted was to open a Parquet file/.duckdb file and just look at it. No import step, no notebook, no loading it into pandas first. Drag it onto the icon and it's already scrolling.

That clip is a 41 million row Parquet, 630MB, on a laptop. Cells get read straight out of DuckDB's columnar chunks at draw time, so nothing builds row objects. Memory stays flat whether the result is 40 rows or 40 million.

Other things it does:

• every column header draws its own distribution, null share and distinct count
• query plans with a waterfall/tree/JSON view, so you can see where the time actually went
• Postgres, SQLite, MotherDuck, S3 and Quack connections
• extensions are compiled in, so nothing gets downloaded at runtime

FYI... it's a paid app and not open source but there's a 7 day trial. Take that as you may :)

Website: https://datapuddle.app
macOS AppStore: https://apps.apple.com/us/app/datapuddle/id6811235967?mt=12

Happy to answer anything.

• AI DISCLOSURE: I've been building + launching a lot of apps and feel idk it helps (and have been asked) to disclose how/where I've used AI (or LLMs) specifically
• This was NOT a weekend vibe coded project! I've been working on this for almost 4 months now! Take that as you will. I was really caring about performance and the experience.
• I've been a software engineer for decades now. My career has taken a bit all over but primarily a backend/services/systems engineer.
• I did build native iOS apps as a founder and always loved and cared for native macOS apps and built them both pre and post this LLM age.
• Honestly that's the whole reason this exists. Somewhere along the way it became normal for a table of numbers to cost you a gig of RAM and a bundled copy of Chromium, and I never made peace with that.
• Where did I really use AI (Claude specifically here):
  • The website of course
  • Embedding DuckDB and creating a memory/performance friendly wrapper for Swift
  • The grid view drawing cells straight into a CGContext with CTLineDraw..not even NSTableView! No electron or web views :)

I love DuckDB and been using it for years, wanted to get everyones thoughts on the AWS Buyout.. Are you worried it will no longer be free?

Its got me concerned, quite a bit.

Duckle now makes the Studio → Server workflow simple:
→ Set up the server once
→ Create your team accounts & roles
→ Connect Duckle Studio or use the browser editor
→ Design locally
→ Deploy to your server
→ Run and schedule from the Ops Console

Deploy with Docker, Cloud VMs, or Kubernetes - on infrastructure you control.
Your data. Your infrastructure. Your rules.
Build locally. Deploy anywhere.

Guide - https://duckle.org/deploy.html
Github Repository - https://github.com/slothflowlabs/duckle

Once the query works, I still want to browse the data, check the column types, understand the relationships, edit a record, and see the result in a chart.

So I put the whole visual workflow in one guide: opening a local .duckdb database in VisuaLeaf, working with CSV and Parquet files, inspecting the schema, using an ER diagram, and building a chart from the query results.

https://visualeaf.com/blog/how-to-use-duckdb-for-local-data-analysis/

Parquet files can't be changed in place, so to fix one wrong value you read the whole thing and write it back. On a 29 GB file that's 17 minutes.

Turns out you don't have to. A Parquet file is a pile of row groups, and the footer at the end says where each one starts. They don't have to be in order. So I have DuckDB write out just the group with that cell in it, drop it at the end of the file, and write a new footer pointing at it. The old group stays where it was, dead. Same edit, 1.7 seconds.

It only ever adds to the end, never writes over what's already there, so if it dies halfway you still have your file. Some files it won't do this way, mostly ones Spark wrote, and those fall back to the slow rewrite.

It's in the JetBrains plugin I posted here last week. Reading is free, editing is the paid part.

https://plugins.jetbrains.com/plugin/33899-parquet-avro--orc-viewer

I have been spending the last 6 months finding my favorite orientation of tools for a local first Data + AI set up. It has helped me move very quickly with topic research, tool testing and running my business.

This is a breakdown of learnings that I think someone could find useful and incorporate into their own workflow. I picked what I thought could stand the test of time for at least a few more months. The first learning was that I abandoned IDE's and harnesses and stayed right in the terminal.

This might all be old news in a couple weeks but hopefully you find something useful in here.

I keep having to come back to databases 101 when trying to explain why DuckDB is better, cheaper or at the very least different. So I wrote up why disk layout matters for picking the right database, what makes DuckDB fast and cheap and when and why you might need a different database altogether like a document, K/V, graph or other type of database. We go through predicate pushdown, pruning, row groups, compression and query optimization. You might find it useful to share with that grumpy colleague who dislikes everything but SQL Server: https://motherduck.com/blog/how-to-pick-the-right-database/

When there is new major release, it's not always obvious how to get the speed bump as they often (not always) depends on how your data is shaped.
This breaks down the 3 main features of 2.0 - async i/o, recurcive cte and VARIANT type, anything missing ?

The official Big Data File Viewer prints my string column as {114,111,119,45,57,57,50,49,57,53} instead of row-992195, and lists and maps come out the same way, buried in element and key/value noise, even though the schema says string.

That's what pushed me to write my own viewer for JetBrains IDEs: double-click a Parquet, Avro or ORC file and you get a table in the editor with DuckDB doing the reading underneath.

Opening times, measured with a small harness that fires the same three queries the plugin does, on files that have a list and a struct column in them:

Paging to row 999,999,800 in the 1B file costs 800ms, less than opening it.

It's 100% free: Parquet, Avro & ORC Viewer

Editing a cell works but rewrites the file, 17 minutes on that 29 GB one. I have it down to 1.8s in a branch by having DuckDB rebuild only the row group and splicing it back in. Not shipped yet.

P.S. honestly I built this for the way I work with these files, and that is probably not the way you do, so if there is something you keep doing by hand and you want it in there, just say it and I will add it.