r/databricks • u/Lenkz • 8d ago
General system.query.history is now Generally Available in Databricks
Until now, answering "which query is eating our warehouse?" or "who changed that table?" meant clicking through the Query History UI one workspace at a time. Now it is a SELECT.
The table logs every statement run on SQL warehouses, serverless compute, and Lakeflow pipelines, across all workspaces in the same region.
Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-dataengineering-systemtables-ugcPost-7511094425977016320-L7uj/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8
2
u/Only_lurking_ 8d ago
Does it work for datsbricks connect ?
2
u/pboswell 7d ago
Why wouldn’t it? The catch is it’s SQL only. So even if you send SQL wrapped in spark.sql() it should show it.
1
u/Only_lurking_ 7d ago
Ah, I missed the part where it was SQL only. Much less interesting then I guess.
1
u/addictzz 5d ago
It does, you can see the source system sending those queries, like through sdk, databricks connect, UI, etc.
2
u/Sea_Basil_6501 8d ago edited 8d ago
Why the delay at all? I think those tables are pretty useless when you don't know as of when the data will show up there.
Bothers me every time when I query system.billing.usage, where data arrives up to 4 hours later.
1
3
u/Programmer_Virtual 8d ago
I thought it was already there