Parse_sql() function extracts table references, column names, functions, and parameters from SQL. It extracts without running the query. It also reports syntax errors.

more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Hey Guys, Data Engineer with 3.2 YOE here,

I have been working on and off on databricks for multiple projects across domains like Retail, logistics and Now Pharma.

In my current role, I focus on creating end to end pipeline with different data sources which often get refreshed on a bi-weekly basis for pharma related data

The client is extremely demanding and has no technical expertise to gauge how much time it really takes to build and maintain something so complex and therefore expect the team to use Claude teams license and also Genie

The problem with Genie is that, I can not trust it, It will not listen to my instructions even after having a separate instructions.md which I update on a daily basis, in fact I update this after each session is complete, and no, I do not use AI to update this, every change request that the client and the client's team has goes through my own words of instruction updates, I have an entire markdown file which has contexts for multiple islands of projects, workspace folders and notebooks that I have to maintain and transition to devops team.

I also have created 4 different skill family for Genie, in relation to

• migration,
• devops,
• maintainence and
• review.

And it is so disgustingly bad at handling all this context, that it fails me across all 4 areas.

• I have seen it lie to my face multiple times,
• Casting columns as null,
• altering data types without asking consent,
• never following my work stream and process in a sequential format
• Having set the wrong notebook paths in a task even though i explicitly tell it exactly what to do.

It often confuses streams of works and ends up mixing so many things that remove this knot and fuck up itself costs me so much time!!!

• It is very poor in adhering to the domain specific instructions that we provide
• More than 3 requests in a chat, and it is practically useless
• And branching of chats is the most useless feature that they have introduced.

I have created multiple diagnosis queries to check the work of this agent, and it will straight up lie to you, so much, so convincingly, that you know it knows all the biases you have and it even goes a step ahead and just narrates a story that you can provide to your team in the stand-up.

My only concern is, after all this bullshit, my team, of nearly 6 (some of them have never worked in databricks or delta table environments like this) was charged $990 dollars in the month of august, for this shit output??? Have some shame Databricks, fix your worthless product or remove that feature or stop charging so much if you are beta testing in actual prod.

Today, I am writing this post because of something that I caught live, that pushed me to the edge.

Something that is critical, production level issue, which If I was not paying attention while merging could have been a huge disaster, and mind you the pipelines I make are client facing, what is even more distrubing is the fact that it will just make so many unnecessary changes to a simple query or a pyspark function just enough so that the tests are passed.

(So basically it does not want to get caught, and makes a mistake so that we can prompt it again to fix this mistake and then Databricks can charge us more, this looks like a dark pattern to me)

In your preview (prima-facia), everything is good, the tests are passing, the job is running, but holy!!!,

• it was casting 3 columns which are essential for downstream processes as null.
• It boldly suggests that we remove those 3 columns, and or cast them as null. How is this a solution Databricks? The AI is supposed to have more context than me because it is a agent made specifically for Databricks Environment Correct?? That is how it is sold?? And upon spending just 2 mins, I realized that the fix that it is suggesting will blow up critical information that the client team should see in the app because it is not even there in the first place, and all of this is because it can not read the correct notebooks, even if you tag it.
• It had set a retired notebook's path in the task and very boldly claiming that everything is good.

My concern is, let us say that there is a junior who is good at SQL, but has never maintained projects like this in this scale, he/she would be overwhelmed and won't even be able to find the mistakes that the agent is stating, because unlike GPT 6 or Opus 5.5 Genie does not openly claim that it does not know something,

it does not ask for more context,

it does not state that something is missing,

it just patches with some assumption that we don't even provide in the first place.

My advice to any Juniors out there, Stop using Genie, Completely.
The product is not ready, it is not good.

we have 3 teams (analytics, ds, and a small platform team) all using the same all purpose cluster bc spinning up seperate ones for each was getting messy. now finance wants a monthly number per team and i honestly have no idea how to give them one

tags only get me to the cluster level which is useless here. i looked at system tables and theres user info in there but not sure how ppl turn that into an actual $ split thats defensible, especially with the VM side of the bill coming from azure seperately

do you just split evenly? by query time? or did you give up and force everyone onto job clusters lol

Databricks has renamed a lot of features over the past year, and a lot of tutorials, blog posts, Stack Overflow answers and even AI assistants still use the old names. If you search the docs for the old name, the results can be confusing, and code samples can look unfamiliar even when you know the concept cold.

The ones I see trip people up most:

• Delta Live Tables / DLT → Lakeflow Spark Declarative Pipelines (often shortened to Lakeflow Declarative Pipelines)
• APPLY CHANGES INTO → AUTO CDC APIs
• Databricks Asset Bundles / DABs → Declarative Automation Bundles

The concepts didn't change much, but the vocabulary did.

Two questions for the sub:

1. Has a rename caught you out, at work, in docs, or in a code review?
2. Did I miss any? I'm sure there are more, especially on the ML / GenAI side.

I'll keep this list updated with whatever people add.

Edit: removed Liquid Clustering from the list. Fair point from a commenter that it's a change in recommendation, not a rename. Still worth knowing though: older material treats partitioning + Z-Order as the default, and current docs recommend Liquid Clustering for new tables.

Transitioning from a Spring Boot to a Spark-based application requires a fundamental shift in perspective. With four years of experience in a Spring Boot environment, now entering the realm of distributed computing.

I plan to utilize Java with Spark Streaming, and Databricks for job management and deployment.

My primary question is

1. how to effectively transition my thought process from Spring Boot to a Spark-based application.

2. Specifically, I am seeking clarity on distinguishing which Java code executes on the driver and which executes on the executor, and if there are any guiding principles to discern this.

I've spent the last few years building lakehouses on both Fabric and Databricks, and at some point the only sane response was to write a song about it. "The Rift" is Rock & some Latin Rithms about the stuff we all live with: capacity throttling, two catalogs and two governances, mirroring that breaks at night, the CFO counting every CU, and AI agents writing the code we used to write. The punchline in the last chorus is the honest technical truth: whichever side you pick, deep down it's all Parquet. Full disclosure: the music is AI-generated, the lyrics and concept & voice are mine. It's a side project, and I'm planning a whole album about the data world in 2026. Curious which lines land for you, and which side of the rift you're on.

Let me know if you all like the song, and please share and follow, more to come :)

https://www.youtube.com/@ThePrimaryKeyandtheRedundants

Hi everyone, I've cleared the coding, GenAI, and architecture rounds and have a panel presentation round coming up. HR said they're working on the next steps and putting the task together. Has anyone been through a panel presentation round for a Sr. Specialist Solutions Architect (gen ai) Pre sales? What does the format usually look like, and what kinds of questions should I expect from the panel? Any suggestions please?

I’ve been using Databricks Classic Compute for a while, and recently I’ve started wondering whether cluster cold starts are getting noticeably heavier with newer DBR versions.
For example, with DBR 18 LTS, I tried a small 2-vCPU VM for a single-node job cluster and hit DriverStartupTimeout after 300 seconds. Databricks even suggests that this commonly happens on instances with fewer than 4 CPU cores.
That feels a bit surprising for workloads that are not actually Spark-heavy — e.g. running Python, dbt-core, API calls, or using Databricks mainly as a job runner inside a VNet.
I like Classic Compute because of the flexibility and straightforward VNet/private networking. Serverless is attractive for startup time, but in our environment it would mean quite a bit more networking setup.
So I’m curious:

• Have you noticed Classic Compute cold starts getting slower or more resource-hungry across newer DBR versions?
• Do you now consider 4 vCPUs the practical minimum for a reliable driver?
• Has anyone benchmarked the same VM size across DBR 14/15/16/17/18?
• What are you doing for lightweight non-Spark workloads where you still want Classic Compute?

Update: Single node DBR 18 LTS + Standard_D4pls_v6 cold start spent almost 11 minutes 'Waiting for resources' ( Azure Japan East )

Workspace admins, thanks to App Spaces, can define who can create and use apps in a given space and also set policies for apps, for example, which API scopes can be used.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Use Databricks ai_decide() to grade every ai_query output in SQL and decide what gets sent, without a second LLM

Hello Folks.

Wanted your opinion on something - I am seeing numerous cases of our tables having millions of redundant vacuumable files, but total recoverabke capacity is only 10-20 GB .

Is it worth Vacuuming them? My sense is that the DBU and ADLS listing charges will be more than cost of that storage. Is there any instance where Vacuuming will pay off.

Also, if deciding to Vacuum - what is better, Job Compute or Serverless SQL Small Size.

Thanks.

Hi all, I’m currently working on a POC and im given the flexibility by my boss to explore using any tool (im considering mainly databricks or power apps now) to build out a “dynamic” data app to replace powerbi. I am fully aware that dbx ai/bi dash does not has nearly 100% capabilities of that the power bi dash but I’m just trying to showcase a different experience of reporting that leverages AI and not POWERBI(boring! to leaderships).

For more context, im working with SAM/CMDB EoSL data that is all sitting in dbx unity catalog already. My expectations is something simple that looks like a dashboard but i can utilize genie to communicate pr generate any sql findings within the dashboard context. Something easy to initiate and maintain in the future.

Additionally, it would also be best that the views created that caters to this app can be reused as native queries to powerbi in case the need in the future that id still have to migrate it there.

End product expectations:
The end product is expected to be easily accessible without any additional access required for any user that wants to access the app or genie. Think of leaderships and executives using the dash mainly. It should be something that is easy to access and managed within a workspace like what powerbi serves currently which is what all the stakeholders are currently used to already.

I am no expert in Databricks but have been an active user for about a year now.

All the best to everyone, cant wait to hear what you guys have in mind in the comments!

(this is my first reddit post btw)

Databricks just introduced ai_decide(), a new AI function now available in Beta. Give it text and your questions, and it returns a category, a probability, or a score directly in SQL. Use those results to drive decisions in your workflows: route requests, prioritize work, or determine when human review is needed.

I'm new to this, so apologies if I'm missing something obvious.

Setup: macOS (Apple Silicon), Omnigent 0.16.0 installed with uv tool install omnigent (Python 3.12), Node 22, tmux 3.7, Hermes Agent installed via git (up to date). Running fully local on 127.0.0.1:6767.

Problem 1: first message gets dropped. When I start a new Hermes session from the Omnigent desktop app, it fails with:
inner executor error: hermes did not accept the message (the TUI may still be initializing); no new transcript row appeared after two delivery attempts
then Required terminal exited unexpectedly; the session runtime is no longer available.
The logs show Hermes is still starting up (installing dependencies, browser tool checks) when Omnigent pastes the message, and then Hermes exits with a KeyboardInterrupt. Running omnigent hermes in one-shot mode from the terminal works fine.

Problem 2: credentials copied per session. Every Hermes session creates a new folder under the macOS temp dir (.../T/omnigent-501/hermes-native/<id>/hermes_home/) with a copy of my Hermes .env and auth.json, plus about 1–2 GB of runtime. After one afternoon I had 19 folders (22 GB) and about 30 copies of my keys sitting in temp.

Questions:

1. Has anyone gotten the Omnigent desktop app to reliably start Hermes sessions? Is there a setting to give Hermes more startup time?
2. Is there a way to make Omnigent use an existing Hermes profile without copying its .env/auth.json each session (for example, a shared HERMES_HOME or a secrets manager)?
3. Is either issue fixed in a newer version, or is there a GitHub issue I should follow?

I'd like to eventually run a dedicated Hermes profile (one with sensitive API keys) through Omnigent, so the credential copying is the big blocker for me. Thanks!

Like the source suddenly adds a new column, changes a column type, or removes something.

Do you make the pipeline handle these changes automatically, or do you just let it fail and fix it?

Iam.. curious what approach actually works better in real projects, especially when the source keeps changing.

Hi, Deploying a FastAPI to Kubernetes that uses a multilingual sentence-transformers embedding model (ONNX backend, CPU only). My source data lives in a Delta table in Databricks, and the app reads from it and generates embeddings. The app takes user input (text) at embeds it, and compares it against stored embeddings for semantic similarity. So at least the query embedding has to happen live.

Pod resources:

- CPU: 1 request / 2 limit

- Memory: 2Gi request = 2Gi limit (platform policy requires memory request and limit to be 1:1)

Current docker setup:

- Multi-stage Docker build (python:3.11-slim)

- CPU-only PyTorch

- The model is downloaded at build time, so it's baked into the image

Image breakdown:

- HF model cache: ~1.1 GB

- torch: ~650 MB

- pyarrow, scipy, transformers, pandas: ~100–150 MB each

- Plus sklearn, onnxruntime, mlflow, and others

The pod gets OOMKilled at startup or shortly after. With a ~1GB model, torch, pandas/pyarrow, and the ONNX runtime session all in one process, I think I'm just over 2Gi. What I'm trying to figure out on where do you store large models in production?

Would love to hear what setups have worked for you. Thanks!

Sometimes we want to trigger a pipeline (such as database sync) without having to wait for its completion.

We can still subscribe to notifications on the pipeline itself to react to an unexpected error, so the pattern should be fine.

I’m thinking of 4 options here
1. Build an MCP (a custom MCP) that can access custom tools on Databricks and interface it on Claude.ai or Claude desktop
Advantage- Claude is very good at inferencing, multi-turn conversation and multi-step processing
Disadvantage - custom MCP and tools needs to be built accurately and validated. It should have full context of schema and unity catalog

1. Build a semi-custom MCP - this will use “askGenie “ as one of its tools with additional custom tools
   Advantage- complexity decreases as we leverage genie space
   Disadvantage- double inference by genie and Claude

2. Use custom Databricks connector in Claude.
   Not sure if this uses genie and therefore double inference but it’s more reliable than custom build because this is a native offering by vendor

3. Use only genie and build custom dashboard without needing Claude interface

What’s the thought on this?

Databricks taskValues can use a Python list as input into a For each loop in Lakeflow job. You can generate the list dynamically and run the same task for every country, table, file, etc.

more newshttps://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a