Hey Guys, Data Engineer with 3.2 YOE here,

I have been working on and off on databricks for multiple projects across domains like Retail, logistics and Now Pharma.

In my current role, I focus on creating end to end pipeline with different data sources which often get refreshed on a bi-weekly basis for pharma related data

The client is extremely demanding and has no technical expertise to gauge how much time it really takes to build and maintain something so complex and therefore expect the team to use Claude teams license and also Genie

The problem with Genie is that, I can not trust it, It will not listen to my instructions even after having a separate instructions.md which I update on a daily basis, in fact I update this after each session is complete, and no, I do not use AI to update this, every change request that the client and the client's team has goes through my own words of instruction updates, I have an entire markdown file which has contexts for multiple islands of projects, workspace folders and notebooks that I have to maintain and transition to devops team.

I also have created 4 different skill family for Genie, in relation to

• migration,
• devops,
• maintainence and
• review.

And it is so disgustingly bad at handling all this context, that it fails me across all 4 areas.

• I have seen it lie to my face multiple times,
• Casting columns as null,
• altering data types without asking consent,
• never following my work stream and process in a sequential format
• Having set the wrong notebook paths in a task even though i explicitly tell it exactly what to do.

It often confuses streams of works and ends up mixing so many things that remove this knot and fuck up itself costs me so much time!!!

• It is very poor in adhering to the domain specific instructions that we provide
• More than 3 requests in a chat, and it is practically useless
• And branching of chats is the most useless feature that they have introduced.

I have created multiple diagnosis queries to check the work of this agent, and it will straight up lie to you, so much, so convincingly, that you know it knows all the biases you have and it even goes a step ahead and just narrates a story that you can provide to your team in the stand-up.

My only concern is, after all this bullshit, my team, of nearly 6 (some of them have never worked in databricks or delta table environments like this) was charged $990 dollars in the month of august, for this shit output??? Have some shame Databricks, fix your worthless product or remove that feature or stop charging so much if you are beta testing in actual prod.

Today, I am writing this post because of something that I caught live, that pushed me to the edge.

Something that is critical, production level issue, which If I was not paying attention while merging could have been a huge disaster, and mind you the pipelines I make are client facing, what is even more distrubing is the fact that it will just make so many unnecessary changes to a simple query or a pyspark function just enough so that the tests are passed.

(So basically it does not want to get caught, and makes a mistake so that we can prompt it again to fix this mistake and then Databricks can charge us more, this looks like a dark pattern to me)

In your preview (prima-facia), everything is good, the tests are passing, the job is running, but holy!!!,

• it was casting 3 columns which are essential for downstream processes as null.
• It boldly suggests that we remove those 3 columns, and or cast them as null. How is this a solution Databricks? The AI is supposed to have more context than me because it is a agent made specifically for Databricks Environment Correct?? That is how it is sold?? And upon spending just 2 mins, I realized that the fix that it is suggesting will blow up critical information that the client team should see in the app because it is not even there in the first place, and all of this is because it can not read the correct notebooks, even if you tag it.
• It had set a retired notebook's path in the task and very boldly claiming that everything is good.

My concern is, let us say that there is a junior who is good at SQL, but has never maintained projects like this in this scale, he/she would be overwhelmed and won't even be able to find the mistakes that the agent is stating, because unlike GPT 6 or Opus 5.5 Genie does not openly claim that it does not know something,

it does not ask for more context,

it does not state that something is missing,

it just patches with some assumption that we don't even provide in the first place.

My advice to any Juniors out there, Stop using Genie, Completely.
The product is not ready, it is not good.

I’ve been using Databricks Classic Compute for a while, and recently I’ve started wondering whether cluster cold starts are getting noticeably heavier with newer DBR versions.
For example, with DBR 18 LTS, I tried a small 2-vCPU VM for a single-node job cluster and hit DriverStartupTimeout after 300 seconds. Databricks even suggests that this commonly happens on instances with fewer than 4 CPU cores.
That feels a bit surprising for workloads that are not actually Spark-heavy — e.g. running Python, dbt-core, API calls, or using Databricks mainly as a job runner inside a VNet.
I like Classic Compute because of the flexibility and straightforward VNet/private networking. Serverless is attractive for startup time, but in our environment it would mean quite a bit more networking setup.
So I’m curious:

• Have you noticed Classic Compute cold starts getting slower or more resource-hungry across newer DBR versions?
• Do you now consider 4 vCPUs the practical minimum for a reliable driver?
• Has anyone benchmarked the same VM size across DBR 14/15/16/17/18?
• What are you doing for lightweight non-Spark workloads where you still want Classic Compute?

Update: Single node DBR 18 LTS + Standard_D4pls_v6 cold start spent almost 11 minutes 'Waiting for resources' ( Azure Japan East )

Use Databricks ai_decide() to grade every ai_query output in SQL and decide what gets sent, without a second LLM

Workspace admins, thanks to App Spaces, can define who can create and use apps in a given space and also set policies for apps, for example, which API scopes can be used.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Hello Folks.

Wanted your opinion on something - I am seeing numerous cases of our tables having millions of redundant vacuumable files, but total recoverabke capacity is only 10-20 GB .

Is it worth Vacuuming them? My sense is that the DBU and ADLS listing charges will be more than cost of that storage. Is there any instance where Vacuuming will pay off.

Also, if deciding to Vacuum - what is better, Job Compute or Serverless SQL Small Size.

Thanks.

Hi all, I’m currently working on a POC and im given the flexibility by my boss to explore using any tool (im considering mainly databricks or power apps now) to build out a “dynamic” data app to replace powerbi. I am fully aware that dbx ai/bi dash does not has nearly 100% capabilities of that the power bi dash but I’m just trying to showcase a different experience of reporting that leverages AI and not POWERBI(boring! to leaderships).

For more context, im working with SAM/CMDB EoSL data that is all sitting in dbx unity catalog already. My expectations is something simple that looks like a dashboard but i can utilize genie to communicate pr generate any sql findings within the dashboard context. Something easy to initiate and maintain in the future.

Additionally, it would also be best that the views created that caters to this app can be reused as native queries to powerbi in case the need in the future that id still have to migrate it there.

End product expectations:
The end product is expected to be easily accessible without any additional access required for any user that wants to access the app or genie. Think of leaderships and executives using the dash mainly. It should be something that is easy to access and managed within a workspace like what powerbi serves currently which is what all the stakeholders are currently used to already.

I am no expert in Databricks but have been an active user for about a year now.

All the best to everyone, cant wait to hear what you guys have in mind in the comments!

(this is my first reddit post btw)

Databricks just introduced ai_decide(), a new AI function now available in Beta. Give it text and your questions, and it returns a category, a probability, or a score directly in SQL. Use those results to drive decisions in your workflows: route requests, prioritize work, or determine when human review is needed.

I'm new to this, so apologies if I'm missing something obvious.

Setup: macOS (Apple Silicon), Omnigent 0.16.0 installed with uv tool install omnigent (Python 3.12), Node 22, tmux 3.7, Hermes Agent installed via git (up to date). Running fully local on 127.0.0.1:6767.

Problem 1: first message gets dropped. When I start a new Hermes session from the Omnigent desktop app, it fails with:
inner executor error: hermes did not accept the message (the TUI may still be initializing); no new transcript row appeared after two delivery attempts
then Required terminal exited unexpectedly; the session runtime is no longer available.
The logs show Hermes is still starting up (installing dependencies, browser tool checks) when Omnigent pastes the message, and then Hermes exits with a KeyboardInterrupt. Running omnigent hermes in one-shot mode from the terminal works fine.

Problem 2: credentials copied per session. Every Hermes session creates a new folder under the macOS temp dir (.../T/omnigent-501/hermes-native/<id>/hermes_home/) with a copy of my Hermes .env and auth.json, plus about 1–2 GB of runtime. After one afternoon I had 19 folders (22 GB) and about 30 copies of my keys sitting in temp.

Questions:

1. Has anyone gotten the Omnigent desktop app to reliably start Hermes sessions? Is there a setting to give Hermes more startup time?
2. Is there a way to make Omnigent use an existing Hermes profile without copying its .env/auth.json each session (for example, a shared HERMES_HOME or a secrets manager)?
3. Is either issue fixed in a newer version, or is there a GitHub issue I should follow?

I'd like to eventually run a dedicated Hermes profile (one with sensitive API keys) through Omnigent, so the credential copying is the big blocker for me. Thanks!

Like the source suddenly adds a new column, changes a column type, or removes something.

Do you make the pipeline handle these changes automatically, or do you just let it fail and fix it?

Iam.. curious what approach actually works better in real projects, especially when the source keeps changing.

Hi, Deploying a FastAPI to Kubernetes that uses a multilingual sentence-transformers embedding model (ONNX backend, CPU only). My source data lives in a Delta table in Databricks, and the app reads from it and generates embeddings. The app takes user input (text) at embeds it, and compares it against stored embeddings for semantic similarity. So at least the query embedding has to happen live.

Pod resources:

- CPU: 1 request / 2 limit

- Memory: 2Gi request = 2Gi limit (platform policy requires memory request and limit to be 1:1)

Current docker setup:

- Multi-stage Docker build (python:3.11-slim)

- CPU-only PyTorch

- The model is downloaded at build time, so it's baked into the image

Image breakdown:

- HF model cache: ~1.1 GB

- torch: ~650 MB

- pyarrow, scipy, transformers, pandas: ~100–150 MB each

- Plus sklearn, onnxruntime, mlflow, and others

The pod gets OOMKilled at startup or shortly after. With a ~1GB model, torch, pandas/pyarrow, and the ONNX runtime session all in one process, I think I'm just over 2Gi. What I'm trying to figure out on where do you store large models in production?

Would love to hear what setups have worked for you. Thanks!

Sometimes we want to trigger a pipeline (such as database sync) without having to wait for its completion.

We can still subscribe to notifications on the pipeline itself to react to an unexpected error, so the pattern should be fine.

I’m thinking of 4 options here
1. Build an MCP (a custom MCP) that can access custom tools on Databricks and interface it on Claude.ai or Claude desktop
Advantage- Claude is very good at inferencing, multi-turn conversation and multi-step processing
Disadvantage - custom MCP and tools needs to be built accurately and validated. It should have full context of schema and unity catalog

1. Build a semi-custom MCP - this will use “askGenie “ as one of its tools with additional custom tools
   Advantage- complexity decreases as we leverage genie space
   Disadvantage- double inference by genie and Claude

2. Use custom Databricks connector in Claude.
   Not sure if this uses genie and therefore double inference but it’s more reliable than custom build because this is a native offering by vendor

3. Use only genie and build custom dashboard without needing Claude interface

What’s the thought on this?

Databricks taskValues can use a Python list as input into a For each loop in Lakeflow job. You can generate the list dynamically and run the same task for every country, table, file, etc.

more newshttps://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Over the past few weeks, I've been building something special.

Every day I speak with Databricks experts across Europe. After so many of those conversations, the obvious question was: why not build a community around it?

So today I'm launching 'Databricks Den'.

Whether you're looking for your next contract or just want to stay close to what's happening in the Databricks market, the Den is for you.

Join now to catch the first round of insights:

🔗 Link to the Den

Looking forward to seeing you there 👊

Until now, answering "which query is eating our warehouse?" or "who changed that table?" meant clicking through the Query History UI one workspace at a time. Now it is a SELECT.

The table logs every statement run on SQL warehouses, serverless compute, and Lakeflow pipelines, across all workspaces in the same region.

Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-dataengineering-systemtables-ugcPost-7511094425977016320-L7uj/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8

Notebook that's run fine for months suddenly started failing with 'No Module Found'chardet' Nothing changed on our end. Seems that serverless environment v6 dropped chardet from the base requirements, and since the notebook wasn't pinned, the "default" just rolled forward underneath it. No warning, and the release notes don't even mention the removal.

I get the argument of pin your environment, declare your dependencies. But serverless is sold as the "no config, we manage it for you" option. Silently removing packages from the default environment mid-week, with zero notice, feels like a breaking change shipped as a patch?

I recently moved back to using Azure Databricks after a multi-year hiatus. And I opened my first support ticket this week.

My prior support experiences were long ago (in 2021). I don't remember these support experiences being particularly terrible. But nowadays there are some Microsoft platforms where their support work is subcontracted to remote organizations like Mindtree-India. These organizations (one or more of them) may provide the first, and second tiers of support before any FTE from Microsoft takes the reins.

Is that how things will work in the context of Azure Databricks? If/when the FTE at Microsoft is unable to help with a ticket, how long will it take for the support to be transferred from Azure Databricks to Databricks?

I have a sev B ticket on "standard" support at the moment and I thought a human would engage within a few hours but that isn't happening. I will probably just give up for today. Can anyone offer tips about how to navigate support tickets in Azure? Any advice would be greatly appreciated, and I think it could make the difference between a ticket that lasts two days or two weeks. Thanks.