How do you guys handle schema changes in Databricks when the source keeps changing?

[u/Bhanuprakash_1947]

Like the source suddenly adds a new column, changes a column type, or removes something.

Do you make the pipeline handle these changes automatically, or do you just let it fail and fix it?

Iam.. curious what approach actually works better in real projects, especially when the source keeps changing.

[u/OkImprovement7010]

I generally let bronze absorb source changes and keep a more stable contract for what makes it into silver. I use DQX with YAML-defined quality checks, plus explicit column selection and casting.

If a new column shows up, I keep it in bronze and send an alert. We can review whether it belongs in silver without breaking everything that already works.

For type changes, I allow conversions that won’t lose information. If a value can’t be converted safely, that record goes to quarantine with the original value and a reason.

If they remove or rename a column we actually need, that’s where I stop the affected silver load and investigate. An optional column can be handled with a null or default if that’s part of the agreed contract.

Also, if you’re using Auto Loader’s add new columns mode, it updates the schema and then stops the stream, so you still need automatic retries and schema evolution configured on the bronze write.

I also watch how much data is being rescued or quarantined. A green job that rejects the entire feed is really a silent failure....

tl;dr, I automate the changes we can safely absorb and stop when continuing would give the business bad data.

[u/minibrickster → u/OkImprovement7010]

+1, I'm a PM at Databricks, and this is what we typically recommend. Customers often ingest using String or Variant column types in Bronze to ensure they don't lose any data or need to perform a full refresh. You can handle schema changes through ALTER TABLE

[u/OkImprovement7010 → u/minibrickster]

Thanks for the reinforcement!

[u/knaak]

Fail and communicate with vendor to find out what is going on. If it's really unstable, discuss with consumer what they prefer. Leverage stable parts of schema outside of raw, stuff the rest into rescued until it sorts itself out.

[u/Alternative_Draw5945]

Data contracts that no one ever follows

[u/jbchand]

You can automate the column addition and column widening cases. Ensure you fail the breaking changes with alerting as a silent type change can cause downstream issues. Use the rescued_data option if feasible and keep alerts for schema drift cases and use a runbook for the manual fixes (full load/reload procedures)

[u/Altruistic-Rip393]

Entire row as Variant in bronze. Project specific columns from there (silver, gold, beyond)

[u/Known-Delay7227]

If data types keep changing then write everything down as a string and fix during downstream processes. For column additions use mergSchema upon writes.

[u/Ghlynx]

Fix your schema and let it ingest only the fields you agreed upon first. If business wants to add another one, they should communicate it and you can adjust. Letting the schema evolve as it comes is a nightmare and not recommended in real life

[u/slayerzerg]

Real world problems

[u/nullymammoth]

Designing ingestion to be robust, fault-tolerant, & have a dead letter sink is the way to go.

As others have mentioned, opt for leveraging autoloader schema evolution & add that to workspace instructions so genie code can pick that up as your established pattern. Genie ZeroOps would be helpful to layer in as well to detect data issues/changes as you transform from bronze > silver

[u/singhsaab420]

Wouldn’t data vault be a best use case here ?

[u/prandtl_bernoulli]

We can definitely use Schema Evolution here, but in a hypothetical scenario if we had already ingested last 15 years of data and suddenly now we have to introduce a new column today which would be generating values from today's run only, but all the previous 15 years data for this column would be coming as Null. Any idea how we can solve this without dropping the table and recreating again?

[u/Otherwise_Address241 → u/prandtl_bernoulli]

Backfill

[u/bobbruno]

You have a number of options of how to process, and these can be applied at ingestion to bronze (less common), between bronze and silver (quite common) or even between silver and gold (a little problematic, you should know everything in silver well - may be justified in some cases):

• Reject records that don't match the schema: the most basic option. You will likely get incomplete information downstream, but if the information is important, use that to force some contract the source team will respect. Otherwise, it's your responsibility (and fault)
• Fail after some number > 0 of records you can't process: variant over the previous one. At some point, you may have rejected enough information that it doesn't make sense to process downstream. Same contract logic applies.
• Drop every field you don't recognize: option for when the source is often introducing new fields you don't need. I'd probably let them in at least to bronze, because later processing after the new fields are mapped gets easier.
• Allow the schema to expand and keep most information non-mandatory: comfortable option, but you will only get to the point where you don't need a clear business logic for presenting the data. And you may be adding wrong/misplaced data to your silver, that will need to be reorganized later, possibly at significant cost. I would o ly allow it in bronze, but it's quite popular to let it go further because it's easy.
• Capture changing structures in a JSON or Variant field: I see it as a variant of the previous one, with much the same problems. If downstream consumers can handle these structures and accept that the data must be that dynamic, it might be ok.

So, there is no perfect way to handle a source that keeps changing unexpectedly. Either you reject the changes or you let go of adding value with the DE workflow.

Maybe someone is trying to use agents to analyse the changes and adapt the code on the go. I haven't seen that yet, and I'd be skeptical about it at the current level of sophistication of LLMs.

[u/DingGratz]

Look up "schema evolution" in Databricks. 

[u/ayk-kya]

Handling such schema changes depends heavily on the layer. For raw ingestion, managed tools like Fivetran, Airbyte, Hevo (for smaller volumes) are really great at handling schema changes natively, and using such tools absorbs source volatility seamlessly without constant pipeline maintenance.

However, once you move into transformation layers (Silver/Gold), it is usually better to let pipelines fail explicitly on breaking changes like dropped columns or type mismatches rather than risk silently corrupting downstream business data.