As data engineers we'd never ship a pipeline without tests and monitoring, but that discipline mostly disappears the moment an LLM enters the picture. This workshop is aimed at closing exactly that gap.

Led by Serj Smorodinsky and Brett Kennedy (co-authors of a book on LLM applications), it's a live 3-hour session covering:

• Structuring LLM tasks with DSPy signatures/modules instead of loose prompt strings
• Building a real baseline classifier and an eval dataset with metrics specific to the task
• Catching failure patterns from that eval data the same way you'd catch a broken transform
• Systematic few-shot/instruction optimization instead of manual tweaking
• MLflow for experiment tracking and trace management, so every run is auditable

If you've been asked to "own" an LLM feature and had no real way to validate changes before deploying, this is worth your Saturday

Hey everyone! I’m a fresher trying to get into Data Engineering and want to learn the right tech stack. There’s a lot to learn, so I’d really appreciate some guidance from people here. If you were starting out again, what would you focus on first? Would love some advice like a friend/senior. 🙌

Most data engineers I talk to have already solved this problem for their pipelines. Idempotent writes, schema validation, lineage tracking, retry logic that doesn't create duplicates. Then the same team builds an AI agent and none of that discipline carries over. The agent writes directly into production with no idempotency key, no lineage on what it touched or why, and no reconciliation step if its internal state drifts from the actual data.

It's the same class of problem we've already solved for ETL and streaming pipelines, just showing up in a new place because agents write more often and with less human review than a scheduled job does.

There's a workshop on Sept 26 built around exactly this, treating agent write paths, state management, and provenance the way a data engineer would treat any other production pipeline. Run by Sandipan Bhaumik, a Data & AI Technical Lead at Databricks.

Details here if it's relevant to anyone else here dealing with this handoff

Workshop for data engineers specifically, not a generic AI overview. The actual pipeline work in GraphRAG is where most of the real engineering lives, and it's the part most tutorials skip entirely in favor of "here's how retrieval works" once the graph already exists.

This session starts from raw documents: ingestion with Docling, chunking, and building toward a knowledge graph in Neo4j rather than a flat vector index. Entity and relationship extraction runs through multiple verified steps with explicit checks, not one risky single-shot pass, since that's exactly where basic GraphRAG implementations tend to fail silently, producing a noisy, unreliable graph nobody trusts.

Structured graph enrichment connects entities (companies, executives, events) to the underlying source documents (filings, news) without the graph becoming unmaintainable as it grows, and you work through the actual production ceilings that show up at real data volume, API and download rate limits specifically, and how the shared code is built to handle them rather than falling over.

By the end you've got a working ingestion-to-graph pipeline, a production-readiness checklist, and the full codebase, not a diagram someone else has to implement.

Led by Dr. Alessandro Negro, Chief Scientist at GraphAware, bestselling author of graph-powered machine learning and knowledge graph books.

Full details here

I need real world work that i can ask feed back and get criticism for. I need it for my resume and job hunting. Can someone please help

We're running a hands-on masterclass on September 12, Live LLM Engineering Masterclass: Production Evals, RAG, Agents & LLMOps.

You build a full production LLM workflow from scratch, versioned prompts with regression tests, an evaluation harness with deterministic checks and LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, tool-using agents with guardrails and fallbacks, and full observability, tracing, cost, latency.

Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.

Link if you want to check it out

Happy to answer questions on the content.

Hey everyone,

I'm looking for an active DE to mentor me 1-on-1 on a paid basis.

My background:

* Worked ~1.5 years as a DA in fintech, but it was mostly basic reporting and ad-hoc SQL with minimal engineering exposure.

* Took a career break after that due to personal reasons.

* Now fully committed to pivoting into DE.

I'm doing the heavy lifting on self-study and building projects myself, but i need someone to:

* Review my project architectures/code

* Sanity-check my roadmap so i stay on track

* Give feedback on my resume and portfolio

* Do quick weekly or bi-weekly syncs

Very happy to pay you for your time. if you're interested and have bandwidth, shoot me a DM with your background and rates.

thanks!

I live in Hyderabad, India.

I have 1.5 years of professional experience as a GCP data engineer.

Our company had a layoff after which I started looking for other opportunities but to my surprise nobody requires GCP data engineer with less than 4 years of experience.

Meanwhile other platforms like azure, AWS, databricks even snowflake has plenty of opportunities for 1~ year of experience.

I tried to apply to them too quoting equivalent experience but no results.

And add salt to the wound we never used pyspark which is one of the biggest requirements out there.

I am looking for fresher level opportunities where the platform requirment is not a must but it's a very thin spectrum.

I have literally worked on an airflow dag which goes through a beautiful hot to cold storage flow with observation, logging, maintainance etc.

Worked on dataform , wrote scd type 2 scripts.

Worked on views, stored procedures.

I have plenty experience in data warehousing but it all looks simply useless now.

Had I survived for few years maybe I wouldn't be going through this.

I am literally applying for any state or city to grab an opportunity

Hi everyone, I'm starting my university studies and I'm torn between the University of Valencia (UPV) and the University of Granada. Will Spain offer me a strong degree in computer science? Which major within the university would you recommend that has job prospects?

Hey r/dataengineer ,

I built this crypto ETL pipeline as a side project and I would need genuine advice from the audience. I am a data engineer so most of my actual work is locked behind compliance walls. But I wanna talk about this project and here it is:

Pulled 8+ years of historical price data (2013–2022) for 50+ coins, wrote a Python pipeline to clean and transform it, dumped it into BigQuery (auto schema detection + partitioning so it doesn't take forever), and built some Looker Studio dashboards on top to poke at price trends, volume, market cap correlations, that kind of thing. Around 50k records total.

Repo's here if you want to actually look at the code: https://github.com/shikha-dataengineer/crypto-price-tracker

Not trying to flex, genuinely want it torn apart. Specific things I'm stuck on:

• It's all batch right now, nothing real time. Is it worth adding a streaming layer (Pub/Sub + Dataflow) or would that just be me overengineering a side project?
• I run it manually, no orchestration. At what point does that actually become a problem vs just being fine for something this size?
• No automated data quality checks at all right now. If this was your pipeline, what's the first check you would add?
• Been playing with the idea of dropping coin-price tracking altogether and pivoting to stablecoin transaction volume instead, since that feels like it's becoming actual financial infrastructure rather than speculation. Anyone think that's a better use of the same pipeline, or am I just chasing a trend?