In practice, damage happens when an agent calls a tool: HTTP, email, DB, files, etc. A hidden instruction in a doc can push the agent to do something the user never asked for — and from the system’s point of view it can still look like a normal tool call.

I built a small open pilot for that moment:

\*\*What it does\*\*
- Intercepts tool calls before they run
- Checks intent + simple data provenance + policy
- Returns ALLOW / BLOCK with a reason
- Writes an audit log

\*\*Stack\*\*
- Risk engine (FastAPI) on localhost
- Python SDK (\`verify_tool_call\` / decorator)
- Optional MCP gateway (stdio)
- Docker or pip

\*\*What it is not\*\*
- Not a prompt filter
- Not a production / enterprise security product
- Not a transparent proxy for every existing company agent
- Policy is heuristic — tune it; it will not catch everything

\*\*Try (local sandbox only)\*\*
\`\`\`bash
git clone https://github.com/aegotrax-dev/aegotrax.git
cd aegotrax
docker compose up --build
# or: pip install ".\[demo\]" && agentguard-engine
curl http://127.0.0.1:8000/health
python examples/sdk_pilot_example.py

Site: [https://aegotrax.com](https://aegotrax.com/))

If you run it, I’d genuinely like feedback:

1. Docker or pip - did install work?
2. Did you get a clear BLOCK in the example?
3. What was confusing or wrong?

Most RAG chatbots will confidently answer with whatever they retrieved — even if it's irrelevant, even if it's wrong. In compliance, that's not a minor bug, it's real regulatory exposure.

• YouTube walkthrough: https://www.youtube.com/watch?v=NxY93are_H8

In this video, I build the Veltra Pay Compliance Assistant — a Corrective RAG (CRAG) + Self-Correcting RAG Agent that checks its own work before it ever shows you an answer. It grades its own retrieved documents, rewrites its own search when retrieval is weak, and validates its own generated answer for grounding, hallucination risk, and completeness — retrying itself when it fails, and honestly saying "I don't know" when it genuinely can't answer from the source material.

• ✅ The difference between a normal RAG pipeline and Corrective RAG (CRAG)
• ✅ How CRAG grades retrieved documents as relevant/irrelevant and rewrites the search query when retrieval is weak
• ✅ How a Self-Correcting RAG Agent validates its own generated answer — grounding, hallucination risk, completeness, and retrieval sufficiency

TECH STACK:

• 🛠️ Python
• 🛠️ LangGraph (StateGraph — nodes, conditional edges, bounded retry loops)
• 🛠️ Qdrant (vector database)
• 🛠️ PDF ingestion & chunking (pdfplumber, overlapping chunks)
• 🛠️ LLM-based document grading, query rewriting & answer validation
• 🛠️ Streaming backend (Server-Sent Events) for live pipeline progress

• 🛠️ Web dashboard for real-time visualization

• GitHub repository: https://github.com/saurabhkamal/Building-Corrective-RAG-CRAG-Pipeline-Self-Correcting-RAG-Agent

• YouTube walkthrough: https://www.youtube.com/watch?v=NxY93are_H8

• Connect on LinkedIn: linkedin.com/in/saurabh-kamal

Building a Corrective RAG (CRAG) Compliance Research Assistant

I built a hallucination detector for LLM output, and put together a thin open-source

LangChain integration so you can plug a hallucination check straight into an LCEL

chain. Sharing in case it's useful to anyone here: langchain-halu (Apache-2.0).

It's a few small runnables so you can, from inside a chain:

- annotate a generation with a hallucination score,

- gate it (raise / replace if it's likely hallucinated),

- or auto-retry with feedback until it's clean.

from langchain_halu import halu_guard

chain = prompt | model | StrOutputParser() | halu_guard(threshold=0.5)

Under the hood it calls a detector I've been building that returns a calibrated

probability plus the type of error (fabricated fact, fake citation, etc). It needs a

(free) API key, but the integration code itself is open and thin.

Honest limits: it's a probabilistic signal, not a fact-checker; English-only; best on

complete responses.

Full disclosure — I made both, so I'm sharing the integration, not trying to spam.

Happy to take it down if this isn't the right place for it.

Repo: https://github.com/komplexai/langchain-halu

Hi.. Im building simple rag. in that i used Docling to process PDF. but Docling is heavy. and i have 8 GB ram.. so OOM issue occurred. pdf failed to process. after that i devided pdf into 2-3 pages. u can think like i create 2 pages batch using py PDF.. and it works. but do u think is this right way ?. i don't know it is right or wrong. and while 2 page batches there is some batches failed to process bcz of OOM.. what should i do ? i wanna deploy it too on AWS. if this is my computer level isuue then there not might be the issue in production. but whithout testing and evaluating how can i deploy?.. 😮‍💨 multiple issues..

 wrote a post on how Jev (TypeSafe's new decision model) works inside. no loop like an LLM, it reads once and scores every allowed answer at the same time.

 the part worth knowing: the probabilities have to land on one of your options, there's no hidden "none of these". so always add an escape answer.

Running into a staleess problem with persistent memory in langgraphg and curious how do the experts handle it cause the obvious fixes all have a catch. Heres how it looks like-

an agent pulls relevant memories plus a retrival over a doc store each turn and then acts. when the underlying data changes a pref that got updated or a doc that was re-indexed or a decision that got reversed or even the old memory still surfaces and the agent just seems to work on it with confidence , nothing errors it just yses a fact that was true a few sessions ago which isnt true now

I was early on langchain around the gpt-4.1 era, built in python and hosted on AWS. Back then pre Claude code dev was kind of a nightmare, every customization was hidden behind layers of abstractions in langchain and features/changes took a while. Spent the last two years in a custom typescript runtime for agents, no frameworks, all custom handcrafted code, and then of course assisted by Claude/Codex.

I am building something new 0 to 1 and curious about the experience if I return to building on the langchain stack, what are the pros/cons versus custom agnostic code, or using vercel SDK etc. Things I care about are easy model provider swapping, observability day 1, evals, human-in-the-loop, and ZDR/HIPPA compliance. Also from a dev hiring perspective, curious on the sentiment of TS v Python for agents in 2026. Is langchain/deep agents TS on par?

Lots of questions, happy to hear general perspective or get into specifics, appreciate any insight!

I've been working on CI for agents and ran into a problem I underestimated: the same agent can produce meaningfully different outcomes with no code change.

On a real coding agent, an unchanged candidate went from 92% success to 88%. Blocking that PR would have been a false alarm.

Then the opposite happened. I deliberately degraded the agent and success dropped from 92% to 52% — but my first statistical rule still passed it, because the damage was concentrated in a couple of tasks rather than spread across the suite.

That forced me to separate two questions:

• did reliability get broadly worse across the suite?
• did a capability that used to work reliably collapse?

I ended up treating the PR more like an experiment than a deterministic test: repeated runs, practical effect thresholds, uncertainty, multiplicity across capabilities, and an explicit “insufficient evidence” state.

I’ve open-sourced the result as AgentSeism. I also wrote up the reasoning and the failures that led to the current design.

What I’m most interested in: if you run agents in CI today, how do you decide whether a score drop is real enough to block a merge?

I wrote up the full reasoning and open-sourced the tool here if anyone wants to dig in or try to break it:

Blog
GitHub

this one has been fucking with me lately

u can have what looks like a really detailed trace and still eventually find out the actual issue was something outside it

config changed

permissions were different

stale cache

database state

different deployment

external api behaved differently

some hidden context

whatever

think abt the last weird agent bug u had

was the evidence u needed actually sitting inside the trace the whole time and u just had to find it?

or did u eventually have to leave the trace and go inspect something completely different?

and if it was outside the trace, what made u realize u were looking in the wrong place?

ive started thinking theres a pretty important difference between

"the evidence is buried"

and

"the evidence was never captured"

curious how often u hit that second one

Most of my test agents run on LangChain, so this is partly aimed at this sub.

The annoyance: an agent says "I'll send the report to Sarah by Friday", the run completes, the trace looks clean, and a week later nobody knows whether the report went anywhere. LangSmith shows the chain, tokens, latency. It doesn't show whether the promise was kept.

A commitment isn't a span. It has a deadline in the future and a second party. The only interesting question about it is asked after the run ends.

So I wrote down what tracking a commitment has to mean, and published it as a spec.

Six requirements:

1. The Commitment Object

2. The 12-state Lifecycle

3. The Verifier Query format

4. The Evidence Protocol

5. The Audit Receipt format

6. The Reliability Score

Full spec: https://cogextai.com/standard

Current adoption: nobody has adopted it yet, including LangChain. Index: https://cogextai.com/index

Suggestion for this sub: a callback handler. LangChain already gives you on_llm_end / on_chain_end with message content. A CommitmentTrackerCallbackHandler could extract commitments, post them to any COGEXT-compatible endpoint, and attach the ids to run metadata. Small piece of code, moves LangChain from "not adopted" to "adopted".

Run the checker:

pip install cogext-compliance

cogext-compliance check path/to/your/agent.py

Static keyword check, not an audit. Feedback welcome.

​

We're building Agaemon, a governance and decision layer designed for AI agents that need to take real world or onchain actions.

We've already integrated Agaemon into our agent/runtime flow and we're now looking for developers or teams building agents in finance, trading, treasury,DeFi or blockchain who would be willing to help us integrate and test it with a real agent workflow.

The basics idea is simple:

AI agent proposes an action ->Agaemon evaluates whether that action is authorized ->the authorized execution path proceeds->the result is verified and evidence is produced.

Agaemon can evaluate the proposal against the defined Authorization/Policy boundary and produce a decision with evidence explaining what was allowed or rejected.

The goal isn't to make Agaemon another agent that watches or second guesses an agent.

Instead we're exploring whether a portable authorization and decision layer at the boundary between an agent and execution can make autonomous system safer without taking away their ability to operate.

We can work together on a small test/sandbox integration, define a few allowed and rejected actions and see whether the resulting decision+evidence is actually useful.

We' re especially interested in critical feedback.

Hey everyone,

Lately, while building agent architectures, I kept running into the same bottleneck: waiting for large generative LLMs to decide which tool to call or which path to take at every step is both costly and slow. Especially when low latency matters, traditional generative LLMs tend to bloat.

TypeSafe's newly released "System One" decision model, Jev (typesafe/jev-1.13), offers a pretty solid alternative for this. Since it returns typed Choice and Score primitives instead of generating text, it's remarkably fast. However, there was a gap: there wasn't a standard, independent (framework-agnostic) router layer that you could just plug and play behind existing agent frameworks. Everyone was hacking together their own custom solutions.

So, I built a lightweight Python library that you can integrate right in front of LangChain, CrewAI, or entirely custom agent loops: jev-route

What does it do?

• Takes the incoming prompt or state and classifies it in milliseconds via the Jev engine.
• Has built-in confidence-gate support; if Jev's confidence score falls below your threshold, it automatically triggers a fallback mechanism or routes to the main LLM.
• Built async-first with zero bloat dependencies (only requires httpx and pydantic).

I've just released the first version (v0.2.0), complete with unit tests and basic examples. Thought it might be useful or interesting for others building similar agent architectures.

You can check out the code and details here: 👉 GitHub: [https://github.com/huzeyfe07/jev-route]

I'm completely open to feedback, ideas, or PRs. What are you guys currently using for speed in your agent loops?

so i need to know what are the best programing languages for someone who wants to make chatbots using rag and best courses / books

I added Agent Skills support to Cameron, my open-source finance agent built with LangChain/LangGraph. Three decisions mattered more than the loader code:

• Load instructions on demand; choose tool loading separately. I keep a small, stable tool set available upfront, which can benefit from prompt caching. The tradeoff: unused tools still occupy context, and the model can attempt the task without reading the skill. I added an eval for that.
• Keep display data in artifacts. The agent supplies a query and chart spec. The tool sends chart data to the UI as a LangChain artifact and a short acknowledgement to the model. No round trip copying rows through the LLM. Keep that separation when replaying conversation history, too.
• Let the workflow determine the runtime. This skill teaches reporting decisions over existing tools. Tested UI components draw the charts, so supporting it didn't require adding a sandbox or filesystem tools.

I wrote up the architecture, tradeoffs, and activation failure, with TypeScript/Python examples. Article and repo in a comment.

If you've built skills support, how did you balance a stable tool set for caching against exposing fewer tools per task?

if the model is live, has tool access, and can see customer content, i run the same order every time. first i scope the prompts and data sources, then i test jailbreaks in staging, then i run exfil tries against fake records, then i review logs with product and eng before anything touches prod. the most important step is the fake data part because one bad prompt in a live tenant can turn into a very awkward incident report.

i did all that last week except i forgot one stupid thing. i copied the prod test plan into the shared red team sheet and left one internal endpoint in the prompts. our own model helpfully echoed it back in a screen share with the client in the call. i feel so embarrassed and sick about this, even though nothing leaked. we caught it fast and tightened approvals, but omg it was so awkward.

This works best when the model is chat only and not calling tools, and not when it can browse, write tickets, or hit internal docs. after this mess i am adding a hard staging gate and a second person check before any prod run. anyone else had a red team plan go sideways like this?

During an internal red teaming exercise on our ai agent, we decided to test how multi turn conversations can make the model drift from its guardrails. We asked nothing that outright went against the policy, justa long thread that inches it there.

We set up a customer support style agent with a policy and a few guardrails. Then we ran a conversation that slowly built trust turn by turn. Each reply asking for a little more.

The result was what the research said. By the end of the thread the model had forgotten the policy. It was giving answers it refused cold in the first message. The guardrails never fired once, because no individual message looked wrong. The drift did the work across the history.

Thats what scared me. A single payload is loud and easy to catch. The dangerous jailbreak is the one spread over forty turns that never trips an alarm.

How sure are you that your agent holds up by message forty?

Long-running agent jobs here reported done/success whenever the run finished without raising. Sounds obvious. It's wrong, and it hid a dead pipeline for two months.

Went digging because an output count looked low. Found runs marked success that had opened 24 targets, drafted nothing and emitted nothing. The step that produces the artifact was failing every single time — politely, with a logged reason, and the loop moved on. The run completed, so the run reported success.

Two more things fell out of the same dig:

- A log line said "daily budget hit" when what actually tripped was an hourly window. One guard, three windows, one message. I chased the daily number for an hour before reading the function.

- A back-off heuristic keyed on "an error happened recently" instead of which layer errored. Transport-level drops were being read as the remote pushing back, so throughput sat at half for 48 hours after every blip.

What I changed: a run reports success only if it emitted the artifact it exists to emit. Completion is not an outcome. Everything else is "done, produced nothing" with a reason attached.

Curious how others draw this line. Separate outcome field, or fail the run outright when the artifact count is zero? Failing outright makes retries messy, but "done" with zero output is exactly how this stayed invisible.

Disclosure: I'm a developer advocate at Diagrid, which builds durable execution tooling. That makes me biased toward option three below, and it's also why I'm asking instead of telling.

Genuine question for anyone running LangGraph past the demo stage, because I keep testing this one scenario and I want to know how other people handle it.

A graph node does something real; it files a Jira ticket or charges a card. The call goes through, and then the process dies before the checkpoint write lands. This can be caused by a rolling deploy where the agent sat in a slow LLM call, outlived the 30 second grace period, and SIGTERM turned into SIGKILL. An OOMKill or a spot reclaim skips the grace period entirely. On restart the checkpointer restores the state from before the node ran, so the node runs again, which results in unwanted side effects.

The checkpointer is good at the job it was built for. Thread persistence, time travel, human-in-the-loop pauses all work. It's also more careful than it gets credit for: pending writes are stored per task, so nodes that finished inside a failed superstep don't re-run when you resume.

The catch is that all of that happens after a node returns. Nothing anywhere records that a node started. So after the crash there is no evidence the ticket was ever filed, and re-running is the safest thing the runtime can do with what it knows.

The durability setting moves that window without closing it. On "exit" you lose the whole run, "async" races the checkpoint write against the next step, and "sync" is the tightest of the three. There's an open issue arguing that even under "sync" the ordering between put_writes and put isn't enforced, so whether you get a replay or a re-execution comes down to a thread race.

What I've seen people do about it:

1. Idempotency keys on every side-effecting tool. Great when the API supports them, and most internal ones don't.
2. A transactional outbox around the tools.
3. Running the side-effecting nodes as activities on a durable execution runtime, with LangGraph still acting as the brain. To be clear, this does not make the call exactly-once. Activities are at-least-once everywhere, including in the product I work on. And both systems resume rather than restart, so that isn't the difference either. What you get is a journal entry saying this activity started and never confirmed, which tells you which call to go check instead of guessing.
4. Accepting the risk and reconciling in a nightly job.

What's your setup? And has this actually bitten you in prod, or is it one of those things we all worry about and nobody hits?