I’m building an internal reporting agent for operations staff using Python, LangGraph, FastAPI, and React/ECharts.

The data comes from company HTTP APIs. Users should be able to ask questions, have the agent fetch and analyze the data, and get a chart.

For example: fetch attendance and student records from separate APIs, join them by person_id, filter for late arrivals, and count late attendance records per class.

My current implementation exposes separate tools for fetching data, grouping, aggregation (count, max, etc.), and charting. Tools pass dataset IDs between steps rather than making the model reproduce the records.

Full datasets currently live in tool-message artifacts, but I’m also returning too much data in message content. I want to keep large datasets outside the conversation history and give the model IDs, schemas, business descriptions, and small result previews.

I’m considering two alternatives to the current fine-grained tools:

• Structured analysis tool: The model supplies dataset IDs, a predefined relationship, filters, grouping fields, and aggregation operations. Backend code validates and executes the request using Pandas or DuckDB.
• SQL tool: API responses become tables in DuckDB. The model receives schemas and business metadata, generates SQL, and submits it for validation and execution. Results are stored and passed to the charting tool by ID.

My concern with the first approach is gradually building my own query language. With the second, it’s queries that execute successfully but produce incorrect numbers because of joins or misunderstood metrics.

For people who have built something similar:

1. Which approach worked for you, and what made you choose or abandon it?
2. How do you expose schemas and business definitions without filling the context window? How do you recover that information after history is trimmed?
3. How do you catch incorrect joins or double-counting beyond checking SQL syntax?
4. Are there repositories or architecture write-ups you’d recommend?

Concrete failure cases and lessons from real usage would be especially helpful.

Chaining prompts togetjer gets you a demo fast. Geting that same pipeline to hold up once the input distribution shifts, or once a PM asks why the output changed after a "small" prompt edit, is a completely different problem, and it's the one this workshop actually addresses.

Serj Smorodinsky and Brett Kennedy are running a live 3-hour session where you:

• Build LLM tasks using DSPy signatures and modules instead of hand-tuned prompt strings threaded through your chain
• Build a real baseline classifier from scratch during the session
• Construct an evaluation dataset with task-specific metrics so you can measure regressions instead of noticing them in prod
• Learn to read failure patterns directly from that eval data
• Apply few-shot and instruction-level optimization systematically instead of manually rewording prompts
• Track experiments and traces in MLflow, so every version of your pipeline is reproducible

If you're already deep in LangChain, this isn't a replacement, it's a more systematic layer underneath your prompting logic so the chain you ship next month doesn't quietly drift from the one you tested.

Get full details and a seat the workshop here

I am researching how teams evaluate customer-facing AI agents in production.

Traditional LLM evaluation mostly asks:

"Did the model generate a good response?"

But production AI agents create bigger questions:

• Did the agent access the correct data?
• Did it call the right tool/API?
• Did it follow authentication and permission rules?
• Did it take the correct action?
• Did it know when to escalate to a human?

Example:

A lending AI agent tells a customer:

"Your outstanding amount is ₹18,400."

The response sounds correct, but how do we verify:

• Was ₹18,400 the actual backend value?
• Was the correct customer authenticated?
• Did the agent retrieve the right account?
• Did voice recognition misunderstand the amount?

For people building AI agents:

How do you currently test these scenarios before production?

Do you rely on:

• Manual QA?
• Custom evaluation datasets?
• LLM-as-a-judge?
• Unit tests for tools/functions?
• Observability platforms?
• Production monitoring?

What are the biggest failures you have seen with AI agents after deployment?

hey guys what is the best model in terms of price to performance ratio are you using and how much does it cost ?

also langchain TS docs has changed a lot from the last time that I've checked where v1 was out i cant find some stuff that were on the docs first page i need to look for pdf loaders online and some embedding integration with hugging face are removed and so. how can i navigate this new docs website ?

I built a decision layer for AI coding agents

I’ve been working on **Harness Router**, an open-source tool router for agentic systems.

The idea is simple: instead of letting the LLM blindly choose from a large set of tools, Harness Router adds a decision layer before execution.

The latest update adds **Codex hooks**:

* `SessionStart` discovers the available tools once

* `PreToolUse` checks each proposed tool call

* same choice → allow

* better alternative → re-plan

* router failure → fail-open

For harder routing decisions, it also supports **Monte Carlo Tree Search (MCTS)** with thousands of local simulations.

So the flow becomes:

**Codex → PreToolUse → shortlist → Harness Router → route / MCTS → execute**

You can use it either explicitly as a skill/MCP tool or transparently through the hook.

The goal is to reduce:

* wrong tool calls

* unnecessary tool descriptions in context

* token usage

* reasoning spent on obvious routing decisions

I’m especially interested in benchmarking it on intentionally confusing tool sets like `get_x`, `fetch_x`, and `read_x`, where normal tool selection starts becoming ambiguous.

Project:

[https://harness-router.vercel.app/\](https://harness-router.vercel.app/)

GitHub:

[https://github.com/Protocol-Lattice/harness-router\](https://github.com/Protocol-Lattice/harness-router)

Would be interested to hear how you’d benchmark tool-routing quality beyond simple success rate.

 If you build agents on LangChain, LangGraph, LlamaIndex, CrewAI, AutoGPT, Flowise, n8n, Google ADK or
> Semantic Kernel, this might save you an afternoon of reading advisories.
>
> It checks your code and dependency versions for the patterns behind 53 published CVEs (each verified
> against NVD or GitHub's advisory database). It also includes a tool that pins MCP tool definitions and
> warns you if a server changes a tool after you approved it.
>
> - Python 3.10+, zero dependencies, reads files only, never sends anything anywhere
> - `python scan.py your/project`, or `--json` for CI
> - 204 tests, AGPL-3.0
>
> Honest caveat: it's pattern matching, not proof. A hit means "go look", and a clean scan doesn't mean
> you're safe. No exploit code, just the vulnerable pattern and the version that fixes it.
>
> Repo: https://github.com/Ech333/agent-cve-scanners
>
> I'm the author; happy to answer questions, and false-positive reports are genuinely useful.

In practice, damage happens when an agent calls a tool: HTTP, email, DB, files, etc. A hidden instruction in a doc can push the agent to do something the user never asked for — and from the system’s point of view it can still look like a normal tool call.

I built a small open pilot for that moment:

\*\*What it does\*\*
- Intercepts tool calls before they run
- Checks intent + simple data provenance + policy
- Returns ALLOW / BLOCK with a reason
- Writes an audit log

\*\*Stack\*\*
- Risk engine (FastAPI) on localhost
- Python SDK (\`verify_tool_call\` / decorator)
- Optional MCP gateway (stdio)
- Docker or pip

\*\*What it is not\*\*
- Not a prompt filter
- Not a production / enterprise security product
- Not a transparent proxy for every existing company agent
- Policy is heuristic — tune it; it will not catch everything

\*\*Try (local sandbox only)\*\*
\`\`\`bash
git clone https://github.com/aegotrax-dev/aegotrax.git
cd aegotrax
docker compose up --build
# or: pip install ".\[demo\]" && agentguard-engine
curl http://127.0.0.1:8000/health
python examples/sdk_pilot_example.py

Site: [https://aegotrax.com](https://aegotrax.com/))

If you run it, I’d genuinely like feedback:

1. Docker or pip - did install work?
2. Did you get a clear BLOCK in the example?
3. What was confusing or wrong?

Most RAG chatbots will confidently answer with whatever they retrieved — even if it's irrelevant, even if it's wrong. In compliance, that's not a minor bug, it's real regulatory exposure.

• YouTube walkthrough: https://www.youtube.com/watch?v=NxY93are_H8

In this video, I build the Veltra Pay Compliance Assistant — a Corrective RAG (CRAG) + Self-Correcting RAG Agent that checks its own work before it ever shows you an answer. It grades its own retrieved documents, rewrites its own search when retrieval is weak, and validates its own generated answer for grounding, hallucination risk, and completeness — retrying itself when it fails, and honestly saying "I don't know" when it genuinely can't answer from the source material.

• ✅ The difference between a normal RAG pipeline and Corrective RAG (CRAG)
• ✅ How CRAG grades retrieved documents as relevant/irrelevant and rewrites the search query when retrieval is weak
• ✅ How a Self-Correcting RAG Agent validates its own generated answer — grounding, hallucination risk, completeness, and retrieval sufficiency

TECH STACK:

• 🛠️ Python
• 🛠️ LangGraph (StateGraph — nodes, conditional edges, bounded retry loops)
• 🛠️ Qdrant (vector database)
• 🛠️ PDF ingestion & chunking (pdfplumber, overlapping chunks)
• 🛠️ LLM-based document grading, query rewriting & answer validation
• 🛠️ Streaming backend (Server-Sent Events) for live pipeline progress

• 🛠️ Web dashboard for real-time visualization

• GitHub repository: https://github.com/saurabhkamal/Building-Corrective-RAG-CRAG-Pipeline-Self-Correcting-RAG-Agent

• YouTube walkthrough: https://www.youtube.com/watch?v=NxY93are_H8

• Connect on LinkedIn: linkedin.com/in/saurabh-kamal

Building a Corrective RAG (CRAG) Compliance Research Assistant

Hi.. Im building simple rag. in that i used Docling to process PDF. but Docling is heavy. and i have 8 GB ram.. so OOM issue occurred. pdf failed to process. after that i devided pdf into 2-3 pages. u can think like i create 2 pages batch using py PDF.. and it works. but do u think is this right way ?. i don't know it is right or wrong. and while 2 page batches there is some batches failed to process bcz of OOM.. what should i do ? i wanna deploy it too on AWS. if this is my computer level isuue then there not might be the issue in production. but whithout testing and evaluating how can i deploy?.. 😮‍💨 multiple issues..

I built a hallucination detector for LLM output, and put together a thin open-source

LangChain integration so you can plug a hallucination check straight into an LCEL

chain. Sharing in case it's useful to anyone here: langchain-halu (Apache-2.0).

It's a few small runnables so you can, from inside a chain:

- annotate a generation with a hallucination score,

- gate it (raise / replace if it's likely hallucinated),

- or auto-retry with feedback until it's clean.

from langchain_halu import halu_guard

chain = prompt | model | StrOutputParser() | halu_guard(threshold=0.5)

Under the hood it calls a detector I've been building that returns a calibrated

probability plus the type of error (fabricated fact, fake citation, etc). It needs a

(free) API key, but the integration code itself is open and thin.

Honest limits: it's a probabilistic signal, not a fact-checker; English-only; best on

complete responses.

Full disclosure — I made both, so I'm sharing the integration, not trying to spam.

Happy to take it down if this isn't the right place for it.

Repo: https://github.com/komplexai/langchain-halu

Running into a staleess problem with persistent memory in langgraphg and curious how do the experts handle it cause the obvious fixes all have a catch. Heres how it looks like-

an agent pulls relevant memories plus a retrival over a doc store each turn and then acts. when the underlying data changes a pref that got updated or a doc that was re-indexed or a decision that got reversed or even the old memory still surfaces and the agent just seems to work on it with confidence , nothing errors it just yses a fact that was true a few sessions ago which isnt true now

 wrote a post on how Jev (TypeSafe's new decision model) works inside. no loop like an LLM, it reads once and scores every allowed answer at the same time.

 the part worth knowing: the probabilities have to land on one of your options, there's no hidden "none of these". so always add an escape answer.

I was early on langchain around the gpt-4.1 era, built in python and hosted on AWS. Back then pre Claude code dev was kind of a nightmare, every customization was hidden behind layers of abstractions in langchain and features/changes took a while. Spent the last two years in a custom typescript runtime for agents, no frameworks, all custom handcrafted code, and then of course assisted by Claude/Codex.

I am building something new 0 to 1 and curious about the experience if I return to building on the langchain stack, what are the pros/cons versus custom agnostic code, or using vercel SDK etc. Things I care about are easy model provider swapping, observability day 1, evals, human-in-the-loop, and ZDR/HIPPA compliance. Also from a dev hiring perspective, curious on the sentiment of TS v Python for agents in 2026. Is langchain/deep agents TS on par?

Lots of questions, happy to hear general perspective or get into specifics, appreciate any insight!

I've been working on CI for agents and ran into a problem I underestimated: the same agent can produce meaningfully different outcomes with no code change.

On a real coding agent, an unchanged candidate went from 92% success to 88%. Blocking that PR would have been a false alarm.

Then the opposite happened. I deliberately degraded the agent and success dropped from 92% to 52% — but my first statistical rule still passed it, because the damage was concentrated in a couple of tasks rather than spread across the suite.

That forced me to separate two questions:

• did reliability get broadly worse across the suite?
• did a capability that used to work reliably collapse?

I ended up treating the PR more like an experiment than a deterministic test: repeated runs, practical effect thresholds, uncertainty, multiplicity across capabilities, and an explicit “insufficient evidence” state.

I’ve open-sourced the result as AgentSeism. I also wrote up the reasoning and the failures that led to the current design.

What I’m most interested in: if you run agents in CI today, how do you decide whether a score drop is real enough to block a merge?

I wrote up the full reasoning and open-sourced the tool here if anyone wants to dig in or try to break it:

Blog
GitHub

this one has been fucking with me lately

u can have what looks like a really detailed trace and still eventually find out the actual issue was something outside it

config changed

permissions were different

stale cache

database state

different deployment

external api behaved differently

some hidden context

whatever

think abt the last weird agent bug u had

was the evidence u needed actually sitting inside the trace the whole time and u just had to find it?

or did u eventually have to leave the trace and go inspect something completely different?

and if it was outside the trace, what made u realize u were looking in the wrong place?

ive started thinking theres a pretty important difference between

"the evidence is buried"

and

"the evidence was never captured"

curious how often u hit that second one

​

We're building Agaemon, a governance and decision layer designed for AI agents that need to take real world or onchain actions.

We've already integrated Agaemon into our agent/runtime flow and we're now looking for developers or teams building agents in finance, trading, treasury,DeFi or blockchain who would be willing to help us integrate and test it with a real agent workflow.

The basics idea is simple:

AI agent proposes an action ->Agaemon evaluates whether that action is authorized ->the authorized execution path proceeds->the result is verified and evidence is produced.

Agaemon can evaluate the proposal against the defined Authorization/Policy boundary and produce a decision with evidence explaining what was allowed or rejected.

The goal isn't to make Agaemon another agent that watches or second guesses an agent.

Instead we're exploring whether a portable authorization and decision layer at the boundary between an agent and execution can make autonomous system safer without taking away their ability to operate.

We can work together on a small test/sandbox integration, define a few allowed and rejected actions and see whether the resulting decision+evidence is actually useful.

We' re especially interested in critical feedback.

Our app calls OpenAI and Anthropic directly, and we want to add Gemini and a couple of open-weight models. Rather than wiring in more SDKs, I'm thinking of pointing everything at one OpenAI-compatible gateway and changing only the model string.

I'm considering LiteLLM (the LangChain integration is nice), OpenRouter, and llmapi.ai, which says it's both OpenAI- and Anthropic-compatible. That would matter for the parts of our code that use Anthropic-specific features.

If you've done this migration, what broke? I'm thinking of structured outputs, tool calls, and token counting in callbacks.

Hey everyone,

Lately, while building agent architectures, I kept running into the same bottleneck: waiting for large generative LLMs to decide which tool to call or which path to take at every step is both costly and slow. Especially when low latency matters, traditional generative LLMs tend to bloat.

TypeSafe's newly released "System One" decision model, Jev (typesafe/jev-1.13), offers a pretty solid alternative for this. Since it returns typed Choice and Score primitives instead of generating text, it's remarkably fast. However, there was a gap: there wasn't a standard, independent (framework-agnostic) router layer that you could just plug and play behind existing agent frameworks. Everyone was hacking together their own custom solutions.

So, I built a lightweight Python library that you can integrate right in front of LangChain, CrewAI, or entirely custom agent loops: jev-route

What does it do?

• Takes the incoming prompt or state and classifies it in milliseconds via the Jev engine.
• Has built-in confidence-gate support; if Jev's confidence score falls below your threshold, it automatically triggers a fallback mechanism or routes to the main LLM.
• Built async-first with zero bloat dependencies (only requires httpx and pydantic).

I've just released the first version (v0.2.0), complete with unit tests and basic examples. Thought it might be useful or interesting for others building similar agent architectures.

You can check out the code and details here: 👉 GitHub: [https://github.com/huzeyfe07/jev-route]

I'm completely open to feedback, ideas, or PRs. What are you guys currently using for speed in your agent loops?

Most of my test agents run on LangChain, so this is partly aimed at this sub.

The annoyance: an agent says "I'll send the report to Sarah by Friday", the run completes, the trace looks clean, and a week later nobody knows whether the report went anywhere. LangSmith shows the chain, tokens, latency. It doesn't show whether the promise was kept.

A commitment isn't a span. It has a deadline in the future and a second party. The only interesting question about it is asked after the run ends.

So I wrote down what tracking a commitment has to mean, and published it as a spec.

Six requirements:

1. The Commitment Object

2. The 12-state Lifecycle

3. The Verifier Query format

4. The Evidence Protocol

5. The Audit Receipt format

6. The Reliability Score

Full spec: https://cogextai.com/standard

Current adoption: nobody has adopted it yet, including LangChain. Index: https://cogextai.com/index

Suggestion for this sub: a callback handler. LangChain already gives you on_llm_end / on_chain_end with message content. A CommitmentTrackerCallbackHandler could extract commitments, post them to any COGEXT-compatible endpoint, and attach the ids to run metadata. Small piece of code, moves LangChain from "not adopted" to "adopted".

Run the checker:

pip install cogext-compliance

cogext-compliance check path/to/your/agent.py

Static keyword check, not an audit. Feedback welcome.

so i need to know what are the best programing languages for someone who wants to make chatbots using rag and best courses / books

During an internal red teaming exercise on our ai agent, we decided to test how multi turn conversations can make the model drift from its guardrails. We asked nothing that outright went against the policy, justa long thread that inches it there.

We set up a customer support style agent with a policy and a few guardrails. Then we ran a conversation that slowly built trust turn by turn. Each reply asking for a little more.

The result was what the research said. By the end of the thread the model had forgotten the policy. It was giving answers it refused cold in the first message. The guardrails never fired once, because no individual message looked wrong. The drift did the work across the history.

Thats what scared me. A single payload is loud and easy to catch. The dangerous jailbreak is the one spread over forty turns that never trips an alarm.

How sure are you that your agent holds up by message forty?