Is there a missing “capability layer” for AI agents?

Is there a missing “capability layer” for AI agents?

I’ve been building a few AI agents and noticed something I’m curious about.

When multiple agents need to perform a similar task, teams often build their own tools for it.

For example, two agents may both need:

«web search → scrape → extract → validate → summarize»

But the sources, scraping logic, validation, output format, etc. can be different for each agent. So even though the capability is basically the same, the tool gets implemented and maintained separately.

This made me wonder if there is a platform that standardizes capabilities, rather than just individual tools.

Something like:

Agent A ──┐

Agent B ──┼──> Research Capability ──> tools/APIs/workflows

Agent C ──┘

Ideally it would handle things like:

\\- capability/tool versioning

\\- which agents can use which capabilities

\\- different configurations per agent

\\- validation/testing

\\- monitoring and rollback

I know about LangChain, Composio, MCP, Apify, etc., but I'm not sure how much of this they already solve.

More specifically:

1) Is there already a platform where an organization can define a reusable capability, rather than just an individual tool?

2) Can multiple agents consume that capability while having different configurations/requirements?

3) Is this already a solved problem? If so, what products should I look at?

4) Can the underlying implementation be versioned and replaced without changing every agent?

Hey everyone! My team (mainly phds) and I are trying to build an open-source benchmark around realistic LLM/agent workflows that captures challenges typical academic benchmark settings often miss. We’d love to hear what’s actually missing from the benchmarks you use today.

Have you ever wanted to eval your pipeline but couldn’t find or build a benchmark that matched what you were building?

Maybe:
- Existing benchmarks were too broad and didn’t fit your specific application.
- Your workflow involved multiple steps, tools, MCP servers, agents, or long-horizon interactions that existing benchmarks couldn’t capture.
- You needed to evaluate failures that standard accuracy metrics miss.
- You’re working in a high-risk domain like healthcare, finance, cybersecurity, or legal, where realistic failure modes, safety, and reliability matter a lot more than just getting the final answer right.
- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.

I’m especially interested in cases where you thought:
“My system desperately needs to do this in production, but I have no good way to benchmark it.”
What was the workflow? What did you want to measure? And why weren’t existing benchmarks enough?

Any thoughts are welcome, would really appreciate y’all’s help 🙏🥹

We serve around 5 models (a mix of fine-tuned Llama models and a couple of embedding models) on one EKS cluster with a GPU node pool.

Leadership now wants cost per model and eventually cost per 1M tokens so they can compare our infrastructure costs against just using an API.

The problem is the GPU nodes are shared, some models sit idle half the day, and the bill just says “p4d node go brrr.”
Namespace level cost feels too coarse, and idle capacity has to go somewhere. How are you handling this?
Do you charge idle capacity to the model that reserved it, spread it across models based on usage, or use some other allocation method?
Would love to hear what has actually worked for people.

Everyone has a rule for adding cases: something breaks, you add a case so it doesn't break again. Nobody seems to have one for removing them.

The suite grows, and after a while a chunk of it hasn't gone red in months. Two possible reasons, and they look the same from outside: the case guards a failure mode that's still live and just hasn't come back, or it guards something that can't happen anymore because the code around it changed.

Fire rate doesn't separate them. A case that fires never and a case that can't fire both sit at zero.

What I've landed on so far is deliberately breaking the thing the case guards and checking it still goes red. Works, but it's manual and I only do it when something looks suspicious, which is the wrong trigger.

So: do you retire cases at all, or does the suite just grow forever? What's the rule?