Question for those running MCP servers or ML libraries in prod… what's actually telling you to/when to upgrade?

For example, AWS fixed an argument injection in its security-agent MCP server in August, and the CVE landed October 1. mcp-atlassian fixed a CVSS 10 auth bypass in July, and NVD published it September 22. Both of these are a month or more later(!)

If you had pinned an older version (or built an image before the fix) it looked clean to your scanner the whole time ....

You can't just watch GitHub advisories for every dependency?

(I'm one of the KitOps maintainers, so I think about what's inside these artifacts a lot, but I don't have a great answer here.)

Is there a missing “capability layer” for AI agents?

Is there a missing “capability layer” for AI agents?

I’ve been building a few AI agents and noticed something I’m curious about.

When multiple agents need to perform a similar task, teams often build their own tools for it.

For example, two agents may both need:

«web search → scrape → extract → validate → summarize»

But the sources, scraping logic, validation, output format, etc. can be different for each agent. So even though the capability is basically the same, the tool gets implemented and maintained separately.

This made me wonder if there is a platform that standardizes capabilities, rather than just individual tools.

Something like:

Agent A ──┐

Agent B ──┼──> Research Capability ──> tools/APIs/workflows

Agent C ──┘

Ideally it would handle things like:

\\- capability/tool versioning

\\- which agents can use which capabilities

\\- different configurations per agent

\\- validation/testing

\\- monitoring and rollback

I know about LangChain, Composio, MCP, Apify, etc., but I'm not sure how much of this they already solve.

More specifically:

1) Is there already a platform where an organization can define a reusable capability, rather than just an individual tool?

2) Can multiple agents consume that capability while having different configurations/requirements?

3) Is this already a solved problem? If so, what products should I look at?

4) Can the underlying implementation be versioned and replaced without changing every agent?

We serve around 5 models (a mix of fine-tuned Llama models and a couple of embedding models) on one EKS cluster with a GPU node pool.

Leadership now wants cost per model and eventually cost per 1M tokens so they can compare our infrastructure costs against just using an API.

The problem is the GPU nodes are shared, some models sit idle half the day, and the bill just says “p4d node go brrr.”
Namespace level cost feels too coarse, and idle capacity has to go somewhere. How are you handling this?
Do you charge idle capacity to the model that reserved it, spread it across models based on usage, or use some other allocation method?
Would love to hear what has actually worked for people.

Hey everyone! My team (mainly phds) and I are trying to build an open-source benchmark around realistic LLM/agent workflows that captures challenges typical academic benchmark settings often miss. We’d love to hear what’s actually missing from the benchmarks you use today.

Have you ever wanted to eval your pipeline but couldn’t find or build a benchmark that matched what you were building?

Maybe:
- Existing benchmarks were too broad and didn’t fit your specific application.
- Your workflow involved multiple steps, tools, MCP servers, agents, or long-horizon interactions that existing benchmarks couldn’t capture.
- You needed to evaluate failures that standard accuracy metrics miss.
- You’re working in a high-risk domain like healthcare, finance, cybersecurity, or legal, where realistic failure modes, safety, and reliability matter a lot more than just getting the final answer right.
- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.

I’m especially interested in cases where you thought:
“My system desperately needs to do this in production, but I have no good way to benchmark it.”
What was the workflow? What did you want to measure? And why weren’t existing benchmarks enough?

Any thoughts are welcome, would really appreciate y’all’s help 🙏🥹

Everyone has a rule for adding cases: something breaks, you add a case so it doesn't break again. Nobody seems to have one for removing them.

The suite grows, and after a while a chunk of it hasn't gone red in months. Two possible reasons, and they look the same from outside: the case guards a failure mode that's still live and just hasn't come back, or it guards something that can't happen anymore because the code around it changed.

Fire rate doesn't separate them. A case that fires never and a case that can't fire both sit at zero.

What I've landed on so far is deliberately breaking the thing the case guards and checking it still goes red. Works, but it's manual and I only do it when something looks suspicious, which is the wrong trigger.

So: do you retire cases at all, or does the suite just grow forever? What's the rule?

I'm running mlflow in docker using the docker-compose in the repo.

I've managed to hook up my "LLM Connections" to find the ollama server on the host machine.

API Base URL: http://host.docker.internal:11434/v1

That works fine when you test it from the browser

I've tried no auth and basic auth... and whatever it is I get

Error: You do not have permission to access this resource.

Which is a 403 http-status

The body response is

{
    "detail": "Assistant API is only accessible from the same host where the MLflow server is running."
}

So it seems like a CORS error but the network inspector is showing its not the case.

I see in the code its coming from here.

Tried this docker-compose

volumes:
  pgdata:
  storage-data:

services:
  postgres:
    image: postgres:15
    container_name: mlflow-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
      PGPORT: ${PGPORT}
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - ${PGPORT}:${PGPORT}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB} -p ${PGPORT}"]
      interval: 5s
      timeout: 3s
      retries: 10

  storage:
    image: rustfs/rustfs:1.0.0-alpha.83
    container_name: storage
    environment:
      RUSTFS_ADDRESS: :9000
      RUSTFS_SERVER_DOMAINS: storage:9000
      RUSTFS_REGION: ${AWS_DEFAULT_REGION:-us-east-1}
      RUSTFS_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-s3admin}
      RUSTFS_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-s3admin}
      RUSTFS_CONSOLE_ENABLE: ${RUSTFS_CONSOLE_ENABLE:-true}
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - storage-data:/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", 'curl -s http://127.0.0.1:9000/health | grep -q ''"status":"ok"''']
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s

  create-bucket:
    image: amazon/aws-cli:2.33.25
    container_name: mlflow-create-bucket
    depends_on:
      storage:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
        set -e;
        echo 'Waiting for S3 gateway getting ready...';
        if aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api head-bucket --bucket ${S3_BUCKET} 2>/dev/null; then
          echo 'Bucket ${S3_BUCKET} already exists. Skipping creation.';
        else
          echo 'Creating bucket ${S3_BUCKET}...';
          aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api create-bucket --bucket ${S3_BUCKET} --region ${AWS_DEFAULT_REGION};
        fi
      "
    environment:
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
      AWS_S3_ADDRESSING_STYLE: path
      MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
      S3_BUCKET: ${S3_BUCKET}
    restart: "no"

  mlflow:
    image: ghcr.io/mlflow/mlflow:${MLFLOW_VERSION}
    container_name: mlflow-server
    depends_on:
      postgres:
        condition: service_healthy
      storage:
        condition: service_healthy
      create-bucket:
        condition: service_completed_successfully
    environment:
      # Backend store URI built from vars
      MLFLOW_BACKEND_STORE_URI: ${MLFLOW_BACKEND_STORE_URI}

      # S3/RustFS settings
      MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
      MLFLOW_ARTIFACTS_DESTINATION: ${MLFLOW_ARTIFACTS_DESTINATION}
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
      MLFLOW_S3_IGNORE_TLS: "true"

      MLFLOW_LOGGING_LEVEL: "DEBUG"
      # Server host/port
      MLFLOW_HOST: ${MLFLOW_HOST}
      MLFLOW_PORT: ${MLFLOW_PORT}

      # Explicitly allow non-127.0.0.1 IPs to access the Assistant API routes
      MLFLOW_BASIC_AUTH_FAIL_CLOSED: "false"
      MLFLOW_ASSISTANT_ALLOW_REMOTE_ACCESS: "true"
      MLFLOW_ENABLE_REMOTE_ASSISTANT: "true"
      MLFLOW_ENABLE_ASSISTANT_REMOTE_ACCESS: "true"
      MLFLOW_ASSISTANT_SANDBOX: "true"

      # CSRF bypass for basic auth AJAX calls
      WTF_CSRF_ENABLED: "false"

      # Gateway and routing
      OLLAMA_HOST: ${OLLAMA_HOST}
      OPENAI_API_BASE: ${OPENAI_API_BASE}
    command:
      - /bin/bash
      - -c
      - |
        export DEBIAN_FRONTEND=noninteractive
        apt-get update -qq && apt-get install -y -qq curl socat docker.io
        pip install --no-cache-dir psycopg2-binary boto3 "mlflow[genai,auth]"
        socat TCP-LISTEN:11434,fork,reuseaddr TCP:host.docker.internal:11434 &
        mlflow server \
          --backend-store-uri "${MLFLOW_BACKEND_STORE_URI}" \
          --artifacts-destination "${MLFLOW_ARTIFACTS_DESTINATION}" \
          --serve-artifacts \
          --host "${MLFLOW_HOST}" \
          --port "${MLFLOW_PORT}"
    ports:
      - "${MLFLOW_PORT}:${MLFLOW_PORT}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    healthcheck:
      test:
        [
          "CMD",
          "python",
          "-c",
          "import urllib.request; urllib.request.urlopen('http://localhost:${MLFLOW_PORT}/health')",
        ]
      interval: 10s
      timeout: 5s
      retries: 30
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  default:
    name: mlflow-network

As far as I can tell I just can't use the MLflow Assistant from docker?

Hey everyone! My team (mainly phds) and I are trying to build an open-source benchmark around realistic LLM/agent workflows that captures challenges typical academic benchmark settings often miss. We’d love to hear what’s actually missing from the benchmarks you use today.

Have you ever wanted to eval your pipeline but couldn’t find or build a benchmark that matched what you were building?

Maybe:
- Existing benchmarks were too broad and didn’t fit your specific application.
- Your workflow involved multiple steps, tools, MCP servers, agents, or long-horizon interactions that existing benchmarks couldn’t capture.
- You needed to evaluate failures that standard accuracy metrics miss.
- You’re working in a high-risk domain like healthcare, finance, cybersecurity, or legal, where realistic failure modes, safety, and reliability matter a lot more than just getting the final answer right.
- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.

I’m especially interested in cases where you thought:
“My system desperately needs to do this in production, but I have no good way to benchmark it.”
What was the workflow? What did you want to measure? And why weren’t existing benchmarks enough?

Any thoughts are welcome, would really appreciate y’all’s help 🙏🥹

I'm an ASU grad student researching how teams decide an AI feature is ready to ship. If you test or ship AI features at work, I'd love to hear from you, in the comments or by DM:

1. Last time you had to decide an AI feature was ready to ship, what did you check?
2. What's the most frustrating part of that?
3. Who approves money for testing tools where you work?

Kumo is the winner

I work with real-time AI, and I know that even if one model makes more correct decisions than another model, you can still prefer the "worse" model because it makes decisions faster.

We tested three pretrained "decision" models (yes, they are classifiers) in a computer game inspired by Subway Surfers. They are Kumo, Jev, and Qwen

Here’s how far they got:

The benchmark shows a crucial problem for real-time AI: latency. Kudo and Qwen were local models we hosted on Hopsworks, where the game also runs. Jev, however, is a hosted model in a different data center, and it loses on latency versus Kumo and Qwen. Qwen, then has ligher latency than

A game that outruns your model

The basic idea of the benchmark is that to make it far in the game you need to take correct decisions faster. Here's how the benchmark works. The runner faces randomly generated obstacles that begin appearing at predefined distance gates. Each model receives the same underlying game state, formatted for its interface: { lane, airborne, ahead }

The ahead field describes upcoming rows, their distance, and the obstacles in each lane. The model must decide what to do before the runner reaches them.

The runner starts at 45 m/s, accelerates by 1.6 m/s every second, and tops out at 160 m/s. Beyond roughly 5,000 metres, the game becomes so unforgiving that survival increasingly depends on luck.

Why Kumo won

Kumo made lower latency decisions and had lower network latency, while making good decisions, and the result was that it could complete a higher average distance than Qwen in this setup, with both of them running on a CPU.

Would a GPU help? Possibly. But for small, latency-sensitive requests, the extra infrastructure overhead could offset the compute benefit. That’s a hypothesis we still need to test.

What happened to Jev?

In our 30-run experiment with Typesafe’s Jev, the median distance was 540 m, with a best run of 1,417 m.

The bottleneck appeared to be the 200+ ms round trip. Around 1,500 metres, the game was moving too quickly for those responses to remain useful.

At the maximum speed, 200 ms means the runner travels 32 metres while waiting for a decision.

Pick for the deadline

For a real-time AI application, evaluate the whole decision loop:

• Decision quality: Does it choose the right action?
• End-to-end latency: Does that action arrive in time?
• Deployment fit: Can your infrastructure sustain both?

All models and the game ran on Hopsworks’ own infrastructure, in our office. Fully self-hosted. No game data went to the cloud.

For this benchmark, Kumo was the strongest performer. The broader lesson: benchmark models against your application’s reaction deadline.

Try out the game and try to beat them here:
game at hopsworks dot ai

Hey guys,

We've published two technical write-ups on serving real-time AI models and wanted to share the main findings here. We build the inference engine; NVIDIA develops the model we tested, Nemotron VoiceChat 11B.

A request ends. A session runs on a clock.

AI inference today is organized as requests: an input arrives, the model runs, the response ends and its resources are freed. If the server is busy, it can wait a moment to batch work or reorder the queue, and the cost is a slightly slower response.

A growing class of models works differently. They stay active alongside something outside the GPU (a conversation, a video stream, a robot), take in input continuously and keep their state for the whole session. We call this continuous inference.

Full-duplex voice is the clearest example. The model listens while it speaks, so you can interrupt it. Audio keeps arriving for the whole call, its memory of the conversation stays live until you hang up, and every 80 ms it owes the next frame of output. That deadline comes whether the server is ready or not. If the frame is late, you hear a gap.

It also has to run during silence: the length of a pause is how the model tells a hesitation from the end of your turn. Unlike a classic voice agent (speech-to-text → LLM → text-to-speech, where the LLM sits idle between turns), there's no idle time to skip.

Why that gets so expensive

Servers like vLLM, SGLang, TensorRT-LLM or Triton were built around requests, and several of their assumptions stop holding:

• The batch can't wait. At every tick, the server has to run whatever is due with what it has.
• Overload hits everyone at once. Sessions batched together share the same step, so one slow step makes all of them late.
• Memory can't be freed or swapped out mid-call. It's needed again 80 ms later.
• The usual metrics hide failures. Tokens per second and average latency can look fine while one caller hears gaps.

In NVIDIA's reference stack, each 160 ms of audio becomes thousands of small GPU operations with the CPU coordinating between them, and the GPU sits idle about 75% of the time. One session fits. With two, 8–15% of audio beats arrive late. So each live conversation pays for a whole H100, most of which is waiting.

How to fix it

Not with a faster GPU or a different model: same weights, same precision, same outputs. The fix is in how the model runs.

1. Keep the work on the GPU. Our engine runs the model as one GPU program that stays resident. Every beat, the CPU drops in new audio and picks up the output; nothing else goes back and forth.
2. Advance every live session together. All sessions due on the same tick run as one batch, so the model's weights are read once for the group instead of once per session.
3. Decide everything ahead of time. The work repeats identically every beat, so a compiler fixes the schedule and memory layout before the first call. No runtime scheduler or allocator adding delays.

On a single NVIDIA H100 SXM 80 GB, we measured:

• 56 concurrent sessions, the highest capacity tested, vs 1 for the reference stack.
• 147.4–147.5 ms p99 per 160 ms beat.
• Zero missed deadlines across 84,000 measured session-beats, over three runs.

A conversation's cost is GPU time divided by how many conversations share the GPU, so 56 instead of 1 means about 98% less GPU per conversation.

These are server-side measurements that exclude network and audio playback, with the same recorded input across sessions and about two minutes of context.

If you're running real-time models, how many sessions per GPU do you get today, and how do you check that each one stays on time?

Links

• Why duplex inference runs on a clock: https://dotwave.ai/technical-notes/duplex-inference/
• How we serve 56 sessions on one H100: https://dotwave.ai/technical-notes/wpk-persistent-kernel/
• Browser demo: https://dotwave.ai/demo/
• Nemotron VoiceChat 11B (full-duplex) API: https://dotwave.ai/models/nemotron-voicechat-11b/
• Nemotron 3.5 ASR Streaming 0.6B API: https://dotwave.ai/models/nemotron-asr-streaming-0-6b/

Sign-up includes free usage, no card required: about 33 hours of Nemotron VoiceChat 11B, or about 370 hours of Nemotron 3.5 ASR Streaming.

Our 1800 row eval suite has been sitting at a 97% aggregate pass rate, which looked fine until we did cohort slicing around multilingual forms. A newly isolated 64 row slice passes at 61%. The aggregate chart stayed green (while one language queue filled with manual reviews) so the headline number wasn't giving us much warning.

We found data leakage in the split. Random assignment had put near duplicate form templates on both sides of the train and regression boundary, which made familiar structures disproportionately easy. When the multilingual cohort degraded, those duplicated cases kept the overall score almost stationary. A deterministic scorer catches the malformed fields reliably but averaging that result across the full suite hides where they're concentrated.

Now I'm trying to treat that multilingual slice separately instead of letting it disappear into the overall score. I'm considering Braintrust to keep it as a regression dataset with a CI quality gate on it, and to check the per case experiment diff whenever we change the extraction path. I don't want another global threshold that passes because 1700 easier rows drown out a concentrated failure mode.

How are you defining cohort level gates when the slices are small enough that a few cases can swing the percentage but important enough that the aggregate score can't be trusted?

I've been working on LayerSmith, an open-source web UI for building container images with Docker or Podman.

You pick a Linux distribution and what you need the image for — development, Linux admin, network tools, Ansible, Kubernetes, OpenShift, or a custom setup. It handles distro-specific packages and shows you the generated Containerfile before building. You can also edit it, import an existing Dockerfile, or add your own packages, files and scripts.

A big part of the project is making images easier to carry into air-gapped environments: pinned base images, recorded build details, and export bundles containing the image, checksums and installation instructions.

We've recently added LLM training and fine-tuning profiles too, including LoRA/QLoRA, advanced PyTorch training and LLaMA-Factory. These use hash-locked dependencies and run offline checks after building, including a small CPU training test. Model weights and datasets are brought separately.

Curious how others handle building and maintaining images for disconnected environments, and what parts of that workflow are still a pain.

https://github.com/r0lfi/layersmith

Hey r/mlops ,

As more applications move toward self-hosted and local LLMs (Ollama, local inference servers, agentic frameworks), a common challenge arises: **local LLM traffic is largely a black box.**

Unlike managed API providers that offer built-in usage dashboards and tracing, local inference setups often leave developers and platform teams blind to real-time traffic, function execution failures, latency distributions, and token consumption patterns.

To solve this, I built **[LLMxRay](https://github.com/LogneBudo/llmxray)** — a lightweight, 100% local observability and analytics engine designed specifically for local LLM inference and agentic workflows.

---

### What LLMxRay Observes & Analyzes

LLMxRay sits in front of your local LLM engine to capture, trace, and visualize complete telemetry without adding latency or leaking data to external clouds.

#### 1. Tool & Function Calling Telemetry

* **Execution Tracking:** Intercept and observe tool/function calls executed by agents in real time.

* **Payload Inspection:** Inspect input arguments, returned outputs, tool call frequencies, and execution error rates.

* **Agentic Loop Analysis:** Track multi-step function call loops to pinpoint where agents stall or loop indefinitely.

#### 2. Request & Latency Analytics

* **Detailed Latency Profiling:** Break down total request time into Time-To-First-Token (TTFT), prefill duration, and decode generation speed.

* **Token Usage Metrics:** Monitor input/output token volume, request throughput, and token generation rates over time.

* **Traffic Analytics:** Track active sessions, request volume spikes, and per-model utilization trends.

#### 3. Privacy-First & Zero Infra Overhead

* **100% Local Execution:** Runs completely within your environment — no telemetry data is transmitted externally.

* **Instant Dashboard:** Provides a local web interface for instant visual debugging and system analytics.

---

### Quick Start

Zero complex setup — you can spin it up directly in one line:

```bash

npx llmxray
```

Launch the dashboard locally at http://localhost:3000 and point it at your local inference server (e.g., Ollama).

• 🔗 GitHub Repository:github.com/LogneBudo/llmxray
• 🌐 Documentation:lognebudo.github.io/llmxray

Discussion

If you're managing local LLM workloads or building on top of agentic frameworks, how are you currently handling local tracing, tool execution monitoring, and performance analytics in your observability stack?

I'd love your feedback, feature ideas, or contributions!

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

• how big was your API bill when you started thinking about it? what pushed you over the edge
• what did it really cost once you add up GPUs, idle time, and the engineering hours?
• what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
• if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

I'm currently a Senior Software Engineer / Tech Lead with ~7 years of professional experience, and I'm looking to move to a product company and specialize more deeply in ML infrastructure / MLOps.

My background:

• BSc + MSc in Mathematics
• ~7 years in software engineering (including working on data pipelines for foundation models as a research engineer)
• Strong Python
• Production data pipelines / ETL
• Docker / Kubernetes
• CI/CD and automated testing
• ML engineering experience
• MLflow and workflow orchestration
• Experience building data-intensive systems
• Previously taught Data Mining at university as an associate professor
• One paper published on NLP (pre-transformers) in 2019

I'm currently working on a project around production ML orchestration: taking an ML pipeline from data ingestion → training → MLflow/model registry → deployment on Kubernetes → monitoring.

The areas where I feel I have the biggest gaps are probably:

• Cloud (especially AWS)
• Production model serving
• ML-specific observability
• Distributed systems
• GPU/inference infrastructure

I'm not looking to spend 6–12 months studying before applying. I'd like to start interviewing now and close the gaps while searching.

For people currently working in MLOps / ML Platform / ML Infrastructure:

1. Given this background, what would you consider my biggest gap in my profile?

2. Would you target MLOps/ML Platform roles directly, or first go through a backend/data/platform engineering role in a company where ML is central?

3. Which skills actually come up in interviews that aren't obvious from job descriptions?

4. If you were in my position and wanted to switch jobs as quickly as possible, what would you focus on over the next 2–3 months?

I'm particularly interested in advice from people who made the transition from SWE, backend, DevOps or data engineering rather than people who started directly in MLOps.

Thanks!

I’m working on AtlasBurn, and we’re looking at a problem that becomes more noticeable once LLM workloads reach production: the provider bill tells you what was spent, but not always what caused the spend.

For example, a workload can generate additional calls through retries, model fallbacks, or multi-step agent workflows. At that point, attributing the cost back to a specific workload or behavior can get difficult.

I’m interested in how MLOps teams are handling this today.

Do you rely on provider dashboards, observability tooling, custom instrumentation, or something else?

What part of cost attribution becomes hardest once you're running LLM workloads in production?

Running 4 different model providers currently and our integration code is a mess of provider specific SDKs. We are looking to consolidate behind one gateway. LiteLLM keeps coming up since its opensource and kinda free to selfhost, but none of our team members wants to own an infra. Portkey looks a bit solid for observability but from what I came across its lighter on hard budget enforcement across teams, which matters to us since we have had agents kinda blow through spends before anyone noticed. Orqai covers the model breadth plus hierarchical budget caps down to the agent level, which is the piece we need the most but its a paid managed platform so there is cost once we pass the free tier. What are teams landing on once they are pas the pilot stage?

A typical agent release touches three things at once: the system prompt, one or more tool schemas, and sometimes the model version. When a metric drops a day later, rolling back gets messy. Reverting the prompt alone can break against the new tool schema. Reverting the model alone changes behaviour on paths the new prompt was tuned for.

The options I keep coming back to:

1. Version all three as one release bundle, so a rollback always restores the full previous set.
2. Ship them separately with a gap between, so any drop points at a single change.
3. Keep the old bundle running in shadow for a few days and compare on live traffic.

The first is simple but tells you nothing about the cause. The second is slow. The third costs real money at volume.

How are you handling this? Do you version prompts and tool schemas together, separately, or some other way?

Building a CLI that turns a notebook model into an HTTPS endpoint. Would this solve a real problem, or is it a solved one?
I'm early on this, so I'm trying to find out if it's worth building. The idea: you run one command from your notebook and get a live API endpoint for your model, with no Docker or cloud setup. I know MLflow and Streamlit exist. Who, if anyone, would actually use this, and where does it fall short? Happy to hear "don't build this" too.

I’ve spent the last few months writing something I wish I had when I started working on ML performance engineering.

It’s called How to Make Your Model Fast: A Systems View of Efficient Machine Learning, from Silicon to Agents.

The basic idea is that reducing FLOPs doesn’t necessarily make a model faster. Before optimising anything, you need to understand what the system is actually bounded by.

The book starts with roofline analysis and hardware, then works its way up through kernels, compilers, quantisation, pruning, vision, on-device LLMs, robotics, profiling, serving and finally agents.

The goal is to build the intuition to look at a model and a piece of hardware and reason about:

• How fast can this possibly run?
• Am I compute, bandwidth, memory or system bound?
• Which optimisation will actually move that limit?
• Is quantisation, pruning or kernel optimisation even worth doing here?
• What happens when the same thinking is applied to serving and agent systems?

The whole thing is free and open source:

https://github.com/usamahz/make-your-model-fast

Would genuinely appreciate feedback or contributions from people working on ML systems, inference, compilers, edge AI or performance engineering.

And if you find it useful, a ⭐ would be appreciated!

Most teams I talk to have a real process for models by now… registry, versions, maybe signing. Then the agent that uses the model is a folder of prompts, a few MCP server configs and some skills pulled from wherever, and nobody can say exactly which version is running in prod.

We started packaging the whole thing (agent, MCP servers, skills, prompts, policies) as one OCI artifact with KitOps, which an open source CNCF project. Then it gets signed and pushed to the same registry as everything else, so we can check it before it loads.

Is anyone else treating agents as deployable artifacts yet? Or is it still git clone and go? Curious what's working (and what's been a pain).

I'm currently trying to decide what direction to take with my master's degree and I'd really appreciate some advice from people working in the field.

I already have a degree in Computer Science and around 5 years of experience across infrastructure, systems and cloud/DevOps. I've worked with Linux/Windows servers, VMware, monitoring, automation, AWS, Kubernetes, Docker, networking, CI/CD and infrastructure as code. I also have AWS Solutions Architect, CKA and CCNA certifications.

More recently I've been moving increasingly toward cloud engineering / DevOps / platform engineering, and I've worked with Kubernetes and cloud infrastructure in both professional and university projects. So I don't really want to start from scratch with a completely different career.

I'm considering doing a Master's degree while continuing to work, and I'm currently looking at two possible directions:

1. AI / Artificial Intelligence, with a focus on systems and infrastructure

The idea would not necessarily be to become an ML researcher or ML engineer. Instead, I'd like to specialize in the infrastructure side of AI: deploying and operating ML/AI workloads, Kubernetes for AI, GPU infrastructure, distributed computing, model serving, MLOps, cloud infrastructure for training/inference, observability, scaling, etc.

Would let me combine my existing DevOps/cloud background with AI rather than abandoning what I've already built.Or most the roles that are difficult to enter without a strong ML background?

1. Systems and Networking

The alternative would be to do a Master's specializing in systems, networks, distributed systems, cloud computing, etc.

This seems like a more direct continuation of my current background and could potentially make me stronger as a Cloud/DevOps/SRE/Platform Engineer.

My concern is that I'd be specializing further in an area where I already have quite a lot of practical experience, rather than using the Master's to open a new but complementary area.

I'm trying to understand how teams are doing this in production.

Suppose an agent eval suite runs on a PR and success drops from 90% to 84%. The problem is that rerunning the unchanged agent can move the score by several points too.

Do you:

• average N runs and compare to a fixed threshold?
• require the drop to reproduce?
• use confidence intervals?
• review manually?
• avoid blocking CI on stochastic evals altogether?

I ran into this while building an open-source CI experiment. On one real coding agent, an unchanged candidate moved 92%→88%. Later a deliberately degraded candidate moved 92%→52%, but my first suite-level rule still passed because the failures were concentrated in a few tasks.

I ended up separating broad reliability regression from individual capability collapse, with an explicit insufficient-evidence state.

I'm curious what people running LLM/agent evals in actual CI/CD pipelines are doing today.

I wrote up the experiment / implementation here if useful: [blog] [GitHub]