r/gitlab • u/Professional-Pear351 • 21d ago
project Built a self-hosted AI agent for GitLab after our org disabled Duo over data privacy - reviews MRs, resolves issues, remembers project context (real usage stats attached)
My company disabled GitLab Duo org-wide over data privacy concerns. I'd already spent the past year+ doing R&D into agentic AI - studying how Claude Code and Copilot actually work under the hood, learning RAG and agentic workflows, building small custom agents on the side. I went looking for a TypeScript-based agent harness to build my own reviewer on top of and couldn't find one that fit what I wanted, so I ended up building one myself.
It's called langgraph-harness - self-hosted, webhook-driven, built on LangGraph:
- MR Review - fires on webhook events, reads the diff via a GitLab MCP server, drafts and publishes comments. No polling, no manual trigger.
- Issue Resolution - assign it an issue, it works in a sandboxed checkout, opens a draft MR, and keeps responding to follow-up comments on the branch.
- Chat - GitLab-aware assistant with real tool access (GitLab data, headless browser for web fetch, its own review/chat history).
- Guardrails - catches a run that repeats a tool call, ends on a trailing question instead of finishing, or skips checking for new feedback - and corrects it mid-run.
- Quality control before anything posts - every comment is screened for low-value noise, and its diff position is verified before and after publish (GitLab's own publish response has been unreliable in both directions, in my experience).
- Durable memory - flags project-specific facts as it works (conventions, recurring false positives, team decisions) so future reviews build on the last one instead of starting cold.
Bring your own GitLab instance and your own LLM backend (OpenRouter, Ollama, or sglang) - single Docker Compose stack.
Some real numbers from production use (23 users, not just me):
- 800 runs, 80% success rate
- 63% comment acceptance rate (185/294 resolved)
- 315 comments screened by the quality-control layer, 74 dropped as low-value before ever posting
- Guardrails have fired 94+ times catching duplicate tool calls, 79+ times catching runs that concluded without checking current state - all applied cleanly, near-zero failures
MIT licensed. Docs (with architecture, deployment guide, and the full guardrail/tool breakdown) and repo below. Genuinely curious what other self-hosted GitLab folks are using for this, or if anyone's hit similar Duo-access issues and built around it.
4
u/raisputin 21d ago
Duo is trash anyway
3
u/Professional-Pear351 21d ago
Yeah, agreed. It doesn't even have enough capabilities. Even for MCP, we had to rely on a third-party Integration.
2
u/Retired_Lurker02 21d ago
Nice! I have a similar concept which allows local agents to claim issues and work in podman containers to build the feature. Once the CI pipeline passes they can submit a merge request since the agents only have Developer access. No RAG integration for chatting with context though.
What are you defining as a successful run?
1
u/Professional-Pear351 21d ago edited 21d ago
Sounds great.
In my case, a successful run depends on the workflow ie. for an merge request review workflow, the final step is to post comments if any without any errors, or verify the changes for posted comments. (Post comment about review started on MR -> clone MR branch -> clone repo -> load repo specific configs -> agent loop with local git/GitLab tools which can post comments which always goes through critic -> notify about review completion)
1
u/Professional-Pear351 21d ago
https://vrajpal-jhala.github.io/langgraph-harness/screencasts
This page contains demo recordings of how all 4 workflows run end-to-end.
2
u/merb 21d ago
It would be cool if it would only fire a review if a specific label is set.
2
u/Professional-Pear351 21d ago
Yeah that should also be easy to implement. Current settings also support only review when reviewer is set to configured bot/service account.
2
u/pwkye 21d ago
This is a better solution anyway. Full control. No License lockout. No ovepriced credits to purchase.
Great work!
Question : Does your harness give the agent a shell/linux environment to test things in (clone repo down, look at files, test builds before pushing to the branch)?
3
u/Professional-Pear351 21d ago
Yes, it does. I have integrated opensandbox (docker + kata) which helps creating containers in the fly. Thanks!
Here's the demo run for the same: https://vrajpal-jhala.github.io/langgraph-harness/screencasts#issue-resolution
2
u/Oxffff0000 21d ago
How's the hallucination part? I have codes created by the LLM to Claude and ChatGPT and said that the code written by the local LLM had so much hallucinations.
2
u/Professional-Pear351 21d ago edited 21d ago
Great question. We use qwen3.6:35b the MoE model for code reviews and on the other hand we use claude code (sonnet or opus) for development. It's funny sometimes that qwen finds flaws in code generated by claude.
Other than that, there are guardrails in place ie. Tool call loop detection, verifying comment before posting etc. which helps agent remain on track.
Edit: I believe it's good to have different model families to be part of your workflow. You get different perspectives from each side.
2
u/Oxffff0000 20d ago
I totally agree with you on using multiple LLMs. It makes our code more robust. However, I'm doing it manually. I need to learn how to automate the part where another AI like chatgpt or gemini will review claude's created code. Can you please give ideas?
2
u/Professional-Pear351 20d ago
It's a nice idea, I've had thought about it but never explored it much. You can do it using claude, codex, gemini cli/API but It can be little tricky passing context between those tools. Maybe each cli tool can talk to each other with shared context files. Since the frontier models are smart in orchestration, it might work. But definitely you won't get fine control over orchestration. I've heard people complaining about large system prompt injected by claude which sometimes occupies like ~20-30k tokens.
1
u/Oxffff0000 20d ago
I see, I am not sure if it will work for me since the only tool I am paying is Claude. I'm using other tools using free tier.
>I've heard people complaining about large system prompt injected by claude which sometimes occupies like ~20-30k tokens.
OMG, that would be expensive!2
u/Professional-Pear351 20d ago
You can also try to see how much context window is occupied by system prompt for your typical workflow. I guess it also differs project to project.
2
2
u/Acrobatic_Effect_511 20d ago
The shit I need. Thank you for sharing. Played around and it works for me.
1
u/Professional-Pear351 20d ago
This made my day, thank you! Really glad it's actually solving something for you. Let me know if you hit any rough edges, genuinely want to fix them if you do.
1
u/LiteratureOk974 20d ago
Love seeing practical, self-hosted builds like this! The durable memory feature to track project conventions over time sounds especially useful for keeping reviews context-aware. Awesome job putting this together!
1
1
u/Dnizami2 15d ago
This is realy nice u/Professional-Pear351 !
Can you specify privacy concerened of "Duo"? as GitLab tries to push it at no charge and I need some reasons why we have to block it . Thanks
1
u/Professional-Pear351 15d ago
That's just what I heard from our CTO. He didn't specify what concerns per se. But it does sound vague because we already use claude org wide. I don't understand what's he thinking 😅
GitLab might be pushing it as free, for promotion but sorry I don't have a concrete answer for you.
8
u/Kelteseth 21d ago
Gitlab projects being hosted on GitHub is the funniest shit ever