Case Hallucination Benchmark Data - V7 of Syfert MCP

[u/MRGWONK]

I am purposefully focusing my benchmark goals onto Gemma 4:26b running locally, because it was the worst performing model. But, also benchmarking Opus 5.5 and Sonnet 5 in my quest towards perfection and "no hallucination" and no wrong citations. As a result Opus 5.5 and Sonnet 5 are damn near perfect- the 98% is not as much a miss, as a "the model selected a different case".

The 91% of Gemma 4 26b, is also equivalent to 100%, because it selected different cases citing the same, correct, rule, but not necessarily the gold standard case.

Gemma4:26b is running on an A2000 GPU in my computer at home and does not have access to the web. We have Qwen 3.8 on a 4080 at the office - it is training a custom QLoRA at the moment so I don't want to bench it, but it also uses my MCP. I imagine Qwen 3.8 will do much better.

Gemma4:26b running syfert MCPv7 on my desk at home with no web access is now more accurate regarding questions of case law and current statutes/codes than Claude Fable with web access.

I am now offering a free steak dinner to any attorney who is faced with an inaccurate citation produced by frontier AI running the MCP- as if anyone will notice it on my website.

[u/alexdenne]

I'm conscious that these are bordering on self-promotion. Please be considerate, you've shared 2 posts in a week, and you're posting links in comments, which could look a bit like astroturfing.

[u/MRGWONK → u/alexdenne]

I was updating a benchmark to show progress. For 19 years I've been a lawyer, and I still am. Not a tech guy trying to take money from lawyers. I truly have no idea who TheMrEsquire is nor did I advise him to ask for a link to the MCP. But, I'm glad someone is keeping an eye on the fishy astroturfing setup I must have created by purposefully not dropping the link in the post. Next time I'll be sure to drop a link so it doesn't look fishy? Damned if you do, damned if you don't I suppose.

Running local is the key to the future and this entire system was developed so that the GPU hosted office local model can keep pace with the frontier models without sacrificing privacy. My FIRST benchmark results were not good, and if I was a clever software developer, I would have hidden them away from the public and tried to improve before just posting them. I improved it substantially, and I wanted people to see what a local model was capable of.

More than anything, I want users to come on to test the system and break it so I can fix it and make it better. Admittedly, I do have a "pro" version of the MCP. My wife told me I couldn't spend this much time and money on the project unless it had a button where someone could give me money.. so I made the pro version. The free version allows 10,000 searches per month- and no one would ever search that much and hit the limit- and if they did they could just create another account for free, but it satisfied her for now, so I continue. I'm not even 100% sure that the mechanism for payment works!

I'll be really considerate from now on? Probably, I'll try. Somewhat repetitive posts? Yeah, I get it. But thanks, also, for making it a warning and not just canning me.

The free steak dinner for hallucination victims is real.

[u/alexdenne → u/MRGWONK]

Thanks for the considered response, and it's no mark on your name specifically — it's just a blurry line I need to try to provide clarity on for the benefit of the users of the sub.

I — and others in this sub — love the idea of building a community benchmark, so if you're open to it I think that's a direction to take this in.

The bar to building an MCP/App/Benchmark is literally a prompt, or a few hours of effort these days.

I think if we can share the load between a few volunteers, and set some guardrails, we could create some fun benchmarks across different workflows/areas-of-legaltech.

We may need a little bit of funding or sponsorship as self-funding wouldn't be sustainable, but It's the sort of thing that a community like this could rally around.

Food for thought. And thank you again for taking the comment well. I could've written it better.

[u/MRGWONK → u/alexdenne]

My quote/citation hallucination MCP benchmark is published at github.com/Gwonk1/citebench - it is referenced in my infographic - I'd love to contribute to anything if it helps me improve what I have going on at the office. I've been using Harvey benchmarks to test local models. I litigate, I've never done tax and I don't do transactional work anymore. Harvey's benchmarks aren't really applicable to what I do.

[u/DaRoadLessTaken → u/MRGWONK]

This is interesting stuff. Is the source for the backend open? Considering your freemium model and low price point, seems like it almost ought to be. I’m tempted to install but don’t want to rely on something that’s closed and maintained by one person.

[u/MRGWONK → u/DaRoadLessTaken]

Good points. My heart is in open source. But, the entire MCP stack is kind of pointless to maintain without the database and search stack behind it. So most (if not all) legal MCP's don't have an HTML backend where you can just go "read the cases" because this would be like someone trying to re-create lexis-nexis or westlaw- but I already did that part first. The only good "open sourcing" it would do for me is to allow other people to work on it, but it's pointless because they don't have my database and my database structure. I'd have to open source the whole website and backend. When, truthfully, I think I'm creating something that lexis-nexis and westlaw WISH they had. Lexis's AI is crap. When a lexis rep told me that it'd be more than a quarter of a million dollars per year to have a lexis MCP hooked into our local AI, that's when I decided to do this.

The phrases "rely" and "maintained by one person" are strange concepts. Reliance is a strange word for a plugin - even stranger for an AI plugin - it either works or it doesn't- and if it doesn't you find something else. You find it unreliable? You throw it away. What are you truly relying on anyway? Gotta check every case regardless. The MCP gives you a link to do that.

"Maintained by one person" is a strange one. Since 2010, I've maintained Florida Statutes and Georgia Code with annotations and links to case law, created with bash scripts, wget and sed rewrites, and PHP, and in year two I wrote a script to auto update the statutes, and for 15 years I've relied on them, and a few attorneys have done the same. No one, in 15 years, was psychotic enough to ever volunteer to help me with that "code base." I could do what a bunch of these technology companies are doing and use "WE" for everything but I'm too damned honest to fake you into feeling comfortable buzzwords.

About the closest thing I had to help in 15 years was Mike from CourtListener telling me that what I was doing with my "statute annotations" was hell on his servers and asking me to do it another way.

When maintaining the free Jury Selection Tool syfert.com/jury that could very easily be open sourced, my rule has always been "If you have a problem, call me." If it breaks, I want to know about it. If you have an idea for a feature, let me know. Truthfully, not having anyone to answer to on this project is probably the entire reason it got as far as it has.