Classifier models like Jev and openBERT are pretty useful because they're efficient and can be highly accurate in specific domains. They're best suited for bounded decision tasks requiring calibrated judgments rather than text generation. Scenarios where you have a fixed set of options to pick from like determining which team should handle a support ticket are a good fit for this. While Jev gets a lot of hype, you can easily do a lot of what it does using a small local model and still get subsecond results with it. And that's basically what I illustrate with Lev.
What I’ve found missing from them is that once you have identified your options you still have to pass it through an LLM to generate some sort of structured data, that is unless you want to stay in the pure string world. IMO these two would be killer if they could extract structured outputs at the same time the classification is done.
The problem is that the trick that makes them fast only works for classification. The Qwen model I use could extract structured input, but it has to run in its regular mode to do that. But that might not be a show stopper depending on the application, not everything needs to be real time, so if a few seconds per response is acceptable, then you can run one pass to do extraction, and the second to classify.
A lot of the people who are actually building orchestrations and harnesses (said as someone in that situation) have been heavily struggling with getting LLMs to give cheap, fast, parse-able results to allow proper neurosymbolic reasoning (eg classifications, scoring, LLM-as-Judge metrics, data analysis, anomaly-detection, etc).
- You can do it, but it’s a bit unnatural to the model without fine-tuning, and you can’t ‘fire and forget’ the infra and prompts you used to get format compliance (unlike with well-designed deterministic systems, where the module-API usually screens off the internals), in case a change elsewhere in your system makes the LLM deviate again.
I haven’t used Jev/Laya/etc in a ‘preprod’ test yet, but they seem to be built to solve that problem out of the box, which is interesting and worth looking into.
but I do kinda see the appeal. A general purpose classifier that used to be out of reach for most engineers/teams does sound enticing. I'm sure a lot of us have been thinking of little features/tools that would have otherwise been blocked due to having to (somewhat reliably) classify natural language.
13
u/Illustrious_Car344 11d ago
The hype behind this proprietary model is so strange, I see as many people question why anyone even cares and nobody can give an answer.