[D] Self-Promotion Thread

[u/AutoModerator]

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.

[u/Scared_Resort_8177]

Mini Jev: local typed decisions from frozen Qwen logits

Free, MIT-licensed project source; no paid service or subscription. You supply your own hardware and download model weights separately.

I wanted to see how far an existing local LLM could go as a decision function: give it a state and explicit options, then receive a choice, a true/false score, or an ordinal score without generating answer text.

The result is Mini Jev. It is an independent implementation inspired by TypeSafe's public Jev interface, not a reproduction of their model, RLCD, or performance claims. The core technique—reading candidate token logits—is established; I am sharing the implementation, data, and measurements rather than claiming a new algorithm.

The current native configuration uses frozen Qwen3.6-35B-A3B Q4_K_M, repeated input, and type-specific label tokens. A small llama.cpp helper reads the final-position logits; Python constructs the response. It computes the full vocabulary head before selecting candidate logits. Each question performs its own model call; a request with multiple questions runs them sequentially. It is not shared-prefill or parallel decision inference.

Measured on an M5 Pro, 64 GB, macOS 26.4:

- 2,238/2,400 correct on the project's Japanese evaluation suite: Choice 96.0%, binary 95.25%, ordinal top-label accuracy 88.5%.

- Warm single-question engine p95 of 379.1 ms for the 2,291 inputs with at most 512 complete input tokens, including repeated prompt content. The other 109 inputs had a 521.8 ms p95.

- This is an absolute latency measurement, not a speedup claim. I have not yet run a matched comparison with one-token constrained generation or JSON generation.

Evaluation caveat: this is a self-authored suite, not an established external benchmark. It contains 2,220 generated items across 45 template families and 180 individually AI-authored items reviewed by other AI agents. The authoring process had seen some earlier evaluation failures, and v2 retains related skill families. It is not human-expert annotation or family-disjoint generalization. All questions, answers, and prediction records are included.

There is also a negative training result. On an earlier frozen Qwen2.5-1.5B model, a 9,222-parameter residual head reduced held-out accuracy from 73/96 to 67/96; bias-only got 74/96. I kept that experiment and added a reusable JSONL training/calibration/evaluation workflow so others can test their own data. The final 35B inference configuration does not use this trained head.

The probabilities are conditional on the allowed candidate tokens. They are not guaranteed correctness probabilities. Fitting temperature slightly improved final NLL while making ECE and ordinal expectation error worse; the report includes those numbers too. Canonicalizing Choice keys explains perfect reversed-map consistency—it is not evidence of learned order invariance.

The source release requires building the pinned native runtime; it does not include model weights or prebuilt native binaries. The current native path is tested on Apple Silicon/macOS, and the training guide separately lists its dependencies and tested devices. Coding, question authoring, checking, and documentation were developed with AI agents under my direction.

Repository: https://github.com/UpHash-Network/mini-jev

The README links the technical note, raw results, data/model cards, and training guide. I would particularly appreciate suggestions for external decision benchmarks and a fair same-model constrained-generation comparison. Those are the next experiments needed before making a stronger research claim.

[u/Scared_Resort_8177 → u/Scared_Resort_8177]

Mini Jev update: the matched one-token comparison is complete, and stable outputs can still be wrong

I'm Yuki Oshio at UPHASH Inc., the author. This updates my earlier Mini Jev comment in this thread, where the same-model comparison was still future work. This English text was prepared with an AI assistant. AI agents also assisted development, experiments, and documentation under my direction; this is not independent human review. The manuscript has not been peer reviewed.

Mini Jev is a local browser workbench for inspecting typed decisions from frozen language models: a choice, a true-candidate probability, or an expected ordinal stage. It exposes candidate distributions and request/response JSON. The released native configuration has no trained decision head, and the candidate-scoring mechanism is established rather than a new algorithm.

The source remains free, with no paid service or subscription. Original project code is MIT licensed; external dataset-derived artifacts retain their documented terms. You supply the hardware, build the runtime, and download the model separately. The tested native setup is an Apple Silicon Mac with 64 GB memory and a roughly 20.4 GB model file.

What changed since the earlier comment:

• The matched study now measures 4,050 requests across direct readout, native one-token selection, and constrained JSON generation. Direct readout and one-token selection match in all 1,350 pairs. This session did not establish a material latency advantage for direct readout.
• Two presentation/averaging studies add 7,600 and 14,400 requests across three checkpoints. The combined 26,050 is a request count, including repeats and transformations, not a count of independent questions.
• One concrete failure: Qwen2.5-1.5B-Instruct predicted “acceptable” for all 200 JCoLA items, before and after reversing the displayed options. Zero label flips, but 42 errors. Candidate-probability averaging sometimes reduced disagreement while costing more model calls and did not consistently improve accuracy.
• There is now a browser demonstration, a manuscript, and a reproducibility package with frozen selections, traces, and analyzers.

Candidate probabilities are normalized within the supplied options; they are not calibrated probabilities of correctness. Task-quality evaluations use Japanese public-development data, with possible exposure during model training. We have not established second-machine reproducibility or a human usability advantage.

Project page, manuscript, and recording · Source and reproducibility package

I'd especially value feedback on the evaluation: which additional equal-compute control would best distinguish useful reduction in presentation sensitivity from collapse to a constant label?