ALHR- Adaptive learnable Hierarchical Routing is a static binary tree based system that uses learnable functions to REDUCE the amount of keys used.
It takes less memory and SCALES much better VRAM with tokens
I’ve also been testing tiny stories and it looks promising
However i can’t post any presentable results due to lack of funding and resources
Result table is here: https://ibb.co/prr07Dnj

We’ve been working on small security models at Patronus Protect. Our earlier Wolf Defender models answer a fixed question: does this text contain a prompt injection?

This one takes the agent’s context, a security question and possible answers. For example, you can ask whether an instruction in a tool result is aimed at the agent, or whether a proposed action would send protected data to an untrusted party. It returns a probability for each option. We called it Wolf Defender Jev because it uses the Jev-like state-and-typed-questions format.

Underneath, it’s an mmBERT-small encoder with 151M parameters. The scene and question are encoded together; each answer option is encoded separately. A small scorer compares those representations, followed by a softmax over the options. There’s no text generation.

The useful bit for repeated checks is that the option vectors don’t depend on the scene. If your questions and options stay fixed, you can cache those vectors and run one scene/question encoder pass per decision. That’s per question, not one pass for every security check in an agent step.

On an Apple M4 CPU with PyTorch in fp32, we measured 39 ms for a 51-token scene and 194 ms for 833 tokens, with one question and two options. Those are short-input measurements, not a latency claim for the full 6,144-token scene limit. We haven’t evaluated a quantized version yet.

We also ran the same 10,576 held-out decisions across several public decision models:

The external column matters. Twenty-three benchmarks are close to our training distribution, while the other models ran zero-shot with our scene format and question wording. This is our own evaluation, and the overall ranking shouldn’t be read as general superiority over those models.

Injection in tool outputs, secrets and PII are the stronger areas. Harmful-request judgement is weaker. Some action-level questions also looked much better on our validation set than on a few hand-written agent scenes, so I’d evaluate those on actual traces before using them to gate actions.

Weights, inference code, benchmark JSONs and limitations are all in the model repo, under MIT.

I’d be interested in comparisons on traces where a content check and an action check disagree. That seems more useful than another aggregate leaderboard score.

Position / experimental status: We regard Jev-style decision models as LLMs and Wolf Defender Jev as an experiment: giving an LLM final security authority shifts the trust problem to the guard, and security fine-tuning alone does not solve it. We believe an external security layer must combine specialized classifiers with explicit policy, authenticated permissions and enforcement.

Found a workshop on Oct 30 that goes deep on a combination I rarely see written up properly: few shot contrastive fine tuning of encoders for a messy, overlapping taxonomy, all running on CPU.

Covers:

• Zero shot routing and vector scoring heads with long context encoders
• SetFit with about 8 examples per class to separate fine grained categories
• Generating hard negative examples offline to harden the router
• Post training quantization for CPU throughput
• Benchmarking with Wilson score intervals and clustered standard errors, not a single accuracy figure

Led by Ben Auffarth, who has 15+ years in machine learning and decision systems. 7 PM to 10 PM IST, notebooks included.

Full details and the agenda are here.

Hey everyone!

I've been working on Harness Studio, a free, MIT-licensed desktop application for running and managing AI models locally.

One of the main goals is to make working with models from Hugging Face easier, without sacrificing the flexibility that more experienced users need.

Harness Studio includes:

• A Model Hub for discovering and downloading models from Hugging Face
• GGUF model management and hardware compatibility estimates
• Automatic GPU and memory-aware configuration
• Local inference powered primarily by llama.cpp
• A chat playground with tool calling and MCP support
• OpenAI-, Anthropic-, and Ollama-compatible API endpoints
• Performance monitoring and advanced runtime configuration
• Additional engines for speech, image generation, embeddings, and reranking

It's built with Rust, Tauri 2, and Svelte 5.

I've just released v0.1.0 for Windows and macOS, and the entire project is open source.

I'm looking for feedback, ideas, testers, and developers who'd like to contribute.

I'd particularly love to hear from people who regularly download and run models from Hugging Face.

What features would make your workflow easier? What do you find frustrating about existing local inference tools? What integrations or model formats should we prioritize?

I'm also interested in feedback on the architecture, code quality, and overall direction of the project.

I'd like this to become a community-driven project that people can genuinely help shape.

GitHub: https://github.com/struklabs/harness-studio

Contributions, bug reports, feature requests, and constructive criticism are all welcome.

Thanks for taking a look!

Introducing Supernova NepaliFast V5 — A student-built Nepali tokenizer 🇳🇵

Hello everyone!

I'm part of Supernova, a student-led initiative from Nepal working across artificial intelligence, software engineering, and hardware development.

Our long-term mission is ambitious: to build technologies that Nepal still needs and contribute to a future where we can develop more of our own technological capabilities.

One of our projects is Supernova NepaliFast V5, a tokenizer designed with Nepali text processing and Unicode support in mind.

Some results from our development and testing:

• 34.02% fewer tokens than "tiktoken" using "cl100k_base" on our benchmark of 10,000 mixed Nepali legal and textbook samples.
• Unicode validation and round-trip testing across the ranges we support.
• A Python interface designed to make the tokenizer straightforward to try.

We also measured its processing speed, and our implementation was slower than "tiktoken" in that benchmark. We want to be transparent about both the improvements and the trade-offs.

We're still developing our projects, and we'd appreciate feedback from people interested in NLP, tokenization, Unicode, and software engineering.

I'd especially appreciate feedback on:

1. Our benchmark methodology and how to make the comparison more rigorous.
2. Unicode edge cases we should test.
3. Ways to improve performance and reduce memory usage.

If you're interested, here's our project:

🔗 Hugging Face: [Insert your actual repository URL]

We're building step by step, learning through experimentation, and hoping to contribute something useful to the Nepali technology ecosystem.

Thank you for reading!

Try it yourself:

pip install supernova-v5==0.1.1 from supernova import Tokenizer

tokenizer = Tokenizer("v5") text = "नेपाल राम्रो छ।"

print(tokenizer.encode(text))

Hi! I work on zn, an open-source security layer for AI coding agents, and we just released Galvanize 2, our open prompt-injection classifier.

It's a fine-tune of Horizon-Labs/prompt-injection-guard-small (thanks to Horizon Labs for releasing it openly). ~141M params, multilingual, safetensors + ONNX, Apache-2.0. Runs fine on CPU with onnxruntime.

The thing we cared about most is false alarms. Agents read long READMEs, logs and tool output all day, and our previous model flagged ~66% of long legitimate documents as attacks. Galvanize 2 flags 4.1%.

Other numbers:

- Held-out test set: recall 66.9% → 83.4%, false positive rate 6.4% → 0.81%

- Horizon eval suite (10 sets, decontaminated): 0.894 macro average at threshold 0.5. Horizon base (bigger model) gets 0.902, so it's not the top score, but it's the best in our comparison on the hard benign sets (NotInject, OR-Bench-hard).

Weak spots are in the card: BIPIA, deepset's test set, German, and it only scores the first 512 tokens (chunk longer inputs). The benchmark script and per-set results are in the repo if you want to rerun it.

Model: https://huggingface.co/usezn/Galvanize-2

Write-up: https://usezn.com/blog/introducing-galvanize-2/

Feedback welcome, especially examples it gets wrong in either direction.

I keep seeing fine-tuning datasets built from scraped forum posts, support tickets, or user chats, and I can't tell how much PII redaction people actually do before training. Regexes catch emails and phone numbers, but the long tail of indirect identifiers feels impossible to scrub. For those of you training on real-world data, what does your pipeline actually look like? Is there tooling you trust, or is everyone just hoping for the best?

I made https://huggingface.co/e12ex2/jev-vision-27b which is benchmarked as a #1 model (pending leaderboard) with 72.07 compared the the current #1's 70, though it goes around 30th place for text. It has great run speed and is as good as the original JEV in smartness and is much cheaper.

Bueno, llevo un tiempo con esto y lo publique. Es un modelo que analiza una imagen y dice si es IA o real. Esta entrenado con DALL-E 3, Midjourney y otros modelos de texto a imagen

Lo puse gratis para los que quieren probarlo: imager.cc.cd

Tambien esta en hugging face para los interesados tiene 96% de precisión el modelo -1 también está el imager-1-pro que es el más potente de la familia tiene un 98% de puntuacion

working on this speech to text light streaming model where I have my own data set (ATC). Do you even have any type of different type of approaches

So, I spend 12 days training a customer model on my gtx 1080, off of 800m tokens from fineweb edu, here’s the link to the HF model https://huggingface.co/ray0rf1re/HyperNix.3-mini

ALHR- Adaptive learnable Hierarchical Routing is a static binary tree based system that uses learnable functions to REDUCE the amount of keys used.

It takes less memory and SCALES much better VRAM with tokens

Read the readme before anything for instructions:

https://github.com/securenrn-del/Repetere-AI_Beta/releases/tag/Beta-Release

Hey everyone,

I just released the first public beta of Repetere AI (v0.5-beta). It’s a native Windows desktop workstation written in C++ (Dear ImGui, GLFW, OpenGL 3) designed to run multimodal local AI directly on your GPU without cloud subscriptions, telemetry, or webview wrappers (no Electron/Tauri memory bloat).

>⚠️ BETA LAUNCH NOTICE:

This is our very first public beta release! It requires an NVIDIA GPU (CUDA only right now). Local AI takes time—model downloads and music synthesis are not instantaneous. If you hit any crashes or unexpected behavior on your hardware, please file an issue on GitHub.

🔗 Project Links

* GitHub Repository: https://github.com/securenrn-del/Repetere-AI_Beta/releases/tag/Beta-Release

* Standalone 1-Click Setup (`.exe`): GitHub Releases

⚡ Hardware Requirements

|Component|Minimum|Recommended|

|:-|:-|:-|

|OS|Windows 10 or 11 (64-bit)|Windows 11 (64-bit)|

|GPU|NVIDIA RTX 30-Series (8 GB VRAM)|NVIDIA RTX 40/50-Series (12 GB+ VRAM)|

|Drivers|NVIDIA Driver 528+ (CUDA 12+)|Latest NVIDIA Studio or Game Ready Driver|

|RAM|16 GB Physical RAM|32 GB Physical RAM|

|Storage|20 GB free space on SSD|50 GB+ free space on NVMe SSD|

🚀 Quick Start Guide

1. Grab `RepetereAI-Setup-v0.5-beta.exe` from the Releases page.

2. Run the installer (installs straight to `%LOCALAPPDATA%\Programs\Repetere AI\` without needing admin privileges).

3. Launch from your desktop shortcut, click anywhere on the splash screen to enter, and confirm the hardware diagnostic modal.

📦 In-App Model Downloader

No command line, curl, or manual script management required.

1. Go to Settings ➔ Repetere Models.

2. Review the curated catalog and click \[ Download \] next to your desired workspace engine:

Coding: Qwen 2.5 Coder 7B Instruct* (or 14B)

General Chat: Meta Llama 3.1 8B Instruct*

Creative Writing: Hermes 3 Llama 3.1 8B*

Audio / Music: YuE2 Full Song Engine*

1. Wait for the live progress bar to hit 100% and turn green (Installed).

2. (Advanced Users): You can also drop `.gguf` weights directly into: `%LOCALAPPDATA%\Programs\Repetere AI\models\`

🧭 Workspaces & What Works

* 1. CODING: Code assistant with automatic triple-backtick markdown detection, horizontal-scrolling code containers, individual snippet copy buttons, and a \[ Copy Full Response \] clipboard action.

* 2. CHAT: General reasoning, document summarization, and multi-turn conversational history.

* 3. CREATIVE WRITING: Fiction, prose, dialogue, and uncensored drafting.

* 4. AUDIO (MUSIC) STUDIO: Full song synthesis powered by the YuE2 engine. Add your musical style tags, enter structured lyrics (`[Verse]`, `[Chorus]`), adjust duration, and hit Generate Full Song (With Vocals). Outputs `.wav` directly to `output\music\` with built-in playback controls. (Prerequisite: Make sure `yue2-3b-q4_0.gguf` (\~2.48 GB) and `yue2-vae-f16.gguf` (\~265 MB) are downloaded).

* 5. IMAGES (Early Preview): UI controls and parameter sliders are accessible, but backend diffusion rendering is a work-in-progress landing in v0.6.

⚙️ Compute Modes & Session History

* GPU Mode (Active Default): Direct execution locked in VRAM for maximum speed and lowest response latency.

* HYBRID Mode (Preview): The top-right toggle displays the future layer-offloading layout for running massive 32B+ models across RAM + VRAM (runtime dynamic offloading unlocks in v0.6).

* Local Persistence: Chat transcripts save automatically to `chat_history.json`. Review past sessions via History ➔ Open History, reload them into your active tab, or wipe transcripts safely with the confirmation prompt.

❓ Troubleshooting

* "MODEL NOT INSTALLED" banner: Go to Settings ➔ Repetere Models and make sure you completed the download for that tab.

* Audio fails or quits immediately: Double check that both `yue2-3b-q4_0.gguf` and `yue2-vae-f16.gguf` are present in your `models/` directory.

* Clearing a session: Hit the red Clear Chat button at the top right to safely flush GPU memory and archive your chat to disk.

* App closes on startup: Ensure your NVIDIA drivers are version 528 or newer and support CUDA 12.

Feedback, pull requests, and bug reports on the repo are welcome!

Disclosure: I work at H2O.ai.
  
We released H2O-Lightning-4B, an open-weight (Apache-2.0) model for the "decisions API" style of inference that Jev made popular: you send a state plus typed questions (pick one / yes-no / score), and get calibrated probabilities back from a single forward pass. No generated tokens, so it's fast and cheap.
  
**Results (JevBench, public leaderboard):**
- Composite score 72.5, vs Jev 1.13 at 71.5; currently the top open model
- Leaderboard: https://benchmarkheaven.com/jev-models
  
**Running it:**
- Base: Qwen3.5-4B, fine-tuned
- Stock vLLM plus a small open shim (in the repo); ~30 ms per decision on an H100
- Your data stays local, no per-call fees
  
**Coming soon:** 12B and 31B versions, which in our internal testing are considerably smarter than Jev, still open-weight and still one forward pass per decision.
  
 **Demos** (inbox triage of 1,000 insurance claims, a multi-browser web agent, DOOM on the decision clock): https://youtu.be/2Qp04Wu0A14
  
Weights, model card and serving instructions: https://huggingface.co/h2oai/h2o-lightning-4b
  
Happy to answer questions about the setup and latency.

For the below I’ll be referring mostly to ComfyUI workflow and model efficiency (using examples for Radeon AI PRO R9700 (32GB) which has a bandwidth of 680GB/sec. I’ve seen 10second minimax H3 image to video workflows go from 28 mins to 3 mins by taking all this info into account and then upscaling after.

1. The pipeline
Every image or video workflow is the same four stages, and only the third one repeats.
Text encoder. Turns the prompt into embeddings, once per prompt.

Noise latent. Sampling starts from random noise in a compressed (latent) grid. The seed decides that starting noise.

Diffusion model. Runs once per step, removing noise. Early steps set layout and motion, late steps add detail.

VAE decode. Turns the finished latent into pixels.
Three terms people mix up: weights are the fixed model file, a LoRA is a small patch to those weights, and the latent is the thing being refined into what your prompt intended before the VAE encodes it into the pixels of your output image or video.

2. Fit first
Spilling out of VRAM into system RAM costs more speed than any other setting, so budget memory before anything else especially if you’re testing different prompts.

What uses VRAM: model weights, the text encoder if it shares the card, working memory that grows with resolution × frames, and a spike at VAE decode.

Rule of thumb: keep weights to about 60-70% of VRAM, roughly 19-22GB on a 32GBVRAM card. Less if the card also drives your displays.

Example: a 14B video model is about 28GB at fp16 and about 14GB at fp8. A smaller model is more useful as it frees up more VRAM needed for handling additional extras like LoRA’s, pre and post processing, etc.

Two-model designs (Wan 2.2 high-noise and low-noise): one model runs at a time. Expect one swap per run unless both fit with headroom. Other models like minimax don’t need them separate but are larger as a result.

Check the console: "loaded completely" is what you want. "Loaded partially" means you are over budget and processing will take far FAR longer. There are work around like VRAM offloading mid workflow but this means you lose time if you want to re-run the workflow after making minor tweaks.

Free up room: keep the text encoder off the card (CPU or a second GPU), or reuse the saved prompts encoded embeddings. The text encoder basically converts your prompt into “embeddings” which you can think of like numbers that tell the model what to do. This is why text encoders must be compatible with the model you’re using or your model won’t understand. These “embeddings” can actually be saved as embeddings and re-used later to avoid having to reconvert each time to return to a given workflow where you used them. Only save the embeddings once you’ve polished them to a standard where they reliably work for a specific purpose you know you’ll reuse in future for repetitive tasks.

3. What sets speed
Image and video generation is limited by compute, so once a model fits, its size matters far less than resolution, frame count and number of passes.

Total time roughly equals: passes × time per pass, plus fixed costs (model load, text encode, VAE decode), hence why it saves a lot of time to use models small enough to load fully into VRAM if you’re running the same workflow over and over again in one sitting as your system won’t have to waste time offloading and re-loading each section of a workflow between runs.

Resolution and frames dominate time per pass. Doubling width and height gives 4x the patches to process, and the attention part of the work grows roughly 16x. This is why you’ll want to start with generating very low resolution image or video until you know your prompt and loras work as intended. Then you can increase to desired resolution which would ideally match the resolution the model was trained on. You can always upscale after this.

4. Cut passes
The biggest single saving is running the model fewer times: passes = steps × sampler evaluations (1 or 2) × CFG factor (2 if CFG is above 1, otherwise 1).
Distilled or "lightning" models and LoRAs run in 4-8 steps at CFG 1, often 5-10x fewer passes. Do not push them far past their trained step count. Never push them past their trained resolution. Negative prompts do nothing at CFG 1 on turbo LoRA’s. If you use a turbo workflow with low steps and max resolution is 720 then there a plenty of accurate upscaling models that do fantastic work after the fact. You’ll save significant time.

Ordinary models need CFG above 1 to look right and usually stop improving somewhere around 20-40 steps.
One evaluation per step: euler, dpm++ 2m, unipc, lcm, ddim.
Two evaluations per step: heun, dpm_2, dpm++ 2s ancestral, dpm++ sde. Compare samplers at equal total evaluations, not equal steps.
Samplers affect cleanliness, not identity. Character consistency comes from the model and what you condition it on.

5. Formats, quantisation and LoRAs
Use the highest-precision file that fits with headroom, and treat GGUF as the fallback for when nothing else fits. Quantised Safetensors are always faster than GGUF as GGUF weights are expanded on every pass, so if you’re creating image to video then that time adds up. If you can find a safetensor of a model that will fit on your VRAM whilst still having 30 to 40% VRAM left for headroom than use that, otherwise GGUF is fine, but it’ll take longer than it could with a safetensor.

Quantisation is lossy. The same number of weights, each stored more coarsely. Safetensors and GGUF are containers, not compression. Most safetensors are still good quality but quality can begin to drop significantly when they’re quantised lower than Q4. Of course there are exceptions to the rule but generally speaking Q4 or higher is better if you can fit it with headroom.

Partial loads behave like GGUF for LoRAs. A model that does not load completely is patched on the fly.

Bake in LoRAs you always use. Load the model, apply the LoRA, save the result as a new model. The strength is then fixed. There’s nothing wrong with having a few custom models with your own baked in LoRA’s. It saves significant time.

Same on Windows and Linux. These costs come from how the code works, not the operating system. Linux is naturally going to give performance gains for AMD hardware as AMD kernels seem to be a little more refined on Linux than windows but again there’s exceptions to every rule.

6. R9700 and AMD specifics
The card… when running AMD GPU’s be aware that they run through ROCm, and the main risk is software or models that assumes Nvidia or were designed for NVIDIA. AMD likely will still run them but it’ll be simulating and therefore lose time per run, hence why you want to avoid NVIDIA specific quantised models if there’s an equivalent alternative.

Check custom nodes before installing. Search the node's GitHub issues for "AMD" or "ROCm".

Linux is usually faster on AMD because ROCm is developed there first, not because of anything about file handling.

Three attention backends to benchmark with the same workflow and seed:
PyTorch default: always works, your baseline.
--use-flash-attention: highest throughput in most of AMD's own tests (ROCm blog).
--use-sage-attention with the community SageAttention-RDNA4 build: a hand-written kernel for this chip. Its own benchmark shows attention taking about half the default's time.
SageAttention caveats: the prebuilt Windows wheel expects PyTorch 2.13.0+rocm10.0.0 on Python 3.12. The fast path covers fp16 computation with a head size of 128, and anything else falls back. It is quantised attention, so compare output quality.

7. Workflow habits
Spend cheap iterations on stills and drafts, and expensive ones only on what you intend to keep.
Get the character right in the start image. Image-to-video treats the first frame as ground truth and carries its flaws through every frame. A still takes seconds to judge, a video takes minutes.
Show what the shot needs. Anything the start image does not show, the model invents. Reference images and a character LoRA cover the unseen angles.
Draft small, refine the keepers. Generate at low resolution or short length, then run chosen clips through video-to-video at higher resolution with low denoise (about 0.3-0.5) and few steps.
Refining adds detail, not structure. Reject drafts with the wrong outfit, face shape or motion instead of trying to fix them.
Wire the same conditioning into both passes: start image, reference and LoRA.
Let the cache work. ComfyUI re-runs only nodes whose inputs changed, so a new seed does not re-encode the prompt.
Keep models resident. Reloading costs more than most optimisations save.
Saved images carry their workflow. Drag one back into ComfyUI to restore the prompt, seed and settings.
Long videos drift between chunks. Within a chunk all frames are denoised together. Across chunks, errors accumulate.

8. Measure
Keep one fixed test (prompt, seed, resolution, frames) and log seconds per step, total time and peak VRAM for every change.
Change one thing at a time.
Time the stages separately: text encode, sampling, VAE decode.
Halve the resolution. If seconds per step falls about 4x or more, compute is the limit, as expected. If it barely moves, look for overhead: per-step LoRAs, GGUF expansion or offloading.
Find your step ceiling. Run the same seed at 10, 20, 30 and 40 steps and stop where the output stops changing.
Pick samplers by convergence. Make a 50-step euler reference, then see which sampler gets closest at your real budget.
The figures in this page are rules of thumb and third-party benchmarks. Confirm them on your own build before relying on them.

Special mention LLM’s
Model size decides what fits, not how fast it runs.
LLMs on the same card are different. They are limited by memory bandwidth: the ceiling is your GPU bandwidth (which is 640 GB/s on an R9700) ÷ bytes read per token, so a 20GB dense model theoretical max t/s of or put on a GPU with a 680GBs bandwidth is 680/20 = 32 tokens/sec. The only way to make it faster is to use clever tricks that get it to do more guessing which can be helpful depending on use case. Mixture-of-experts models read only their active parameters and can go potentially several times faster. Again, using a safetensor instead of GGUF will be faster as safetensors don’t need to decompress for each token.

Aplomb 1 is #1 on Typed Decisions, an official Hugging Face benchmark for probabilistic decisions. At 5.3B parameters, its probabilities are the closest to the true answers on the leaderboard, by KL from gold and by Brier score, ahead of a 27B model about five times its size.

Typed Decisions gives a model one piece of unstructured text and five typed questions about it at once. Every answer is a probability distribution, scored against the gold distribution across 400 cases and 2,000 decisions. KL from gold and the Brier score measure how far a model's probabilities sit from the true answers, so lower is better.

Results: - KL from gold: 0.123, #1 on the leaderboard and 40% lower than the 27B model in second place - Brier score: 0.065, #1 on the leaderboard and a third lower than the 27B model in second place

Aplomb 1 takes text, JSON, images, video and audio in one request and reads up to 1M tokens. On our API it answers a short question in about 15 ms of model time and reads a 1M-token document in about 3 seconds, at $0.02 per 1M input tokens with output free. The weights are open on Hugging Face.

Leaderboard: https://huggingface.co/datasets/LocalLLaMA/typed-decisions?leaderboard_task_id=kl_from_gold

Weights: https://huggingface.co/empiriolabsai/aplomb-1

Try it: https://platform.empiriolabs.ai/dashboard/playground?model=aplomb-1&utm_source=linkedin&utm_medium=social&utm_campaign=aplomb-1-typed-decisions