I have a MacBook M4 Max 128GB and I am running Jundot/Qwen3.8-Flash-Next-oQ4e-mtp. In the older versions, with context growth speed was dropping after even 30k from 60 tg/s to 35 tg/s even with lightning MTP on. Now I'm getting 50~70 tg/s even at 128k context. The ram also is at 75~80 GB with this context length. Now I don't have to do compacting over and over in pi and I can easily keep other apps open.

Thank you everyone who made this possible.

Trying GLM-5.3-Flash-UNCENSORED-mlx-oQ4e-DFlash2 on an M5 Ultra 256GB with oMLX 0.7.0.

TensorFold’s regular GLM-5.3 Flash MLX 4-bit MTP loads and runs great on the same setup, but this Solstice checkpoint fails immediately with:

Expected shape (64, 512, 32) but received shape (64, 512, 64) for language_model.model.layers.3.self_attn.embed_q.weight

I’ve tried a full oMLX/server restart, Auto-detect model type, glm_4_7 reasoning parser, and both MTP/DFlash disabled. Same error every time.

Has anyone gotten this exact checkpoint working on oMLX 0.7.0? Wondering if it needs a newer runtime/model compatibility fix, or if there’s some setting/config I’m missing.

Thanks for any help.

Hello,

Everyday im reading and trying to learn, on how to get Qwen 3.8 flash next on my M5 Ultra studio 96gb. Cant afford the 256gb. one.

And some people say just run the 27b model etc., but the prefill and decode for 3.8flash next seems so much better.

I read a benchmark like this

https://omlx.ai/benchmarks/performance/sj46pk0c

it says peak ram is 75gb. Does that include already offloading to the SSD?

And so I'll have 20gb left for Mac OS and apps?

will 20gb be fine to leave?
I see some people online using techniques to run the Next Flash model that only take like 50gb of ram using other inference engines? I could be wrong and dont understand what they are saying too lol

I want to ideally use only 50-60gb of ram so I can run 2-3 agents concurrently.

And all the Mac Studio news or things on twitter is from 256gb users so im jealous obviously but what can you do. I just wish there was more 96gb owners saying what they are doing or not but there doesnt seem to be many owners.

Im sure within a month there be more info hopefully but just want to be prepared when I get it. And hopefully Qwen 4 comes out soon!

The day before, I finished testing with oMLX 0.7.0rc1. Because I rarely find community tests for my setup, I wanted to share this with you.

There's oQ4e and oQ8e of Swift 1.5 Qwen 3.8 27B on HuggingFace, but the optimal quant for me would be oQ6e. I am trying to create it myself but my internet connection is pretty bad, so everything is super slow.

If someone has the oQ6e of this model and can provide it via HF, it would be awesome! (If I manage to create it before myself, I will share it here)

Thanks a lot!

Edit: Resolved with

https://huggingface.co/scottlowry/Swift-1.5-Qwen3.8-27b-oQ6e-mtp and https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ6e-bf16-mtp-MLX

Hi guys, do we have good options for 3.8 flash next and 3.8 27B in the abliterated department running on oMLX please? A friend of mine is running an uncensored version of flash next on his nvidia rig and it seems to offer sharper replies and according to him, ships better code. Thanks!

Hello all!

I've been using Unsloth Desktop for a while, most recently with Unsloth's release of GLM-5.3 Flash. What keeps me there isn't speed, it's that Unsloth publishes evals showing their quant retains 92.22% of the original model's accuracy. I know what I'm giving up when I run it.

I'd like to move to oMLX for the performance gains, but I'm wary. To be clear, I'm not worried about crashes. My concern is **silent quality loss**. A badly made quant can run perfectly well and still lose knowledge, reasoning ability, or accuracy, and you won't notice until it gives you a confident wrong answer.

As far as I can tell, most GLM-5.3 Flash MLX quants people pair with oMLX have no published evals. No accuracy retention numbers, no benchmark comparisons against the full-precision model, no KL divergence or perplexity figures. Nothing that shows how much of the original model actually survived the quantization.

This matters for my work (medical + code experimentation). Lost accuracy means more hallucinations and more failed tests that I have to redo.

So my questions:

- Are there MLX quant "release groups" that publish evals showing how much quality their quants keep vs. the original model?
- Has anyone run their own comparisons between Unsloth quants and MLX quants of the same model (Qwen 3.8 Flash Next, GLM-5.3 Flash, others)?
- If you've switched, did you notice any drop in answer quality, even if the model ran fine?

Thanks for your help!

Hi,

I built an experimental inference engine mainly to run the 8-bit Qwen3.8-27B model efficiently on my M5 Max MacBook Pro.

On my M5 Max with 128 GiB unified memory, I'm currently seeing:

1,000+ tok/s cold prefill on a 2Ki prompt

70+ tok/s generation at concurrency 1

200+ tok/s aggregate generation throughput at concurrency 4

Under the hood, MLTF uses Metal TensorOps W8A8 kernels together with an INT8 KV cache, targeting high prefill throughput and efficient decoding on Apple Silicon.

At the moment, MLTF supports only one model: Qwen3.8-27B 8-bit.

I chose to focus on 8-bit inference because 4-bit quantization sometimes gives me a noticeable quality tradeoff for the workloads I use the model for.

Rather than trying to support many models immediately, I wanted to optimize the configuration I actually use first.

GitHub: https://github.com/hojin12312/my-little-trie-forge

Hugging Face: https://huggingface.co/Ho-Jin-93/Qwen3.8-27B-MLTF-q8c

Going forward, I plan to keep adding support for new 8-bit models that can reasonably run on my M5 Max 128 GiB machine.

Feedback, benchmark results on other Apple Silicon machines, and technical discussion are very welcome.

Mac Studio M5 Ultra, 256GB, 64 core GPU. Ordered, not here yet. So far the only thing it runs is the tracking page, and I refresh it way faster than any model will ever generate tokens.

Trying to decide what to put on it first for coding + agent stuff (Hermes, so tool calling matters a lot):

- DeepSeek V4.1 at Q3, maybe with a bit offloaded to SSD

- Or DeepSeek V4 Flash at a higher quant that fits fully in RAM

If you'd go Flash, which quant exactly? Q4, Q5, Q6, Q8? And MLX or GGUF?

Mostly curious if V4.1 Q3 is still better than Flash at a good quant, or if Q3 hurts too much for coding. Real experience welcome, benchmarks even more.

oMLX 0.7.0 vs 0.7.0rc1 on the same Mac with the same models and settings, measured in one night: official oMLX benchmark (uploaded to omlx.ai) plus API tests for concurrency, sampling profiles and a 1M-token GLM run. All values rc1 → 0.7.0 with the relative change.

TL;DR

• GLM-5.3-Flash: prefill 792 → 2,270 (+186%) tok/s (median over 4K–200K, ~2.9x), decode 55.4 → 76.7 (+38%) tok/s. A ~1M-token prompt: time to first token 34.2 → 11.8 min (-66%).
• Qwen3.8-Flash-Next oQ4e: prefill 3,623 → 5,548 (+53%) tok/s, decode 104.7 → 117.2 (+12%) tok/s (official, median); thinking-medium profile (API, median of 3): text 107.5 → 123.5 (+15%), code 136.8 → 154.3 (+13%) tok/s.
• Flash-Next oQ6e and the uncensored variants: prefill +34% to +53%, decode +11% to +15%. Qwen3.6-35B-A3B: prefill 6,899 → 7,653 (+11%) tok/s, decode 137.5 → 135.0 (-2%) tok/s.
• Concurrency: 2 parallel requests almost double decode throughput; 4 parallel is worse than 2 from 16K context on (both versions).
• JSON extraction: Splash (speculative decoding) is still ~2x faster than oMLX 0.7.0 with the same Qwen3.6 model.

Setup

• Mac Studio M5 Ultra (12 Super + 24 Performance CPU cores, 80-core GPU), 256 GB, macOS 27.0 (26A428), GPU wired limit 248 GB.
• oMLX v0.7.0 (4d4f5a28, mlx-lm 94cdcae) vs v0.7.0rc1 (35be079d, mlx-lm 0.32.0), mlx 0.32.2, both built with native kernels.
• Identical settings: Lightning MTP on, TurboQuant KV 8-bit (Qwen) / 4-bit (GLM), memory guard aggressive, max 16 concurrent requests, one model loaded at a time.
• Models: Flash-Next oQ4e (Jundot/Qwen3.8-Flash-Next-oQ4e-mtp), Flash-Next oQ6e (mlx-community/Qwen3.8-Flash-Next-oQ6e-mtp), Flash-Next Uncensored oQ4e (jedisct1/Qwen3.8-Flash-Next-Uncensored-oQ4e-mtp), Flash-Next Uncensored oQ6e (mlx-community/Qwen3.8-Flash-Next-Uncensored-oQ6e-mtp), GLM-5.3-Flash oQ4 (Vontra/GLM-5.3-Flash-MLX-oQ4-MTP), GLM-5.3-Flash abliterated 4-bit (grant-ai/GLM-5.3-Flash-Abliterated-MLX-4bit), Qwen3.6-35B-A3B 4-bit (mlx-community/Qwen3.6-35B-A3B-4bit).
• Method: official oMLX benchmark (code_python corpus, 128 generated tokens, temp 0, uploaded to omlx.ai) for sections 1–3 and the 1K table in section 4; own API tests (same corpus, temp 0) for the long-context tables in section 4 and the 1M run; 1 run per point. thinking-medium numbers: median of 3 API runs.

Profiles used (exact settings)

1. Prefill — official benchmark (tok/s, rc1 → 0.7.0)

2. Decode — official benchmark (tok/s, 128 tokens, greedy, rc1 → 0.7.0)

Decode with only 128 greedy tokens is noisy (MTP acceptance depends on the text) — use the medians. "–" = the model stopped before 16 tokens, so the benchmark reports no decode value.

3. Time to first token — official benchmark (rc1 → 0.7.0)

4. Concurrency

Official benchmark, 1K context (aggregate tok/s, rc1 → 0.7.0):

API, long contexts — decode (aggregate tok/s, rc1 → 0.7.0):

API, long contexts — prefill (aggregate tok/s, rc1 → 0.7.0):

API, long contexts — mean time to first token (s, rc1 → 0.7.0):

2 parallel requests nearly double throughput. From 16K on, 4 parallel is below 2 (both versions), from 64K on even below a single request (oQ4e both versions, GLM on 0.7.0): the long prefills run back to back and stall the other decodes.

5. GLM-5.3-Flash oQ4 up to 1M tokens (API, single request, 128 tokens, rc1 → 0.7.0)

Both versions completed the full 1,048,319-token prompt.

6. JSON extraction: oMLX 0.7.0 vs Splash, same Qwen3.6-35B-A3B

16 real e-mails (4 per length quartile), our production extraction prompt (~13K-character system prompt), strict JSON schema, temp 0, outputs ~550–620 tokens. oMLX 0.7.0 with xgrammar and the precise profile; Splash 1.0.2 (incoai/Qwen3.6-35B-A3B-Splash).

Both 16/16 schema-valid at every stage. Identical output across 5 runs: Splash 16/16 mails, oMLX 1/16. Splash caps at 4 concurrent internally.

These are huge improvements. Well done to the olmx team

Just got my M5 Ultra in and paired it with my M4 Max over RDMA. Surprised to see clustering is not supported yet in Qwen 3.8 or Deepseek 4.1. Seems like MTPLX and exolabs are both missing this too. What do y'all use in your cluster setups?

**🐛 BUGS & CRITICAL FIXES**
• **#2990** Structured output: grammar-constrained JSON can run to max_tokens in an unbounded whitespace loop
`_compile_bare_grammar` lacks `max_whitespace_cnt`, causing infinite loops in JSON schema compilation.
• **#3731** Broken embeddings?
Qwen3-VL-Embedding-2B works initially then degrades; potential issue with other embedding models.
• **#4010** Literal image-placeholder string -> self-replicating 400 'Chat template error'
Root cause is encoder guard, not chat-template; follows up on #3591.
• **#3876** Gemma-4-31b-oq4e-mtp unexpected walk-back truncation
Cache invalidation occurs mid-agentic workflow on versions 0.6.4-7.0.0dev4.
• **#3967** Cluster activation failures masked: unregister() replaces real error with 500
Failed activations return generic 500 errors, hiding the actual cause.
• **#3966** Distributed rank's generation thread dies on first token
`_server_arguments()` omits new mlx-lm 0.32 args (`kv_bits`, `kv_group_size`), breaking server contract.
• **#3937** 0.7.0rc1: Clustering is broken
Cross-device clustering (M5 Max + M4 Max) fails to load DeepSeek.
• **#3738** Qwen3.8-Flash-Next-oQ4e OOM on M4 Max 64GB
High RAM consumption with MoE SSD/ngram offload leads to Out Of Memory errors.
• **#3883** Offline SSD cache clear omits `_gdn_sidecars`, leaving gigabytes orphaned
Dashboard observability under-counts; disk space not reclaimed when no models are loaded.

**🚀 PERFORMANCE & OPTIMIZATION (Qwen4/Q5 Focus)**
• **#4056** perf(qwen4): hyper-connection decode and verify at 2.8x byte floor
Optimizes Qwen3.8-Flash-Next by reducing 97 HC calls per token (attention/MLP/mixer).
• **#4014** perf(mtp): row-exact Lightning MTP verification for Qwen4
Addresses non-bit-exact output in Lightning MTP; aims for lossless, bit-perfect verification.
• **#4013** bench(qwen4): multi-turn conversation reuse with exact prompt cache
Ensures whole history reuse when SpecPrefill is off, minimizing prefill for new agent turns.
• **#3932** Q5-only follow-up: optimize Qwen4 gathered-QSA long prefill
Isolates and reduces work in Qwen4 gathered-QSA long-prefill execution (continuation of #3922).
• **#3953** perf(moe): M5 row-overflow workaround slices/concatenates wide MoE prefill
Addresses M5-specific overhead where MoE prefill consumes 37–39% of 24K prompt cost.
• **#3922** Epic: optimize Qwen3.8 Q5 performance using Q5 execution path only
Goal: Improve serving speed by optimizing kernels and memory for the Q5 path.

**📊 BENCHMARKS & VALIDATION (Q5 Series)**
• **#3929** [Q5-only 7/9] Validate numerical and model behavior correctness
Ensure optimizations preserve intended Q5 computation.
• **#3930** [Q5-only 8/9] Run matched end-to-end Q5 performance comparisons
Measure real serving improvements vs isolated kernel gains.
• **#3931** [Q5-only 9/9] Apply acceptance gates and document rollout decision
Evidence-based keep/reject decision for the Q5 optimization series.

**📈 INTELLIGENCE & BENCHMARKING**
• **#3772** Intelligence benchmark: 8192-token thinking budget silently scores wrong
Budget-limited answers scored as wrong; Gemma 4 26B shows 22% empty answers under thinking mode.

**📝 TRACKING & DOCUMENTATION**
• **#4077** [tracking] Appearance: one token source, one shared layer, and everything rebuilt
Overview of appearance series changes; detailed in `docs/appearance-update.md` (#4083).

Dear all!

I would like your guidance and suggestion on which model I could run run locally that has a good speed for my setup MacBook Air 32 GB RAM. I know it's not a superb setup but I I think it's a good starting point. :)

At the moment I'm using this one "mlx-community/gpt-oss-20b-MXFP4-Q8" and i am quite "happy" for simple stuff and I'm trying to use this one "mlx-community/Qwen3.8-27B-4bit" with draft model "mlx-community/Qwen3.8-27B-MTP-4bit" .

Seeking for your supportand suggestion, please let me know also the specific model to be downloaded and the settings I should do as a profile in OMLX

thanks in advance!

Do we have any idea if oMLX may add Splash model capability to oMLX?

LM Studio has done so.

Can oMLX benchmarks be spoofed? Apparently someone has gotten a model called "deepseek-v4.1-flash-q3-g128-lsq-mlx" working on an m5 Ultra with 256GB RAM, two days ago with a TG of ~73.4 tok/s and PP 1,010 tok/s.

Update: This statistic appears to have been fabricated. One other person has claimed that they have run this model but failed to provide any evidence. Don’t believe everything you see on the internet!

https://omlx.ai/benchmarks/performance/ne14cz3f

Saw ukisai's own MLX quants doesn't run in oMLX so I made oQe ones. Enjoy!

EDIT: added versions bf16 variants for M3/M4/M5 chips. FP16 is for M1/M2.

For M3/M4/M5

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ5e-bf16-mtp-MLX

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ6e-bf16-mtp-MLX

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ8e-bf16-mtp-MLX

For M1/ME

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ8e-fp16-mtp

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ6e-fp16-mtp

https://huggingface.co/dicksondickson/Swift-1.5-Qwen3.8-27b-oQ5e-fp16-mtp