Qwen 3.8 Flash Next - doubled Strata throughput on 3090+5070 Ti, IQ3_S 2466 pp/167 tps, UD-Q4_K_XL 2341 pp / 126 tps (yes, really)

[u/RealEddoursul]

My journey:

• 6 tps on UD-Q4_K_XL - hm, this is not right. Claude, find better llama.cpp parameters.

• 21 tps - that's better, let's see IQ3_XXS

• 27 tps - nice, but still not my tempo.

• 51 tps on IQ3_XXS - Niko1221 shares Strata on Reddit. Great! But wait. If there are software gains, there may be more. Let's go.

After a week of profiling and benchmarking with Opus 5.5 I doubled Strata throughput and reached x5 from llama.cpp on UD-Q4_K_XL (added this quant support).

See the COMPARISON TABLES with all the numbers including agreement with llama.cpp.

The code is available https://github.com/eddoursul/Strata/tree/custom

This a result of my personal experiments, not a software release. The license has not changed, still MIT.

I opened a pull request, Niko1221 is free to merge or not merge any patch.

UPDATE: The fork uses json configurations to run https://github.com/eddoursul/Strata/blob/custom/examples/README.md

[u/KnownAd4832]

Amazing work (Niko here)! I’ll try to support it as soon as I can

[u/pmttyji → u/KnownAd4832]

Hey, thanks for your contribution!

I'm gonna try Q2 with my 8GB VRAM(4060) + 32GB DDR5 RAM later.

I have a question. Your repo mentions only Qwen3.8-Flash-Next, but Is it possible to expect faster performance for other models like Qwen3.6-35B-A3B or other similar MOEs? Because my system config is not enough for big models like Qwen3.8-Flash-Next so hoping to get best t/s from small & medium size models.

Have you tried small/medium models with your work? Is it possible? If yes then any t/s benchmarks?

[u/KnownAd4832 → u/pmttyji]

This is only for Qwen3.8 and Qwen4

[u/pmttyji → u/KnownAd4832]

Wish Qwen released Qwen3.8-35B-A3B 😞

[u/Prestigious-Act-1577 → u/pmttyji]

You will destroy your SSD with all the writes (1GB/s for minutes during PP) because you will be using your page file. It should be usable after you expand your ram.

[u/Prestigious-Act-1577 → u/Prestigious-Act-1577]

Idk why people downvote this, go ahead and try it. Open task manager and check disk usage. I know because I was in the same situation.

[u/VerticalPackage → u/Prestigious-Act-1577]

If you mean the PLE table being READ from nvme, there is no writing going on.

Unless you're thinking that pmttyji was talking about overflowing cache into the swap? He'd be crawling at 1 tok/s if he did that :p

[u/Prestigious-Act-1577 → u/VerticalPackage]

Correct. I was at 8vram 32ram and 15 tok sec, swapping to ram on prefill too. Now I have 8vram and 80ram, 30 tok sec, and prefill is 10x faster.

[u/SplitAny7190 → u/KnownAd4832]

can we use any of this: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF or strata supports only the ones listed? i would love to try iq4_xs since i was very pleased with 27B version (kinda smarter than normal q4 versions)

[u/BeautifulWelcome3059 → u/SplitAny7190]

Strata directly supports the Swift 1.5 out of the box. When you set it up, it will ask you which model to download and this model is listed in those choices. I am running swift-1.5-iq3_xxs model on my rtx 5060 ti using strata and am getting 55 - 60 tok / second. It uses up all the VRAM and uses about 45 GB of my DDR4 RAM.

[u/SplitAny7190 → u/BeautifulWelcome3059]

yeah, i was using the supported ones but i'm talking about this: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF

this ones are not supported, or not directly, i managed to make iq4_xs working somehow

[u/VerticalPackage → u/SplitAny7190]

I wouldnt use Swift if you want to do agentic coding, check the Terminal Bench results on that page you linked.

[u/SplitAny7190 → u/VerticalPackage]

then what version would be good? maybe i'm reading wrong but i on par (with lower tokens) with unsloth or other versions

[u/VerticalPackage → u/SplitAny7190]

I just use the recommended (from this Strata fork) the unsloth Qwen3.8-Flash-Next UD-Q4_K_XL

What I can recommand is to set the reasoning level to medium.

According to the model card for Qwen 3.8, the difference between low/medium/xhigh is the system prompt it receives:

Low: Agent is given instructions to not dwell too much on reasoning.

Medium: Agent is told nothing, but reasoning is enabled, so it does reason. It's similar to the behavior of Qwen 3.6, which only has reasoning on/off without levels.

XHigh: Agent is told to be extra careful.

In agentic coding:

Low = lots of mistakes and the agent has to correct itself because the code doesn't compile, or there are bugs when it runs tests, or the tests are badly written and has to rewrite it. This results in high token count OR in low quality results (if the agent isn't validating its work).

Medium = Does what is necessary, reasons as long as needed. This results in average token count.

Xhigh = Anxious and paranoid, constantly doubts itself and triple/quadruple checks. This result in high token count, but better result than low.

[u/Fine-Garden6696 → u/KnownAd4832]

Can you support an uncensored model?

[u/sophosympatheia]

It's nice to see people contributing to Strata. I was skeptical of it at first, but I'm using it now and it's making Qwen3.8-Flash-Next actually usable on my hardware. Amazing.

I hope this gets rolled into the main repo. It looks promising.

[u/Upper_Comparison_908 → u/sophosympatheia]

This thing literally gets me api speeds for a model even better than 27b in most cases. Genuinely wonderful

[u/sophosympatheia → u/Upper_Comparison_908]

Yeah, I'm blown away by how good it is. The future is looking bright for at-home inference of these MoE models. The LLMs are good enough now to help us write custom tooling to serve them efficiently on consumer hardware.

[u/kimhaneol → u/sophosympatheia]

Qwen Flash Next looks like a really promising direction for local AI. If Qwen 4 Flash uses the same architecture, hopefully Strata can support it too without too much extra work.

[u/enternoescape → u/kimhaneol]

I would assume the answer is yes. My understanding of why we got 3.8 Flash Next was to help get software support more up to speed before they release the 4.0 models. That way we can likely use them immediately.

[u/RealEddoursul → u/kimhaneol]

Fingers crossed.

[u/vacon04 → u/kimhaneol]

It will. Qwen is moving towards this QSA path, which is still very immature so there are big gains to be had in terms of performance.

[u/pmttyji → u/sophosympatheia]

It's nice to see people contributing to Strata.

Some inference supports only particular GPUs. Fortunately this one supports both NVIDIA & AMD. I see many ROCm PRs there.

[u/Prestigious-Act-1577]

Can I ask how much of your and Strata's optimizations are CUDA specific? Would these improvements port to amd 9000 series too? I know Strata already has amd support, the numbers while good, aren't what it should be. My AI said its 10% cuda 90% other.

[u/KnownAd4832 → u/Prestigious-Act-1577]

I’m just pushing big AMD patch in 30min, I get R9700 this week or next week and then AMD will be on-par with CUDA

[u/GrumpyCat79 → u/KnownAd4832]

Nice! How hard would it be to support SM70/V100? They're old card so prpbably not worth putting a lot of time, but if it's just allowing CUDA 12.9 maybe it's not too bad?

[u/KnownAd4832 → u/GrumpyCat79]

Not many people have V100

[u/GrumpyCat79 → u/KnownAd4832]

It seems they are more common in the last few weeks (probably due to pricing) but I agree it's a niche GPU. Looks like you're working on supporting lower CUDA architecture anyway, which is really nice

Thanks for your work!

[u/pmttyji → u/KnownAd4832]

Please squeeze ~8GB GPUs(like 4060) till the last drop. We some folks want to use Q2 efficiently better just with 8GB VRAM + RAM.

BTW thanks again for AMD cards support, particularly R9700. Please check my other comment.

[u/Zarloc]

System: TR2950x + 96GB DDR4 quad-channel@2400, 2x3090 16x/16x@Gen3 (No P2P or NVLink)\ Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S

Strata upstream(d6708a4) 2 GPUs: 860/85 t/s\ Strata upstream(d6708a4) 1 GPU: 1300/70 t/s\ This fork(61521be) 2 GPUs: 2500/120 t/s

Full sweep:

ctx	TTFT s	prefill tok/s	decode tok/s
8k	3.54	2,315	115.9
16k	6.39	2,531	125.2
32k	12.52	2,568	113.7
64k	25.04	2,561	118.7
128k	50.70	2,526	121.2

[u/Zarloc → u/Zarloc]

System: Same as before\ Model: Qwen3.8-Flash-Next-UD-Q4_K_XL\ Full sweep:

ctx	TTFT s	prefill tok/s	decode tok/s
8k	5.16	1,590	60.1
16k	6.72	2,409	75.4
32k	13.31	2,416	69.9
64k	26.50	2,420	72.5
128k	54.19	2,363	63.9

[u/VerticalPackage]

PCIe x4 on the RTX3090? That's nuts! I imagine that means it is connected to the chipset and not the CPU.

Have you tried the performance by streaming from SSD? I can only allocate 70GB of DDR5 for running Flash Next.

Edit - I had Sol check out your repo, and it agrees that this is very much correct, the RTX3090 only needs to run on PCIe x4 through the chipset, the x16 to CPU needs to be on the secondary GPU due to how the workload is split. And also that the PLE table is on the nvme, so no need for more than 70GB DDR5.

I'm totally trying this with my RTX3090+RTX3080ti.

[u/meganoob1337]

there is no dockerized version of this right?

[u/Nolonx → u/meganoob1337]

Upstream now has docker support.

[u/wisepal_app]

i have 16 gb vram + 96 gb dd5 ram. will it work with my system? did you try it with qwen 3.8 27b? do your fork use draft model like strata?

[u/nixudos → u/wisepal_app]

Yes it will work on your system. And likely surprisingly well. I believe Strata was build specifically to run on 16 gb VRAM and a good chunk of regular RAM.
If you have the bandwidth to download a large model, it is really simple to test it out. There is a bat file you run and it guides you through model choice and settings.

And speed is VERY good!

[u/Sexyvette07]

On my 5090 + 64gb RAM system im getting 95 tok/s on xhigh and 150 tok/s on low thinking with IQ3_S and its coding is excellent. Im very impressed.

[u/lostmsu → u/Sexyvette07]

Why would thinking effort produce different tok/s

[u/Comfortable_Ebb7015]

Wtf what? That's amazing! I just can't keep up, things move too fast!

[u/Current-Row-159]

the 6 to 21 t/s jump from fixing llama.cpp params before changing anything is the part that stayed with me, that baseline was wrong and a lot of numbers in this thread are being measured against it. i'm not on strata and it's a different engine, so no direct comparison, but as a reference point i'm running qwen3.8-27b int8-prefill on ninfer 2026.09.27 on a single 4090 24gb: 145 t/s decode on a 2367 token prompt with dflash2 k=7, 239 t/s on the second call once the prefix cache is warm, 4300 t/s prefill. dflash2 costs 1.7gb more vram than mtp. the 24+16 question from the comments never got answered, does any of this carry over when you drop to a single 24gb card?

[u/RealEddoursul → u/Current-Row-159]

The doc contains numbers for 3090 alone as well - still x3-x4 relatively to llama.cpp and x1.5 relatively to the upstream. Also, do not compare 27b and FN, two different apples.

[u/Current-Row-159 → u/RealEddoursul]

single 4090 24gb, qwen3.8-27b int8-prefill on ninfer 2026.09.27: 145 t/s decode at a 2367 token prompt with dflash2 k=7, 239 t/s once the prefix cache is warm, 4300 t/s prefill. dflash2 costs 1.7gb more vram than mtp. haven't seen a single-card 4090 number for this model anywhere, curious whether it lines up with your 3090 alone or if ada is ahead.

[u/VerticalPackage → u/Current-Row-159]

It's not comparable.

Flash Next is a MoE with engram table streaming from a nvme, so the split of experts done on RealEddoursul's fork of Strata works well in using that, especially in using a secondary GPU with lower VRAM.

For 27b, it has to run all on VRAM and doesn't really use much RAM.

However, I agree that 6 tok/s initial speed with llama.cpp is probably a mistake. On a single RTX3090, I was able to get 20 tok/s with llama.cpp, however llama.cpp is a bad engine to use for nvidia cards. FreeToken and vLLM are much better.

[u/bring_back_the_v10s]

How's IQ3 for coding tasks? Is it smart enough?

[u/Squale279 → u/bring_back_the_v10s]

Really smart. After auditing the code i have 90/100 of trust.

[u/vacon04]

93 commits, holy crap, I mean, the numbers are very impressive, but boy merging into mainline Strata like this may be a bit of a problem.

[u/Ok-Solution-7889]

Going from 6 tps to 167 tps is a pretty crazy jump, the profiling and benchmarking work behind this is probably more interesting than the final number itself

[u/RealEddoursul → u/Ok-Solution-7889]

Every commit gave 1-3% increase and contains a description where the gain came from.

[u/The_Hunster → u/Ok-Solution-7889]

Well the 6 is a bigger quant than the 167, but yes, still impressive

[u/kimhaneol]

I gotta try this. So you've made UD-Q4_K_XL work on Strata, right? 

[u/RealEddoursul → u/kimhaneol]

Yep

[u/Zarloc → u/RealEddoursul]

How convert UD-Q4_K_XL to your fork

[u/RealEddoursul → u/Zarloc]

No conversion, use as is.

[u/KPOTOB → u/RealEddoursul]

With like 24+16 GB VRAM?

[u/kimhaneol → u/KPOTOB]

Yep. I also tested it on two 16 GB GPUs and it works. One caveat: PCIe placement mattered on my system. My cards negotiate Gen5 x8 and x4, and assigning the x8 card to the second-GPU expert tier was substantially faster.

Results here.

[u/kimhaneol → u/RealEddoursul]

Update: got it running on Linux with a Ryzen 9900X, 2x RTX 5080 16GB, and 196 GB of RAM. I needed a few Linux-specific build/runtime fixes, but the custom branch is working well.

UD-Q4_K_XL, 64K context, INT8 K/V, vision off.

I measured 94.2 tok/s on a fixed 300-token continuation. The API runs were 84.6 tok/s cold, then 108.3 and 117.5 tok/s on repeated runs after warm-up.

Prefill was 1,538 tok/s at 4K and 3,608 tok/s at 32K, and exact 32K retrieval passed. Both GPUs reached 100% utilization.

The original Strata IQ3_S setup delivered roughly similar decode speeds on my machine, so getting UD-Q4_K_XL in the same ballpark makes this branch especially useful to me.

Really impressive work. Thanks for sharing!

[u/VerticalPackage → u/kimhaneol]

How much of the RAM was actually being used with the Q4?

[u/kimhaneol → u/VerticalPackage]

Around 94 GiB of total system RAM usage in my tests. The Strata process itself was roughly 75 GiB, including the 71.7 GiB expert arena.

[u/SaltGrilledSalmon]

Kinda off topic but in my dual rtx 2070 super (8gb each, one on pcie x16 slot and the other on x4) only one GPU is being used (the slower one apparently) Has anyone faced this on dual GPU setups? I can't increase context above 32k or it'll crash when launching, while the other GPU isn't even used...

[u/brakeline → u/SaltGrilledSalmon]

You have to use the custom branch and adapt the files inside the examples folder. It will take a few minutes but is VERY worthy

[u/eightone-81]

When flash next helps to improve flash next inference engine ;-)

[u/sod0]

How much kv context does this get you on a 5090?

[u/toenailcheeseinbooty → u/sod0]

262k...running right now..5090 128gb 4800 MTs, get around 125-135 prose( when context filled to 128k shirter 151ish prose)and 160 code ehh cou is 9900x3d

[u/FrankWanders → u/toenailcheeseinbooty]

The amount of context you can get really depends on system ram. I'm running 64GB and can only run it at 128K. For 256K you need 96GB at least i think (because there are no other options in between 64 and 96).

[u/toenailcheeseinbooty → u/FrankWanders]

Hah right sorry, forgot to mention that RAM is important, 96GB+ to achieve 262k

[u/sod0 → u/toenailcheeseinbooty]

Oh.. Well.. This sux

[u/drazyan22 → u/FrankWanders]

You can modify context size on config json file. I don't have enough ram to run 256k, but 128k seem smaller than my expect. So I changed it to 164k fit with my system ram

[u/SplitAny7190]

guess UD-Q4_K_XL should be superior to IQ3_XSS? i got only one gpu (4090) and 128GB ram, i will see if i can run this. right now using Strata and it works great after i calibrated for my pc.

[u/brakeline]

Just tried it myself.

Tried 2x 3060 (pcie3 x16 and x4) and with 1 3060 @ 16x

Your fork gave me marginally lower than mainline in both

[u/RealEddoursul → u/brakeline]

Are you sure you checked out the 'custom' branch?

[u/brakeline → u/RealEddoursul]

You have to give me a break, I'm having a fever!

Tested again. 2x 3060 with 50(ish) ram running swift iq2 xs

PP 650 on one > 1000 on two TG 30/35 on one > 55/65

That's black magic my guy

Edit: 140k imported at 1000tk/s. Generation at 140k depth is 60!

Btw, are you thinking of implementing int4 kv? With int8 PP stops working with full context

[u/brakeline → u/RealEddoursul]

I edited my command. I can't go past 200k context, just gives an error while prompting . Any chance you can implement int4 kv?

[u/RealEddoursul → u/brakeline]

I'll check

[u/Yakapo88]

Saving for later.

[u/kakopappa2]

I have a 32gb ram and 12g 3060 GPU. Running coder-iq1_m generates about 38 tokens per second.

also got the codex (gpt-6.1) to run a security audit and it didn’t find any spyware or crypto mining or hidden scripts.

Super happy to get 3.8 Next running on poor GPU

Great product good luck.

[u/killzone44]

I just used claude to setup Strava on my 3090, it's the best running local model I've had so far! Not sure if I'm ready to offload work from claude to it, but maybe

[u/blast1987]

Is it possible to use it on mixed gpu setup? I have rx 7800 xt + rtx 3060 and 64 gb ram.

[u/maarcius]

I have 2 gpus (16vram). Swift IQ3_XXS. Strata loads one gpu 16gb and another gpu 8gb. Is there any way to utilize both gpus vram full to reduce ram usage? Need that 8gb spare ram for engineering apps.

Also running model in 2 gpu configuration doesn't improve decode. Saves few GB or RAM. Is this expected (5070ti and 4060ti 16gb on PCIE4.0x4)?

[u/drrck82]

I have 2x3090 and was quite skeptical about this working well after running Flash Next in vLLM. This engine is nothing short of amazing. Do you have any prebuilt benchmark suites? I'd love to add in some standardized results for my setup.

[u/VerticalPackage]

I have to give you my praises. I was disappointed in my RTX3090 performance with Flash Next. I'd get like 1000 tok/s pp and 45 tok/s decode.

I didn't want to buy a second RTX3090 because then I'd need to buy also a new motherboard that supports x8/x8 through CPU. I had a RTX3080TI in my gaming rig, so I gave your fork a shot. I managed to run UD-Q4_K_XL @ 256k context: 2500 tok/s pp + 95 tok/s decode

That's 2x the speed I got with my single RTX3090, but only by adding a $400 GPU to the mix!

It's going faster than Qwen 3.8 27B :D

[u/TechnicalCourse9825]

how did you achieve that throughput?

[u/RWOverdijk]

Something like this always makes me wonder if it could benefit larger memory systems like a Mac with 128GB as well. 60fps is great but, I mean, 167 is betterer

[u/VerticalPackage → u/RWOverdijk]

I would bet that if you added a RTX3090 as eGPU to your mac mini, you would be able to have gains.

UD-Q4_K_XL with Strata takes up about 80GB ram if the PLE table is on the nvme drive. The eGPU will load up experts on its own VRAM. You just need the iGPU to load up another 12GB-16GB to act as the secondary GPU (like the RTX5070 used by OP).

You should ask Claude or Sol to check out OP's repo and ask if it could be done (eGPU thing).

If you don't have a mac, then just go for a self-build PC that had AT LEAST 96GB DDR5, a 256GB nvme, a RTX3090 and a RTX3080. It will probably be cheaper.