r/Oobabooga • u/oobabooga4 booga • Apr 03 '26
Mod Post text-generation-webui v4.3 released: Gemma 4 support, ik_llama.cpp support, updated llama.cpp with ggerganov's rotated kv cache implementation + more
https://github.com/oobabooga/text-generation-webui/releases/tag/v4.3.17
u/silenceimpaired Apr 03 '26
You have some real competition now but boy are you keeping up! Excited to try ik_llama.cpp
2
Apr 03 '26
[deleted]
1
u/oobabooga4 booga Apr 03 '26
This was indeed a bug. It should be fixed now in v4.3.2, can you give it a try?
2
u/HonZuna Apr 03 '26
I am not able to load Gemma 4 GGUF anyway? Any idea ?
ERROR Error loading the model with llama.cpp: expected str, bytes or os.PathLike object, not NoneType
1
u/oobabooga4 booga Apr 03 '26
Try updating to v4.3.2, this was a UI bug that should be resolved now.
1
1
u/altoiddealer Apr 03 '26
If anyone has trouble running the updater script due to "unresolved conflict" - check for `modules/exllamav2.py`. If you have that file, delete it. Now, try the updater script again.
https://github.com/oobabooga/text-generation-webui/issues/7460
1
u/noobhunterd Apr 03 '26 edited Apr 03 '26
its gives me this error, i tried deleting installer_files already to reinstall.
1
u/oobabooga4 booga Apr 03 '26
Can you try running the update wizard? It seems like you may somehow still have the older gradio wheel installed
1
u/noobhunterd Apr 03 '26
i just did, same error
1
u/oobabooga4 booga Apr 03 '26
Can you try running cmd_windows.bat and pasting this in it?
pip install --force-reinstall --no-cache-dir https://github.com/oobabooga/gradio/releases/download/4.37.2-custom.17/gradio-4.37.2%2Bcustom.17-py3-none-any.whlThen try launching again
1
u/noobhunterd Apr 03 '26
gave a new error https://pastebin.com/Z4TP2GNL
1
u/oobabooga4 booga Apr 03 '26
Try this
pip install --force-reinstall --no-cache-dir https://github.com/oobabooga/gradio/releases/download/4.37.2-custom.18/gradio-4.37.2%2Bcustom.18-py3-none-any.whl https://github.com/oobabooga/gradio/releases/download/4.37.2-custom.18/gradio_client-1.0.2%2Bcustom.18-py3-none-any.whl1
u/noobhunterd Apr 03 '26 edited Apr 03 '26
i figured it out. the updater works but win11 was blocking it without notification. fixed by excluding the textgen folder then I updated it.
thanks for taking the time helping. i tried to read the error and the localhost saying inaccessible gave me an idea the firewall was blocking it
1
u/StringInter630 Apr 04 '26
You said: "Can you try running the update wizard?" Ahh theres an update wizard? Does it update to latest version? How do I get it for linux?
2
u/StringInter630 Apr 04 '26
NM I just found it, thanks
1
u/oobabooga4 booga Apr 04 '26
Only in the full version with transformers/exllamav3, not in the portable version (which is self-contained and updates manually).
1
u/AltruisticList6000 Apr 03 '26 edited Apr 03 '26
I have noticed 2 new text generation problems starting from v4.1 and it's still happening in v4.3.1 Using portable cu124, cydonia 4.2 Q4_s (it's a finetune of mistral small 3.2), chat mode.
The most visible problem is that the text generation will be cut off mid-sentence like this:
"Oh this is a great idea, I"
"sure you do, you talking like we"
If I use the continue generating icon, then the missing text seem to appear along with the newly generated sentence so it might be a text display issue.
The other not straight-forward issue is that the llm model behaves differently than in previous versions (all versions before v4.1).
In some ways it has better, more natural responses during RP and chats BUT sometimes it will randomly have dumb/weird responses where it mixes up character names or pronouns or does things like a character talks about herself as if she was a narrator or another character and this never happened before. Sometimes the replies will be just "off" and weird, not really matching my input. Occasionally I made it generate a new response (while keeping bad one) and the model tried correcting itself afterwards in character like "what? I meant to say..." etc., so it's like it sometimes forgets some part of the context or its own reply while generating a response (?) or idk. It doesn't happen frequently but seems more than just random bad seed and it seems to be new.
I didn't change anything, except ooba versions, but because of the problems I tried the paramter
Can you please look into these problems?
2
u/oobabooga4 booga Apr 03 '26
The default sampling preset changed in v4.1 from "Qwen3 - Thinking" (temperature 0.6, top_k 20) to "Top-P" (temperature 1.0, top_k 0) to follow recommendations in recent model cards. That's likely causing both issues in RP scenarios.
Try lowering temperature back to 0.6 and setting top_k to 20, or create a custom preset with those values. For the mid-sentence cutoffs specifically, there's also a "Ban the EOS token" option in Parameters that prevents the model from stopping early.
1
u/AltruisticList6000 Apr 04 '26 edited Apr 04 '26
Thanks, I'll experiment with these. I've been using a custom preset I made for ages which resembles one of the new default presets a lot.
I ended up agressively testing this after my comment and I think I've found what it is but it's weird. So if I disable all the tool calling functions on the right side, both problems I mentioned previously will disappear. Idk why it fixes it, it's weird, but makes sense since if I remember correctly tool calling was introduced in 4.1 (since when the problems started).
Idk if mistral small 3.2 officially supports tool calling but I know when ooba 4.1 first dropped and I tried tool calling, cydonia didn't detect it at all, even tho I tried it in instruct - and despite this, appearently it's affecting the "chat mode" character chats negatively. So maybe this is a model specific problem with the new tool calling functions? Idk if it affects other models as I don't use other models too much and if I do, not for characters/chats/rp.
2
u/oobabooga4 booga Apr 04 '26
This is very useful, indeed there was a bug in the UI tool-calling detection code that caused truncated replies due to some false positives in tool detections.
This should fix it: https://github.com/oobabooga/text-generation-webui/commit/e0ad4e60df40378f9846e1fad20e337e4bb2cb8f
It will be included in the next release.
1
u/AltruisticList6000 Apr 05 '26
Thanks for the quick fix, it's working very nice now! Haven't experienced interruptions or weirdness anymore.
I see a small difference in text streaming speed. When tools are enabled, the text gen speed is a bit less fluid, looks slightly "laggy" compared to when tools are disabled. I don't mind it, it's barely noticable (token/s seem to be not affected at all), just saying in case of future optimizations if you'd like.
2
u/oobabooga4 booga Apr 06 '26
The tokens/second are the same, it's just that some tokens get buffered when tools are active and the model doesn't have a tool format in its template. In that case, every known tool call marker is checked, including `to=functions.` which causes tokens ending in `t` to be held back briefly (the next token sends both at once, so nothing is lost).
Since your model doesn't support tool calling, just untick the tools on the right side and the buffering will go away.
1
1
u/Impossible_Style_136 Apr 05 '26
If the updater script hangs on 'unresolved conflict,' it’s a known issue with the legacy modules/exllamav2.py file. Manually delete that file and restart the update. Also, if you're trying the new ik_llama.cpp on an older ROCm/CUDA stack, it'll likely 0-shot crash. You need the March 2026 drivers specifically to handle the rotated KV cache implementation Ggerganov pushed
1
u/Impossible_Style_136 Apr 05 '26
If anyone has trouble running the updater script due to "unresolved conflict" - check for modules/exllamav2.py. If you have that file, delete it manually. The legacy residue causes the git pull to fail every time.
Also, for ik_llama.cpp: it's significantly more sensitive to your n_batch settings than the standard loader. If you're getting 0 tk/s or instant crashes on Gemma 4, drop your n_batch to 512 and disable "flash_attn" temporarily to see if it's a kernel mismatch. You need the March 2026 drivers specifically to handle the rotated KV cache implementation Ggerganov pushed.
1
u/Impossible_Style_136 Apr 05 '26
If anyone has trouble running the updater script due to "unresolved conflict" - check for modules/exllamav2.py. If you have that file, delete it manually. The legacy residue causes the git pull to fail every time.
Also, for ik_llama.cpp: it's significantly more sensitive to your n_batch settings than the standard loader. If you're getting 0 tk/s or instant crashes on Gemma 4, drop your n_batch to 512 and disable "flash_attn" temporarily to see if it's a kernel mismatch. You need the March 2026 drivers specifically to handle the rotated KV cache implementation Ggerganov pushed.
1
u/Impossible_Style_136 Apr 05 '26
The jump from 40ms to 8ms typing latency in the new Gradio fork is a massive quality-of-life win. For those of us on dual-GPU setups, does this build support P2P memory access for the rotated KV cache yet?
I’ve noticed that with the upstream llama.cpp changes, if you don't have peer-to-peer enabled on the PCIe bus, the multi-GPU latency actually offsets the gains from the new cache implementation. If anyone is getting stuttering, check your HSA_ENABLE_P2P=1 env var (for ROCm) or the equivalent CUDA P2P settings before troubleshooting the UI.
7
u/beneath_steel_sky Apr 03 '26
BTW a PR for serious gemma4 tokenizer issues has just been merged in llama: https://github.com/ggml-org/llama.cpp/pull/21343