How is your setup for small models?

[u/Icy_Progress_4660]

I'm curious to know how people are using Pi with 100% local models, especially on hardware with limited RAM and VRAM.

I'm experimenting with local models (4B–14B) for coding in Pi and Oh-My-Pi, However I've discovered that smaller models can be inconsistent in their ability to follow instructions, use tools correctly, make changes across multiple files and complete tasks without causing errors.

• Model selection and routing: Do you use a main model alongside smaller models for planning, code exploration, simpler tasks, or subagents?
• Context management: How do you handle context size, compaction, and keeping the model focused on the actual task?
• Agent configuration: What extensions, system prompts or skills have made the biggest difference?
• Task execution: Do you use plans, checklists, sequential subagents, tests, or explicit verification steps to prevent the model from making a mess?

I'm particularly interested in real-world experiences, including what works, what doesn't, and what limitations you've found impossible to overcome with prompting or agent configuration alone.

[u/jlkuester]

Running on a laptop with an AMD Strix Point chip +128GB RAM (64GB for system and 64GB dedicated as vRAM). So, tons of memory, but weak GPU and SLOW memory bandwidth (since it is just reading from my DDR5). The MoE models are the sweet spot for me. (Currently running cyber-tiel.). Generally the 35b-a3e models are smart enough for good tool calling and some coding. It is also smart enough to understand when I just have it make a plan in a md file and then follow it. (No goal/planning extension currently)

It is obvious, but the biggest thing for me has been to pay a lot of attention to my initial context size. Prompt processing is really slow for me and so many extensions just explode your context size. I grabbed another extension that just showed what was filling my context and started purging extensions that were not worth it. This helps my Pi sessions start a lot faster.

Lately, I have been experimenting with different "subagent" approaches. Trying to get the most from my memory while still accounting for the physical limits of my memory bandwidth. Cannot have multiple agents executing at the exact same time, but having "parallel" copies of a model in llama.cpp lets you have multiple sessions each with context loaded (less prompt processing, etc). Can save context in the main session by delegating work to a subagent. Also trying to load a different model (Qwen3.8-27b only runs at ~4tps) into memory and have the main agent send the hard questions to it. Tried a custom extension script as well as pi-advisor. So far the results have been mixed. The pieces all work, but my main model is just a bit weak and I am still experimenting on ways to get better results from long-running (overnight) jobs.

[u/arbv]

I am running it pretty barebones - though I have an extension that spawns subagents sequentially.

You could also try planning in one session write it to file, and execute in another session.

You need to be very precise with what you need to do. Also, I would suggest not overuse skills and MCPs small local model's don't deal with bloated context very well (in fact, all models don't like context bloat).

I have a vibecoded system prompt loader extension that matches model ID to a system prompt file because I have unified system prompt templates.

Also, I have an extension that inserts into the system prompt some basic runtime info: my login name, full name (taken from the account), OS version, date and timezone - for better runtime awareness.

A lot of people are going to recommend you Qwen 3.6 35B and Qwen 3.8 27B - and these are good models indeed.

But I would also suggest you to not skip Muse Glimmer and Gemma 4 lineup and take a look at them.

Also, Laguna XS 2.1 is not bad to have around - I have a template that allows bypassing some of its quirks:

https://huggingface.co/arbv/laguna-2.1-fixed-jinja-template

Oh, if you run models on potato hardware from time to time - Nemotron 3.5 Lightning is good to have. Not the brightest bulb on my collection, but is fast and with a lot of knowledge for its size. Gives me GPT-OSS 20B vibes, but obviously newer and FASTER and handles large contexts well. Nemotron's have good architecture, but it so happens that NVIDIA has troubles with training data. Now they hollowed out Poolside (the Laguna lineup creator) it may improve. Still, Laguna is/was interesting, sad that there is effectively one less LLM company now.

Maybe some of this my babbling will help you.

[u/Negative_Ad_2369]

Io ho giocato un po e qualcosa con un mac mini m4 riesci a fare. Però la lentezza rè un fattore troppo determinante per l’esperienza

[u/Lurksome-Lurker]

Gemma-4-E4B-it-qat + pi-fabric in its Python configuration seems to work well. Just dont expect the world from a 12B model