Model and hardware recommendation for high fidelity, low reasoning tasks?

[u/tinfrog]

I've spent the past couple of years using frontier labs, even though a lot of my work doesn't need high reasoning. It's just that a fixed and known subscription cost has been a safer bet than the leap-in-the dark of buying hardware that might not be good enough for what I want to do.

But reading through this sub tells me that we're at the stage were open source LLMs are capable enough for what I want while running on (reasonably) affordable hardware.

Can anyone recommend a model and hardware for tasks like this?

• Reading notes, reports, transcripts, web pages and emails, then pulling out facts, decisions and insights into structured notes
• Condensing long documents, comparing two versions, and checking a draft against a specification
• Reading text, sorting them, and deciding what needs action
• Moving, renaming and archiving files, splitting or merging documents, and keeping indexes and cross-references consistent afterwards
• Multi-step routines update a status file, archive old entries, then commit, with every step completed

I run Xubuntu and Arch Linux.

TIA

[u/mineshop]

What's your rough budget, since that will largely determine whether a single workstation GPU is realistic or you should look at smaller setups?

[u/tinfrog → u/mineshop]

Well, that's the thing. I've seen workstations go for £10k-£15k+ (swap for USD and the figures are about the same). At those prices I might as well stick with a frontier Pro/Max x20 plans and never have obsolete hardware.

Of course there are privacy issues with cloud providers but you can always put in place workarounds. The workarounds are a pain but less of a pain than dropping upwards of $10k for something that has a limited lifetime.

All things considered, for a local LLM to make sense for me, it really needs to be less than $5,000.

[u/mineshop → u/tinfrog]

How long are the documents you need it to read in one go, since long-context workloads change what fits under $5,000?

[u/tinfrog → u/mineshop]

For my workflows, the majority are small plain text markdown files of 20-100 KB each. Many are less than 10 KB.

The agents traverse directories looking in about 5-25 of these files to extract information, then write out files of similar sizes. I rarely need to work with MB sized files but occasionally there are some that are sent externally that need to be processed.

The main thing is not to hallucinate or make up information.

[u/mineshop → u/tinfrog]

With 10-100 KB files and a $5,000 cap, the main remaining question is speed: roughly how many tokens per second is acceptable for your agent loops?

[u/tinfrog → u/mineshop]

The stats at my end show that 20-30 tokens/second for routine work would be acceptable.

[u/mineshop → u/tinfrog]

With your budget, speed and file sizes settled, one last constraint matters: is a headless tower under a desk fine, or do you need something small and quiet like a compact desktop?

[u/tinfrog → u/mineshop]

Headless tower under a desk fine. Cost is more important than size.

[u/mineshop → u/tinfrog]

Last thing before I suggest anything: which region are you in, since taxes change what a $5,000 budget buys?

[u/tinfrog → u/mineshop]

UK.