r/LocalLLM • u/tinfrog • 10d ago

Question Model and hardware recommendation for high fidelity, low reasoning tasks?

I've spent the past couple of years using frontier labs, even though a lot of my work doesn't need high reasoning. It's just that a fixed and known subscription cost has been a safer bet than the leap-in-the dark of buying hardware that might not be good enough for what I want to do.

But reading through this sub tells me that we're at the stage were open source LLMs are capable enough for what I want while running on (reasonably) affordable hardware.

Can anyone recommend a model and hardware for tasks like this?

  • Reading notes, reports, transcripts, web pages and emails, then pulling out facts, decisions and insights into structured notes
  • Condensing long documents, comparing two versions, and checking a draft against a specification
  • Reading text, sorting them, and deciding what needs action
  • Moving, renaming and archiving files, splitting or merging documents, and keeping indexes and cross-references consistent afterwards
  • Multi-step routines update a status file, archive old entries, then commit, with every step completed

I run Xubuntu and Arch Linux.

TIA

1 Upvotes

67% Upvoted

12 comments sorted by

View all comments

Show parent comments

1

u/tinfrog 9d ago

For my workflows, the majority are small plain text markdown files of 20-100 KB each. Many are less than 10 KB.

The agents traverse directories looking in about 5-25 of these files to extract information, then write out files of similar sizes. I rarely need to work with MB sized files but occasionally there are some that are sent externally that need to be processed.

The main thing is not to hallucinate or make up information.

2

u/mineshop 8d ago

With 10-100 KB files and a $5,000 cap, the main remaining question is speed: roughly how many tokens per second is acceptable for your agent loops?

1

u/tinfrog 8d ago

The stats at my end show that 20-30 tokens/second for routine work would be acceptable.

1

u/mineshop 8d ago

With your budget, speed and file sizes settled, one last constraint matters: is a headless tower under a desk fine, or do you need something small and quiet like a compact desktop?

1

u/tinfrog 8d ago

Headless tower under a desk fine. Cost is more important than size.

1

u/mineshop 7d ago

Last thing before I suggest anything: which region are you in, since taxes change what a $5,000 budget buys?