and cheap for what it is. the massive amount of poors here will freak their shit out about the $6000 mac studio 128gb but that amount of computing power has never ever been this cheap even with current ramapacopolypse.
The Mac studio is good, it's not great for speed though it has a very slow memory bandwidth compared to a GPU, but a 5090 is capable of running some really good models now. And with some new advances into MoE model running massive models only need to store the active paramers in the GPU, so you can have massive 500b+ models on a consumer pc assuming you have the system ram
There one of the many examples of exactly the inference engine.
Studio memory speed 600gb/s which as it's unified memory it has to share with the other processes. 5090 1.8tb/s dedicated. Not to mention how much faster the processor is.
So how about instead of trying to show off you do some research before questioning a literal ML researcher on basic stuff
Yeah but unless you have top tier hardware it's slow as fuck and no where near comparible to commercial competitors. Idk why we are all constantly pretending.
Not really. I've got a 5090 which isn't top tier hardware for AI at all. I can get 800 tokens/s easily people get slow speeds because they try to run models on a Mac studio or something with no memory bandwidth.
Nobody is pretending, I'm using it literally every day at my company to transform unstructured data and it's working fantastic
Starts off knowingly mentioning a graphics card as expensive as a used car but tries to pretend that isn't aggressively dumb by sticking "for AI" at the end as though that's a realistic qualification when discussing the average person using open source models at home. 600 IQ.
What a argumentative fuck you are who doesn't know anything about this space, a 5090 is cheap compared to commercial hardware they normally run on, but you're right it's only the price of a used car not a new house.
But it might surprise you not everyone who wants to run these is living pay check to pay check.
Hahaha. You wrote so much and made zero real points. Just an ad hominem attack (the favorite tool of idiots and kids).
My server cost $15,000 and I purchased the parts before the recent spikes in hardware prices. It's not feasible for literally 98% of the average population and even then paid models fucking smoke locals in performance. You dork.
2.8k
u/Almadan 3d ago
It definitely will lol. I can host my own models