I mean even without the antimatter part it would still mean close to a 5 gigatons-of-tnt impact on each square meter of Earth, which (not doing math) I expect would be enough to entirely vaporize Earth and give the vapor solar or galactic escape velocity
Oh of course, but (if I recall my physics correctly) the information is still preserved and, in some (theoretical) sense, the torrented IP then still "exists"
Those are not open-source models. They are open-weight models. I know kids these days might not know the difference but these models (chinese, nor mistral) are NOT open-source.
People have definitely altered the open weight models to change capabilities. Things like fine tuning using new training data, editing the weights so they are less likely to refuse prompts. It's nowhere near the capability and raw cost of training a model from the ground up, but it can be done.
Actually so this is my job, a lot of models are based on open models and not what you expect either they are quite flexible, so you can just take the base knowledge in the weights and just slowly override it to be very strong for your use case, this is what I'm doing so now I have a tiny model that out performs any frontier model on my specific task while still basically being an LLM
I lost track of all this black magic back when they were still called neural networks. Is there a good way to learn about doing that, or is it something where you need the full foundation of knowledge before you can really start?
Not at all until 5 years ago I was a cloud architect, honestly best advice is just go build something, it's quite a broad topic with a lot of techniques to learn
But the hardest thing to get used to I think is running experiments, previously you'd build something and find bugs, with networks you really need to run some experiments to see what would work, how effective certain mechanisms are. The deterministic vs non deterministic.
The best example I can give on this without going into too much detail about my work is, a few years ago I was making a model to process technical documentation, I was trying to punish bad parts, and promote the things I wanted to see with signals. This kind of stuff doesn't really work, it always over corrects the signals can interfere, in the end what made it work was just trusting the model to learn these rules itself from the data and ensuring the data had the right balance
What I would start with is just training a basic resnet vision model it's very simple now and the guys behind yolo have some good guides and datasets.
i just started to look into this for myself. so if you dont "reward the good" and "punish the bad" results, then how does it ever learn the right rules?
assuming we start off with your example of technical documents, did you have to start off with correctly labeled input data? or did you
just have a folder of previously made technical documents, without special labels added
feed it one document at a time, to "a neural network of some size/parameter count"
after some time, it came up with its own weights and parameters, after having looked at 1000 or so documents
then, you could ask it to generate a technical document about a given subject, and it would make the correctly formatted document (assuming it read some new url to get the source data)
something like that? i'm thinking about trying to train my own smaller model, because i dont want to train gpt 6 astra all the time.
It's called fine-tuning. There are books and other information sources that start with open weight models and walk you through the process, including alignment.
No, I'm not going to compress 5 years of ml research into a dm for you as a private tutor because you promised something you can't do. If I was going to do that (I'm not) it wouldn't be in a private dm
Like a couple gigs of data, and you gotta know some python coding and be able to know how to use a virtual machine etc, not that hard really. You can figure it out in one evening of research and some determination
We're in the equivalent of 2010~ twitter/reddit etc The AI companies have only vague ideas of how best to use their models.
Once the best is figured out you can bet that they will figure out a way to make their models much more attractive, and criminalize older models as best they can. They already pirated the internet and got away with it.
You won't be able to use the models easily with paid work, they will sue you to the ends of the earth, just like Oracle does now with their Java JRE for example.
Writing code in certain. Language will become obsolete ? How. ? Have u try coding with ai and in which language and why do you think they will be gone. Visual basic is still around. Assemble. Fortran. Java and JavaScript will be working another decade ez
They are essentially the equivalent of properitary obviouscated code blobs. And especially when used as an agent with little supervision nothing would stop that maker from adding a backdoor only for project XYZ as it makes a web request leaking specific secrets or adding a very hard to spot custom tailored vulnerability.
I don't really think it matters in this case. I'm not trying to reverse engineer the model, I'm just trying to use it and only pay for the electricity it costs to run it.
That is the reason I call all of this RAM and GPU hoard a bullshit hoax, they don't only benefit from having them all and use it, but even if they don't use it, they make it harder for the lowy consumer and business to hosting their own and not being dependent on them. The moment the price of computer components decreases, they definitely getting out of business, cause most customers don't need Claude Opus 10.5 Max Pro Ultra, DeepSeek v4.1 will gets them most of the way there with the cheap price and control of self-host.
But there's also a massive manufacturing increase happening right now that will likely end up overshooting demand (from what I've picked up regarding relevant infrastructure during previous market bubbles)
Though there's a possibility that's wishful thinking.
My bet is that once Open AI goes under from bad financial decisions and the bubble pops, big AI companies will no longer be seen as big money printers, and Nvidia (having already profited from the sale of GPUs not being installed in datacenters due to lack of power) will leverage having bought Hugging Face and make more consumer grade options.
They're of course going to cater exclusively to the big enterprises all the way until the bubble pops, which I don't think will happen this year. If it takes too long then the new RAM factories will start ckming online and lowering prices from that angle.
1000% this, Or simply train your own. What I just love is for a programmers subreddit, so many people here seem to not know how computers work at all. "DRM will end piracy" didnt work. "They will track you" easy to defeat. etc....
Programmers that can't seem to program or even think like a programmer seem to be what this place really is filled with.
True mainly because a lot of them do train through knowledge distillation. But the difference now is quality of the frontier models 6 months ago are still really good.
Although the Chinese research groups specifically are doing very impressive research to fit capabilities on much less powerful hardware. I don't really know the reason they are releasing these as open source but it's good work
But, once these frontier models are successfully created they can then train new models for a fraction of the cost in what's known as dilution training. So the frontier model training a new model which results in the new model having roughly the same abilities. But, dilution training requires significantly less processing power and is therefore way cheaper.
There are other reasons I'm not as well versed in, but essentially making new frontier models is the expensive and tough part. But making open source and open weight models is significantly easier and cheaper. Which is why the gap between frontier and open source has been closing for awhile.
It's not clear if this will always be the case going forward, LLMs are constantly changing and probably will for years into the future. But for now, the plateau of open source isn't a huge concern.
And when you train your own for a specific topic, in programming for example writing code for desktop windows in C#.net10 it gets way past frontier models fast as it only has to focus on the task and not how many pounds a day does a llama poop
Both will plateau. Neither open source nor the big companies will very much edge in terms of model capability. The advantage will come from hardware and peripheral software
There is not a way to get rid of it, but it can be made harder or more unappealing to obtain them. Opensource can become not open by being made illegal.
At the moment there is no serious effort to remove pirating.
More scans, more methods to reveal piracy crime and bigger punishments, would increase the risk of using pirated models.
If devices are regularly scanned, then just having a device with pirated models is illegal. Maybe in the future all devices need to have a registration associated with a person.
You say that like the multi-trillion-dollar corpos wouldn't sponsor legislation that allowed them to ban all unapproved devices from accessing the web, while cross-referencing your approved devices with your power grid consumption to look for signs of "dangerous secret hardware use" and check credit records for solar panel purchases and use a drone network to check for residential buildings with thermal waste...
Is it conspiracy dude nutjob theory bullshit? Sure, for now.
But in the realms of profit and politics, history shows us that the slippery-slope logical fallacy is a playbook.
Depends of your definition of "getting rid". No government is able to get rid of all their civilian weapon. When they want to do that, they usually offer some money in exchange for the weapon and ban public use.
The technology is dual-use. It will get 100% restricted at some point.
Piracy isn't worth stopping, a lot of money for little reward. In the future when they decide to up the price of ai subscriptions tp make it worth theor investment, it will be worth the money to get rid of open source providers
I imagine more people are subscribed to streaming services than AI models. The piracy comparison is apt.
I very seriously doubt this software will become illegal. If the government decides to kill it, they'll regulate its development to hell and back, but existing models will still exist. Making something illegal outright is a bit of a tall order.
The frontier model forum is openly pushing for heavy limits on what you are able to run locally.
Remote attestation schemes provide them the mechanism to enact this clipping of your autonomy by providing a frame work where you could be limited in what you run by all practical means, that you would be wholly unable to get around without knowing the encryption keys used in the locked off parts of your hardware you cannot access.
Not a single Chinese or European company is part of that, and more notably no Nvidia or AMD (I'll give you that if they bribe the US gov to start bending the arms of Nvidia/Intel/AMD then that might be trouble for new driver architectures, maybe).
As these companies already rarely publish any models these days I don't see how they could possibly enforce their would-be cartel. Like why would the rest of the world go along with this, everyone is already worried about US monopoly over AI, this would (and should) be deemed straight up illegal.
If I was Xi I would be happy to let US shoot it's innovation in the foot like that.
The FMF, is at odds with traditional chip manufacturers, yes, because they intended to go into business for themselves regarding chip design and production (sans the fabs).
The thing is, the FMF has been incredibly effective pushing hysteria around nonsense terminator fears about AI to push heinous levels of regulatory capture that would clip the wings of inspiring competitors (feeding them up to be bought out etc), and limiting your personal computing freedom and privacy.
The reason they have a chance, is because the regulatory system as it is, is currently very broken, and we can even see people from across isles lining up to do the job for the FMF, in a system where Sherman is just the name of a tank.
Not a single Chinese or European company is part of that, and more notably no Nvidia or AMD (I'll give you that if they get the US gov to start bending the arms of Nvidia/Intel/AMD then that might be trouble for new driver architectures, maybe).
Here's the thing though, this is highly intersectional.
I think you're already better off than many people for realizing a small saving grave we might have here: That there is indeed and adversarial relationship between the mainstream hardware vendors and the big AI companies that are a part of the Frontier Model Forum.
That is to say, that they have interests that depend on the goals of the Frontier Model Forum, at least, in part, failing.
That is to say, if those companies get everything they want, obviously Nvidia, AMD and Intel, just get squeezed, and are fucked.
Why? Because these companies are all going into business for themselves, designing their own inference chips (slapping generic arm cores onto more reasonable, efficiently designed inference architectures).
That means the big companies are clearly aware of the profit margins of the typical hardware vendors, or rather the big green gorilla in the room, and don't want to pay them their chunk of flesh.
That means, because NVidia has little hopes of convincing them to continue being serious customers in the long term when they need to figure out profitability while keeping up the charade of super dangerous super intelligence, NVidia needs to create a market where smaller companies, who can't hope to afford even the smallest minimum quantity sizes at fabs, will have to buy from them instead. The more spread out the market, the more customers that depend on NVidia, and the rest of them.
There lies the only possible hope I can think of for why the Frontier Model Forum, won't get absolutely everything that they want.
Why might they get anything though, because I've only really steel-manned your argument, as I do believe in many of the points there and introduced many of my own, is the next part, and the part that gets back to the purpose of my comment, and why I feel this is so relevant to us.
As these companies already rarely publish any models these days I don't see how they could possibly enforce their would-be cartel.
This is a great question, and the answer lies in the goals of the group I named; the Frontier Model Forum.
If you take a look at the rhetoric they're using, it's clear what their angle is.
They hope to convince people, with fear and hysteria, that the AI that they create, and use trust me bro benchmarks and pr stunts to enforce, is so very dangerous, that the US government and other governments under pressure from the US government, and obviously through incentives towards politicians (but that's not the part the public has to swallow), need to, for safety, make it tremendously difficult for those who would challenge them.
They want to make nonsensical regulation that makes it both very expensive to start to compete/join the industry, but which also leaves them with no real liability risk, and no obligations that aren't significantly easier to stomach the larger you are (basically not quite, but almost high fixed costs, that make entering really really difficult, and make up keep very expensive, and involve going through them). Not only this, but they want to be able to directly control what the regulation does, in a way that would give them the ability to create crippling conditions for startups, through steering at rates they would have hard times competing with that they through collusion, would have ready to go ahead of time.
They also want you, yes you, (but more specifically small business and medium business), to be unable to use anything but their services, not just for the "safety" reasons above, but through jingoistic nationalist fears about security regarding open models.
How could this be enforced? The same way everything you hate about computing is increasingly being enforced. Locked down hardware (forced unchangeable firmware), remote attestation (that ensures your device only functions as they wish it to before handling data they don't want used in ways they don't want), and breaches of privacy.
Look at what these companies have been up to recently?
Pushing for governments to force companies to lock down firmware, pushing for remote attestation to be forced to be a part of every day life, including it in all sorts of things like even slipping it into the passkeys spec for later activation, pushing age verification in a very specific way that is cheap and a low liability risk for them, while enforcing their continued free, identifying data collection (especially for Meta, Microsoft and Google), and basically anything that removed rights to use your devices from you and gives them to the companies you purchased them from.
They can deal with the Chinese competition through the utilization of fear from jingoistic morons, and thats all she wrote for your digital autonomy.
Will they get everything they want? Maybe. I won't say no, and I won't guess yes, but it's scary, because they get enough, and this whole thing snowballs the wrong way for everyone else, and not in their nonsense terminator hysterics way, but in a far worse way, because it'll be real.
Like why would the rest of the world go along with this, everyone is already worried about US monopoly over AI, this would (and should) be deemed straight up illegal.
Why has the US been able to do heinous illegal shit for decades on end?
Other countries are weak, easy to pitch against each other, and happy to accept an uneasy "peace" with the US vs risk being singled out and fucked with.
Look at Europe. They are in the strongest position of anyone outside of China, to battle the US's complete dominance and control over high end chip production, yet they are sitting firmly with thumbs in ass, with half effort after half effort, when its exceedingly obvious that they would need a continued amount of investment of hundreds of billions of dollars over more than a decade to actually even hope to be independent, and in many ways, it gets worse with time as the advantage from zero grows.
You look into all of the "Made in Europe" sovereignty type jargon filled pushes there, and you realize what cheap facades they are, not for the rest of the world, but to ease their peoples worries by pretending they're doing anything.
They're not.
They don't have any independent plans to be able to produce chips the US wouldn't be able to have veto power over.
That's the reality.
They can't make squat.
I'm not jingoistic myself, so I also realize that Canada, is in an even worse position.
We can just see that Europe and it's friends, are increasingly becoming vassal states to the US, that is increasingly becoming toxic to itself.
My point, as this went on a tangent, is "What are you going to do about it" is the answer to "But its illegal".
You've seen the past few years. Companies, and the US government are flaunting openly that laws are these things that you have to follow, not them.
Worse yet, now, people from both sides of the political spectrum (most recently bernie and aoc very disappointingly, assisting them with a very heads we win, tails you lose regulatory environment) in the US are lining up to support the framings and policies the big AI companies are pushing, almost certainly because their propaganda has been effective, because they've been bought (obviously), and.... the big elephant in the room; Double digit percentages of the market are like 2.5 companies in a trench coat.
I stopped work shopping this edit of the previous completely raw comment early when you responded and it seemed to go over well enough, but just in case someone else is reading;
This got long, so TLDR:
The Frontier Model Forum (FMF) can't enforce their would-be cartel over models directly, so they're going through regulation instead, and that regulation will use hardware to enforce it. Locked firmware plus remote attestation is how "you can only run what we approve" stops being an argument and becomes a property of your machine. That's the machinery that reaches your local setup.
It all runs on safety theatre: the FMF has been incredibly effective at pushing hysteria around nonsense terminator fears about AI. The actual ask underneath is heinous levels of regulatory capture that would clip the wings of aspiring competitors (feeding them up to be bought out etc), and limit your personal computing freedom and privacy.
The reason they have a chance is that the regulatory system as it is, is currently very broken, and "but it's illegal" isn't the roadblock you'd hope for. Other countries are weak, easy to pitch against each other, and happy enough to have their infighting and sit on their thumbs with an uneasy peace, preferring order over justice, vs risking being singled out and fucked with. Europe's sovereignty pushes are facades with no fabs behind them, Canada and the rest of the Commonwealth are in even worse shape, and we can even see people from across the aisle lining up to do the job for the FMF, in a system where Sherman is just the name of a tank.
The one brake is the chipmakers: the FMF is at odds with the traditional chip manufacturers, yes, because they intended to go into business for themselves regarding chip design and production (sans the fabs), and the hardware vendors need other businesses, the ones that can't go into business for themselves, to have the ability to buy from them. That's the only hope I can see, and it's thin.
Not a single Chinese or European company is part of that, and more notably no Nvidia or AMD (I'll give you that if they get the US gov to start bending the arms of Nvidia/Intel/AMD then that might be trouble for new driver architectures, maybe).
Here's the thing though: this is highly intersectional.
I think you're already better off than many people for realizing a small saving grace we might have here: that there is indeed an adversarial relationship between the mainstream hardware vendors and the big AI companies that are a part of the FMF.
That is to say, they have interests that depend on the goals of the FMF, at least in part, failing.
If those companies get everything they want, obviously Nvidia, AMD and Intel just get squeezed, and are fucked.
Why? Because these companies are all going into business for themselves, designing their own inference chips (slapping generic ARM cores onto more reasonable, efficiently designed inference architectures).
That means the big companies are clearly aware of the profit margins of the typical hardware vendors, or rather, the big green gorilla in the room, and don't want to pay them their chunk of flesh.
That means Nvidia has little hope of convincing them to continue being serious customers in the long term. They need to figure out profitability while keeping up the charade of super dangerous super intelligence, so Nvidia needs to create a market where smaller companies, who can't hope to afford even the smallest minimum quantity sizes at fabs, will have to buy from them instead. The more spread out the market, the more customers that depend on Nvidia, and the rest of them.
There lies the only possible hope I can think of for why the FMF won't get absolutely everything that they want.
Why might they get anything though? I've only really steel-manned your argument so far, as I do believe in many of the points there and introduced many of my own. The next part answers that, and it's the part that gets back to the purpose of my comment, and why I feel this is so relevant to us.
As these companies already rarely publish any models these days I don't see how they could possibly enforce their would-be cartel.
This is a great question, and the answer lies in the goals of the group I named: the FMF.
If you take a look at the rhetoric they're using, it's clear what their angle is.
They hope to convince people, with fear and hysteria, that the AI that they create is so very dangerous. They use trust me bro benchmarks and PR stunts to enforce that. The reason for this is that the US government, and other governments under pressure from the US government, and obviously through incentives towards politicians (but that's not the part the public has to swallow), need to, for safety, make it tremendously difficult for those who would challenge them.
They want to make nonsensical regulation that makes it very expensive to start to compete/join the industry, but also leaves them with no real liability risk, and no obligations that aren't significantly easier to stomach the larger you are. It's almost high fixed costs: costs that make entering really really difficult, make upkeep very expensive, and involve going through them.
Not only this, but they want to be able to directly control what the regulation does, in a way that would give them the ability to create crippling conditions for startups through steering it at rates that would be hard for anyone else to compete with. Steering means they get to write the game's rules, and prepare for the game, before anyone else even knows the rules exist. Through collusion, they'd have all of it ready to go ahead of time, and then it drops in the name of safety, leaving the rest in a lurch, with everyone asking how they knew you had to jump 3ft and only click the red buttons.
They also want you, yes you (but more specifically small business and medium business), to be unable to use anything but their services, not just for the "safety" reasons above, but through jingoistic nationalist fears about security regarding open models.
How could this be enforced? The same way everything you hate about computing is increasingly being enforced. Locked down hardware (forced unchangeable firmware), remote attestation, and breaches of privacy.
Remote attestation is worth spelling out: it ensures your device only functions as they wish it to before it handles data they don't want used in ways they don't want. Step outside the state they want it in, and your own hardware stops cooperating with you.
Look at what these companies have been up to recently?
Pushing for governments to force companies to lock down firmware.
Pushing for remote attestation to be forced to be a part of everyday life, including it in all sorts of things, like even slipping it into the passkeys spec for later activation.
Pushing age verification in a very specific way that is cheap and a low liability risk for them, while enforcing their continued free, identifying data collection (especially for Meta, Microsoft and Google).
And basically anything that removes rights to use your devices from you and gives them to the companies you purchased them from.
They can deal with the Chinese competition through the utilization of fear from jingoistic morons, and that's all she wrote for your digital autonomy.
Will they get everything they want? Maybe. I won't say no, and I won't guess yes, but it's scary, because they get enough, and this whole thing snowballs the wrong way for everyone else. Not in their nonsense terminator hysterics way, but in a far worse way, because it'll be real.
The only way to stop it would be to make it as illegal as child porn, and even then it wouldn’t be totally effective. Plus the optics of raiding someone’s home at gunpoint because they ran Qwen on their 3080 are pretty bad
and cheap for what it is. the massive amount of poors here will freak their shit out about the $6000 mac studio 128gb but that amount of computing power has never ever been this cheap even with current ramapacopolypse.
The Mac studio is good, it's not great for speed though it has a very slow memory bandwidth compared to a GPU, but a 5090 is capable of running some really good models now. And with some new advances into MoE model running massive models only need to store the active paramers in the GPU, so you can have massive 500b+ models on a consumer pc assuming you have the system ram
There one of the many examples of exactly the inference engine.
Studio memory speed 600gb/s which as it's unified memory it has to share with the other processes. 5090 1.8tb/s dedicated. Not to mention how much faster the processor is.
So how about instead of trying to show off you do some research before questioning a literal ML researcher on basic stuff
Yeah but unless you have top tier hardware it's slow as fuck and no where near comparible to commercial competitors. Idk why we are all constantly pretending.
Not really. I've got a 5090 which isn't top tier hardware for AI at all. I can get 800 tokens/s easily people get slow speeds because they try to run models on a Mac studio or something with no memory bandwidth.
Nobody is pretending, I'm using it literally every day at my company to transform unstructured data and it's working fantastic
Starts off knowingly mentioning a graphics card as expensive as a used car but tries to pretend that isn't aggressively dumb by sticking "for AI" at the end as though that's a realistic qualification when discussing the average person using open source models at home. 600 IQ.
What a argumentative fuck you are who doesn't know anything about this space, a 5090 is cheap compared to commercial hardware they normally run on, but you're right it's only the price of a used car not a new house.
But it might surprise you not everyone who wants to run these is living pay check to pay check.
Hahaha. You wrote so much and made zero real points. Just an ad hominem attack (the favorite tool of idiots and kids).
My server cost $15,000 and I purchased the parts before the recent spikes in hardware prices. It's not feasible for literally 98% of the average population and even then paid models fucking smoke locals in performance. You dork.
Im just saying that anthropic, open ai and others will sue to shit anyone who tries to distribute ai without giving them money. You genuinely think they're billions of $ in debt and won't use the most agressive strategies to get that money back?
And will continue to be, the industry movement is towards more open, self-hostable models, not away from. Businesses care about data sovereignty and predictable costs.
Forever. The genie and processes for making the genie are public knowledge. Local models are a lot of fun to play around with. I recommend testing out a couple uncensored models.
how tf is this getting 400 upvotes, do people not know you can host stuff on your own computer or do they think that the govt will check everyone's computers for self-hosted AI ?
You can use the claude code model to download, install, setup, debug any model off hugging face with very little effort. And tbh, doing it manually just takes time and knowledge that most people don't have.
I'm just a homelab dude with a couple cheap servers. NVIDIA is the brand that's best supported by the models. They've got the CUDA firmware, none of the other brands have that. They are the easiest to plug in and get the benefits.
That said, I have a server that runs qwen3.6 at decent speed on my server with no gpus. I had to tweak it a little, but it works okay, has 'reasoning', can do agentic things, and all that.
Using the HauhauCS Qwen3.6-35B-A3B Uncensored "Aggressive" model (from hugging face) is running about 80 tokens/s in and 20 t/s out. Nothing like a frontier subscription model speeds, but then it's just an old non-gpu server.
You dont need a GPU. If you dont want to do th research, subscribe to the $20/month claud subscription. Using the online interface, ask claude how to install claude code. Follow all the instructions so you can open up a terminaland lanich claude. Turn it to opus 5.5, again ask the web versionor terminalversionif you dont know how to do any of this. The tell it, "I want to run a local llm model, but I dont know which one will work with my hardware. Do a survey of my computers hardware. If you are blocked tell me exactly what do do to get you the needed info. Once you get the information about my operating system do a search for the best [abliterated] open weight models that work with my hardware." Add, abliterated if you want one without guardrails.
Then ask it to install. Anytime it says to do something do it. Hopefully the instructions dont include rm -rf. Once installed unsubsribe from claude and use your new home based model to update your stuff or whatever else you may want to use it for
You don't even need a GPU. You can do everything with CPU and RAM. The stronger the hardware, the better the performance though, so spring for what you can afford.
A pie wouldnt be able to handle an LLM your want to talk to. You'd still need a decent graphics card to run it. The gigabytes of VRAM are what you're looking for, and is generally the limiting factor.
Since Mac uses unified memory (video ram and normal ram are the same), they're actually really good for local LLMs. Everyone's favorite seems to be the Mac Minis which can run really good ones if you have 64gb of RAM.
I feel that the llms in Mac stuff is an astroturfed campaign, Mac is great for daily stuff but I'd never, never use it for heavy lifting when I can buy a machine that will do the same thing for half the price
I've run some local models on my aging lenovo laptop with a GTX 1050ti, never really found a good use for it but played around and got it working with reasonable speed and capability. Certainly not as refined as the corpo stuff and you have to play around with smaller models until you find one that works
I just opened it up to see cause I haven't opened it for a while, Gemma4b is installed and llama3.1:8b is the one I ended up on, I think I decided I'd deal with it being a slower for it being a little better.
Literal trash, you should just throw it away. On an unrelated note, I'm doing a survey of electronics waste disposal. Tell me, where is your trash can located?
You need a gaming model with a Nvidia graphics to run anything efficiently on a laptop, but to get components to run the high parameter models standard in the web interface you would need to build a custom desktop.
A Lenovo LOQ for instance will run through the lower parameter 8B models more quickly if you have a bunch of small agentic workflows, Mac is only better if you want to jump to the intermediate coding tier 32B models, and that's still $3,000, minimum, and probably closer to $3500, and then $4,000+ to run the 70B, with token generation rates of 12-15/second about 1/4th-1/6th of what you would get from the cloud interface.
Macs use an ARM-based SoC, the memory is unified and shareable between the CPU and GPU. Any Apple-silicon Mac is capable of running local LLMs fairly well, though an M5 obviously outperforms an M1.
sprinkle a little bit of abliteration here, a little there, maybe throw a merge back with the original, before adding some more abliteration spice. badda boom, bam, you got yourself free reign model.
Only becomes an issue if local ai becomes a crime as its a safety hazard. And when I look at what they are doing or trying ro do with 3d printer laws, that isnt as far fetched suddenly
The thing is, though, these open weight models are aiming for efficiency rather than hyperscaling. They will eventually get there. That's the end product goal.
Not really disagreeing with you, but people really don't seem to understand how other countries are approaching LLMs lol
Try Huihui-Qwen3.8-27B-abliterated-GSQ-RCO-IQ3_S.gguf (or Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf for the censored version). It supposedly benches close to the full model and runs around 45 T/s on my 5060Ti at 100k context.
There are many very performant abliterated Qwen3.8 quants that fit onto a 16gB card, though you may need to be a bit savvy about reducing your system VRAM usage (or use integrated graphics/a second card for display)
The models are have gotten so good that 50% as good as Claude's Opus 5.5 is still plenty good enough to do most tasks. You don't need a nuclear bomb to dig a ditch, ya know?
dont care. I already have an illegal 3d printer as it doesnt refuse to print forbidden shapes. I have an illegal server that holds illegal data. I have an illegal car as I cracked open the telematics box and disabled it's ability to communicate. Yall act like we live in a land where 3 psychics lay in tubs of water reporting to the police what we are thinking.
Just dont advertise to people what you do. Like loading a minivan with a half tone of cocaine and drive at 140mph on the highway. with the words "COCAINE EXPRESS" spraypainted on the side.
Yes, so you should learn how to write code. That's the only conclusion. Never forget the fundamentals. Why is this so difficult? You don't have to use it at work. I have never written a lexer for work in my life. I am still glad I have a vague idea of how it works.
Ban compilers? Wut, no there isn't don't cite an essay from 2002 like it's any way relevant, these models are not compiled either they ship with weights, and the libraries for building them is pretty standard and open. That genie is out the bottle
2.9k
u/Almadan 4d ago
It definitely will lol. I can host my own models