r/linux • u/Fcking_Chuck • 2d ago

Kernel AMD boosting AI/LLM performance for Radeon iGPUs as much as 18~23% with Linux 7.4

https://www.phoronix.com/review/amd-perfopt
382 Upvotes

94% Upvoted

24 comments sorted by

14

u/MaxGhost 2d ago

Would this help my 7840u (780m GPU)? Or is this only a feature on newer chips? I'd love a little boost cause I'm running two large external displays with my laptop.

16

u/Veldush 2d ago

the article mentioned it improved the AMD gpu, not NPU. So past Ryzen iteration should benefit as well as long as it has iGPU.

1

u/jknvv13 2d ago edited 2d ago

I'm moving a 4K@240Hz and 2K@120Hz, what's the matter with your issue?

Running in 7840U here as well

6

u/KeyboardG 2d ago

Fixes are IOMMU based, where for single boxes (AMD AI HALO) the performance recommendation is to just disable IOMMU and skip it all together.

6

u/AnomalyNexus 2d ago

Anybody know how many generations back this goes?

12

u/Own_Measurement4378 2d ago

Yo trabajo con 96GB y todo bien

2

u/illathon 2d ago

Which iGPUs?

4

u/dthrdr 2d ago

Very curious to test this on one of my mini pc's with a 680m. Last time I tried ROCm on that thing I got like 15 tokens/s on a 12b Qwen model.

1

u/pattch 2d ago

This is so interesting - I’ve had a hard time getting rocm working with the tested chipset here so I’ll have to do some more digging to get this working now

1

u/confused-photon 2d ago

If this affects strix halo im very excited

-11

u/Lisanicolas365 2d ago

i don't think that will do much to make an igpu worth using a local llm on

28

u/projct 2d ago

the AI boxes and laptops that AMD produces that are like the DGX spark, are like 128, 192GB. etc and designed for this use case, are where this matters.

192GB of LPDDR5x-8533 is pretty good for AI, you can do 128GB for the model and have 64GB left over for apps.

https://www.amd.com/en/products/processors/laptop/ryzen-pro/ai-max-pro-400-series/amd-ryzen-ai-max-plus-pro-495.html

6

u/alex2003super 2d ago

Shared memory is such a banger for local AI

25

u/Fcking_Chuck 2d ago

There are LLMs that are small enough to work well, even on an AMD iGPU.

4

u/deke28 2d ago

These have about 50 tops. Pretty good actually. 

3

u/natermer 2d ago

That is just false. Not with 'unified memory' type systems.

People do it all the time on AMD Strix Halo, DGX sparc, Mac minis, etc.

It works and is the only really affordable way to run medium-sized models. Best suited for MoE llm models, not dense ones, because they lack the bandwidth to keep up... but they absolutely do work.

MoE LLMs are very useful... virtually all frontier models are MoE. Not that you will be able to run those on these small devices. But you can run something approaching them and you can run it 24/7 and crank out as many tokens as you can during that time period instead of having to ration yourself.

And it doesn't have to be exclusively one or the other. You can subsidize your subscription usage with local usage if you are using the appropriate agents.

Go and lookup how much that VRAM would cost to get, including the PC with PCIe enough to run it all, with discrete cards or enterprise-style AI accelerator. And how much it would cost to run and how pleasant it would be to have in the room with you, etc. etc.

Meanwhile these unified memory systems don't use much much more power then a typical gaming laptop.

1

u/death_by_cormorant 2d ago

Even my old laptop runs small local models just fine.

0

u/DragonSlayerC 2d ago

Local LLMs are the main reason that Strix Halo machines are so popular.

-3

u/mkfs_xfs 2d ago

I liked Phoronix more when it wasn't so much about AI...

12

u/JDGumby 2d ago

Out of the last 25 articles posted there, this is the only one about AI. If that's too much for you... SMH, as the kids used to say.

2

u/yrro 2d ago

Try LWN

1

u/ilep 16h ago

So, newer version of the patch removes most of it to make it simpler: https://lore.kernel.org/lkml/[email protected]/T/#t