My level of worry hasn’t lowered in years…

But honestly? Low on the totem pole. Even with Trumpy governments.

Things like engagement optimized social media warping people’s minds for profit, the internet outside of apps dying before our eyes, Sam Altman/OpenAI trying to squelch open source generative models so we’re dependent on their Earth burning plans, blatant, open collusion with the govt, everything turning into echo chambers… There are just too many disasters for me to even worry about the government spying on me.

If I lived in China or Russia, the story would be different. I know, I know. But even now, I’m confident I can given the U.S. president the middle finger in my country, but I’d really be more scared for my life in more authoritarian strongman regions.

Presumably you will advance along with humanity though, or failing that, just figure out the transcendence thing yourself with so much time?

I don’t think anyone would choose to stay ‘meatbag human’ for trillions of years.

Almost all of Qwen 2.5 is Apache 2.0, SOTA for the size, and frankly obsoletes many bigger API models.

These days, there are amazing “middle sized” models like Qwen 14B, InternLM 20B and Mistral/Codestral 22B that are such a massive step over 7B-9B ones you can kinda run on CPU. And there are even 7Bs that support a really long context now.

IMO its worth reaching for >6GB of VRAM if LLM running is a consideration at all.

I am not a fan of CPU offloading because I like long context, 32K+. And that absolutely chugs if you even offload a layer or two.

For local LLM hosting, basically you want exllama, llama.cpp (and derivatives) and vllm, and rocm support for all of them is just fine. It’s absolutely worth having a 24GB AMD card over a 16GB Nvidia one, if that’s the choice.

The big sticking point I’m not sure about is flash attention for exllama/vllm, but I believe the triton branch of flash attention works fine with AMD GPUs now.

Basically the only thing that matters for LLM hosting is VRAM capacity. Hence AMD GPUs can be OK for LLM running, especially if a used 3090/P40 isn’t an option for you. It works fine, and the 7900/6700 are like the only sanely priced 24GB/16GB cards out there.

I have a 3090, and it’s still a giant pain with wayland, so much that I use my AMD IGP for display output and Nvidia still somehow breaks things. Hence I just do all my gaming in Windows TBH.

CPU doesn’t matter for llm running, cheap out with a 12600K, 5600, 5700x3d or whatever. And the single-ccd x3d chips are still king for gaming AFAIK.

I still have to keep Windows around for (ironically) performance reasons.

Some sim games like Rimworld and Stellaris just have a big hit on linux for me, native or Proton. And in a sim game, that means slow turns and stutters you can’t avoid instead of slightly lesser graphics. And it’s not sublte, native stellaris is like a good 30%-40% slower with even higher spikes last time I benched them back to back.

It totally kills interest in the device though. Its like they think every single buyer is an impulse buyer who just looks at the demo in Best Buy without even looking it up or trying it.

It’s really sad, as it kills the end product for no good reason. Just theoretical pennies.

I mean, Windows would be fine if the OEM stripped it down instead of bloating it even more. They can totally do with with group policies.

I honestly don’t know what they were thinking here.

They just can’t help themselves, lol.

All that bloat is bad enough on a laptop, but its the absolute last thing a handheld needs, both for performance/battery and ergonomics.

Yeah. Normally I wouldn’t care, but when the colonies/galaxies get big, even a 30% TPS hit or whatever starts to feel very painful.

I would note that Rimworld and Stellaris (for me) run much slower in linux than they do on windows, and they are kind of perforamnce sensitive games.