I am looking at resintalling my OS (currently Mint) which I have been running for a few years.
Mint is great and I may end up going back; but was thinking of trying something new.
I want to be able to play with local models and agents; I have a pretty decent system.
- AMD 3700X
- 64 GB DDR4
- AMD RX9070 16GB GPU
- 2TB NVME SSD
- A few TB of HDD’s
What are some good options for easily running models and agents; I have been using ollama to run models; but I’m sure things have moved on since I begun playing with things a year ago.
Give me a few tips; I am going to have a bit of time on my hands over the next few months.
Look at rolling releases. I’ve used Mint but at the time (maybe still today) some projects required the latest ROCm drivers, and Mint’s repositories are just too slow to keep up with pace of AI. I decided to go with EndeavourOS since it’s a fork of Arch so it’s a rolling release, and you can keep up with the latest experimental AI hacks that are developed. The downside to a rolling release is that sometimes you update your computer and something stops working, like your sound drivers or Steam. But someone always finds a fix a few hours later. It doesn’t happen every time. I generally update my system when I need to, so about every month or two. I use EndeavourOS as my daily driver actually, so it’s not unstable at all. Anyway, look at rolling releases that have more than 1 maintainer. Look at their forum, see how helpful it is, etc.
Maybe also consider moving away from Ollama:
I agree with the sentiment in general, ollama sucks and is problematic, but the obstacle that stands in the way of that is reflected pretty much right here:
llama-swap handles multi-model orchestration, loading, unloading, and hot-swapping models on demand behind a single API endpoint. Pair it with LiteLLM and you get a unified OpenAI-compatible proxy that routes across multiple backends with proper model aliasing.
The article is just vaguely asserting people can go ahead and install 3 different applications and then figure out how to configure them to talk to each other and do all the things you want it to do seamlessly, without ever even attempting to explain how to actually do that, just to get what ollama does in 1 installer.
Yeah, pick your poison. And mind this article is a few months old. I’m not sure where we’re at as of today, but plain llama.cpp has multi model routing built in as well. Not sure if all of those extra steps are still necessary to do.
That’s one alternative they describe. They also mention other all-in-one alternatives. Ollama is not the only one of its kind.
Ignoring the other comments around complexity. If anybody is considering running LiteLLM in their lab. I strongly recommend looking at GOModel instead. LiteLLM was nothing but bugs, slow page loads, and security issues for me. GOModel just worked and was simple.
Interesting; what do you suggest to run in its place?
I started with ollama then moved to llama.cpp directly and just use it through llama-server and my own custom code now. The lack of ability to use mmproj for community models (meaning things like vision processing in uncensored qwen did not work at all!) plus the fact that I had to edit a modelfile every time I want to change temperature pushed me over the edge to figure out the rough edges around llama.cpp. There have been a few bumps in the road, but it’s been worth doing overall.
The blog article recommends you go straight to the source and use llama.cpp and it’s llama-server. And they mention a few other options as well. I did that for a while, but I’m mostly using KoboldCPP. Guess it depends on what people are trying to accomplish, but it’s packed with features, fairly easy to set up and operate and does images and audio as well. (But then I don’t use AI all that much, so maybe don’t take advice from me.)
I used a mix of Ubuntu, Fedora, and Windows for a decade or so but recently switched over to CachyOS and it’s pretty nice, installing / updating graphics drivers is easy as and having access to the AUR is pretty neat.
Switching to a new distro would also be a good time to explore better alternatives to ollama too, plain llama.cpp is decent (got it running on my laptop rtx 3060) and Strata is quite neat for supported MoE models - runs a bit faster than llama.cpp on my oculink 9060 xt + 64gb ddr4.
Just stay with mint. It’s fine.
How is switching OS related to local models? Are there any OS that is more efficient? Is there some software that serves certain OS better? I’m curious because I’m also using Mint, have been for about a year now after ditching Windows and I was actually exploring potential benefits of other distros for the first time earlier this week. I’d love to learn if there is something to keep in mind, or is it related to your AMD hardware?
I was going to do a reinstall; it has been a few years of inplace upgrades and a lot of extra cruft that inevitably gets built up. I was asking mainly to see if there was a clear recommendation for which system is best for experimenting with local LLM’s etc…Looks like there isn’t any specific advantages to any system so I’ll probably just stick with Mint, likely wait until Mint 27 is released before doing an upgrade.
Ah, I was worried that there was some local model distro meta that I wasn’t aware of. I have no knowledge of AMD but according to DeepSeek we seem to have somewhat comparable hardware for local LLM. I just noticed the 64GB RAM and figured you’d be on a whole different level, but apparently that setup is in the mid-high tier, spanning from 7-32B? What kind of models have you tried so far, and did any stand out? I’ve only been able to get a 13B Qwen3.5 model as most myself (which was way too slow to be worth it). From what I have picked up >24B is where you get a generally good model that goes beyond a niche purpose.
It has been a while, but last model I ran was gemma4:e4b 9.6GB. It ran very quickly; also have run deepseek-coder-v2:16b 8.9GB, again very fast…
I am looking to get back into it once I have a bit of time on my hands; I’ll look into the state of the art models again and see what I can run.
No suggestions, but keep us updated on your experience with ROCm. I’d love to migrate over to AMD GPUs for local model training, but the architecture just doesn’t seem to have the toolkit support I need to feel secure about such a decision.
I have been running models on it for a year…seems to be getting better and better; I never ran on Nvidia so I can’t compare.
I’m pretty heavily invested into training image and video generation models, so my use cases might be more particular. I recently did a huge deepdive and decided I should just stick to CUDA for the time being, given the specific toolkits and software I leverage in my process.
A lot of the software targets Ubuntu compatibility so that would be the easy distro choice. But, you should be able to get this stuff running on any/most distros as long as you’re comfortable building/compiling the software as needed. Or using containers like Docker, etc. I doubt you’d have any issue doing this with Mint if that’s your preference.
I’m just tinkering with this stuff so other people here will have way more experience. But as a point of reference I have no trouble running llama.cpp / sd.cpp on current Debian Trixie (with/without an updated Linux Kernel). This is with an AMD R9700 (32 GB VRAM) in an older Intel based system. Everyone says that running with Vulkan is faster but I’ve found it’s good to test your own setup - My overall inference speed is actually faster using ROCm. ROCm gives me less tokens per second but for whatever reason the RAM-VRAM operations are much faster when using ROCm which leads to the overall time from prompt to result being shorter. There’s likely more tweaking I could try to get more speed but I’m not heavily using AI at the moment so am happy with how it works now.
Before that had the same setup with a old Nvidia gaming GPU (6 GB VRAM), it sort of worked but was slow as hell and I could only get it to run with Vulkan. That GPU supported an older CUDA version (7.5 I think) and I never figured out how to get that working with llama.cpp + Nvidia software + Debian.
deleted by creator
I would say that the best os/distro is the one you’re comfortable with. With that said, I would really iron out why you want to switch, like what you’re looking for. Stability? Debian. Newest features asap? Arch. Mix? Fedora. General support? Probably any of them, but fedora or Debian if you want properly supported packages and not rely on the AUR. (You can use any of the derivatives, but I can’t say they do anything special for me, so I stick to the core distros)
