Running local AI on a Linux PC can be a powerful tool, but it can also put a strain on the system. A writer who uses local AI on their desktop has found that with the right hardware, software, and setup, local AI can be a valuable asset. The writer’s reasons for using local AI include privacy, environmental, and economic concerns, as they don’t have to hand over their information to a third party, cause harm to the environment, or put a strain on the electrical grid.
The writer has experienced situations where their computer came to a grinding halt when using local AI, but after working with local AI systems for some time, they’ve found ways to run AI tasks without worrying about their primary desktop machine becoming unresponsive. One key factor is using an Nvidia GPU, which runs AI exponentially better than a CPU or other video hardware thanks to parallel processing and CUDA Cores.
Hardware Requirements
Nvidia GPUs contain tiny cores designed to execute thousands of calculations simultaneously, making them perfectly tailored for AI. The writer experienced this firsthand after adding a machine to their network with an Nvidia GPU and adopting it as their local AI server. Every task they sent to that machine ran faster and more reliably, thanks to the Nvidia hardware. They also installed the latest proprietary drivers for Fedora, the OS they chose for that machine, and verified that the GPU was active and visible to the system.
While it’s possible to get enough performance from an AMD GPU, especially with the installation of ROCm, an open-source software stack created by AMD to run AI workloads on AMD hardware, an Nvidia GPU is the better choice for running AI tasks smoothly. The writer has AMD hardware and found that installing ROCm makes a big difference, allowing them to run more intensive AI tasks with minimal impact on their machine.
Optimizing System Performance
Enabling ZRAM on a Linux machine can also help prevent system lockups and keep the desktop responsive when running local AI tasks. ZRAM expands usable memory on the system by creating a compressed block device in physical RAM, allowing the machine to pack roughly 1.5 to 3 times as much compressed data into the same physical space. This helps when loading small to medium-sized models that would normally strain physical RAM, and it’s considerably faster than traditional swap, smoothing out memory spikes that can occur when running other apps alongside AI.
Read Also: Google Fitbit Edge offers slimmer design and sleep upgrades
The writer enabled ZRAM on their recently deployed Fedora server with an Nvidia GPU and noticed a slight improvement in response times and fewer pauses during AI processing. While ZRAM won’t allow running more demanding AI models, it makes a big difference with smaller and medium-sized models, and it’s a useful tool for optimizing system performance when running local AI tasks.
Selecting the right model for the job is important to achieving goals with local AI. The writer notes that small, medium, and large models are available, and the name of the model usually gives away its size, with small models labeled 1B, 2B, 7B, or 8B, medium models labeled 30B and 70B, and large models typically called LLMs or frontier models, measured in trillions of parameters. For local AI, it’s best to stick with small models when possible, or medium models if the hardware is more powerful.
Using a dedicated server for AI purposes is another approach the writer recommends. Instead of installing AI on their primary desktop machine or laptop, they install it on a server within their LAN and point any AI app they use to that server. As long as the server has enough resources to handle the task, everything runs smoothly, and the desktop machines never even notice when AI is in use. This method takes a bit more up-front work, but the payoff is worth the time involved, and it’s a reliable way to prevent system crashes when running local AI tasks.
The writer’s server, which is used for AI purposes, has an Nvidia GPU and is running Fedora, with ZRAM enabled to optimize system performance. The server is connected to their LAN, and they use it to run AI tasks, which are then accessed from their desktop machines. This setup has allowed the writer to run AI tasks smoothly and efficiently, without putting a strain on their desktop machines.
