Many of us have been there: working with Claude, ChatGPT, Copilot, etc. and BOOM, the token allowance runs out and you have to wait a few hours for it to reset. All you want to do is finish your project.

Well, you are on your personal computer which can compute things. Perhaps your machine can even play some fairly recent games? If so, you have free compute right at your fingertips.

Some of the AI models used by the big companies are available for free. This is true. The trouble is knowing which model to get, where to get it, what to run it with and how to tune the model to your machine. If this sounds complicated, it's less complicated than installing current games and likely takes up less space. But you'll want to follow this guide for some tips on how to do this.

There are many models and many ways to run them. I'll show you some of what I have done as an example for you to do similar.

What You'll Be Running On

Hardware

This was one of the trickiest parts of my home computer journey. I have a fairly beefy machine but it's 6 years old. For what I do it's enough but for heavy modeling it's not the best. I have an AMD Threadripper 3960X 24 core/48 thread CPU and 128GB RAM and an nVidia RTX2080 Super GPU with 8GB VRAM. Models love GPUs and mine is dated with only 8GB VRAM. The 128GB system RAM is good but not shared with the GPU.

My machine is not a top tier system like buying a $5000 dedicated machine or having an RTX5090 GPU. So running the best models is not an option, as even if they ran, the measure of success, tokens/second, would be very low. 

You should start by gathering up your hardware specifications. Write down your CPU model number, your system RAM, and your graphics card information. If you only have integrated graphics, there might me a chance, so note that.

The Upgrade Conundrum

After writing down your hardware specifications, do you already have regrets, thinking your machine might not run do what you want? 

Here are some thoughts on that:

• Look for another machine in your house that might work better
• Upgrade your current machine (which is expensive in today's market but likely possible)
• Buy a new machine. This is beyond the scope of this note as I cannot possibly inform you on what hardware you need. Consider your requirements and budget, do some reading, and look to make an informed decision.

What Do You Want the LLM to Do?

When looking for a model, it helps to know what you might want to do with it. Do you want help coding? Writing? Research? Using other programs with an LLM ("Tool Use"), computer vision? Write down what you need and what you'd like along with your system specifications.

Software: Finding the Right Model Fit

Finding the right model to run given the hardware I had was a a large roadblock for me. Then a colleague found the free website canirun.ai. You can enter the hardware parameters of your machine and get a list of models and how well they may perform on your hardware. Do note it is rather GPU-centric, although it does have Raspberry Pi computers in the hardware list if that's what you're looking for.

This is likely not the only tool to help you choose, but it is the one standalone site I know of.

Here is output for my graphics card:

The top chart shows, I believe, more current models and how they may run. At right is a confidence on their running well or not and to the left of that how many tokens per second it could generate. 

So for my graphics card, only three more recent models might work well. Then there is the second table which has all models. You'll likely find many of these may run fine on your computer. The trade-off is they are likely older models or models trimmed down to fit small hardware budgets.

Looking at the models that will fit, check at the tags after the model name. They include:

• Wrench/spanner: Use tools
• Star - popular
• Light bulb - thinking/reasoning
• Eye - vision
• Graph - mixture of experts

Finally there is a box with the model's license. MIT or Apache is basically free to use. Other models might have conditions. Note that depending on your use case. You can click that tag to get additional information. For home use only, likely you're fine with any of them.

Making the Leap: How to Run Models

There are many ways to run models. I looked for a model interface, sometimes called an inference engine, basically a front end. The ones I found:

• LM Studio (free for home and work use, terms)
• Ollama (free but running multiple "cloud models" costs extra) 
• llama.cpp (MIT license)

I chose LM Studio. It runs on mac, Windows, and Linux in a GUI. It seemed more straightforward and no cost, so I chose it. You can do your research on the one you might want to use.

I'm sure I see someone in the back asking if these are "harnesses." No. The model interface is what you use to see the LLM. A downloaded file is the LLM itself. The harness is the scaffolding between the two that makes it able to remember, interact, and use agents. I'm sure there is a harness on my machine but when I use LM Studio, it handles things so I don't have to make harness connections (I think).

See this video on what a harness is and how all the pieces relate to each other.

Using LM Studio

Here is where I get more specific as LM Studio is the only model interface I have tried. If you want another interface (command line, Docker, etc.) you'll want to find a specific guide for that. You can always try LM Studio (as it's free) and go in another direction later.

Download and install LM Studio for your machine and CPU architecture. When I started this guide, the latest version for Windows x64 was 0.4.12 (Build 1). They have great Getting Started documentation.

You'll want to download one or more LLMs to your computer. Click on the Robot/Magnifying Glass icon at left (circled in red).

And here is the screen that pops up:

Search for the model(s) you were looking for on canirun.ai. Or look through the list. The same icons for model strengths are used except reasoning here is a green brain. The "Best Match" says "Only include staff picks that can fit on this device. Based on an estimated model capacity of xxxx.xx GB". 

Do not select such a big model that your hardware couldn't do well with it. On my 8GB GPU, I think models with more than 9B are likely too big.

You can read about the models. When you click download, it will download the model data, which is usually several gigabytes.

After downloading, you can clock on the "My LLMs" icon circled below to see your models.

I have four models downloaded. I have been trying out the qwen 3.5 and 3.6 models. Gemma 4 and Trellis 2 are there also.

How about trying one out! Click the "Use in New Chat" button circled below.

I type "Hello" and start a conversation:

I have successfully used it that mode; to discuss CircuitPython coding and more. All with zero cloud costs.

Note that it's best to use a new project for new topics and not switch in the middle of a conversation. I tried after starting with CircuitPython and every answer to the new line of questioning led to it wanting to use a solution to the previous line of questioning.

Tuning Your Model

If you get an error, something about 4096 tokens or some such, you need to tune your model and you probably want to anyway to be sure the model's thinking time is kept to a minimum.

Click the My Models icon on the left then the Load button on the right. There will be three sliders:

1. Context Length - this is the cause of the error if you had one. You will need to try different values to see how to minimize thinking time. Definitely higher than a default 4096 unless you do not have a GPU. With K and V Cache Quantization Type F16 and checked, I get my best performance at Context Length at 11486 for my 8GB card.
2. GPU Offload - will allow the model to use your graphics card if you have one. Set it almost max, leave a bit for your computer to render your screens. I got my best performance at 30 (out of 32?).
3. CPU Thread Pool Size - even though it says thread (I have 48 threads) it seems to be limited to CPUs in your microprocessor. I have 24, so I set it at 21 to let it use most of the processing power. Bumping it up seemed to hurt performance some.

These will have a big impact on your model seeming to be as snappy as your cloud AI provider (if you have one) so spend some time in tuning to get good results for that model. If no setting helps, perhaps look to use a less taxing model to get better responses at the cost the model may have been trained on less data.

I would change a setting, give it a greeting, see how many tokens per second were generated and how many tokens were used. Change one setting, give it a similar greeting, see the results, and iterated like that. Likely not the best.

Tool Use

Out of the box, models will not likely be using tools to do local or remote operations. If you don't need that' you're golden. If you do want to use an LLM to do things, you'll need a tool package added in.

I found Beledarians-lm-studio-tools. It adds a number of common tools for accessing your computer locally, fetching web content, and more. And there are security settings to enable, ask, or disable use of each one. Go to the page linked above and click the "Run in LM Studio" button.

This will give you many more settings in the right toolbar:

You'll want to go through the list to change from Ask to Allow or Deny depending on your trust in the model using tools responsibly.

Reference:

• Yes, local LLMs are ready to ease the compute strain 5/2026
• How to use Unsloth as an API endpoint 
• After a year of self-hosting LLMs, I realized the real bottleneck isn’t the GPU
• Claude Code with a local LLM running offline is the hybrid setup I didn't know I needed
• Run AI Locally: The Best LLMs for 8GB, 16GB, 32GB Memory and Beyond 11/2025