> ## Content Index > Fetch the complete content index at: https://blog.matthewbrunelle.com/llms.txt > Use this file to discover other available public pages before exploring further. # Using Self-hosting Language Models So You Can Evaluate Claude Code - URL: https://blog.matthewbrunelle.com/using-self-hosting-language-models-so-you-can-evaluate-claude-code/ - Published: 2025-11-13T16:16:00.000Z - Updated: 2025-11-13T16:16:00.000Z - Description: Because why would I pay Anthropic when I could do this myself with Qwen3-coder, Ollama, and LiteLLM on NixOS. Especially if I just want to try the tool out and see if its worth using. - Author: Matthew Brunelle - Tags: Self-hosting, NixOS #### On this page I've avoided hopping on the AI hype train on this blog, but this topic is in line with what I usually write about for self-hosting. My work had a nice workshop that I want to write about. However, that requires using Claude Code and that in turn requires paying Anthropic money. So I first wanted to do a little guide on a self-hosted alternative for running a language model that works with Claude Code. Normally, I self-host all my services on my Unraid server. I explicitly built it with no dedicated GPU to reduce power consumption. The Intel Core i5-13500 CPU I have is more than enough for video transcoding and surprisingly power efficient. Technically, for my [self-hosted dog camera](https://blog.matthewbrunelle.com/self-hosting-a-dog-camera-for-away-from-home-monitoring/) I also added a Coral Dual Edge TPU, but that also sips energy. However, in my desktop I have a AMD Sapphire 7900 XT [\[1\]](#fn1) so I will install everything for Claude code there. This works out well, because I do most of my personal projects on my desktop anyways and I don't need the model to always be available, only when I'm working. Additionally, I run NixOS on my desktop so I will share my nix configuration, but the ideas here will work more broadly. Essentially all I need is: - Something to run the model on. - Something to make the model look like it's an Anthropic model. - Some configuration changes for Claude Code. --- ## On model choice [MiniMax M2](https://www.minimax.io/news/minimax-m2?ref=blog.matthewbrunelle.com) released last week, and on benchmarks it compares really well to Claude's official models. The model is also specifically designed for use in agentic coding. It is self-hostable in the sense that the weights are open, but the [requirements](https://huggingface.co/MiniMaxAI/MiniMax-M2/blob/main/docs/vllm%5Fdeploy%5Fguide.md?ref=blog.matthewbrunelle.com#:~:text=OS%3A%20Linux,context%20tokens.) state: > Memory requirements: 220 GB for weights, 240 GB per 1M context tokens > > - 4x 96GB GPUs: Supported context length of up to 400K tokens. > - 8x 144GB GPUs: Supported context length of up to 3M tokens. So self hosting is not particularly feasible. Especially since I am trying to be considerate of power usage. I think conceptualizing the requirements for these models is important for understanding just how much resources these tools are sucking up. Additionally, Ollama does not even support MiniMax-M2 yet. The only supported inference options are SGLang, vLLM, and MLX. Of those MLX is Apple Silicon only. Of the remaining two, vLLM has better LiteLLM integration, but getting vLLM working is a giant can of worms currently. So for models that could reasonably be run on a desktop single GPU, there's GPT-OSS 20B or [Qwen3-Coder 30B](https://ollama.com/library/qwen3-coder?ref=blog.matthewbrunelle.com) which are both good options. I chose the later because of the larger context window, which will be relevant in the next post I make. Qwen is also nice since it uses Mixture-of-experts which reduces the number of active layers. Finally, Claude Code needs a model that supports [tool calling](https://docs.ollama.com/capabilities/tool-calling?ref=blog.matthewbrunelle.com) which Qwen checks the box on. --- ## Setting up Ollama [Ollama](https://docs.ollama.com/?ref=blog.matthewbrunelle.com) is a very quick and easy way to download and run language models. The NixOS wiki has a nice [guide for Ollama](https://wiki.nixos.org/wiki/Ollama?ref=blog.matthewbrunelle.com). In my case I added the following configuration: ```nix services.ollama = { enable = true; loadModels = ["qwen3-coder:30b"]; acceleration = "rocm"; }; ``` The example from the wiki uses cuda for acceleration, but I searched nixpkgs for the [services.ollama.acceleration option](https://search.nixos.org/options?channel=25.05&show=services.ollama.acceleration&query=ollama.acceleration&ref=blog.matthewbrunelle.com) and see that I can use "rocm" instead. Typically on Linux, AMD hardware will get better performance with a Vulkan backend, but Ollama only [merged support last month](https://github.com/ollama/ollama/pull/11835?ref=blog.matthewbrunelle.com) and [nixpkgs is still waiting](https://github.com/NixOS/nixpkgs/issues/456454?ref=blog.matthewbrunelle.com) to add support. Also, note that I preloaded the Qwen model, but models can also be downloaded with: ``` $ ollama run qwen3-coder:30b ``` I do not use `services.ollama.openFirewall` since everything will run on the same machine. With a nixos-rebuild I then should see: ``` ❯ ollama list NAME ID SIZE MODIFIED qwen3-coder:30b 06c1097efce0 18 GB About a minute ago ``` --- ## LiteLLM Proxy Server setup LiteLLM can [be configured to work with ollama](https://docs.litellm.ai/docs/providers/ollama?ref=blog.matthewbrunelle.com). [\[2\]](#fn2) Additionally, LiteLLM provides a proxy server for models and NixOS supports running the proxy server as a service. I couldn't find a wiki entry for LiteLLM, but I found all the options using nixpkgs search. Normally, the LiteLLM proxy settings are written in yaml: ```yaml model_list: - model_name: "qwen3-coder" litellm_params: model: "ollama/qwen3-coder:30b" api_base: http://localhost:11434 model_info: supports_function_calling: true ``` On NixOS they can be set using [services.litellm.settings](https://search.nixos.org/options?channel=25.05&show=services.litellm.settings&query=litellm&ref=blog.matthewbrunelle.com). The settings field is a type "yaml 1.1 value" which maps into a [nix attribute set](https://nix.dev/manual/nix/2.18/language/values.html?highlight=attribute+set&ref=blog.matthewbrunelle.com#attribute-set). So I converted the above yaml into the following: ```nix services.litellm = { enable = true; settings = { model_list = [ { model_name = "qwen3-coder"; litellm_params = { model = "ollama/qwen3-coder:30b"; api_base = "http://localhost:11434"; model_info = { supports_function_calling = true; }; }; } ]; environment_variables = { LITELLM_MASTER_KEY = "sk-1234"; }; }; }; ``` Alternatively, I could have mapped the model to a Anthropic model name: ```yaml - model_name: claude-sonnet-4-20250514 litellm_params: model: ollama/qwen3-coder api_base: http://localhost:11434 ``` Also, the [default Ollama port on nixos](https://search.nixos.org/options?channel=unstable&show=services.ollama.port&query=ollama&ref=blog.matthewbrunelle.com) is 11434. Weirdly, the LiteLLM docs reference 8000 as the port for the web UI, but on NixOS the [port defaults to 8080](https://search.nixos.org/options?channel=unstable&show=services.litellm.port&query=litellm&ref=blog.matthewbrunelle.com): ![litellm_web_ui.png](https://blog.matthewbrunelle.com/content/images/2025/11/litellm_web_ui.png) When I attempted to go to the admin panel I was redirected to `http://localhost:8080/sso/key/generate`. LiteLLM needs a master key set and above I used an example key, but any random string starting with "sk-" works. For example, `openssl rand -hex 32` to generate an appropriate string. Hitting the /models endpoint gets a response similar to: ```json { "data": [ { "id": "qwen3-coder", "object": "model", "created": 1677610602, "owned_by": "openai" } ], "object": "list" } ``` That confirms LiteLLM is configured with the Ollama model. The model can be tested quickly with: [\[3\]](#fn3) ```bash curl -X POST http://0.0.0.0:8080/v1/messages \-H "Authorization: Bearer $LITELLM_MASTER_KEY" \-H "Content-Type: application/json" \-d '{ "model": "qwen3-coder", "max_tokens": 1000, "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` --- ## Claude Code Setup All that's left now is to make Claude code work with the qwen. Claude Code has [open issue for self hosting models](https://github.com/anthropics/claude-code/issues/7178?ref=blog.matthewbrunelle.com). So until they support that I have to trick it into using the model. This works because LiteLLM will conform to the shape of [Anthropic's Gateway API](https://code.claude.com/docs/en/llm-gateway?ref=blog.matthewbrunelle.com). So after [installing Claude Code](https://code.claude.com/docs/en/setup?ref=blog.matthewbrunelle.com) or just adding the `claude-code` package on NixOS, I just needed to [point it at LiteLLM](https://docs.litellm.ai/docs/tutorials/claude%5Fresponses%5Fapi?ref=blog.matthewbrunelle.com#4-configure-claude-code): ```bash export ANTHROPIC_BASE_URL="http://0.0.0.0:8080" export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY" export ANTHROPIC_MODEL="qwen3-coder" ``` Now, there is a chance it may not work due to a a [known bug](https://github.com/BerriAI/litellm/issues/13429?ref=blog.matthewbrunelle.com). If that happens, OpenRouter could be used as an alternative. --- ## Final thoughts I'm not convinced that Claude Code or tools like it will be necessary for writing code in the future. However, I do think there is value in trying to evaluate new tools to figure that out for yourself. That way you can have a more informed opinion. Normally, you would have to spend $20 to get a month's access to the models you would need to run Claude Code in order to do the evaluation. Self-hosting is a nice opportunity to just try it out and see how it goes. Even if the models you run locally aren't quite as good as the official ones. --- 1. Newegg somehow accidentally sent me this at one point, and never noticed... [↩︎](#fnref1) 2. MiniMax-M2 support for LiteLLM was [added last week](https://github.com/BerriAI/litellm/pull/15990?ref=blog.matthewbrunelle.com) if you are somehow able to run it.\] [↩︎](#fnref2) 3. This example is taken from the [LiteLLM docs](https://docs.litellm.ai/docs/tutorials/claude%5Fresponses%5Fapi?ref=blog.matthewbrunelle.com#3-verify-setup). [↩︎](#fnref3)