Technical Documentation

Walkthrough: Fine-Tune & Ship

Step-by-step instructions to optimize a small weights model locally and serve it as a production endpoint on Render.

Why this track exists

A Stanford study found 71.3% of real-world ChatGPT queries could be accurately answered by a small, local model instead of a frontier API — a figure Hugging Face CEO Clement Delangue has publicly pointed to as concrete rationale for local-first AI. That's the whole reason this track exists: fine-tune small, cheap, and private instead of routing everything through an expensive frontier API by default.

Render doesn't offer GPU instances, so training happens on your own machine or a cloud GPU — Render is for serving the result. That's why this walkthrough splits fine-tuning (Steps 1–4, wherever you have compute) from deployment (Step 5, on Render).

Before You Start

Could your device run this locally?

Checking for a WebGPU adapter…

This only checks browser-side inference (WebGPU) — it's separate from fine-tuning. Training with llmtune still requires a real NVIDIA CUDA GPU (see Prerequisites below), regardless of what this shows — a Mac's Metal/WebGPU adapter doesn't count for that step.

Prerequisites

Python 3.9+

The toolkit itself is a Python CLI — everything below assumes this is already on your PATH.

python3 --version

Install Python

pipx

Installs llm-toolkit into an isolated environment — what Step 1's pipx install command uses.

pip install pipx
pipx ensurepath

pipx docs

Hugging Face account + token

Needed to pull base model weights, and required for gated models like Llama-3 (Advanced track).

export HF_TOKEN="hf_..."

Create a token

llama.cpp

Converts and quantizes your fine-tuned model to GGUF for CPU serving — used in Step 4.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && pip install -r requirements.txt

llama.cpp repo

A Render account

Free tier, no credit card required — needed for Step 5's one-click deploy.

GPU with CUDA

llmtune's config uses bitsandbytes 4-bit quantization (QLoRA) — there's no CPU-only training path, a real CUDA GPU is required. No local GPU? Google Colab's free tier gives you a real T4 (16GB VRAM), enough for this Quickstart track. Kaggle Notebooks (30 free GPU-hrs/week) works too. Skip Flash Attention 2 in the Advanced track on either, though — it needs Ampere or newer, and T4 isn't supported.

Open Google Colab

Install Georgian's LLM Fine-Tuning Toolkit

The real CLI: a config-based tool, installed from PyPI. "llmtune generate config" scaffolds a starter YAML you edit rather than passing flags.

pipx install llm-toolkit
llmtune generate config

Configure Your Experiment — The Local-First Router

This walkthrough's flagship use case: a classifier that decides whether a query is answerable by a small local model or needs a frontier API. The toolkit is config-driven — data ingestion, base model, and LoRA hyperparameters all live in one YAML file.

# config.yml
data:
  file_type: "json"
  path: "data/router_dataset.json"
  prompt:
    ### Instruction: Classify this query as "local" or "frontier".
    ### Input: {query}
    ### Output:
  prompt_stub: { label }

model:
  hf_model_ckpt: "microsoft/Phi-3-mini-4k-instruct"
  quantize: true
  bitsandbytes:
    load_in_4bit: true

lora:
  task_type: "CAUSAL_LM"
  r: 32
  target_modules: [q_proj, v_proj, k_proj, o_proj]

Why this track exists

The 71.3% local-answerable figure driving this dataset's two labels ("local" vs "frontier") is cited in full in the "Why this track exists" callout above.

Read the paper

Run the Fine-Tuning Job

One command reads the whole config and runs data ingestion, training, and QA tests together. Artifacts land under experiment/[hash]/ — a hash of the config, so reruns resume instead of restarting.

llmtune run ./config.yml

# Outputs:
# experiment/<hash>/dataset   -> HF datasets format
# experiment/<hash>/model     -> PEFT (LoRA) weights, HF format
# experiment/<hash>/qa        -> automated QA test results

Export for Serving

llmtune's own output is a PEFT adapter in HF format, not a servable single file. To run it through llama.cpp on Render's no-GPU infrastructure, merge and convert to GGUF — the same conversion pipeline behind this app's actual reference model (see the callout above).

python convert_hf_to_gguf.py experiment/<hash>/model --outfile model.gguf
llama-quantize model.gguf model_q4.gguf Q4_K_M

Deploy to Render

Click the button to launch a pre-configured serving endpoint on Render — the llama.cpp server blueprint, no GPU dependencies. It ships with a Hermes-3 LoRA fine-tune trained on a support-ticket-triage dataset as the reference model; swap the env vars to point it at your own fine-tuned model instead.

*Deploys this repo's render.yaml blueprint, including the llama.cpp inference service (GGUF, no GPU) currently serving the Hermes-3 support-triage fine-tune. Swap the LLAMA_ARG_HF_REPO/LLAMA_ARG_HF_FILE env vars to point it at your own fine-tuned model.

Verify the Router's Routing Decision

Test the router's local/frontier decision live, right here — a separate, always-on, zero-GPU classifier (TF-IDF + logistic regression) that makes the routing call in milliseconds, distinct from the LLM you just deployed in Step 5. In a full build, this classifier decides whether to answer locally with a deployed model like that one, or forward to a frontier API.

# Works locally right now:
curl -X POST http://localhost:3000/api/classify \
  -H "Content-Type: application/json" \
  -d '{"query": "Summarize this paragraph in one sentence."}'

# Works the same way once you deploy this app to Render:
curl -X POST https://<your-render-service>.onrender.com/api/classify \
  -H "Content-Type: application/json" \
  -d '{"query": "Design a fault-tolerant, multi-region database strategy."}'

Common Troubleshooting

Ecosystem Resources

Toolkit GitHub Repository Render Deploy Blueprints Docs Base Weights Hub (HF) Serving Engine (llama.cpp) Stanford Study: 71.3% Local-Answerable