Lessons Learned when Fine-Tuning LLMs Locally
This note identifies the lessons from fine-tuning a LLM against a small technical specification. Overall, the main takeway is that fine-tuning does not overwrite anything: it simply shifts the probability distributions, however base-model priors (prior base model knowledge), dataset gaps, and inference configuration still massively influence the final output.
Context
The main aim of the experiment was to understand how an LLM can answer questions and generate code syntax on a specific technical domain using parameter-efficient fine-tuning (PEFT). The experiment built the pipeline on-prem rather than a cloud deployed service so that hardware constraints, dependency issues, adapter configuration, and evaluation failure modes could be observed directly (As well as costs...).
The training setup focused on a QLoRA-style adaptation. This essentially means quantising the base model to reduce memory constraints on my PC while training a small set of low-rank adapter weights. This made the experiment possible on constrained GPU hardware, but the tradeoff on dataset quality and training configure were highly focused on.
Key Lessons
1. Fine-tuning dictates behaviour; it does not replace the base model knowledge
The fine-tuned model was more likely to use domain-specific terminology and answer in the expected format. BUT, prior base knowledge was retained. This was massively obvious as when the prompt moved outside the coverage of the training dataset, the model fell back to generic answers.
2. Dataset quality dominated model quality
The strongest improvements came from removing ambiguous examples, making answer formats consistent, and increasing coverage around edge cases. More examples did help but only when they reduced uncertainty. Repetitive or weak examples just taught the model style in how to respond as opposed to correctness.
3. Adapter size is a trade-off, not some random leaderboard metric
Larger adapters did give the model far more capacity to take on domain behaviour, however it increased the risk of overfitting a small dataset. Smaller adapters were far more easier to train and even cheaper to store, but could still underfit complex domain rules. The practical target was not the biggest adapter; it was the smallest adapter that improved evaluation performance without making generalisation worse.
| Configuration Choice | Observed Effect | Risk |
|---|---|---|
| Small LoRA rank | Fast, cheap, easier to iterate | May not capture enough domain behaviour |
| Larger LoRA rank | Higher adaptation capacity | Can overfit narrow or repetitive datasets |
| More training epochs | Improves memorisation of target format | May reduce robustness on unseen prompts |
| Lower temperature at inference | More stable outputs | Can still produce confidently wrong answers |
4. CUDA compatibility is vital, not just a setup pain
Local fine-tuning is masssively sensitive to CUDA, PyTorch, bitsandbytes, GPU driver versions, and operating-system details. A large part of the engineering work was not model design; but dependency alignment and reproducibility.
# Example environment checks
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
nvidia-smi
python -c "import bitsandbytes as bnb; print(bnb.__version__)"Evaluation Notes
Evaluation should include both successful examples and failure examples. A model that answers common domain questions well can still fail on slight prompt variations, missing concepts, or callable constructs that were not represented in training.
- Positive tests: questions directly represented in the training data.
- Paraphrase tests: semantically equivalent prompts written differently.
- Boundary tests: prompts close to, but outside, the domain specification.
- Negative tests: prompts where the correct answer is to reject, clarify, or state uncertainty.
Final Summary
Fine-tuning is valuable for adapting tone, terminology, and response structure. It is far less reliable as a standalone method for enforcing correctness. For serious production AI systems, it is best to treat fine-tuning as one layer in a wider architecture: retrieval for facts, guardrails for policy constraints, evaluation harnesses for regression testing, and observability for monitoring real-world behaviour.