Quick start

1. Install

Create a virtual environment, then install the SDK with its training extras, Langtune and the datasets package the training script uses.

1python -m venv .venv
2source .venv/bin/activate
3pip install "langtrain-ai[train]" langtune datasets

A good install can see your GPU. This should print True:

1python -c "import torch; print(torch.cuda.is_available())"

2. Log in to Hugging Face

This guide trains Llama 3.1 8B, a gated model. Open its page on Hugging Face, accept the licence, create a read token in your account settings, then log in so the download works.

1huggingface-cli login

Using an open model such as Qwen/Qwen2.5-7B-Instruct instead? You can skip the licence step.

3. Prepare your data

Create a .jsonl file. Each line is one conversation: a messages array in the same format as the OpenAI chat API.

1{"messages": [{"role": "user", "content": "What is Langtrain?"}, {"role": "assistant", "content": "A platform for fine-tuning open models."}]}
2{"messages": [{"role": "user", "content": "How do I start?"}, {"role": "assistant", "content": "Write a JSONL file like this one, then train."}]}

For a first run, 50 to 100 examples are enough to change style and tone; a specific task needs 500 to 1,000. Put about one in ten aside in a second file, test_data.jsonl, and never train on it: step 5 uses it.

4. Train

Load the model in 4-bit, add a QLoRA adapter and train. Only the adapter learns; the base model stays frozen, so an 8B model fits on a single 24 GB GPU.

1from datasets import load_dataset
2from langtrain import FastLanguageModel
3
4model, tokenizer = FastLanguageModel.from_pretrained(
5    "meta-llama/Llama-3.1-8B-Instruct",
6    load_in_4bit=True,
7)
8model = FastLanguageModel.get_peft_model(model, r=16, method="qlora")
9
10dataset = load_dataset("json", data_files="training_data.jsonl", split="train")
11FastLanguageModel.train(model, tokenizer, dataset, output_dir="./output")

While it trains, the loss is printed every few steps. It should fall over the first few hundred steps; if it stays flat or turns into nan, stop and check your data.

5. Test it

Ask it a question from test_data.jsonl, one it has never seen. A right answer to a training question proves nothing; a right answer here shows it learned.

1import json
2
3held_out = [json.loads(line) for line in open("test_data.jsonl")]
4question = held_out[0]["messages"][0]["content"]
5
6FastLanguageModel.for_inference(model, tokenizer)
7print(FastLanguageModel.generate(model, tokenizer, question))
8print("Expected:", held_out[0]["messages"][1]["content"])

6. Save and run

Merge the adapter into the base model. The result is a standard Hugging Face model folder.

1FastLanguageModel.save_pretrained(model, tokenizer, "./my-model", merge_before_save=True)

llama.cpp and Ollama run GGUF files, so convert the folder first. From a llama.cpp checkout:

1python convert_hf_to_gguf.py ./my-model --outfile my-model.gguf
2llama-cli -m my-model.gguf -p "How do I start?"

For Ollama, point a Modelfile at the GGUF file:

1echo "FROM ./my-model.gguf" > Modelfile
2ollama create my-model -f Modelfile
3ollama run my-model

vLLM serves the folder directly, behind an OpenAI-compatible API:

1pip install vllm
2vllm serve ./my-model

If something goes wrong

CUDA out of memory

The model does not fit on your GPU. Keep load_in_4bit=True and start from a smaller model, such as Qwen/Qwen2.5-1.5B-Instruct.

401, or "Access to model … is restricted"

The model is gated. Accept its licence on the Hugging Face model page, then run huggingface-cli login again with a token from the same account.

torch.cuda.is_available() is False

PyTorch cannot see the GPU. Check that nvidia-smi lists it, then install the PyTorch build for your CUDA version inside the virtual environment.