Fine-tune vision models
Langvision is an open-source (MIT) Python library for fine-tuning vision-language models on your own images and text.
- Runs locally
- Images + text
- MIT
Install
Install Langvision on a machine with an NVIDIA GPU. It is also included in langtrain-ai[vision].
python
1pip install langvision
Supported models
LLaVA 1.5, Qwen-VL, InternVL, CogVLM and Phi-3 Vision, loaded from Hugging Face.
Prepare your data
One JSON line per image: the image path and the conversation about it.
python
1{"image": "photos/0001.jpg", "conversations": [{"from": "human", "value": "What's wrong with this part?"}, {"from": "assistant", "value": "A hairline crack near the left mount."}]}
Train
The vision encoder stays frozen by default and the adapter is trained on the language side. Methods include lora, qlora (the default), dora, and preference methods such as dpo.
python
1from langvision import FastVisionModel23model, processor = FastVisionModel.from_pretrained("llava-hf/llava-1.5-7b-hf", load_in_4bit=True)4model = FastVisionModel.get_peft_model(model, r=16, method="qlora")5FastVisionModel.train(model, processor, dataset, output_dir="./output")
Ask it about an image
Check the trained model on images it hasn't seen.
python
1from PIL import Image23answer = FastVisionModel.generate(4 model, processor,5 images=Image.open("invoice.jpg"),6 prompt="What is the total amount due?",7)
Export
Langvision can save the trained model as ONNX or safetensors to run it elsewhere.