Features:
- GitHub Repo access.
- Ability to post Issues.
Suitable for:
- Fine-tuning avatar video models (with lip sync and head motion)
- Fine-tuning diffusion models for image generation, FLUX Schnell or FLUX dev.
- Fine-tuning vision models for custom image OR text datasets:
- Qwen VL 2.5
- SmolVLM
- Florence 2 (including bounding box fine-tuning)
- Pixtral [buy individual scripts here]
- Older models: IDEFICS, LLaVA, Moondream (early v2)
Purchase Options
Vision Repo Access (Join 303+ members)
LIFETIME access, plus support via Github Issues, for one individual:
For TEAM access (i.e. repo used by multiple engineers), kindly post a comment below. It will be submitted privately and I’ll respond by email.
Trelis Multi-Repo Bundle
LIFETIME access to all SEVEN Trelis Repos (Voice, Vision, Fine-tuning (LLMs), Inference, Evals and Time-Series, Robotics) + support via Github Issues PLUS Trelis Discord Access for one individual.
Want lifetime access for a team? Purchase lifetime access here.
Testimonials from Youtube Comments
“Awesome content. Thank you! Purchased your gated repo and learning loads from it.”
“Thank you! It’s a great investment :)”
“I had Whisper&CoLab running a few months ago, but it broke. Your video and notebooks showed me why, and taught me several new tricks! Keep it up please.”
Repo Screenshots
Frequently Asked Questions
Q: Can I get a formal invoice with my tax number so I can reimburse this expense?
Yes, purchases come with an invoice and receipt of payment. You can include your tax/VAT number (not applicable for personal purchases or those using the LEARN discount). If you require modifications to the invoice, just respond to your email receipt.
Q: Is support included in my repo purchase?
Yes, if you purchase repo access, you will be able to post Issues in the corresponding Github repo.
Q: Can I buy individual scripts for one video?
Yes, if you only wish to purchase scripts for one video, you can post a comment and I will add a purchase link. Purchases of individual scripts do not provide GitHub repo access or the ability to post issues to get support.
Q: Is there a student discount available?
Yes. Use LEARN2025 at checkout if EITHER of the following applies:
a) You are currently a registered student at an academic institution.
b) You will use these materials only for non-commercial purposes.
After your purchase, reply to your receipt with a quick note indicating whether a or b applies. In the case of a, kindly provide a link to your LinkedIn page. Purchases using a company email/tax number/Github are not eligible for this discount. This discount applies to repos or the repo bundle, but not to individual scripts.
I was hoping this was included in the Advanced Fine-tuning Scripts repo. Is there a discount for people that bought the lifetime access to the Advanced Fine-tuning Scripts repo?
Howdy Dustin, there are separate repos to allow customers to buy access according to what they need. I’m leaning away from creating discounts in favour of keeping automation and pricing simple.
Hello! This is a great source of information. I am still a student and I would like to access this repository for acquiring knowledge. Is there any discount available for students like me?
Students can use the code STUDENT2024 for a discount. After checking out, please respond to the receipt with a link to your linkedin page so I can verify. Thanks.
Does the student discount code still work? Where do I input the code? The checkout page doesn’t seem to have an option for that. thanks
Thanks Tony, yeah I just enabled the field for promo codes when purchasing the vision repo.
Hi, before buying I just want to ask if this is the right repo I need for the full process of fine-tuning Llava on custom dataset? I saw many links in your tutorial and got confused!
Correct!
Hi, can I intend to implement this on the GCP Vertex AI VM. Can I use the repo based on the following configuration:
4 x A100 (40GB) GPUs
CUDA version 11.8
Howdy! Yes, for the 7B Llava 1.6 model that looks like it will work. Here are the GPU specs required:
### GPU requirements for training
LLaVA 1.5
– liuhaotian/llava-v1.5-7b will run on a single A6000
– liuhaotian/llava-v1.5-13b REQUIRES VRAM OF 80 GB+. Will run on 2x A6000
LLaVA 1.6
– liuhaotian/llava-v1.6-mistral-7b takes about 100 GB of VRAM in bf16. 3x A6000 or 2x A100 (80GB).
– liuhaotian/llava-v1.6-vicuna-7b takes about 100 GB of VRAM in bf16. 3x A6000 or 2x A100 (80GB).
– liuhaotian/llava-v1.6-34b takes about 300 GB of VRAM in bf16. 7x A6000 or 4x A100 (80GB).
Can you briefly summarise the costs associated with fine tuning, thank you.
Two main costs:
– Running the GPU for fine-tuning (free if you use colab, but can be more practical/faster to pay $0.7/hr for a runpod or vastai gpu)
– Generating synthetic data (if you need it), for which you would need openai credits, unless you want to manually make the data with free chatgpt.
Thanks for the breakdown. How long do I need to run, runpod for to get some decent results for example with 200 training images.
I did fine-tuning on some 40 or so images and I recall each run was taking me around 10 mins for the larger 34B Llava 1.6 model.
200 training images is not a lot.
Hi Ronan,
You’re doing great work. I’m interested in buying access but wanted to know if you have a roadmap for adding new models?
E.g. the new version of idefics, moondream2, qwenVL, etc.
The space is developing very quickly and Id like to work with something that’s using SigLIP and a newer and smaller LLM
Howdy, yeah working on both moondream and idefics right now.
QwenVL I’m unlikely to cover as it’s worse than LlaVA 1.6 and badly supported by inference libraries. IDEFICs will soon be TGI supported.
Hello Ronan,
I really like your great job at fine-tuning LLaVA. I was wondering… Would it be possible to purchase your repo, follow your whole work pipeline, and later deploy locally one of my fine-tuned LLaVA models with a 24GB NVIDIA GeForce RTX 3090 Ti? If feasible, do you think it would be slow at inference?
Also, any advice regarding quantization?
Thanks in advance!
Yes, that is all possible. See my video posted today (22nd April) for a fast inference option with MoonDream. Quantization isn’t recommended for production grade because it hurts quality and can also slow inference.
Hello
I need to train the model to recognize Windows UI, App UI, Web/Browsers UI.
I want to build an AI agent to auto use my pc like a human.
Is this suitable for my usecase?
This repo can help you with the recognition of the UI, yes. It does not cover building an agent, you’ll want to build that separately.
Also how much that would cost actually to host a custom finetuned model? Is there any free solution?
check out the latest video that covers server setup – it’s on this page now. you could host for about $0.35/hour on runpod. I don’t think you’ll find a free option.
Hi Ronan
I want to purchase this now but just want to confirm if possible first.
Can I train this vision model to detect a button on a image and respond with the coordinates? I have a big amount of images but want to see what approach I could take with this.
Kind regards
Jonathan
Yes, that is the kind of problem that should be fine-tunable. Whether the smallest moondream model is strong enough is a bit uncertain, but using a larger model in the model it should be (although I recommend starting small).
Hi Ronan,
before buying the repo, i want to mention about my project..
i want to use vlm model to extract fashion product attributes, for example collar type, sleeve type etc…. Will your solutions help me about it ?
and i want to use my vlm on locallay… i will have resources for it.
i wonder your comments and advices. thank you already.
ps: this is great job what u did, thank you.
yes, that sounds like a good application for fine-tuning, if you have some data to train on.
Hi Ronan
So for my use case is a bit complicated, I am trying to train model with SLDPRT/SLDASM(solidworks files) or train model with images and convert picture image to a 3d object using SLDPRT/ASM files. Is it possible to do it using LLAVA or Multi Modal models?
Atish
In principle yes, although I would say this is a challenging task and you will need to play around with the datasets, and probably have quite a lot of good data to get good results.
Hi Ronan,
You are a complete legend. I was wondering if you have any thoughts on if/how these vision models can be fine-tuned to accept multiple images as input (eg with a text question asking to compare/contrast)?
Thanks,
Dane
cheers Dane, hopefully will get to this next week – there won’t be a vid out next week but maybe the following.
ohh interesting case, would be nice.
Any idea of fine-tuning Llava using Apple Silicon, like a M2 with 64GB RAM?
Llava 7B model could be fine tuned on 64GB RAM?
I want to classify Wikicommons images in about 30 category classes, have a NSFW label/probability and some description fields.
I’m thinking about fine-tuning Wikicommons images and adding better meta data fields, to improve image search on Wikicommons.org. It’s a lot of images, but resolution would be low-is so bandwidth should be manageable.
It’s a hobby project to make an open-source Wikicommons image dataset on HF.
I see this tutorial on inference on mac (not fine-tuning) – https://www.youtube.com/watch?v=2Tv5ZfPabGM – but perhaps you can extend or dig in more to the docs.
Easier really to just use colab if you can, or runpod.
Hi. I am interested in buying. Just wanted to check if any of the models are good enough for financial documents key information extraction such as name, date, address, amount, payee names etc. I need the VLM / solution to run at close to 1 second or lower latency. GPU i can use is max a T4 for inference.
Howdy.
Yes, you could fine-tune on financial docs. To run with fast latency, you’ll want to use a model like Moondream, which will fit on a T4.
Hi, I have watched your fine-tuning LLAVA video. I am currently working on a project where I would like to fine-tune the latest LLaVA One Vision model with traffic scenes data for autonomous driving scenarios. I am a university student and would like some guidance from you. I am not sure if this repo would be useful for me. If you don’t mind, is it possible to connect and get some guidance from you. I am willing to compensate as well. Thanks so much!
Howdy, yes you could try this with traffic scenes. Best to look at the latest video about Pixtral on Trelis YouTube, as it is more recent.
For questions/support see the options on Trelis.com/About
Hi Ronan,
i have a project about grouping people in an event..
people have group or individual photos,
so every person wants to see just own photos when they go to album,
like iphone photos gallery,
i wonder your advices, recommendations or any new developments
(detect faces and cluster them etc.)
thank you.
I was hoping to get access to the script from the following video: https://www.youtube.com/watch?v=1N76mJ1pMro&ab_channel=TrelisResearch
Also, I am a student, is there any discount available?
Howdy, I’ll try to get to this and upload a script soon. Regarding student discount, yes there is but not on individual scripts – only on repos – see above.
Hi Andrew, I’ve now put up a link to buy the pixtral scripts.
how can i use this for detection of myself on any camera? Where do i begin? The camera should be programmed to follow me in the room where i work to.. any suggestions?
yes, you can, and you need to use a small model so that it inferences fast (like the small qwen or smolvlm models) and write a script to pass the video to the model as an image every few seconds. Potentially pass multiple images together as a history.
Comments are closed.