Features:
- GitHub Repo access
- Ability to post Issues
Repo Content = all you need to set up servers and APIs for inference:
- GPU Benchmarking Scripts incl. time-to-first-token, throughput, TPS and cost.
- Nvidia chips
- Google’s TPU chips (v6e)
- Allow agents to run code in a sandbox with “mcp-run-python”
- Build and Publish MCP Servers
- MCP config setup script [NEW June 2025]
- Create python or javascript/typescript servers from scratch
- Publish MCP services to NPM or PyPI package registries
- Edit an existing MCP service
- Using MCP (Model Context Protocol)
- Setting up MCP with Cursor, Windsurf, Claude Desktop, OpenAI Agents
- Using MCP with any LLM (via LiteLLM)
- Working with MCP Services for Github, Stripe and Web browsing (playwright, puppeteer)
- Memory Management (read-only and read-write memory for agents)
- Keyword (bm25) vs vector search
- Agents for Retrieval (RAG).
- Multi-LoRA Inference Servers [purchase individually here]
- Enterprise RAG with PostgreSQL [purchase individually here]
- Build an auto-scaling inference service
- Output Predictions / Speculative Decoding with OpenAI and vLLM.
- Test time compute, Part 1: Sampling and Chain of Thought.
- Test time compute, Part 2: Verifiers
- DSPy – programmatic prompt generation and optimization [purchase individually here].
- Fact Checking (incl. structured generation with Gemini, Claude and GPT-4o) [purchase individually here]
- Server Setup Guides: EC2 instances (e.g. AWS), Runpod or Vast.AI
- API Setup Guides:
- Platforms: RunPod, VastAI or your own laptop
- Inference Libraries: SGLang, Nvidia NIM, vLLM, TGI, llama.cpp
- Also, Serverless deployment with Runpod.
- API-Calling Scripts (OpenAI, Runpod and VastAI):
- Simple API calling scripts
- Context Caching with Gemini, Claude and vLLM/SGLang [purchase individually here]
- Function calling API calling scripts (handles function calling, execution and response) [purchase individually here]
- Speed tests (simple and concurrent requests)
- Data extraction scripts (JSON and YAML) [purchase individually here]
- ALL-SORT (LLM assisted retrieval) scripts [purchase individually here]
- RAG Citation Verification [purchase individually here]
- Code for Embeddings and BM25
- Prompt formatting to solicit citations
- Citation verification code
- Sensitive Data Anonymization Scripts [purchase individually here]:
- Named Entity Extraction with Presidio
- Sensitive Info Extraction with Phi-3 Mini and Outlines
- Monte Carlo Simulation for Improved LLM Accuracy [purchase individually here]
- Security [purchase individually here]:
- Jailbreaking attacks (responses that cause reputational damage)
- Off-topic use (wasting your server resources)
- Injection attacks (stealing data from your server, manipulating responses)
- Malicious code execution (running code that may overwhelm your server, or grant admin access)
Purchase Options
Inference Repo Access (Join 391+ members)
LIFETIME access, plus support via Github Issues, for one individual:
For TEAM access (i.e. repo used by multiple engineers), kindly post a comment below. It will be submitted privately and I’ll respond by email.
Trelis Multi-Repo Bundle
LIFETIME access to all SEVEN Trelis Repos (Voice, Vision, Fine-tuning (LLMs), Inference, Evals and Time-Series, Robotics) + support via Github Issues PLUS Trelis Discord Access for one individual.
Want lifetime access for a team? Purchase lifetime access here.
Testimonials from Youtube Comments
“Awesome content. Thank you! Purchased your gated repo and learning loads from it.”
“Thank you! It’s a great investment :)”
“I had Whisper&CoLab running a few months ago, but it broke. Your video and notebooks showed me why, and taught me several new tricks! Keep it up please.”
Repo Screenshots
Frequently Asked Questions
Q: Can I get a formal invoice with my tax number so I can reimburse this expense?
Yes, purchases come with an invoice and receipt of payment. You can include your tax/VAT number (not applicable for personal purchases or those using the LEARN discount). If you require modifications to the invoice, just respond to your email receipt.
Q: Is support included in my repo purchase?
Yes, if you purchase repo access, you will be able to post Issues in the corresponding Github repo.
Q: Can I buy individual scripts for one video?
Yes, if you only wish to purchase scripts for one video, you can post a comment and I will add a purchase link. Purchases of individual scripts do not provide GitHub repo access or the ability to post issues to get support.
Q: Is there a student discount available?
Yes. Use LEARN2025 at checkout if EITHER of the following applies:
a) You are currently a registered student at an academic institution.
b) You will use these materials only for non-commercial purposes.
After your purchase, reply to your receipt with a quick note indicating whether a or b applies. In the case of a, kindly provide a link to your LinkedIn page. Purchases using a company email/tax number/Github are not eligible for this discount. This discount applies to repos or the repo bundle, but not to individual scripts.
If I want to deploy codelamma on AWS, does the repo provide instructions and setup files for that scenario?
Yes, if you are running an EC2 server on AWS that has a GPU, this repo provides install instructions.
Separately, the repo explains how you can use runpod (which is cheaper per hour, unless you have lots of free AWS credits).
Hello Ronan,
Hope you are doing well.
Quick question. If I were to purchase all of your repos, would you be willing to offer a bundle/better price? I believe your work will expedite our research.
Thanks
Krishna
Howdy Krishna, no discounts but I adjust the price of the repos as I add more content – so there’s a benefit to those who are earlier buyers.
Cheers, Ronan
Hi Ronan,
Apologies I seem to have purchased the QLora_Fine_Tuning unsupervised notebook while I thought it was access to the github repo. I suppose I have now seen the content of the notebook but definitely isn’t what I was looking for. Any chance to redirect the order to have the github repo?
howdy, you can go ahead and buy the repo access and I’ll just refund the notebook, cheers, r
Hi Ronan
I have been watching your youtube videos, They are very good an informative.
I was wondering if is possible to split the Mixtral LLM over multiple GPUS, I have some old 6gb ethereum mining rigs that i wanted to use for AI projects, They can have up to 6 gpu, So i can run 36gb of Vram split over 6 cards
Many Thnaks
Andrew
Cheers Andrew,
I think you can run Mixtral locally across multiple GPUs using Text Generation Inference. A few steps:
– Install TGI (you can take a look at my youtube video on Enterprise Server Setup)
– Run TGI (you can maybe borrow the params I use in the Mixtral runpod template – https://github.com/TrelisResearch/one-click-llms). Use quantization (–quantize bitsandbytes-nf4) to fit the model on your 6×6 GB setup.
Now, this setup probably will work, but may not be optimal. Optimal would maybe be having one expert per GPU, but anyway you have 6 GPUs so that won’t be possible. Either way, I think this overall should work.
I want to buy this script, but i wonder whether i can get invoice, and how to do?
Yes, no problem. You get a receipt when you buy something, and if you need further info on the receipt just respond to that email. (I see you did respond and I’ll send over an updated receipt shortly).
Do your script also handle JSON structures output?
Like guidance or outline or something of that kind?
Yes, the inference repo does handle json outputs for function calling and use that information to make function calls and feed them back to the model.
Is that what you are asking?
If you look at the latest function calling video you’ll see me using the inference repo and how JSON outputs are handled.
No, I don’t use the guidance library. Guidance is a good idea in principle but just isn’t supported well for integrating with a lot of open source models. I just use the python json library.
I have purchased access to the repo but how do I access that repo? Where is it?
Howdy, we were in touch by email but I’m answering here for the benefit of others.
Access is automatically provided to your github username. If you don’t get an email, go to github and click on the github logo to the top right to see your activity.
Any trouble, just respond to the receipt from your purchase and I’ll help.
Hello Treils, thanks for this hard work.
My question is if I pay 99$ do I have access to all your repos or just to these Advanced Inference Scripts ? I would like to have access to this repo too: Advanced Fine-tuning Scripts.
One more question: I would like to manage data locally on my PC and send it to the Docker image created on Runpod: can I do that with your API scripts ? Is there any video of yours showing the API call in details ?
Thanks a lot for you answer.
Each repo is purchased separately. This page describes and provides a link for ADVANCED Inference Scripts. The ADVANCED Fine-tuning repo is separate.
If you are inferencing a runpod API, then yes you can make calls from your local PC (including sending your data). The video “Serve a Custom LLM for Over 100 Customers” goes through this.
Dear Ronan,
thanks a trillion for this reply of yours and, again, for all this effort.
Am I correct in buying the Advanced Inference Script to use API calls ?
One quick thing, I have seen three times your video: would you mind doing a short one about Runpod and not Vast.ai ? To use the API with MiXtral, am I correct in setting up a pod using the template called “Mixtral Instruct API by Trelis” ?
Again and once more, thanks a million for any help and assistance you may want to give me about this topic. Priceless.
Hello Treils,
i have finetuned mistral-instruct-7b-gptq model, now i want to speed up its inference time, i’m using runpod.
So, could you clarify which repository is preferable for my use case?
Thanks
Howdy! ok yeah so if you have a gptq model, then your best option is probably to just run that model with vllm and/or runpod. You can just go to the free https://github.com/TrelisResearch/one-click-llms/ repo, find a template for mixtral, and then edit the template for your model name AND change awq to gptq (and, for vllm, I think you may need to get rid of dtype half).
gptq is quite well supported but the quality is a lot worse than awq for the same speed. Ideally you would have fine-tuned in bf16 and then quantized with awq (see the youtube video).
Thanks for the replay Ronan.
i want script to finetune mistral-instruct-7b on my custom data with bf16, then need ti convert in AWQ, How can i do that.
Thank you
You can go back to Trelis.com and check out the ADVANCED fine-tuning page. You’ll see vids for each of those.
Hi Ronan,
i have fintuned model(mistral-instruct-7b) in bf16 and then quantized it in AWQ, now i want to setup or make template for this model using vLLm on runpod, so i can use this model as a api.
How can we setup, any reference for the same?
Yup, try out https://github.com/TrelisResearch/one-click-llms/ and find a vLLM template. Then modify for your use. BTW, TGI is faster than vLLM.
I’m trying to download youtube videos for transcription and I watched your video using Whisper and I’ve used it no problem. But now there is the Distil -whisper model. Do you have a solution for Distil and what would it cost? I’m really just trying to use an end solution, not really understand how it works. I’m not that smart.
Ross
I haven’t tried, but probably you can sub in “model_id = “distil-whisper/distil-small.en”” to the code
Hey Ronan,
I was wondering, are there any video courses or just github access?
I was really interested to learn the concepts like finetuning, you teach on this channel.
Thanks,
Prateek
Howdy! yes all the video courses/tutorials are FREE on youtube! There are almost 100 videos there, probably you saw them, and if you want to follow along more closely, that’s where buying the repos can be useful.
How do we know which repos will be tied to which youtube videos? I purchased a lifetime on your ADVANCED-fine-tuning repo as it was the only one mentioned in your videos at that time thinking it would be the sole source of your youtube code then this repo came out. I am thinking about purchasing this repo and just noticed you have 2 other repos for sale too. If i purchase the full bundle to save, then what is the likely hood a 5th repo will come out?
I really wish you just had one repo for Transformer-based LLMs/GPTs even if it was more expensive. As a customer it becomes very difficult and a little discouraging to keep seeing the code in different repos from the ones I purchased previously. Especially as there can be some overlap between the 4 repos when dealing with these transformer based projects.
Howdy BJ, let me try to answer each question in turn – I appreciate the comments.
Q: How do we know which repos will be tied to which YouTube videos?
A: This is the ADVANCED-inference product page and videos tied to it are shown at the bottom part of the page. To know what is included in advanced-vision, go to trelis.com/advanced-vision and scroll down to see the included videos. The same applies for each repo (advanced-fine-tuning, advanced-inference, advanced-vision, advanced-transcription (audio)). You can find the list of repos on the home page Trelis.com .
Q: If i purchase the full bundle to save, then what is the likely hood a 5th repo will come out?
A: If you purchase the full bundle right now, you get lifetime access to content I upload to ADVANCED- fine-tuning, inference, audio and transcription. I’m careful not to make any commitments regarding future products because it risks over-promising.
Q: Do you have one repo for Transformer-based LLMs/GPTs?
A: All repos essentially involve transformers so this is too broad to just have one repo. So far, I have split things into language fine-tuning, language inference, vision, and audio/transcription. However, you can buy a bundle of all of those repos. The bundle is linked on the page for each individual repo AND linked on the home page. If you have already bought an individual repo, and you then buy the bundle, email me back in response to the receipt and I’ll refund previous individual repo purchases you made.
Putting everything in one repo makes it expensive for people who only want to buy part of the content. Probably it’s around 50-50 split between people who buy the bundle vs people who buy individual repos.
You make a fair point around there being some overlap between repos. I try to keep the separation clean (to keep the option for customers to only buy parts), but ultimately there are trade-offs in deciding what to include where. My guiding principles are to:
– Segment products into chunks that people want, allowing customers buy what they need.
– Offer a bundle for those who want the best value.
– Increase price steadily over time as content is added and each product gets more valuable (this benefits early buyers, since access is lifetime).
Hello, love your work. If I purchase a repo, can I use it for commercial purposes? Couldn’t see anything with a license mentioned.
Howdy, it’s a bit obscure but you can see it in the Terms referenced on payment links.
The rough answer is that you cannot re-publish the content, but you can use it to build or provide a commercial service. See here for full details: https://trelis.com/terms-of-service/
Hi Ronan,
Thanks for the reply.
To clarify: I’d like to reference parts of the code (e.g. functions, structure) while building my own implementation for a commercial product. This might involve copying and adapting individual functions or structural ideas, but the final codebase would be my own and not a republish of yours.
Is this permitted under the licence? And can the licence be modified to allow integration into a broader product?
Thanks,
Seán
Hi Seán,
Our standard licence (see trelis.com/terms-of-service) already lets you run the code (including if modified, or if your code with Trelis code incorporated) on your own servers,
including as part of a SaaS product, provided:
• The Trelis source/binaries never leave your servers
(no redistribution in customer downloads or SDKs).
• You hold enough licences for every engineer who touches the repo.
If that matches your plan, you’re covered.
If you ever need to ship Trelis code inside a product (desktop plug-in,
mobile app, on-prem package), you can reply to your receipt to email me and we can sort out something for redistribution.
Cheers, Ronan
Not relevant to ADVANCED-inference, but the Trelis Mart repo has an extra restriction, namely a hosted service would also require a separate agreement.
Hi Ronan,
Thanks for getting back to me! I want to clarify if the annual subscription to the Advanced Inference Repo includes the same access as the lifetime option, just for one year? The payment options aren’t entirely clear on that when I click into them both.
Seán
Correct! Let me know if there’s a way I can make things more clear. You should see annual subscription on the annual link, and just a one-off payment with the word lifetime on the lifetime one.
Comments are closed.