Important: As of July 2024, the strongest open source models support function calling and tool use out of the box. Check out this video tutorial.
Products:
- Dataset: Function calling dataset v3 (Purchase here).
- Pre-trained models (v3, latest):
- Phi 3 (full collection on HuggingFace).
- Llama 3 8B and 70B (full collection on HuggingFace).
- Trelis Tiny 1.3B tuned for function calling. Visit here.
- Llama 2 7B tuned for function calling. Free here.
- OpenChat 3.5 7B tuned for function calling. Visit here.
- Mistral 7B tuned for function calling. Visit here.
- Yi 34B 200k context tuned for function calling. Visit here.
- DeepSeek 67B tuned for function calling. Visit here.
- CodeLlama 70B tuned for function calling. Visit here.
- Browse the full collection on HuggingFace.
- Don’t see a model? Request one by commenting on this post.
- Script/Notebook to train for structured responses:
OR Buy access to the ADVANCED-fine-tuning repo, which includes the fine-tuning scripts for function calling.
- Function-calling inference scripts:
- Pre-trained models (v2) – DEPRECATED:
- Yi 200k context 34B (purchase here) and 6B (purchase here) models.
- Llama 2 70B tuned for function calling. Purchase here.
- Mistral 7B tuned for function calling. Visit here.
- Deepseek Coder 1.3B, 6.7B and 33B fine-tuned for function calling. Visit here.
- CodeLlama 34B tuned for function calling. Purchase here.
- Llama 2 13B tuned for function calling. Purchase here.
- v2 dataset – DEPRECATED. Purchase here.
Hello Ronan McGovern,
Hope you are doing well!
I have gone through your youtube video for fllama2 (function calling) and looks awesome. I would like to go through more about the fllama2 model. If you don’t mind can you provide free trail access for fllama2 – 13 model with limited requests so that we can test it and opt for purchasing the model.
Thanks,
Santhosh
Howdy Santhosh, thanks for the comment.
You can try out Llama 2 with function calling for free on the 7B model [here](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2). I’ve just pushed some improvements today. Check out also the performance video that is linked in the repo – it gives a sense for 7B versus 13B. Cheers, Ronan
Hello,
Could you do a function calling version of ‘Yi-34B-200k’? I’d like to pay for it.
Best,
Zack
Howdy! I’ve put in a request for a license as this model is only licensed for non-commercial use without permission.
btw, I’m getting very poor results on the model. I’ll see what the authors say, but seems a supervised fine-tuned is required at the very least.
Cool~ Looking forward to what you can get out of this model. Since this new pretrained model ranks top1 on Huggingface, I think it will be highly recommended. Considering its 34B size, if you can get this one with function calling works good, it could be a great sell.
Yi 6B and 34B are live now for function calling with 200k context length. See here: https://huggingface.co/Trelis/Yi-34B-200K-Llamafied-function-calling-v2
Great!
1. What’s the differences between Trelis/Yi-34B-200K-Llamafied-function-calling-adapters-v2 and Trelis/Yi-34B-200K-Llamafied-function-calling-v2?
2. If I want to learn how to finetune the function calling, Dataset and Script/Notebook to train for structured responses are what I need to buy, aren’t they?
3. You said “btw, I’m getting very poor results on the model. I’ll see what the authors say, but seems a supervised fine-tuned is required at the very least.” What happend? Is the result good neough now?
Thanks for your work!
1. You’ll get access to both. The adapter model allows you to load the base model and then apply the adapter. It can be useful if you wish to apply multiple adapters. It’s a specific advanced use case.
2. Yes, the Dataset and the notebook to train.
3. Yes, the prompt format wasn’t clear and that was affecting the model. I’ve put the correct prompt format now in the model card.
Hey I love your work,
could you consider creating a function-calling version of: https://huggingface.co/jphme/em_german_leo_mistral
We would love paying for that.
Cheers Simon
Howdy, I’ve just posted a question about the license: https://huggingface.co/jphme/em_german_leo_mistral/discussions/2 so I can understand whether it’s allowed to sell fine-tunes or not.
As a backup option, you could fine-tune yourself using the function-calling script and dataset. For a 7B model, you could do it in Colab with QLoRA on a T4 if you turn off flash attention.
But let’s see what the authors say first.
Okeydoke, the model is ready here Simon: https://huggingface.co/Trelis/em_german_leo_mistral-function-calling-v2
Clicking on dataset shows that the link is no longer active.
Thanks! Fixed now
I purchased license for the dataset – “Trelis/function_calling_v3”.
I am getting the following error –
“Trelis/function_calling_v3 is awaiting a review from the repo authors.”
How long does it take to authorize a new user?
Howdy, access is immediate.
I saw you entered a name rather than HuggingFace username, so the access failed. No problem, I just manually gave you access, cheers, Ronan
I am interested to offer function calling in a SaaS application. Can you explain how the “per user” license is to be understood in this context? Is every end user of the SaaS application an user? Thanks a lot
Howdy!
The simple way to think about “user” is who has access to the weights.
– If you (the user) is building a SaaS application containing a model where you are hosting the model from your server (or a server you rent), then your customers don’t need a license.
– If you are in a company where there are multiple devs working on a project using the weights, they would each buy access.
– If your customers are hosting the model (as opposed to just inferencing or using SaaS that makes uses of your hosted endpoint or service), then they would need to buy access.
Hi Ronan, have you tried/considered of doing a function-calling version of TinyLlama?
Thanks!
Best,
Giuseppe
Howdy Giuseppe, I love the question:
– Tiny Llama is very weak in chat format. It has trouble finishing sentences. Phi 2 is a better model (but not permitted for commercial use, although I have petitioned Microsoft about that).
– I have tried fine-tuning tiny models and I find that they can either a) function call or b) chat. I’ve tried a lot of things and when they can function call they can no longer answer basic queries (like how many planets in our solar system).
Would it even be useful if the model can ONLY function call? What do you think?
Thank you for sharing your insights!
I think it could be interesting to also have an ONLY function calling model, especially considering local llm for small devices.
Perhaps an approach to test could be to always have (and maybe finetune) a “conversational_response” function with an “answer” parameter. Do you think it could work?
Best,
Giuseppe
That’s a really interesting idea. I’ll try to play around with that.
Does selling LLMs for a fixed fee work as a business model? I might try it myself.
Yup, works.
I’ll likely put out a new channel soon to help with doing so.
For now, you can check:
– https://trelis.com/monetize-your-ai-models-and-datasets/
– https://trelis.com/trelis-mart-server-lifetime-repo-access/
Which of the fine tuned models for function calling would be the best to run on my local machine? I can run dolphin-2.6-mistral-7b.Q3_K_M.gguf decently on my Macbook, I was hoping that your mistral 7b fine tuned for functions would work as well.
You could try Mistral, but OpenChat 3.5 is also 7B (with Mistral as a base) and much stronger.
Ah Ok, I didn’t realize that OpenChat 3.5 has Mistral as the base. Thank you very much, I am purchasing now. :)
Do these work with autogen? Have you tested some models with autogen? what would be the best model for autogen?
x
I still need to dig in on autogen, but the v3 models will work with any framework that allows a chat_template to be specified (which is most frameworks/libraries).
How long does it take to get access to notebooks once payment is done?
Access is granted immediately after payment.
The most common reason for delays is in the case of repo purchases, where a wrong GitHub username is entered.
While training openchat function calling I’m getting this error
TypeError: MistralForCausalLM.forward() got an unexpected keyword argument ‘loss_mask’
Can you post on the GitHub repo if you have purchased access. If you only bought the scripts, then please share here:
– what specific script you are using for training
– what you changed versus the base script
– where the error occurs, specifically
I’m using fine_tuning_function_calling_v3 notebook.
I have changed the following:
FROM:
base_model = “meta-llama/Llama-2-7b-chat-hf”
config = LoraConfig(
r=8, #typically use 8 for models 7B or larger and use 128 for smaller models
lora_alpha=32, #typically use 8 for models 7B or larger and use 128 for smaller models
# r=128,
# lora_alpha=512,
target_modules=[
“q_proj”,
“k_proj”,
“v_proj”,
“o_proj”,
“gate_proj”,
“up_proj”,
“down_proj”,
“lora_magnitude_vector”, #required for DoRA
],
lora_dropout=0.1,
bias=”none”,
task_type=”CAUSAL_LM”,
# use_dora=True
)
B_INST, E_INST = “[INST] “, ” [/INST]” #Llama 2 or Mistral style
TO:
base_model = “openchat/openchat_3.5”
config = LoraConfig(
r=8,
lora_alpha = 32,
target_modules = [
“self_attn.q_proj”,
“self_attn.k_proj”,
“self_attn.v_proj”,
“self_attn.o_proj”,
“mlp.gate_proj”,
“mlp.up_proj”,
“mlp.down_proj”,
],
lora_dropout = 0.1,
bias = “none”,
task_type = “CASUAL_LM”
)
B_INST, E_INST = “GPT4 Correct User: “, “GPT4 Correct Assistant:”
And I’m getting the following error
TypeError: MistralForCausalLM.forward() got an unexpected keyword argument ‘loss_mask’
Seems you have a typo here: task_type = “CASUAL_LM”
Should be CAUSAL_LM not CASUAL
Thanks!!!
Could you do a function call fine-tuning of the Aya 7B or 13B models? That could be quite useful for people who need multilingual function calling. Not sure how well this might work but right now GPT-3.5 seems to be the only model for this use case.
Could you share a link to the specific model? (choose the one that’s most important for you)
And in what language do you plan to use the model in?
Model: https://huggingface.co/CohereForAI/aya-101
Paper: https://cohere.com/research/aya/aya-model-paper.pdf
Planning to use it for Central and Eastern European languages, especially Hungarian and Romanian.
hmm, looks like the base model is mt5 https://arxiv.org/abs/2010.11934, which is quite old and I worry is weak…
Yeah, not sure how good real world performance is. But I don’t know of any other open source model supporting these languages either… maybe doing some language specific fine-tuning on Mistral 7B or Phi or similar models could work? And then train it for function calling?
I am using your model to send email using the function calling feature, and I want it to use some info from my database like sender name, email, and phone number when crafting the email. It performs well when it comes to following the user’s prompt and creating the email body, but it ignores the sender information that I am providing as a system prompt. I tried to use “user”, and “system”, but it still doesn’t work. What should I do.
What specific model are you using? Can you comment please in the issues/community section of that model on huggingface?
Regarding the system message, it is probably best to include that in the user’s first message instead. any further Qs – just post on the relevant HF repo
if i buy full repo bundle ,
can i have access to new updates to the same repos
or need to buy again
That’s correct! If you buy access to any repo (or all four repos, via the bundle), you get life-time access including future updates to that repo.
Hi Ronan, Great work on the repo and resources. I am a little confused, could you please help me with getting the access link to all repos (bundle link). I am getting a little overwhelmed by the different options at this website and on the Huggingface.
Thanks in advance, keep up the good work!
!
Howdy, you can find the multi-repo bundle on this page: https://trelis.com/advanced-fine-tuning-scripts/ – just scroll down.
This page here is for function calling datasets, note that the datasets are sold separately (from the corresponding huggingface page OR from mart.trelis.com). That said (and I know this is a bit confusing), the scripts for function-calling fine-tuning are included in the ADVANCED-fine-tuning repo – which is the link I just gave there (and if you buy the multi-repo bundle, that includes the ADVANCED-fine-tuning repo).
Comments are closed.