Gorilla Files

Highlights

🏆 Stable release of Berkeley Function Calling Leaderboard V3 with Multi-step and Multi-turn function call evaluation

What's Changed

• Gorilla README and repo structure revamp by @CharlieJCJ in https://github.com/ShishirPatil/gorilla/pull/799
• [BFCL] Fix live_parallel_multiple_9-8-0 copy-paste issue by @pkesseli in https://github.com/ShishirPatil/gorilla/pull/865
• [BFCL] Fix Typo in multi_turn_base_34 Ground Truth by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/876
• Adding New Model Haha-7B by @ZydHaha in https://github.com/ShishirPatil/gorilla/pull/858
• [BFCL Chore] Implement retry_with_backoff for Amazon Nova Handler by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/880
• [BFCL] Fix live_simple_183-108-0 by @pkesseli in https://github.com/ShishirPatil/gorilla/pull/872
• [BFCL] Fix live_simple_165-98-0 by @pkesseli in https://github.com/ShishirPatil/gorilla/pull/871
• [BFCL] Fix live_simple_44-18-0 and live_simple_45-18-1 by @pkesseli in https://github.com/ShishirPatil/gorilla/pull/870
• [BFCL] Fix Nova Handler for Consecutive User Prompt Issue by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/881
• Add support for QwQ and Sky-T1-32B-Preview by @SumanthRH in https://github.com/ShishirPatil/gorilla/pull/888
• add handler for Bielik by @dominikabasaj in https://github.com/ShishirPatil/gorilla/pull/887
• [BFCL Chore] Align Score File id with Result File Test Case IDs by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/893
• Fix minor typo in default system prompt without func by @canyon289 in https://github.com/ShishirPatil/gorilla/pull/895
• Falcon3 support by @kirill-fedyanin in https://github.com/ShishirPatil/gorilla/pull/894
• [BFCL] Update tool construction for Palmyra models by @samjulien in https://github.com/ShishirPatil/gorilla/pull/897
• Added compute_exchange_rate to multi_turn_base entry 180 ground truth by @Raymond112514 in https://github.com/ShishirPatil/gorilla/pull/892
• [BFCL] Add New Model o3-mini-2025-01-31 and o3-mini-2025-01-31-FC by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/898
• Add CALM models by @jgreer013 in https://github.com/ShishirPatil/gorilla/pull/900
• [BFCL] Add New Model gemini-2.0-flash-001, gemini-2.0-flash-lite-preview-02-05, gemini-2.0-pro-exp-02-05. by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/902
• chore: added snippet for hf datasets compatibility by @alt-glitch in https://github.com/ShishirPatil/gorilla/pull/906
• Update model_metadata.py by @jgreer013 in https://github.com/ShishirPatil/gorilla/pull/907
• Rename CALM to CoALM by @jgreer013 in https://github.com/ShishirPatil/gorilla/pull/913
• Bitagent 8b submission by @VectorForger in https://github.com/ShishirPatil/gorilla/pull/917
• Bitagent 8b Metadata Change by @VectorForger in https://github.com/ShishirPatil/gorilla/pull/919
• [BFCL] Add New Model gpt-4.5-preview-2025-02-27, gpt-4.5-preview-2025-02-27-FC by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/922
• [BFCL] fix bug in how score_dir is handled for bfcl evaluate by @liamcli in https://github.com/ShishirPatil/gorilla/pull/924
• [BFCL] Add New Model DeepSeek-R1 by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/901
• Make all import paths absolute. by @fvisin in https://github.com/ShishirPatil/gorilla/pull/935
• Move logic to eval a task in a separate function. by @fvisin in https://github.com/ShishirPatil/gorilla/pull/933
• Fix Gorilla Paper requirements.txt Location to Remove Global Dependency Confusion by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/937
• [BFCL] Add _unused Suffix to Unused Dataset Files in the BFCL Benchmark by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/938
• [BFCL] Support Local Inference for deepseek-ai/DeepSeek-R1 by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/926
• [BFCL] Add Support for Qwen2.5 Models in Function Calling Mode by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/925
• [BFCL] Add New Model claude-3-7-sonnet-20250219, claude-3-7-sonnet-20250219-FC by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/923
• [BFCL] Add handler and meta info for ToolACE-2-8B by @XuHwang in https://github.com/ShishirPatil/gorilla/pull/941
• [BFCL] Reorganized All constant.py Files to a constants Folder by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/944
• [BFCL] Add New Models gemini-2.0-flash-lite-001, gemini-2.0-flash-thinking-exp-01-21 by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/942
• [BFCL] Add Google Gemma-3 Series Models by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/939
• [BFCL] Move model_metadata.py to constants folder by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/949
• Add Cohere Command A by @harry-cohere in https://github.com/ShishirPatil/gorilla/pull/951
• Reformatted Supported Model Table by @JasonHuang1103 in https://github.com/ShishirPatil/gorilla/pull/961
• [BFCL] Use HTTPS instead of HTTP for OMDB by @hrshtv in https://github.com/ShishirPatil/gorilla/pull/960
• [BFCL] Fix ambiguity in exec_parallel_10 question by @amitojsingh2022 in https://github.com/ShishirPatil/gorilla/pull/962
• [BFCL] Fix API Keys Handling by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/959
• [BFCL] Fix wrong date in live_simple_205-116-13 by @amitojsingh2022 in https://github.com/ShishirPatil/gorilla/pull/963
• [BFCL] Moved Ground Truths for Executable Tests to ./data/possible_answer Folder by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/953
• [BFCL] Reorganizing Codes in ./bfcl/eval_checker/executable_eval/data/ by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/954
• [BFCL] Add gemini-2.5-pro to the Leaderboard by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/974
• [BFCL] Update Retry Logic for Gemini Models by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/976
• [BFCL] Fix Typo in multi_turn_base_166 Ground Truth. by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/979
• Add Salesforce xLAM-2 series of model handlers and update vLLM version from 0.6.3 to 0.6.5 by @zuxin666 in https://github.com/ShishirPatil/gorilla/pull/972
• [BFCL] Retire Executable Categories from Leaderboard by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/943
• feat. Add Novita LLM Models API by @novita-viktor in https://github.com/ShishirPatil/gorilla/pull/980
• [BFCL] Add New Models Llama-4-Scout, Llama-4-Maverick by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/981
• [BFCL] Add Support for Fully Offline Model Inference via --local-model-path by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/985
• Fix Typo in Model Name for xLAM-2-8b-fc-r by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/992
• Add ThinkAgents/ThinkAgent-1B by @0xayman in https://github.com/ShishirPatil/gorilla/pull/928
• [BFCL] Add Grok 3 Models to the Leaderboard by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/987
• [BFCL] Add mistral-large-2411 and mistral-small-2503 by @pracheeti12 in https://github.com/ShishirPatil/gorilla/pull/988
• Add xiaoming-14B by @kevin2016 in https://github.com/ShishirPatil/gorilla/pull/977
• [BFCL] Retire Outdated Models from the Leaderboard by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/997
• [BFCL] add support for microsoft/Phi-4-mini-instruct by @RobotSail in https://github.com/ShishirPatil/gorilla/pull/967
• [BFCL] Add microsoft/phi-4 to the Leaderboard by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/1000
• [BFCL] Add GPT 4.1 Series Models to the Leaderboard. by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/1002
• Bump writer-sdk Dependency Version by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1006
• [BFCL] add model config by @itea1001 in https://github.com/ShishirPatil/gorilla/pull/999
• [BFCL] Add Validation for Model Names by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/1008
• [BFCL] Update Error Message for New Handler Mappings by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/1013
• [BFCL] fix entry id typo in live_multiple_1052-279-0 by @itea1001 in https://github.com/ShishirPatil/gorilla/pull/1022
• Update QwQ-32b api by @CostaliyA in https://github.com/ShishirPatil/gorilla/pull/1014
• Migrate to correct testing API by @emmanuel-ferdman in https://github.com/ShishirPatil/gorilla/pull/1029
• Add gemini-2.5-pro-preview-05-06 Models by @Guangyu-Joshua-Feng in https://github.com/ShishirPatil/gorilla/pull/1031
• [BFCL] Add Qwen 3 Series Models to the Leaderboard by @catherineruoxiwu in https://github.com/ShishirPatil/gorilla/pull/1015
• [BFCL] Remove latency data for open source models by @errorfourten in https://github.com/ShishirPatil/gorilla/pull/1033
• fix treesitter setup by @CharlieJCJ in https://github.com/ShishirPatil/gorilla/pull/1045
• [BFCL] Added support for Mistral Medium 3 by @errorfourten in https://github.com/ShishirPatil/gorilla/pull/1040
• New colab links for gorilla hosted and openfunctions hosted by @ShishirPatil in https://github.com/ShishirPatil/gorilla/pull/1036
• [BFCL] Add version to bfcl CLI by @ShishirPatil in https://github.com/ShishirPatil/gorilla/pull/1038
• Add DM-Cito-8B by @kevin2016 in https://github.com/ShishirPatil/gorilla/pull/1017
• fix: loosen openai requirements to be >= 1.76.0 by @TheFloatingString in https://github.com/ShishirPatil/gorilla/pull/1050
• [BFCL] Packagerize for PyPI Distribution by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1054
• [BFCL] CI: Add “Publish to PyPI” workflow with CalVer-serial auto-versioning by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1055
• [BFCL] Replace Exception with SyntaxError for Java and JavaScript Parsers by @TheFloatingString in https://github.com/ShishirPatil/gorilla/pull/1057
• [BFCL] Support DashScope API Inference for Qwen3 Series by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1061
• [BFCL] Add type hinting by @TheFloatingString in https://github.com/ShishirPatil/gorilla/pull/1058
• [BFCL] Added support for DeepSeek-R1-0528 and DeepSeek-V3-0324 by @errorfourten in https://github.com/ShishirPatil/gorilla/pull/1063
• [BFCL] Add support for Ling-Lite-V1.5 by @fengzhu1 in https://github.com/ShishirPatil/gorilla/pull/1056
• [BFCL] Omit Reasoning Content from Chat History for Function-Calling Models by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1064
• Add support for llama-3.1-nemotron-ultra-253b-v1 to BFCL by @AdityaGhai18 in https://github.com/ShishirPatil/gorilla/pull/1032
• _get_item() can not handle the "." directory in path string by @YJ3329 in https://github.com/ShishirPatil/gorilla/pull/1060
• [BFCL] Multi-turn TravelAPI book_flight() Fix by @amitojsingh2022 in https://github.com/ShishirPatil/gorilla/pull/966
• [BFCL] Fix prompt concatenation bug in Qwen template by @nehcgs in https://github.com/ShishirPatil/gorilla/pull/1068
• Add Qwen handler by @zhangyingerjelly in https://github.com/ShishirPatil/gorilla/pull/1072
• Add traceback logging to json outputs by @imradawoodani in https://github.com/ShishirPatil/gorilla/pull/1074
• [BFCL] Fix is_fc_model config propagation by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1082
• feature: Added RZN-T to the suite of models. by @KevinDayve in https://github.com/ShishirPatil/gorilla/pull/1079
• Fix typo in month parameter (Febuary -> February) by @Gnav3852 in https://github.com/ShishirPatil/gorilla/pull/1084
• Update irrelevance_232 question by @Gnav3852 in https://github.com/ShishirPatil/gorilla/pull/1085
• [BFCL] Fixed missing airport route entries by @amitojsingh2022 in https://github.com/ShishirPatil/gorilla/pull/1087
• [BFCL] Resolve duplicated 'live-relevance_3-3-0' test entry id by @gumgizoa in https://github.com/ShishirPatil/gorilla/pull/1086
• Added support for Claude 4 family models to BFCL by @Swordscore in https://github.com/ShishirPatil/gorilla/pull/1034
• Add DM-Cito-8B-v2 by @kevin2016 in https://github.com/ShishirPatil/gorilla/pull/1088
• update ground truth for multi turn base by @Daniel-Mash in https://github.com/ShishirPatil/gorilla/pull/956
• Restrict GitHub Actions Workflow to Run Only on Source Repository by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1089
• [BFCL] Add support for Arch-Agent by @nehcgs in https://github.com/ShishirPatil/gorilla/pull/1078
• [BFCL] Introduce OpenAI Responses API Handler + o4-mini/o3 models by @errorfourten in https://github.com/ShishirPatil/gorilla/pull/1062
• nit(docs): Improve README Clarity on Sample Filename by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1093
• [BFCL] Add support for granite-3.1-8b-instruct and granite-3.2-8b-instruct by @RobotSail in https://github.com/ShishirPatil/gorilla/pull/1041
• [BFCL] Replace system role with developer role for OpenAI models by @errorfourten in https://github.com/ShishirPatil/gorilla/pull/1090
• [BFCL] nit: Print traceback on generation error by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1100
• [BFCL] Migrate Gemini Inference to Google AI Studio by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1099
• [BFCL] Update Gemini model checkpoints to stable 2.5 releases by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/1102
• [BFCL] Reintroduce latency stats for local models, update cost calculation by @Gnav3852 in https://github.com/ShishirPatil/gorilla/pull/1098
• BitAgent Bounty Model Submission by @VectorForger in https://github.com/ShishirPatil/gorilla/pull/1096
• [BFCL] Contact Customer Support Multi Turn & Vehicle Control [#914] by @amitojsingh2022 in https://github.com/ShishirPatil/gorilla/pull/1110

New Contributors

• @pkesseli made their first contribution in https://github.com/ShishirPatil/gorilla/pull/865
• @ZydHaha made their first contribution in https://github.com/ShishirPatil/gorilla/pull/858
• @SumanthRH made their first contribution in https://github.com/ShishirPatil/gorilla/pull/888
• @dominikabasaj made their first contribution in https://github.com/ShishirPatil/gorilla/pull/887
• @canyon289 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/895
• @kirill-fedyanin made their first contribution in https://github.com/ShishirPatil/gorilla/pull/894
• @jgreer013 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/900
• @alt-glitch made their first contribution in https://github.com/ShishirPatil/gorilla/pull/906
• @VectorForger made their first contribution in https://github.com/ShishirPatil/gorilla/pull/917
• @liamcli made their first contribution in https://github.com/ShishirPatil/gorilla/pull/924
• @fvisin made their first contribution in https://github.com/ShishirPatil/gorilla/pull/935
• @catherineruoxiwu made their first contribution in https://github.com/ShishirPatil/gorilla/pull/944
• @JasonHuang1103 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/961
• @hrshtv made their first contribution in https://github.com/ShishirPatil/gorilla/pull/960
• @novita-viktor made their first contribution in https://github.com/ShishirPatil/gorilla/pull/980
• @0xayman made their first contribution in https://github.com/ShishirPatil/gorilla/pull/928
• @pracheeti12 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/988
• @kevin2016 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/977
• @RobotSail made their first contribution in https://github.com/ShishirPatil/gorilla/pull/967
• @itea1001 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/999
• @CostaliyA made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1014
• @emmanuel-ferdman made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1029
• @Guangyu-Joshua-Feng made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1031
• @errorfourten made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1033
• @TheFloatingString made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1050
• @fengzhu1 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1056
• @AdityaGhai18 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1032
• @YJ3329 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1060
• @nehcgs made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1068
• @zhangyingerjelly made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1072
• @imradawoodani made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1074
• @KevinDayve made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1079
• @Gnav3852 made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1084
• @gumgizoa made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1086
• @Swordscore made their first contribution in https://github.com/ShishirPatil/gorilla/pull/1034
• @Daniel-Mash made their first contribution in https://github.com/ShishirPatil/gorilla/pull/956

Full Changelog: https://github.com/ShishirPatil/gorilla/compare/v1.2...v1.3