Gorilla Files

Highlights

🏆 We are thrilled to announce the stable v1.0 release of the Berkeley Function Calling Leaderboard data-set and eval-pipeline! A heartfelt thank you to all our contributors and users for your enthusiastic engagement and support throughout v1. We are just getting started! Buckle-up for v2 🚀 🚀 🚀

What's Changed

• better handle float value comparison by @vandyxiaowei in https://github.com/ShishirPatil/gorilla/pull/407
• Bump pymysql from 1.1.0 to 1.1.1 in /goex by @dependabot in https://github.com/ShishirPatil/gorilla/pull/453
• Fixes For NexusHandler by @VenkatKS in https://github.com/ShishirPatil/gorilla/pull/437
• [BFCL] PR#407 Evaluation Pipeline Robustness Patch by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/462
• Add firefunction-v2 to the leaderboard by @pgarbacki in https://github.com/ShishirPatil/gorilla/pull/470
• [BFCL] Add Claude 3.5 Sonnet Function Calling Infernece Inference by @Fanjia-Yan in https://github.com/ShishirPatil/gorilla/pull/480
• [BFCL] Standardize Model Name Among handler_map and eval_runner_helper by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/439
• Remove redundant tokens from GPT-handler by @hellovai in https://github.com/ShishirPatil/gorilla/pull/490
• [GoEx] Undo Minor Bug Fix + README Minor Improvement by @royh02 in https://github.com/ShishirPatil/gorilla/pull/468
• [BFCL] Add ability to evaluate Nemotron-4-340B-Instruct by @Fanjia-Yan in https://github.com/ShishirPatil/gorilla/pull/489
• fix some data issues in parallel/parallel multiple answers by @vandyxiaowei in https://github.com/ShishirPatil/gorilla/pull/423
• [BFCL] Add Support for GLM-4-9B function calling inference by @Fanjia-Yan in https://github.com/ShishirPatil/gorilla/pull/474
• [BFCL] Sanity check is now optional by @ShishirPatil in https://github.com/ShishirPatil/gorilla/pull/496
• [BFCL] Improved tree-sitter java, javascript installation by @CharlieJCJ in https://github.com/ShishirPatil/gorilla/pull/505
• [BFCL] Fix Possible Answer for AST Parallel and Parallel_Multiple Category by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/503
• [BFCL] Add Test Dataset to Repository by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/504
• [BFCL] Support Category-Specific Generation for OSS Model, Remove eval_data_compilation Step by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/512
• [BFCL] Fix Double-Casting Issue in model_handler for Java and JS category. by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/516
• [BFCL] Fix Dataset Issue for executable_parallel_multiple Category by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/522
• [BFCL] add ibm-granite-20b-functioncallling model by @MayankAgarwal in https://github.com/ShishirPatil/gorilla/pull/525
• [BFCL] Overhaul apply_function_credential_config.py for Enhanced Usability by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/508
• Fixed the warning message "Setting pad_token_id to eos_token_id:1… by @dineshkumarsarangapani in https://github.com/ShishirPatil/gorilla/pull/110
• [BFCL] Specify package version in requirements.txt by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/515
• [BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/506
• fix line return by @fantasist in https://github.com/ShishirPatil/gorilla/pull/531
• [BFCL] Apply Fix to Newly Introduced Model Handler Missed in Previous PR Merge by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/536
• [RAFT] Fix Datapoint Field in Formatter for Data Generation by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/535
• [BFCL] Fix language_specific_pre_processing for Java and JavaScript Test Category by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/538
• [BFCL] Patch Generation Script for Locally Hosted OSS model by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/537
• [BFCL] Support Multi-Model Multi-Category Generation; Add Index to Dataset; Handle vLLM Benign Error by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/540
• Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models by @alonsosilvaallende in https://github.com/ShishirPatil/gorilla/pull/542
• [BFCL] Fix Dataset Pre-Processing for Java and JavaScript Test Category, Part 2 by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/545
• Add Salesforce xLAM handler and fix minor issues by @zuxin666 in https://github.com/ShishirPatil/gorilla/pull/532
• Add NousResearch/Hermes-2-{Pro-Llama-3-80B,Theta-Llama-3-80B} by @alonsosilvaallende in https://github.com/ShishirPatil/gorilla/pull/556
• Add Yi Handler by @fantasist in https://github.com/ShishirPatil/gorilla/pull/543
• Add more descriptive error message in eval_runner.py by @alonsosilvaallende in https://github.com/ShishirPatil/gorilla/pull/552
• [BFCL] Fix JS type converter to handle dictionaries with array values by @CharlieJCJ in https://github.com/ShishirPatil/gorilla/pull/549
• [BFCL] Handling rate limits by @ShishirPatil in https://github.com/ShishirPatil/gorilla/pull/559
• [BFCL] Fix Dataset and Possible Answer Issue by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/557
• [BFCL] Dataset Question Fix for Executable Parallel Category by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/568
• [BFCL] Add New Model gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/569
• [BFCL] Add New Model open-mistral-nemo-2407, open-mixtral-8x22b, open-mixtral-8x7b by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/570
• [BFCL] Improve Warning Message when Aggregating Results by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/517
• [BFCL] Add New Model functionary-small-v3.1, functionary-small-v3.2, functionary-medium-v3.1; Update Token Price by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/573
• [BFCL] Set Model Temperature to 0.001 for All Models by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/574
• [BFCL] Support Parallel Inference for Hosted Models by @HuanzhiMao in https://github.com/ShishirPatil/gorilla/pull/571
• [BFCL Chore] Fix Functionary Medium 3.1 model name & add readme parallel inference by @CharlieJCJ in https://github.com/ShishirPatil/gorilla/pull/577

New Contributors

• @dependabot made their first contribution in https://github.com/ShishirPatil/gorilla/pull/453
• @VenkatKS made their first contribution in https://github.com/ShishirPatil/gorilla/pull/437
• @pgarbacki made their first contribution in https://github.com/ShishirPatil/gorilla/pull/470
• @hellovai made their first contribution in https://github.com/ShishirPatil/gorilla/pull/490
• @MayankAgarwal made their first contribution in https://github.com/ShishirPatil/gorilla/pull/525
• @dineshkumarsarangapani made their first contribution in https://github.com/ShishirPatil/gorilla/pull/110
• @fantasist made their first contribution in https://github.com/ShishirPatil/gorilla/pull/531
• @alonsosilvaallende made their first contribution in https://github.com/ShishirPatil/gorilla/pull/542

Full Changelog: https://github.com/ShishirPatil/gorilla/compare/v0.3...v1.0