ContextGem Files

Added

• output_language of DocumentLLM and DocumentModelGroup now takes a BCP 47 language tag of any language, with its script or region where they matter, e.g. "ja", "zh-Hant" or "pt-BR", besides "en" and "adapt": string values are translated into that language and justifications are written in it, whatever the languages of the document, and the default system message tells the LLM to answer in it. It took "en" or "adapt" only, so the output of a document in several languages could be in English or in the language that appears most often in it. The tag is stored in its standard form, e.g. "zh-TW" for "zh_tw" and "en-GB" for "en-uk", and a value that is not a tag of a language raises ValueError, with the tag that a language name or a region code likely stands for, e.g. "ja" for "Japanese" or "jp". The prompts of "en" and "adapt" are unchanged. See Choosing the output language.
• New dependency: langcodes, with its data extra (language-data and marisa-trie), which validates the tags and gives the names of their languages in the prompts.