model2Vec Files

What's Changed

• fix: Updated semantic chunking tutorial by @bhavnicksm in https://github.com/MinishLab/model2vec/pull/205
• rewrite backend by @stephantul in https://github.com/MinishLab/model2vec/pull/207
• fix bibtex by @stephantul in https://github.com/MinishLab/model2vec/pull/208
• feat: Added py.typed file by @Pringled in https://github.com/MinishLab/model2vec/pull/214
• fix: pretokenize tokens before checking vocabulary by @stephantul in https://github.com/MinishLab/model2vec/pull/215
• feat: add dimensionality during loading by @stephantul in https://github.com/MinishLab/model2vec/pull/216
• feat: add quantization by @stephantul in https://github.com/MinishLab/model2vec/pull/217
• feat: save load subfolder by @stephantul in https://github.com/MinishLab/model2vec/pull/218
• feat: Added quantization for from_sentence_transformers by @Pringled in https://github.com/MinishLab/model2vec/pull/219
• feat: faster inference for large vocab by @stephantul in https://github.com/MinishLab/model2vec/pull/221
• feat: track token provenance by @stephantul in https://github.com/MinishLab/model2vec/pull/222
• fix: typing issues, bug in infernece by @stephantul in https://github.com/MinishLab/model2vec/pull/224
• fix: issues with unk and pad by @stephantul in https://github.com/MinishLab/model2vec/pull/225
• bug: fix 0 score in evaluate by @stephantul in https://github.com/MinishLab/model2vec/pull/226
• fix: precision during training by @stephantul in https://github.com/MinishLab/model2vec/pull/228
• fix: issue with unk in unigram by @stephantul in https://github.com/MinishLab/model2vec/pull/227
• docs: add info about quantization and dimensionality reduction by @stephantul in https://github.com/MinishLab/model2vec/pull/231
• increment version by @stephantul in https://github.com/MinishLab/model2vec/pull/232

New Contributors

• @bhavnicksm made their first contribution in https://github.com/MinishLab/model2vec/pull/205

Full Changelog: https://github.com/MinishLab/model2vec/compare/0.4.1...v0.5.0