Search Results for "tokenizer" - Page 2
Sort By:
Showing 63 open source projects for "tokenizer"
View related business solutions-
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
1
LLaMA
Inference code for Llama models
“Llama” is the repository from Meta (formerly Facebook/Meta Research) containing the inference code for LLaMA (Large Language Model Meta AI) models. It provides utilities to load pre-trained LLaMA model weights, run inference (text generation, chat, completions), and work with tokenizers. Tokenizer utilities, download scripts, shell helpers to fetch model weights with correct licensing/permissions. Includes example scripts for chat completions and text completions to show how to call the models in code. This repo is a core piece of the Llama model infrastructure, used by researchers and developers to run LLaMA models locally or in their infrastructure. ... -
2
llm
An ecosystem of Rust libraries for working with large language models
llm is an ecosystem of Rust libraries for working with large language models - it's built on top of the fast, efficient GGML library for machine learning. The primary entry point for developers is the llm crate, which wraps the llm-base and the supported model crates. Documentation for the released version is available on Docs.rs. For end-users, there is a CLI application, llm-cli, which provides a convenient interface for interacting with supported models. Text generation can be done as a... -
3
LaMDA-pytorch
Open-source pre-training implementation of Google's LaMDA in PyTorch
Open-source pre-training implementation of Google's LaMDA research paper in PyTorch. The totally not sentient AI. This repository will cover the 2B parameter implementation of the pre-training architecture as that is likely what most can afford to train. You can review Google's latest blog post from 2022 which details LaMDA here. You can also view their previous blog post from 2021 on the model. -
4
Tensorflow Transformers
State of the art faster Transformer with Tensorflow 2.0
...Auto-Batching tf.data.dataset or tf.ragged tensors. Everything is dictionary (inputs and outputs) Multiple mask modes like causal, user-defined, prefix. tensorflow-text tokenizer support. Supports GPU, TPU, multi-GPU trainer with wandb, multiple callbacks, auto tensorboard. -
Stop vibe-debugging.AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.
-
5
GPT Neo
An implementation of model parallel GPT-2 and GPT-3-style models
An implementation of model & data parallel GPT3-like models using the mesh-tensorflow library. If you're just here to play with our pre-trained models, we strongly recommend you try out the HuggingFace Transformer integration. Training and inference is officially supported on TPU and should work on GPU as well. This repository will be (mostly) archived as we move focus to our GPU-specific repo, GPT-NeoX. NB, while neo can technically run a training step at 200B+ parameters, it is very... -
6
GPT2 for Multiple Languages
GPT2 for Multiple Languages, including pretrained models
...The contents in this repository are for academic research purpose, and we do not provide any conclusive remarks. Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC) Simplifed GPT2 train scripts（based on Grover, supporting TPUs). Ported bert tokenizer, multilingual corpus compatible. 1.5B GPT2 pretrained Chinese model (~15G corpus, 10w steps). Batteries-included Colab demo. 1.5B GPT2 pretrained Chinese model (~30G corpus, 22w steps). -
7
javalang
Pure Python Java parser and tools
javalang is a pure Python library for working with Java source code. javalang provides a lexer and parser targeting Java 8. The implementation is based on the Java language spec. -
8
YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
YouTokenToMe is a fast and efficient unsupervised text tokenization library designed for training subword embeddings, particularly useful for NLP models. -
9
Ganja.js
Javascript Geometric Algebra Generator for Javascript, c++
...Mathematically, an algebra generated by ganja.js is a graded exterior (Grassmann) algebra (or one of its subalgebras) with a non-metric outer product, extended (Clifford) with geometric and contraction inner products, a Poincare duality operator and the main involutions and morphisms. Technically, ganja.js is a code generator producing classes that reificate algebraic literals and expressions by using reflection, a built-in tokenizer and a simple AST translator to rewrite functions containing algebraic constructs to their procedural counterparts. ganja.js now has a nodejs based templated source generator that allows the creation of arbitrary algebras for C++, C#, python and rust. The generated code provides in a flat multivector format and operator overloading. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
JSMN
Jsmn is a world fastest JSON parser/tokenizer
JSMN (pronounced "jasmine") is a minimalistic JSON parser in C. It's tailored for resource-constrained environments, making it ideal for embedded systems where efficiency and low memory footprint are paramount. -
11
cyqlite
enhanced SQLite
100% Upwards compatible variant of SQLite. Provides win32/win64 versions of sqlite3.dll, which work better (smaller/faster/longer paths) than the dll's provided by sqlite.org. -
12
WikiSQL
A large annotated semantic parsing corpus for developing NL interfaces
...Regarding tokenization and Stanza, when WikiSQL was written 3-years ago, it relied on Stanza, a CoreNLP python wrapper that has since been deprecated. If you'd still like to use the tokenizer, please use the docker image. We do not anticipate switching to the current Stanza as changes to the tokenizer would render the previous results not reproducible. -
13
flex: the fast lexical analyser
flex is a tool for generating scanners
flex is a tool for generating scanners. A scanner, sometimes called a tokenizer, is a program which recognizes lexical patterns in text. The flex program reads user-specified input files, or its standard input if no file names are given, for a description of a scanner to generate. The description is in the form of pairs of regular expressions and C code, called rules. Flex generates a C source file named, "lex.yy.c", which defines the function yylex(). -
14
Persianp Toolbox
A toolbox for Persian texts preprocessing
A toolbox for preprocessing Persian texts including: Normalizer Tokenizer Sentencizer POS tagger Lemmatizer Stopword detector For more information please visit: www.persianp.ir/toolbox.html -
15
We have implemented a core summarizer of scientific articles written in Spanish, with the following components: a tokenizer, a grammar checker, a clarity checker, a cohesion-coherence checker, a common-topic extractor and an output formatter.
-
16
slug_dev
developing plattform for SLUG projects
...Here we are hosting our projects related to solr and lucene. log4jSolr - log4j appender (and more) to index all log events solr_core - solr analysis extensions such as filters oder tokenizer -
17
tokenizer
Transforms arithmetic expressions (cstrings) into a sequence of tokens
A c-string that represents an arithmetic expression ist transformed into a sequence of tokens ( functions, constants, variables, operators, brackets, commas ) and stored on a stack. -
18
BASIC interpreter for the 16bit PIC microcontroller 24FJ64GA002. The interpreter runs on the chip only, no compiler/tokenizer is needed. Communication with PC is done by USB-to-serial converter cable.
-
19
This is a simple tokenizer for converting source code in ascii text files into a ZX Spectrum loadable image file
-
20
EWBTokenFactory for PHP is a whitespace inclusive tokenizer. Most tokenizers will grab each word as a token and drop the whitespace. This tokenizer will also tokenize the spaces in between into their own tokens. Also supports associated token data.
-
21
Similar to StringTokenizer already in the Java standard library, but capable of discerning different types of data as tokens (e.g. "a=1" will yield 3 tokens) as it doesn't rely on whitespaces or some single delimiter (as StringTokenizer does).
-
22
Stamp Basic compiler based on the tokenizer shared library (provided by Parallax Inc., Rocklin, CA USA.) for the Linux platform.
-
23
This is my attempt to write a very simple parser in C++ in my (very infrequent) free time. Please ignore the tokenizer as I cheated a bit.
-
24
NLPTools-ES is a Spanish plugin for GATE (General Architecture for Text Engineering). It includes a tokenizer, sentence splitter, gazetteer, pos tagger.
-
25
Namespaced Template Engine (PHP extension) - An extremely powerful Tokenizer driven Template engine with XSL-like syntax. Nstag works based on the idea to have special tags in a separate namespace to apply view related logic or just assignments.
