Search Results for "decoder" - Page 7

...Its architecture combines sliding-window and global attention, a 681M-parameter vision encoder, dedicated audio encoders, and a five-layer multi-token speculative decoder. It targets coding, general and visual agents, tool use, cybersecurity, long-horizon reasoning, etc.

t5-base is a pre-trained transformer model from Google’s T5 (Text-To-Text Transfer Transformer) family that reframes all NLP tasks into a unified text-to-text format. With 220 million parameters, it can handle a wide range of tasks, including translation, summarization, question answering, and classification. Unlike traditional models like BERT, which output class labels or spans, T5 always generates text outputs. It was trained on the C4 dataset, along with a variety of supervised NLP...

facebook/bart-large-cnn is a large-scale sequence-to-sequence transformer model developed by Meta AI and fine-tuned specifically for abstractive text summarization. It uses the BART architecture, which combines a bidirectional encoder (like BERT) with an autoregressive decoder (like GPT). Pre-trained on corrupted text reconstruction, the model was further trained on the CNN/DailyMail dataset—a collection of news articles paired with human-written summaries. It performs particularly well in generating concise, coherent, and human-readable summaries from longer texts. Its architecture allows it to model both language understanding and generation tasks effectively. ...

...The model uses a sparse Mixture-of-Experts architecture with 276B total parameters and 12B active per token, enabling strong performance with lower inference cost than a fully dense model of similar scale. Its 42-layer decoder routes each token through six of 256 specialized experts plus two shared experts, while hybrid local and global attention supports efficient processing. Inkling-Small performs strongly across software engineering, tool use, mathematics, vision, and audio benchmarks, including 80.2% on SWE-Bench Verified and 95.5% on AIME 2026.