SmolVLM

LLaVA (Large Language-and-Vision Assistant) is an innovative multimodal model that integrates a vision encoder with the Vicuna language model to facilitate comprehensive visual and language understanding. Through end-to-end training, LLaVA exhibits impressive chat capabilities, emulating the multimodal functionalities of models like GPT-4. Notably, LLaVA-1.5 has achieved state-of-the-art performance across 11 benchmarks, utilizing publicly available data and completing training in approximately one day on a single 8-A100 node, surpassing methods that rely on billion-scale datasets. The development of LLaVA involved the creation of a multimodal instruction-following dataset, generated using language-only GPT-4. This dataset comprises 158,000 unique language-image instruction-following samples, including conversations, detailed descriptions, and complex reasoning tasks. This data has been instrumental in training LLaVA to perform a wide array of visual and language tasks effectively.

Magma is a cutting-edge multimodal foundation model developed by Microsoft, designed to understand and act in both digital and physical environments. The model excels at interpreting visual and textual inputs, allowing it to perform tasks such as interacting with user interfaces or manipulating real-world objects. Magma builds on the foundation models paradigm by leveraging diverse datasets to improve its ability to generalize to new tasks and environments. It represents a significant leap toward developing AI agents capable of handling a broad range of general-purpose tasks, bridging the gap between digital and physical actions.