VideoPoet

Marengo is a multimodal video foundation model that transforms video, audio, image, and text inputs into unified embeddings, enabling powerful “any-to-any” search, retrieval, classification, and analysis across vast video and multimedia libraries. It integrates visual frames (with spatial and temporal dynamics), audio (speech, ambient sound, music), and textual content (subtitles, overlays, metadata) to create a rich, multidimensional representation of each media item. With this embedding architecture, Marengo supports robust tasks such as search (text-to-video, image-to-video, video-to-audio, etc.), semantic content discovery, anomaly detection, hybrid search, clustering, and similarity-based recommendation. The latest versions introduce multi-vector embeddings, separating representations for appearance, motion, and audio/text features, which significantly improve precision and context awareness, especially for complex or long-form content.

Mirage is the world’s first real‑time, autoregressive video‑to‑video transformation model that instantly turns any live video, game, or camera feed into a new digital world without pre‑rendering. Powered by Live‑Stream Diffusion (LSD) technology, it processes inputs at 24 FPS with under 40 ms latency, ensuring smooth, continuous transformations while preserving motion and structure. Mirage supports universal input, webcams, gameplay, movies, and live streams, and applies text‑prompted style changes on the fly. Its advanced history‑augmentation mechanism maintains temporal coherence across frames, avoiding the glitches common in diffusion‑only approaches. GPU‑accelerated custom CUDA kernels deliver up to 16× faster performance than traditional methods, enabling infinite streaming without interruption. It offers real‑time mobile and desktop previews, seamless integration with any video source, and flexible deployment.