Search Results for "transformer"

...It is designed to handle bipeds, quadrupeds, birds, marine creatures, insects, serpentine rigs, and articulated rigid objects with one unified model. Its topology-aware diffusion transformer conditions generation on skeleton structure rather than requiring a fixed character template. The accompanying UniML3D dataset contains 13,006 text-paired motion sequences with unified skeletal canonicalization. Inference can generate motion for unseen skeleton topologies without per-skeleton retraining or test-time optimization. ...

...Natural-language instructions can drive subject edits, local changes, background replacement, style transfer, motion changes, and reference-guided transformations. Its architecture combines a multimodal condition encoder, causal video VAE, and a 16B-parameter multimodal diffusion transformer. Autoregressive diffusion and bounded KV-state inference are used to keep computation stable across long streams. The released deployment reaches high-throughput 720p editing and also supports real-time operation on selected consumer GPUs. Updated checkpoints improve identity preservation, reference conditioning, and temporal consistency for reference-image-guided editing.

...Along these two audio streams, Moshi predicts text tokens corresponding to its own speech, its inner monologue, which greatly improves the quality of its generation. A small Depth Transformer models inter codebook dependencies for a given time step, while a large, 7B parameter Temporal Transformer models the temporal dependencies.

ProPainter is a video inpainting project based on the ICCV 2023 paper “Improving Propagation and Transformer for Video Inpainting.” It is designed to remove objects, complete missing regions, and fill masked areas in videos while preserving temporal consistency. The project accepts video input or split frames along with frame-wise masks that define the areas to reconstruct. It provides pretrained models, example inputs, inference scripts, and an interactive demo workflow for object removal. ...

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch. The main novelty seems to be an extra layer of indirection with the prior network (whether it is an autoregressive transformer or a diffusion network), which predicts an image embedding based on the text embedding from CLIP. Specifically, this repository will only build out the diffusion prior network, as it is the best performing variant (but which incidentally involves a causal transformer as the denoising network) To train DALLE-2 is a 3 step process, with the training of CLIP being the most important. ...

TimeSformer is a vision transformer architecture for video that extends the standard attention mechanism into spatiotemporal attention. The model alternates attention along spatial and temporal dimensions (or designs variants like divided attention) so that it can capture both appearance and motion cues in video. Because the attention is global across frames, TimeSformer can reason about dependencies across long time spans, not just local neighborhoods.

JAVT or Just Another Voice Transformer (formerly, it is called Just Another Video Transcriber) is a Speech Recognition software that also support text to Speech and simple media conversion. JAVT allows you to convert from video files to audio wav file using ffmpeg, and then transcribe the audio file to text using either Microsoft SAPI or CMU Sphinx. You can also open a text file and allow JAVT to read it out for you through text to speech conversion.