How can the effectiveness of vision transformers be leveraged in diffusion-based generative learning? This paper from NVIDIA introduces a novel model called Diffusion Vision Transformers (DiffiT), which combines a hybrid hierarchical architecture with a U-shaped encoder and decoder. This approach has pushed the state of the art in generative models and offers a solution to…

How can Neural Radiance Fields (NeRFs) be improved to handle scale variations and reduce aliasing artifacts in scene reconstruction? A new research paper from CMU and Meta addresses this issue by proposing PyNeRF (Pyramidal Neural Radiance Fields). It improves neural radiation fields (NeRFs) by training model heads at different spatial grid resolutions, which helps…

Recently, there have been significant advancements in video editing, with editing using Artificial Intelligence (AI) at its forefront. Numerous novel techniques have emerged, and among them, Diffusion-based video editing stands out as a particularly promising field. It leverages pre-trained text-to-image/video diffusion models for tasks like style change, background swapping, etc. However, The challenging part in…

An essential function of multi-view camera systems is novel view synthesis (NVS), which attempts to generate photorealistic images from new perspectives using source photos. The subfields of human NVS have the potential to significantly contribute to real-time efficiency and consistent 3D appearances in areas such as holographic communication, stage performances, and 3D/4D immersive scene capture…

Researchers developed the CoDi-2 Multimodal Large Language Model (MLLM) from UC Berkeley, Microsoft Azure AI, Zoom, and UNC-Chapel Hill to address the problem of generating and understanding complex multimodal instructions, as well as excelling in subject-driven image generation, vision transformation, and audio editing tasks. This model represents a significant breakthrough in establishing a comprehensive multimodal…

Human posture is crucial in overall health, well-being, and various aspects of life. It encompasses the alignment and positioning of the body while sitting, standing, or lying down. Good posture supports the optimal alignment of muscles, joints, and ligaments, reducing the risk of muscular imbalances, joint pain, and overuse injuries. It helps distribute the body’s…

The problem of video understanding and generation scenarios has been addressed by researchers of Tencent AI Lab and The University of Sydney by presenting GPT4Video. This unified multi-model framework supports LLMs with the capability of both video understanding and generation. GPT4Video developed an instruction-following-based approach integrated with the stable diffusion generative model, which effectively and…

Human Activity Recognition (HAR) is a field of study that focuses on developing methods and techniques to automatically identify and classify human activities based on data collected from various sensors. HAR aims to enable machines like smartphones, wearable devices, or smart environments to understand and interpret human activities in real-time. Traditionally, wearable sensor-based and camera-based…

Researchers from the University of Southern California, the University of Washington, Bar-Ilan University, and Google Research introduced DreamSync, which addresses the problem of enhancing alignment and aesthetic appeal in diffusion-based text-to-image (T2I) models without the need for human annotation, model architecture modifications, or reinforcement learning. It achieves this by generating candidate images, evaluating them using…

In generative modeling, diffusion models (DMs) have assumed a pivotal role, facilitating recent progress in producing high-quality picture and video synthesis. Scalability and iterativeness are two of DMs’ main advantages; they enable them to do intricate tasks like picture creation from free-form text cues. Unfortunately, the many sample steps required for the iterative inference process…