TRIP: Temporal Residual Learning with Image Noise Prior
for Image-to-Video Diffusion Models

CVPR 2024


1 University of Science and Technology of China     2 HiDream.ai Inc    

Norway night aurora perform

Flying over sea waves on a tropical beach

Beautiful sunflowers blooming in the field

A woman standing on the beach at sunset

A hand holding a model of a wooden house, through door of which setting sun is visible, background of coast

Fluffy panda cub is cleaning her fur

Aerial view of a vineyard in the back mountain Tavor, Israel

Aerial view of San Francisco at sunset

Greyface moray eel negros philippines

Magical sunset and rocky coast, Cinque Terre National Park, Liguria, Italy, Europe

Honey bees on flower pollination, macro shot

Seascape of coral reef in the Caribbean Sea around Curacao

Istanbul, ship, transportation turkey Bosphorus city ocean

Snail, close up

Norway night aurora perform

Flying over sea waves on a tropical beach

Beautiful sunflowers blooming in the field

A woman standing on the beach at sunset

A hand holding a model of a wooden house, through door of which setting sun is visible, background of coast

Fluffy panda cub is cleaning her fur

Aerial view of a vineyard in the back mountain Tavor, Israel

Aerial view of San Francisco at sunset

Greyface moray eel negros philippines

Magical sunset and rocky coast, Cinque Terre National Park, Liguria, Italy, Europe

Honey bees on flower pollination, macro shot

Seascape of coral reef in the Caribbean Sea around Curacao

Istanbul, ship, transportation turkey Bosphorus city ocean

Snail, close up

Abstract

Recent advances in text-to-video generation have demonstrated the utility of powerful diffusion models. Nevertheless, the problem is not trivial when shaping diffusion models to animate static image (i.e., image-to-video generation). The difficulty originates from the aspect that the diffusion process of subsequent animated frames should not only preserve the faithful alignment with the given image but also pursue temporal coherence among adjacent frames. To alleviate this, we present TRIP, a new recipe of image-to-video diffusion paradigm that pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning and ease the coherent temporal modeling via temporal residual learning. Technically, the image noise prior is first attained through one-step backward diffusion process based on both static image and noised video latent codes. Next, TRIP executes a residual-like dual-path scheme for noise prediction: 1) a shortcut path that directly takes image noise prior as the reference noise of each frame to amplify the alignment between the first frame and subsequent frames; 2) a residual path that employs 3D-UNet over noised video and static image latent codes to enable inter-frame relational reasoning, thereby easing the learning of the residual noise for each frame. Furthermore, both reference and residual noise of each frame are dynamically merged via attention mechanism for final video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate the effectiveness of our TRIP for image-to-video generation.

Pipeline

An overview of (a) our Temporal Residual learning with Image noise Prior (TRIP), and (b) the conceptual comparison between typical frame-wise denoising and our temporal noise fusion. During training, the input video clip is encoded into a sequence of image latent codes via a pre-trained 2D VAE. The latent code of first frame is then concatenated with noised video latent code as the input of the 3D-UNet. Then, TRIP performs a residual-like noise prediction along two pathways, i.e., shortcut path and residual path. The shortcut path computes the image noise prior as the reference noise based on the first frame latent codes and the noised video latent codes through a one-step backward diffusion. The residual path leverages 3D-UNet to predict the residual noise with reference to the image noise prior. By dynamically merging the reference and residual noise via a Transformer-based Temporal Noise Fusion (TNF) module, the target noise vector for video denoising is attained. Both 3D-UNet and TNF module are jointly optimized with the typical MSE loss for I2V generation.

Image-to-Video Results on WebVid-10M

Norway night aurora perform

Wild hungry hedgehog eating domestic pet food

Close-up view of the red february caucasian ladybug crawling on the green leaf

Happy woman at cascade

The pieces of coconut on a plate with leaves and halves of coconut on a wooden table

Wild deer in the Pacific Northwest, USA, eating vegetation in dense rainforest

Customized Image Animation Combined with 2D Diffusion Model

Two parrots are sitting on a branch in the forest(SD-XL)

A castle in the middle of a forest with a river running through it(SD-XL)

WALL-E on the beach with a plant in his hand(SD-XL)

A fox is walking in the woods at night(SD-XL)

A ship sails in the sea(SD-XL)

Crabs with bubbles crawls on the beach with sunset in the background(SD-XL)

A bear is playing a guitar in front of a tank in the snow(SD-XL)

A giant robot with pumpkins on it(SD-XL)

Halloween pumpkin with glowing eyes and smoke coming out of it(SD-XL)

A black swan swims on the pond(Ori image)

A white duck swims on the pond(InstructPix2Pix)

A colorful duck swims on the pond(ControlNet)

comparisons

             Image                     T2V-Zero              VideoComposer                TRIP

A male fencer adjusts his epee mask and prepares to duel with his sparring partner in slow motion

             Image                     T2V-Zero              VideoComposer                TRIP

Glass of red wine being poured

             Image                     T2V-Zero              VideoComposer                TRIP

Giant spider crab

             Image                     T2V-Zero              VideoComposer                TRIP

Few big purple plums rotating on the turntable water drops appear on the skin during rotation isolated on the white background, close-up macro

             Image                     T2V-Zero              VideoComposer                TRIP

Asian elephants, thailand

             Image                     T2V-Zero              VideoComposer                TRIP

Close-up view national flag of Iceland waving in the wind on a blue sky background without clouds, national symbol consists of a blue background bearing a red cross