![\"\"](\"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2024\/11\/Screenshot-2024-11-22-at-07.52.27-1024x602.jpg\")
<\/p>\n*Even by the elevated standards of contemporary generator-jargon, that’s pretty strange.<\/p>\n
<\/p>\n
At a high level, we turn videos into patches by first compressing videos into a lower-dimensional latent space, and subsequently decomposing the representation into spacetime patches.<\/p>\n
Video compression network
\n‘
\nWe train a network that reduces the dimensionality of visual data.20 This network takes raw video as input and outputs a latent representation that is compressed both temporally and spatially. Sora is trained on and subsequently generates videos within this compressed latent space. We also train a corresponding decoder model that maps generated latents back to pixel space.<\/p>\n
Spacetime latent patches<\/p>\n
Given a compressed input video, we extract a sequence of spacetime patches which act as transformer tokens. This scheme works for images too since images are just videos with a single frame. Our patch-based representation enables Sora to train on videos and images of variable resolutions, durations and aspect ratios. At inference time, we can control the size of generated videos by arranging randomly-initialized patches in an appropriately-sized grid…..<\/p>\n