{"id":2779,"date":"2022-11-17T19:56:54","date_gmt":"2022-11-17T19:56:54","guid":{"rendered":"https:\/\/toshareproject.it\/artmakerblog\/?p=2779"},"modified":"2022-11-17T19:56:54","modified_gmt":"2022-11-17T19:56:54","slug":"infinite-nature-generating-3d-flythroughs-from-still-photos","status":"publish","type":"post","link":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/","title":{"rendered":"Infinite Nature: Generating 3D Flythroughs from Still Photos"},"content":{"rendered":"<p>*Generated &#8220;natural&#8221; photos that are even more endless than the imperilled planet.<\/p>\n<p>&nbsp;<\/p>\n<p>https:\/\/ai.googleblog.com\/2022\/11\/infinite-nature-generating-3d.html<\/p>\n<p>&nbsp;<\/p>\n<h2>Background: Learning 3D Flythroughs from Videos<\/h2>\n<p>To establish the basics of how such a system could work, we\u2019ll describe our first version, \u201c<a href=\"https:\/\/infinite-nature.github.io\/\">Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image<\/a>\u201d (presented at <a href=\"https:\/\/iccv2021.thecvf.com\/home\">ICCV 2021<\/a>). In that work we explored a \u201clearn from video\u201d approach, where we collected a set of online videos captured from drones flying along coastlines, with the idea that we could learn to synthesize new flythroughs that resemble these real videos. This set of online videos is called the <a href=\"https:\/\/infinite-nature.github.io\/\">Aerial Coastline Imagery Dataset<\/a> (ACID). In order to learn how to synthesize scenes that respond dynamically to any desired 3D camera path, however, we couldn\u2019t simply treat these videos as raw collections of pixels; we also had to compute their underlying 3D geometry, including the camera position at each frame.<\/p>\n<p>The basic idea is that we learn to generate flythroughs step-by-step. Given a starting view, like the first image in the figure below, we first compute a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Depth_map\">depth map<\/a> using <a href=\"https:\/\/ai.googleblog.com\/2021\/02\/the-technology-behind-cinematic-photos.html\">single-image depth prediction methods<\/a>. We then use that depth map to <em>render<\/em> the image forward to a new camera viewpoint, shown in the middle, resulting in a new image and depth map from that new viewpoint.<\/p>\n<p>However, this intermediate image has some problems \u2014 it has holes where we can see behind objects into regions that weren\u2019t visible in the starting image. It is also blurry, because we are now closer to objects, but are stretching the pixels from the previous frame to render these now-larger objects.<\/p>\n<p>To handle these problems, we learn a neural <em>image refinement<\/em> network that takes this low-quality intermediate image and outputs a complete, high-quality image and corresponding depth map. These steps can then be repeated, with this synthesized image as the new starting point. Because we refine both the image and the depth map, this process can be iterated as many times as desired \u2014 the system automatically learns to generate new scenery, like mountains, islands, and oceans, as the camera moves further into the scene&#8230;.<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>*Generated &#8220;natural&#8221; photos that are even more endless than the imperilled planet. &nbsp; https:\/\/ai.googleblog.com\/2022\/11\/infinite-nature-generating-3d.html &nbsp; Background: Learning 3D Flythroughs from Videos To establish the basics of how such a system could work, we\u2019ll describe our first version, \u201cInfinite Nature: Perpetual View Generation of Natural Scenes from a Single Image\u201d (presented at ICCV 2021). In that [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2779","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v17.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog<\/title>\n<meta name=\"description\" content=\"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog\" \/>\n<meta property=\"og:description\" content=\"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog\" \/>\n<meta property=\"og:url\" content=\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/\" \/>\n<meta property=\"og:site_name\" content=\"Artmaker Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-17T19:56:54+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bruce Sterling\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/\",\"name\":\"Artmaker Blog\",\"description\":\"on Toshareproject.it - curated by Bruce Sterling\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#webpage\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/\",\"name\":\"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog\",\"isPartOf\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\"},\"datePublished\":\"2022-11-17T19:56:54+00:00\",\"dateModified\":\"2022-11-17T19:56:54+00:00\",\"author\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\"},\"description\":\"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog\",\"breadcrumb\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/toshareproject.it\/artmakerblog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Infinite Nature: Generating 3D Flythroughs from Still Photos\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\",\"name\":\"Bruce Sterling\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#personlogo\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"caption\":\"Bruce Sterling\"},\"description\":\"Art director at Share Festival, author and journalist\",\"sameAs\":[\"http:\/\/toshareproject.it\/tomorrowart\"],\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","description":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/","og_locale":"en_GB","og_type":"article","og_title":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","og_description":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","og_url":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/","og_site_name":"Artmaker Blog","article_published_time":"2022-11-17T19:56:54+00:00","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bruce Sterling","Estimated reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/toshareproject.it\/artmakerblog\/#website","url":"https:\/\/toshareproject.it\/artmakerblog\/","name":"Artmaker Blog","description":"on Toshareproject.it - curated by Bruce Sterling","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#webpage","url":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/","name":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","isPartOf":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#website"},"datePublished":"2022-11-17T19:56:54+00:00","dateModified":"2022-11-17T19:56:54+00:00","author":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085"},"description":"Infinite Nature: Generating 3D Flythroughs from Still Photos | Artmaker Blog","breadcrumb":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/toshareproject.it\/artmakerblog\/infinite-nature-generating-3d-flythroughs-from-still-photos\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/toshareproject.it\/artmakerblog\/"},{"@type":"ListItem","position":2,"name":"Infinite Nature: Generating 3D Flythroughs from Still Photos"}]},{"@type":"Person","@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085","name":"Bruce Sterling","image":{"@type":"ImageObject","@id":"https:\/\/toshareproject.it\/artmakerblog\/#personlogo","inLanguage":"en-GB","url":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","caption":"Bruce Sterling"},"description":"Art director at Share Festival, author and journalist","sameAs":["http:\/\/toshareproject.it\/tomorrowart"],"url":"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/"}]}},"_links":{"self":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/2779","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/comments?post=2779"}],"version-history":[{"count":1,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/2779\/revisions"}],"predecessor-version":[{"id":2780,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/2779\/revisions\/2780"}],"wp:attachment":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/media?parent=2779"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/categories?post=2779"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/tags?post=2779"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}