https:\/\/sites.google.com\/view\/stablediffusion-with-brain\/<\/a><\/p>\n![\"\"](\"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM-1024x481.jpg\")
<\/p>\n
Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straightforward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs….<\/p>\n
Code Coming Soon!…<\/p>\n","protected":false},"excerpt":{"rendered":"
*That’s not an artwork, but it is about human vision, and it’s really quite out-there. https:\/\/sites.google.com\/view\/stablediffusion-with-brain\/ Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3170","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"yoast_head":"\n
Stable Diffusion's brain visualization | Artmaker Blog<\/title>\n<meta name=\"description\" content=\"Stable Diffusion's brain visualization | Artmaker Blog\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Stable Diffusion's brain visualization | Artmaker Blog\" \/>\n<meta property=\"og:description\" content=\"Stable Diffusion's brain visualization | Artmaker Blog\" \/>\n<meta property=\"og:url\" content=\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/\" \/>\n<meta property=\"og:site_name\" content=\"Artmaker Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-05T07:36:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM-1024x481.jpg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bruce Sterling\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/\",\"name\":\"Artmaker Blog\",\"description\":\"on Toshareproject.it - curated by Bruce Sterling\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#primaryimage\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM.jpg\",\"contentUrl\":\"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM.jpg\",\"width\":2462,\"height\":1156},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#webpage\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/\",\"name\":\"Stable Diffusion's brain visualization | Artmaker Blog\",\"isPartOf\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#primaryimage\"},\"datePublished\":\"2023-03-05T07:36:11+00:00\",\"dateModified\":\"2023-03-05T07:36:11+00:00\",\"author\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\"},\"description\":\"Stable Diffusion's brain visualization | Artmaker Blog\",\"breadcrumb\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/toshareproject.it\/artmakerblog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Stable Diffusion’s brain visualization\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\",\"name\":\"Bruce Sterling\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#personlogo\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"caption\":\"Bruce Sterling\"},\"description\":\"Art director at Share Festival, author and journalist\",\"sameAs\":[\"http:\/\/toshareproject.it\/tomorrowart\"],\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Stable Diffusion's brain visualization | Artmaker Blog","description":"Stable Diffusion's brain visualization | Artmaker Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/","og_locale":"en_GB","og_type":"article","og_title":"Stable Diffusion's brain visualization | Artmaker Blog","og_description":"Stable Diffusion's brain visualization | Artmaker Blog","og_url":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/","og_site_name":"Artmaker Blog","article_published_time":"2023-03-05T07:36:11+00:00","og_image":[{"url":"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM-1024x481.jpg"}],"twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bruce Sterling","Estimated reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/toshareproject.it\/artmakerblog\/#website","url":"https:\/\/toshareproject.it\/artmakerblog\/","name":"Artmaker Blog","description":"on Toshareproject.it - curated by Bruce Sterling","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"ImageObject","@id":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#primaryimage","inLanguage":"en-GB","url":"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM.jpg","contentUrl":"https:\/\/toshareproject.it\/artmakerblog\/wp-content\/uploads\/2023\/03\/Screenshot-2023-03-05-at-8.34.10-AM.jpg","width":2462,"height":1156},{"@type":"WebPage","@id":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#webpage","url":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/","name":"Stable Diffusion's brain visualization | Artmaker Blog","isPartOf":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#primaryimage"},"datePublished":"2023-03-05T07:36:11+00:00","dateModified":"2023-03-05T07:36:11+00:00","author":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085"},"description":"Stable Diffusion's brain visualization | Artmaker Blog","breadcrumb":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/toshareproject.it\/artmakerblog\/stable-diffusions-brain-visualization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/toshareproject.it\/artmakerblog\/"},{"@type":"ListItem","position":2,"name":"Stable Diffusion’s brain visualization"}]},{"@type":"Person","@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085","name":"Bruce Sterling","image":{"@type":"ImageObject","@id":"https:\/\/toshareproject.it\/artmakerblog\/#personlogo","inLanguage":"en-GB","url":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","caption":"Bruce Sterling"},"description":"Art director at Share Festival, author and journalist","sameAs":["http:\/\/toshareproject.it\/tomorrowart"],"url":"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/"}]}},"_links":{"self":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/3170","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/comments?post=3170"}],"version-history":[{"count":1,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/3170\/revisions"}],"predecessor-version":[{"id":3172,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/3170\/revisions\/3172"}],"wp:attachment":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/media?parent=3170"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/categories?post=3170"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/tags?post=3170"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}