{"id":1338,"date":"2021-12-08T10:01:53","date_gmt":"2021-12-08T10:01:53","guid":{"rendered":"https:\/\/toshareproject.it\/artmakerblog\/?p=1338"},"modified":"2021-12-08T10:01:53","modified_gmt":"2021-12-08T10:01:53","slug":"transformers","status":"publish","type":"post","link":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/","title":{"rendered":"Transformers"},"content":{"rendered":"<p><a href=\"https:\/\/e2eml.school\/transformers.html\">https:\/\/e2eml.school\/transformers.html<\/a><\/p>\n<p>Transformers were introduced in this 2017 paper as a tool for sequence transduction\u2014converting one sequence of symbols to another. The most popular examples of this are translation, as in English to German.<\/p>\n<p>          It has also been modified to perform sequence completion\u2014given a starting prompt, carry on in the same vein and style. They have quickly become an indispensible tool for research and product development in natural language processing.<\/p>\n<p>          Before we start, just a heads-up. We&#8217;re going to be talking a lot about matrix multiplications and touching on backpropagation (the algorithm for training the model), but you don&#8217;t need to know any of it beforehand. We&#8217;ll add the concepts we need one at a time, with explanation.<\/p>\n<p>          This isn&#8217;t a short journey, but I hope you&#8217;ll be glad you came.<\/p>\n<p>            \u25ba One-hot encoding<\/p>\n<p>            \u25ba Dot product<\/p>\n<p>            \u25ba Matrix multiplication<\/p>\n<p>            \u25ba Matrix multiplication as a table lookup<\/p>\n<p>            \u25ba First order sequence model<\/p>\n<p>            \u25ba Second order sequence model<\/p>\n<p>            \u25ba Second order sequence model with skips<\/p>\n<p>            \u25ba Masking<\/p>\n<p>            \u25ba Rest Stop and an Off Ramp<\/p>\n<p>            \u25ba Attention as matrix multiplication<\/p>\n<p>            \u25ba<br \/>\n              Second order sequence model as matrix multiplications<\/p>\n<p>            \u25ba Sequence completion<\/p>\n<p>            \u25ba Embeddings<\/p>\n<p>            \u25ba Positional encoding<\/p>\n<p>            \u25ba De-embeddings<\/p>\n<p>            \u25ba Softmax<\/p>\n<p>            \u25ba Multi-head attention<\/p>\n<p>            \u25ba Single head attention revisited<\/p>\n<p>            \u25ba Skip connection<\/p>\n<p>            \u25ba Multiple layers<\/p>\n<p>            \u25ba Decoder stack<\/p>\n<p>            \u25ba Encoder stack<\/p>\n<p>            \u25ba Cross-attention<\/p>\n<p>            \u25ba Tokenizing<\/p>\n<p>            \u25ba Byte pair encoding<\/p>\n<p>            \u25ba Audio input<\/p>\n<p>            \u25ba Resources and credits&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/e2eml.school\/transformers.html Transformers were introduced in this 2017 paper as a tool for sequence transduction\u2014converting one sequence of symbols to another. The most popular examples of this are translation, as in English to German. It has also been modified to perform sequence completion\u2014given a starting prompt, carry on in the same vein and style. They have [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1338","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v17.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Transformers | Artmaker Blog<\/title>\n<meta name=\"description\" content=\"Transformers | Artmaker Blog\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Transformers | Artmaker Blog\" \/>\n<meta property=\"og:description\" content=\"Transformers | Artmaker Blog\" \/>\n<meta property=\"og:url\" content=\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/\" \/>\n<meta property=\"og:site_name\" content=\"Artmaker Blog\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-08T10:01:53+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bruce Sterling\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/\",\"name\":\"Artmaker Blog\",\"description\":\"on Toshareproject.it - curated by Bruce Sterling\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#webpage\",\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/\",\"name\":\"Transformers | Artmaker Blog\",\"isPartOf\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#website\"},\"datePublished\":\"2021-12-08T10:01:53+00:00\",\"dateModified\":\"2021-12-08T10:01:53+00:00\",\"author\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\"},\"description\":\"Transformers | Artmaker Blog\",\"breadcrumb\":{\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/toshareproject.it\/artmakerblog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Transformers\"}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085\",\"name\":\"Bruce Sterling\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/toshareproject.it\/artmakerblog\/#personlogo\",\"inLanguage\":\"en-GB\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g\",\"caption\":\"Bruce Sterling\"},\"description\":\"Art director at Share Festival, author and journalist\",\"sameAs\":[\"http:\/\/toshareproject.it\/tomorrowart\"],\"url\":\"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Transformers | Artmaker Blog","description":"Transformers | Artmaker Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/","og_locale":"en_GB","og_type":"article","og_title":"Transformers | Artmaker Blog","og_description":"Transformers | Artmaker Blog","og_url":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/","og_site_name":"Artmaker Blog","article_published_time":"2021-12-08T10:01:53+00:00","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bruce Sterling","Estimated reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebSite","@id":"https:\/\/toshareproject.it\/artmakerblog\/#website","url":"https:\/\/toshareproject.it\/artmakerblog\/","name":"Artmaker Blog","description":"on Toshareproject.it - curated by Bruce Sterling","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/toshareproject.it\/artmakerblog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#webpage","url":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/","name":"Transformers | Artmaker Blog","isPartOf":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#website"},"datePublished":"2021-12-08T10:01:53+00:00","dateModified":"2021-12-08T10:01:53+00:00","author":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085"},"description":"Transformers | Artmaker Blog","breadcrumb":{"@id":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/toshareproject.it\/artmakerblog\/transformers\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/toshareproject.it\/artmakerblog\/transformers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/toshareproject.it\/artmakerblog\/"},{"@type":"ListItem","position":2,"name":"Transformers"}]},{"@type":"Person","@id":"https:\/\/toshareproject.it\/artmakerblog\/#\/schema\/person\/6f20726ed2761431f3e0ff4e096c3085","name":"Bruce Sterling","image":{"@type":"ImageObject","@id":"https:\/\/toshareproject.it\/artmakerblog\/#personlogo","inLanguage":"en-GB","url":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c390e8ed4db57a34278dcf667f928a643cf769a865c8a8632dcd310412bb9a99?s=96&d=mm&r=g","caption":"Bruce Sterling"},"description":"Art director at Share Festival, author and journalist","sameAs":["http:\/\/toshareproject.it\/tomorrowart"],"url":"https:\/\/toshareproject.it\/artmakerblog\/author\/brucesterling\/"}]}},"_links":{"self":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/1338","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/comments?post=1338"}],"version-history":[{"count":1,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/1338\/revisions"}],"predecessor-version":[{"id":1339,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/posts\/1338\/revisions\/1339"}],"wp:attachment":[{"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/media?parent=1338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/categories?post=1338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/toshareproject.it\/artmakerblog\/wp-json\/wp\/v2\/tags?post=1338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}