{
" Vision Transformer (ViT)": " \u89c6\u89c9\u53d8\u538b\u5668 (ViT)",
"<h1><a href=\"https://nn.labml.ai/transformer/vit/index.html\">Vision Transformer (ViT)</a></h1>\n<p>This is a <a href=\"https://pytorch.org\">PyTorch</a> implementation of the paper <a href=\"https://arxiv.org/abs/2010.11929\">An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale</a>.</p>\n<p>Vision transformer applies a pure transformer to images without any convolution layers. They split the image into patches and apply a transformer on patch embeddings. <a href=\"https://nn.labml.ai/transformer/vit/index.html#PathEmbeddings\">Patch embeddings</a> are generated by applying a simple linear transformation to the flattened pixel values of the patch. Then a standard transformer encoder is fed with the patch embeddings, along with a classification token <span translate=no>_^_0_^_</span>. The encoding on the <span translate=no>_^_1_^_</span> token is used to classify the image with an MLP.</p>\n<p>When feeding the transformer with the patches, learned positional embeddings are added to the patch embeddings, because the patch embeddings do not have any information about where that patch is from. The positional embeddings are a set of vectors for each patch location that get trained with gradient descent along with other parameters.</p>\n<p>ViTs perform well when they are pre-trained on large datasets. The paper suggests pre-training them with an MLP classification head and then using a single linear layer when fine-tuning. The paper beats SOTA with a ViT pre-trained on a 300 million image dataset. They also use higher resolution images during inference while keeping the patch size the same. The positional embeddings for new patch locations are calculated by interpolating learning positional embeddings.</p>\n<p>Here's <a href=\"https://nn.labml.ai/transformer/vit/experiment.html\">an experiment</a> that trains ViT on CIFAR-10. This doesn't do very well because it's trained on a small dataset. It's a simple experiment that anyone can run and play with ViTs. </p>\n": "<h1><a href=\"https://nn.labml.ai/transformer/vit/index.html\">\u89c6\u89c9\u53d8\u538b\u5668 (ViT)</a></h1>\n<p>\u8fd9\u662f\u8bba\u6587\u300a\u56fe\u50cf<a href=\"https://arxiv.org/abs/2010.11929\">\u503c\u5f97 16x16 Words\uff1a\u5927\u89c4\u6a21\u56fe\u50cf\u8bc6\u522b\u7684\u53d8\u5f62\u91d1\u521a\u300b\u7684 PyTorc</a> <a href=\"https://pytorch.org\">h</a> \u5b9e\u73b0\u3002</p>\n<p>\u89c6\u89c9\u53d8\u6362\u5668\u5c06\u7eaf\u53d8\u6362\u5668\u5e94\u7528\u4e8e\u6ca1\u6709\u4efb\u4f55\u5377\u79ef\u5c42\u7684\u56fe\u50cf\u3002\u4ed6\u4eec\u5c06\u56fe\u50cf\u62c6\u5206\u4e3a\u8865\u4e01\uff0c\u7136\u540e\u5728\u8865\u4e01\u5d4c\u5165\u4e0a\u5e94\u7528\u53d8\u6362\u5668\u3002<a href=\"https://nn.labml.ai/transformer/vit/index.html#PathEmbeddings\">\u8865\u4e01\u5d4c\u5165</a>\u662f\u901a\u8fc7\u5bf9\u9762\u7247\u7684\u6241\u5e73\u5316\u50cf\u7d20\u503c\u5e94\u7528\u7b80\u5355\u7684\u7ebf\u6027\u53d8\u6362\u6765\u751f\u6210\u7684\u3002\u7136\u540e\u5c06\u6807\u51c6\u53d8\u538b\u5668\u7f16\u7801\u5668\u4e0e\u8865\u4e01\u5d4c\u5165\u4ee5\u53ca\u5206\u7c7b\u4ee4\u724c\u4e00\u8d77\u9988\u9001<span translate=no>_^_0_^_</span>\u3002<span translate=no>_^_1_^_</span>\u4ee4\u724c\u4e0a\u7684\u7f16\u7801\u7528\u4e8e\u4f7f\u7528 MLP \u5bf9\u56fe\u50cf\u8fdb\u884c\u5206\u7c7b\u3002</p>\n\u5411@@ <p>\u53d8\u538b\u5668\u63d0\u4f9b\u8865\u4e01\u65f6\uff0c\u5b66\u4e60\u7684\u4f4d\u7f6e\u5d4c\u5165\u4f1a\u6dfb\u52a0\u5230\u8865\u4e01\u5d4c\u5165\u4e2d\uff0c\u56e0\u4e3a\u8865\u4e01\u5d4c\u5165\u6ca1\u6709\u5173\u4e8e\u8865\u4e01\u6765\u81ea\u4f55\u5904\u7684\u4efb\u4f55\u4fe1\u606f\u3002\u4f4d\u7f6e\u5d4c\u5165\u662f\u6bcf\u4e2a\u9762\u7247\u4f4d\u7f6e\u7684\u4e00\u7ec4\u5411\u91cf\uff0c\u8fd9\u4e9b\u5411\u91cf\u901a\u8fc7\u68af\u5ea6\u4e0b\u964d\u4ee5\u53ca\u5176\u4ed6\u53c2\u6570\u8fdb\u884c\u8bad\u7ec3\u3002</p>\n<p>VIT \u5728\u5927\u578b\u6570\u636e\u96c6\u4e0a\u8fdb\u884c\u9884\u8bad\u7ec3\u65f6\u8868\u73b0\u826f\u597d\u3002\u672c\u6587\u5efa\u8bae\u4f7f\u7528 MLP \u5206\u7c7b\u5934\u5bf9\u5b83\u4eec\u8fdb\u884c\u9884\u8bad\u7ec3\uff0c\u7136\u540e\u5728\u5fae\u8c03\u65f6\u4f7f\u7528\u5355\u4e2a\u7ebf\u6027\u5c42\u3002\u8be5\u8bba\u6587\u57283\u4ebf\u5f20\u56fe\u50cf\u6570\u636e\u96c6\u4e0a\u9884\u5148\u8bad\u7ec3\u4e86ViT\uff0c\u51fb\u8d25\u4e86SOTA\u3002\u5b83\u4eec\u8fd8\u5728\u63a8\u7406\u8fc7\u7a0b\u4e2d\u4f7f\u7528\u66f4\u9ad8\u5206\u8fa8\u7387\u7684\u56fe\u50cf\uff0c\u540c\u65f6\u4fdd\u6301\u8865\u4e01\u5927\u5c0f\u4e0d\u53d8\u3002\u65b0\u9762\u7247\u4f4d\u7f6e\u7684\u4f4d\u7f6e\u5d4c\u5165\u662f\u901a\u8fc7\u63d2\u503c\u5b66\u4e60\u4f4d\u7f6e\u5d4c\u5165\u6765\u8ba1\u7b97\u7684\u3002</p>\n<p><a href=\"https://nn.labml.ai/transformer/vit/experiment.html\">\u8fd9\u662f\u4e00\u4e2a\u5728 CIFAR-10 \u4e0a\u8bad\u7ec3 ViT \u7684\u5b9e\u9a8c</a>\u3002\u8fd9\u6837\u505a\u4e0d\u592a\u597d\uff0c\u56e0\u4e3a\u5b83\u662f\u5728\u4e00\u4e2a\u5c0f\u6570\u636e\u96c6\u4e0a\u8bad\u7ec3\u7684\u3002\u8fd9\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u5b9e\u9a8c\uff0c\u4efb\u4f55\u4eba\u90fd\u53ef\u4ee5\u8fd0\u884c\u548c\u73a9VIT\u3002</p>\n"
}
wehub-resource-sync