This is an implementation of nucleus sampling, introduced in the paper The Curious Case of Neural Text Degeneration.
\nThe paper discusses the problems with other sampling methods such as Beam Search, Pure sampling, Temperature sampling, and Top-k sampling. The paper introduces the idea of nucleus sampling, which practically performs better than other sampling methods for text generation.
\nNucleus sampling first picks a subset of the vocabulary _^_0_^_, where _^_1_^_ is smallest set of tokens such that
\n_^_2_^_
\nThat is, we pick the highest probable tokens until the sum of their probabilities is less that _^_3_^_.
\nThen we sample from the selected tokens.
\nHere's an experiment that uses these sampling techniques.
\n": "\u8fd9\u662f\u539f\u5b50\u6838\u91c7\u6837\u7684\u4e00\u79cd\u5b9e\u73b0\uff0c\u5728\u8bba\u6587\u300a\u795e\u7ecf\u6587\u672c\u53d8\u6027\u7684\u597d\u5947\u6848\u4f8b\u300b\u4e2d\u8fdb\u884c\u4e86\u4ecb\u7ecd\u3002
\n\u672c\u6587\u8ba8\u8bba\u4e86\u5176\u4ed6\u91c7\u6837\u65b9\u6cd5\uff08\u4f8b\u5982\u5149\u675f\u641c\u7d22\u3001\u7eaf\u91c7\u6837\u3001\u6e29\u5ea6\u91c7\u6837\u548cT op-K\u91c7\u6837\uff09\u5b58\u5728\u7684\u95ee\u9898\u3002\u672c\u6587\u4ecb\u7ecd\u4e86\u539f\u5b50\u6838\u91c7\u6837\u7684\u6982\u5ff5\uff0c\u5728\u6587\u672c\u751f\u6210\u65b9\u9762\uff0c\u6838\u91c7\u6837\u7684\u6548\u679c\u5b9e\u9645\u4e0a\u6bd4\u5176\u4ed6\u91c7\u6837\u65b9\u6cd5\u8981\u597d\u3002
\nNucleus \u91c7\u6837\u9996\u5148\u9009\u62e9\u8bcd\u6c47\u7684\u4e00\u4e2a\u5b50\u96c6_^_0_^_\uff0c\u5176\u4e2d_^_1_^_\u662f\u6700\u5c0f\u7684\u4ee4\u724c\u96c6\u5408
\n_^_2_^_
\n\u4e5f\u5c31\u662f\u8bf4\uff0c\u6211\u4eec\u9009\u62e9\u53ef\u80fd\u6027\u6700\u9ad8\u7684\u4ee3\u5e01\uff0c\u76f4\u5230\u5b83\u4eec\u7684\u6982\u7387\u603b\u548c\u5c0f\u4e8e\u8be5\u503c\u4e3a\u6b62_^_3_^_\u3002
\n\u7136\u540e\u6211\u4eec\u4ece\u9009\u5b9a\u7684\u4ee4\u724c\u4e2d\u62bd\u6837\u3002
\n\u8fd9\u662f\u4e00\u4e2a\u4f7f\u7528\u8fd9\u4e9b\u91c7\u6837\u6280\u672f\u7684\u5b9e\u9a8c\u3002
\n", "\n": "\n", "
Sample from logits with Nucleus Sampling
\n": "\u4f7f\u7528 Nucleus \u91c7\u6837\u4ece logits \u4e2d\u63d0\u53d6\u6837\u672c
\n", "Find the cumulative sums less than _^_0_^_.
\n": "\u627e\u51fa\u5c0f\u4e8e\u7684\u7d2f\u8ba1\u603b\u548c_^_0_^_\u3002
\n", "Get log probabilities and mask out the non-nucleus
\n": "\u83b7\u53d6\u5bf9\u6570\u6982\u7387\u5e76\u63a9\u76d6\u975e\u6838
\n", "Get probabilities _^_0_^_
\n": "\u83b7\u53d6\u6982\u7387_^_0_^_
\n", "Get the actual indexes
\n": "\u83b7\u53d6\u5b9e\u9645\u7d22\u5f15
\n", "Get the cumulative sum of probabilities in the sorted order
\n": "\u6309\u6392\u5e8f\u987a\u5e8f\u83b7\u53d6\u6982\u7387\u7684\u7d2f\u79ef\u603b\u548c
\n", "Prepend ones so that we add one token after the minimum number of tokens with cumulative probability less that _^_0_^_.
\n": "\u5728@@\u524d\u9762\u52a0\u4e00\u4e2a\uff0c\u8fd9\u6837\u6211\u4eec\u5c31\u53ef\u4ee5\u5728\u7d2f\u79ef\u6982\u7387\u5c0f\u4e8e\u8be5\u503c\u7684\u6700\u5c0f\u4ee3\u5e01\u6570\u91cf\u4e4b\u540e\u6dfb\u52a0\u4e00\u4e2a\u4ee4\u724c_^_0_^_\u3002
\n", "Sample from the sampler
\n": "\u6765\u81ea\u91c7\u6837\u5668\u7684\u6837\u672c
\n", "Softmax to compute _^_0_^_ from the logits
\n": "\u8981\u6839\u636e\u5bf9\u6570\u8ba1\u7b97_^_0_^_\u7684 softmax
\n", "Sort probabilities in descending order
\n": "\u6309\u964d\u5e8f\u5bf9\u6982\u7387\u8fdb\u884c\u6392\u5e8f
\n", "