Byte-pair encoding tokenizerCodingImplement byte-pair encoding: repeatedly merge the most frequent adjacent pair until the vocabulary reaches its target size, then tokenize with the learned merges. Reported on…
Last asked Sep 2026
#tokenization
Tokenizer design and KL-regularized fine-tuningML designDiscuss how you would design a tokenizer for a language model and the trade-offs involved. Then explain supervised fine-tuning with a KL penalty that keeps the model close to its…Last asked Jun 2026
Speed up a byte tokenizer and estimate token countsCodingYou are given a slow but correct reference tokenizer over bytes. Explain precisely what it does, then write a faster version that returns identical output. Finally, estimate the…Last asked Jan 2026
Merge known bigrams in a sentence, leftmost firstCodingWrite a function that splits a sentence into tokens using a dictionary of known two-word phrases (bigrams). - text is a non-empty string of lowercase words separated by spaces. -…- Apply ordered token merges to a stringCodingMany language models read text as a list of token ids. In this scheme, every token after the single characters is built from two earlier tokens, and turning text into ids means…