← Questions
Speed up a byte tokenizer and estimate token counts
0
I was asked this
No reports yet. Sign in to say where you were asked it.
You are given a slow but correct reference tokenizer over bytes. Explain precisely what it does, then write a faster version that returns identical output. Finally, estimate the token count of a large input within a fixed sampling budget without tokenizing all of it.
No official answer yet. Write yours on the right and run it.
Nothing published yet.
Sign in to submit your code; every attempt is kept here.
Write and test your solution herePython in your browser, the question's tests level by level, every submission kept for you.Sign in to start coding