Ha, I like that, but to a layman it looks like the author is trying to say "it can't count the R's in strawberry because the letters are overlapping", which is funny.
Cute idea, but is this fixed width words, and not fixed width tokens? Isn’t an ‘ing’ ending normal a separate token from the root? Isn’t ‘strawberry’ two tokens?
I think it is: look at the "in" which is likely one token, "strawberry" looks like it is 2x the length of "in". Same for the last word, "ing" seems less compressed than "bear" right before it.
Looking at the first several examples, shorter (one, two, three-) letter words are not stretched to the full ‘fixed’ width. It’s the whitespace around them that is fixed. The only example in the first three screenshots of splitting a word into tokes is ‘vibes’ which turns into ‘vib’ and ‘es’. I’m skeptical of the -ing and -tion words being single tokens as shown.
3 comments