You know what's also faster to roughly get the amount of tokens? string.length/5 | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		janwilmake 7 months ago \| parent \| context \| favorite \| on: Show HN: TokenDagger – A tokenizer faster than Ope... You know what's also faster to roughly get the amount of tokens? string.length/5

_flux 7 months ago | [–]

It is not helpful in actual tokenization, though.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact