GitHub BlogNEWS
GitHub BlogNEWS
So many tokens, so little time: Introducing a faster, more flexible byte-pair tokenizer
We released a new open source byte-pair tokenizer that is faster and more flexible than popular alternatives.
Read at original source ↗
Comments
No comments yet. Be the first to contribute.
✎ Write a comment