Tag: tiktoken
-
Simon Willison’s Weblog: A warning about tiktoken, BPE, and OpenAI models
Source URL: https://simonwillison.net/2024/Nov/21/a-warning-about-tiktoken/#atom-everything Source: Simon Willison’s Weblog Title: A warning about tiktoken, BPE, and OpenAI models Feedly Summary: A warning about tiktoken, BPE, and OpenAI models Tom MacWright warns that OpenAI’s tiktoken Python library has a surprising performance profile: it’s superlinear with the length of input, meaning someone could potentially denial-of-service you by sending you…