In the context of LLMs, what is a token?

In the context of LLMs, what is a token?

• A. Individual characters only
• B. Complete sentences treated as atomic units
• C. Common words or word pieces that the model uses as units
• D. Numbers only

Answer: (C)

In LLMs, a token is the basic unit of text that the model processes. Tokens are not limited to single characters or entire sentences; instead, text is broken into pieces from the model’s vocabulary. This means common words can be stored as one token, while rarer words are split into smaller subword units the model has seen during training. The tokenizer may also include punctuation or numbers as tokens, depending on the system. The key idea is that a token is a piece of text the model uses to understand and predict the next piece.

In LLMs, a token is the basic unit of text that the model processes. Tokens are not limited to single characters or entire sentences; instead, text is broken into pieces from the model’s vocabulary. This means common words can be stored as one token, while rarer words are split into smaller subword units the model has seen during training. The tokenizer may also include punctuation or numbers as tokens, depending on the system. The key idea is that a token is a piece of text the model uses to understand and predict the next piece.