Identify tokens

Tokens identifies and separates each word into one or more tokens through advanced statistical modeling. A token is an atomic element such as a word, number, possessive affix, or punctuation. The resulting token output minimizes index size, enhances search accuracy, and increases relevancy.