Effer für Wissenschaft

A Single event which happens once and whose identity is limited to that one happening or a Single object or thing which is in some single place at any one instant of time, such event or thing being significant only as occurring just when and where it does, such as this or that word on a single line of a single page of a single copy of a book, I will venture to call a Token.

By the term "type/token ratio", we understand the ratio of the vocabulary in a given text, or sample from it, to the total number of words comprised in the text or the sample.

In the first stage of the automatic language translation process, each kana in an input text will be transformed into two tokens for two roman letters so as to preserve a one-to-one correspondence between kanas and their correspondent roman letters. In a kana-kanji input text, however, each kanji will be transformed into an irreducible unit token.