Token (parser) Guide, Meaning , Facts, Information and Description
A token, in computing, is a segment of text, regardess whether it be readable or comprised of symbols. Tokens are generally defined abstractly in a context free grammar, which is fed into a program such as yacc which checks the stream of tokens for conformity to this grammar. The tokens themselves are detected, from a larger block of text, by a lexical analyzer which attempts to match regular expressions against the text.Older interpreterss and older computers with BASIC in ROM, for example, would tokenize the program, turning the instructions from the programmer into byte-codes that were more easily processed by a virtual machine or processor. By doing so they would gain a bit more speed and faster execution. Some examples of this are the, O-code, P-code, Z-machine, and byte-codes used in different VM systems.