US Patent 9208134 Methods and systems for tokenizing multilingual textual documents

Methods and systems for tokenizing multilingual textual documents are provided. A method implemented in a computer infrastructure, includes determining an attribute of a current character in input text, the attribute of the current character indicating one or more classes of characters the current character is assigned thereto. The method further includes determining one or more attributes of one or more next characters in the input text, the one or more attributes of the one or more next characters indicating the one or more classes the one or more next characters are assigned thereto. The method further includes constructing a token of the input text that includes the current character and the one or more next characters, the attribute of the current character and the one or more attributes of the one or more next characters intersecting with each other.