Development of Thai word segmentation technique for solving problems with unknown words

Abstract

© 2015 IEEE. This research has an objective to develop an efficient technique for Thai word segmentation, especially those nonexistent in dictionaries. The researchers developed a model for Thai word segmentation by relying on grammar and rules to solve the problems with words not found in dictionaries. The model was intended to be used as the best approach of word segmentation, which applied the segmentation technique developed by the researchers called PTTSF (Parsing Thai Text with Syntax and Feature of Word). The system of this technique operates by starting from finding the boundary of each word in Thai sentences. If the system finds a word that does not exist in the dictionary or a meaningless word, it would not be able to solve the problem with the method of longest-matching algorithm. Therefore, rules need to be specified to solve such problems. In this study, 28 rules were created and Digraph method was used to find a pattern of word segmentation with the highest probability based on the grammatical principle. After the procedure of finding boundary of the word, the result from correct word segmentation can be used for further processes. In analyzing efficiency of the system, its accuracy in word segmentation was the main point of concern. The results revealed that the derived mapping technique could solve the problem concerned with segmentation words that do not exist in the dictionary with an average accuracy over 90% of the whole document. However, the researchers encountered with ambiguous words problem. Although this problem rarely occurs, it could affect accuracy of word segmentation.