Thai word safe segmentation with bounding extension for data indexing in search engine

Abstract

Word segmentation ambiguity in Thai language affects data indexing process by creating the inverted index relatively to the segmentation results. This phenomenon leads to unreasonable search result. This article proposes Thai word Safe segmentation algorithm using dictionary to solve this problem so that all different terms in an ambiguous part of the sentence are queryable. Next, it shows the bounding extension to improve Safe segmentation performance. It also compares several off-the-shelf implementations of the trie data structure which -we believe- is the best data structure for dictionary-based Thai word segmentation and compares the efficiency of serializable libraries for de-serializing trie in the analyzer’s initial state. Finally, it evaluates the Safe segmentation with several implementations called Safe Analyzer. The experimental results also show that the linked-list Trie and Protostuff library give the outstanding results. The Safe segmentation can definitely solve the ambiguity problem but still it could not solve the misspell within text accurately.