A Comparison of Thai Sentence Boundary Detection Approaches Using Online Product Review Data

Abstract

© 2021, Springer Nature Switzerland AG. In Natural Language Processing (NLP), the goal of sentence boundary detection (SBD) is to identify sentence boundaries in a phrase, paragraph, or document, which can be used in current NLP applications, including sentimental analysis, contextual chatbot, and machine translation, etc. Previous studies and existing NLP libraries often provide a straightforward approach to the task; for instance, they assume that a sentence always ends with certain punctuation symbols such as a period, a semicolon, a exclamation mark, or a question mark. The mentioned approach is impractical for other languages, such as Thai, where there is no symbol to designate where a sentence ends. With regard to developing an effective sentimental analysis or machine translation for the Thai language, a solid effort in detecting sentence boundary is needed. There is also as a need validating the SBD model against a real-world dataset, by involving the use of an online textual corpus. This paper attempts to compare Condition Random Fields (CRF) and Bidirectional Long-Short Term Memory with CRF layer (BiLSTM-CRF) on the online textual dataset. We scraped our own corpus from the top Thai web forums through the use of a Scrapy web-crawling framework. In the paper, 2,496 comments related to beauty product reviews were manually segmented by a Thai linguistic expert. Our experimental results revealed that the CRF based on the word-based labelling approach with widow size outperformed the BiLSTM-CRF.