A Comparison of Open Source Web Crawlers for E-Commerce Websites

Abstract

© 2020 IEEE. Web crawlers are important tools for retrieving data such as text or image from the internet. It is an automated program that can retrieve information from the websites. E-commerce websites are essential areas for crawler applications. Moreover, large-scale crawling involves many problems on the internet. Poor performing web crawlers can waste many resources for development and maintenance. Thus, choosing a suitable open source crawler becomes a huge challenge. This paper attempts to review the published previous studies on open source crawlers. The paper focuses on summarizing the performance evaluation methods of open source web crawlers, possible research trends and related research gaps. In addition, a proposed framework of the open source crawler evaluation was presented.