A study about the influence of text specificity in the perceived helpfulness classification of online reviews

Abstract

Online reviews are valuable sources of information to support the decision-making process, both for individuals and companies. Nevertheless, the large volume of reviews that have a low quality hinders the process of gathering helpful information from those reviews. Several retailers’ websites provide a voting system to allow customers to evaluate product reviews as helpful or not. However, those votes are often biased and most of the reviews do not receive any votes at all. Besides that, several websites do not even have this voting mechanism or any other component for organizing the reviews in terms of their helpfulness. Therefore, classifying reviews according to their helpfulness has paramount importance in facilitating access to truly informative content. In this context, previous studies have unveiled several features and architectures that are beneficial for the perceived helpfulness prediction. In the present work, we argue that text specificity, defined as the level of details expressed in a text, can influence the perception of review helpfulness and, consequently, can also be a novel useful linguistic aspect for modeling the helpfulness prediction. We proposed two approaches to incorporate the specificity aspect into helpfulness classification models: i) using hand-crafted features based on text specificity and ii) using the review specificity prediction as an auxiliary task in a Multitask Learning (MTL) setting. First, we conducted an unsupervised domain adaptation approach [Ko, Durrett and Li 2019] to label text specificity scores on sentences from online reviews automatically. To evaluate the different trained models using this approach, we proposed a measure named Specificity Prediction Evaluation (SPE), which is based on the assumption that, on average, reliable specific sentences tend to be longer than reliable general sentences [Li and Nenkova 2015]. For the 18 collections of online reviews used in all of our experiments, we could achieve more reliable specificity predictions, according to SPE, by varying only the training set size and the number of training epochs. Finally, we performed experiments to assess the performance of the helpfulness classification models using two neural architectures: Convolutional Neural Network (CNN) [Kim 2014] and Bidirectional Encoder Representations from Transformers (BERT) [Devlin et al. 2019]. In summary, using balanced datasets, the perceived helpfulness classification models, embodied with text specificity - either as features or MTL - showed significantly higher precision results in comparison to a popular SVM baseline when using CNN. With BERT, the experiments showed that MTL outperformed the single-task models for most of the 18 datasets and both accuracy and precision were improved compared to the SVM baseline.