Active learning using pessimistic expectation estimators

Active learning is the process in which unlabeled instances are dynamically selected for expert labelling, and then a classifier is trained on the labeled data. Active learning is particularly useful when there is a large set of unlabeled instances, and acquiring a label is costly. In business scenarios such as direct marketing, active learning can be used to indicate which customer to approach such that the potential benefit from the approached customer can cover the cost of approach. This paper presents a new algorithm for cost-sensitive active learning using a conditional expectation estimator. The new estimator focuses on acquisitions that are likely to improve the profit. Moreover, we investigate simulated annealing techniques to combine exploration with exploitation in the classifier construction. Using five evaluation metrics, we evaluated the algorithm on four benchmark datasets. The results demonstrate the superiority of the proposed method compared to other algorithms.