Cheap Learning: Maximizing Performance of Language Models for Social Data Science Using Minimal Data

Abstract

The field of machine learning has recently made significant progress in reducing the requirements for labeled training data when building new models. These “cheaper” learning techniques hold significant potential for the social sciences, where development of large labeled training datasets is often a significant practical impediment. In this article we review three “cheap” techniques that have developed in recent years: Weak supervision, transfer learning and prompt engineering. For the latter, we also review the particular case of zero-shot prompting of large language models. For each technique, we provide a guide of how it works and demonstrate its application and the presence of systematic biases across two different and realistic social science tasks paired with three different dataset makeups. We show good performance for all techniques and we demonstrate how prompting of large language models can achieve high accuracy at very low cost, but biases must be considered.