Artificial intelligence (AI) chatbots powered by large language models (LLMs) such as ChatGPT are rapidly gaining popularity as labour-augmenting tools. This paper is for sociologists seeking to make the best use of this technology in their work. It presents a practice-oriented framework for using AI chatbots in sociology, building on considerations of the technical conditions of LLMs to introduce both a task categorization and the concept of a ‘knowledge funnel’. This model illustrates the relationship between the scope of knowledge and accuracy in outputs to guide sociologists in evaluating the reliability and applicability of AI-generated content in their research. The main argument driving this article is to establish a paradigm of ‘augmented sociology’ that focuses on human–AI interaction and understands LLMs as a resource rather than as a replacement. This augmentation manifests itself clearly in dialogic ideation, enhancing research by bridging domains, and broad methodological assistance. The paper's primary contribution lies in introducing specific terminologies and actionable strategies for sociologists to integrate LLM chatbots creatively and effectively in their work, filling a significant gap in the current academic understanding of generative AI's role in sociology.
AdamsGFabbriALadhakF, et al. (2023) From sparse to dense: GPT-4 summarization with chain of density prompting. Available from: http://arxiv.org/abs/2309.04269 (accessed 25 September 2023).
2.
AdessoG (2023) Towards the ultimate brain: Exploring scientific discovery with ChatGPT AI. AI Magazine44(3): 328–342.
3.
AlasadiEABaizCR (2023) Generative AI in education and research: Opportunities, concerns, and solutions. Journal of Chemical Education100(8): 2965–2971.
4.
AyersJWPoliakADredzeM, et al. (2023) Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Internal Medicine183(6): 589–596.
5.
AzariaAAzoulayRRechesS (2023) ChatGPT is a remarkable tool – for experts. Available from: http://arxiv.org/abs/2306.03102 (accessed 15 November 2023).
6.
BangYCahyawijayaSLeeN, et al. (2023) A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity (arXiv:2302.04023). arXiv. https://doi.org/10.48550/arXiv.2302.04023
7.
BarrosAPrasadAŚliwaM (2023) Generative artificial intelligence and academia: Implication for research, teaching and service. Management Learning54(5): 597–604.
CaoYZhouLLeeS, et al. (2023) Assessing cross-cultural alignment between ChatGPT and human societies: an empirical study. arXiv pre-print. Available from: http://arxiv.org/abs/2303.17466 (accessed 13 November 2023).
ChiuTKF (2023) The impact of generative AI (GenAI) on practices, policies and research direction in education: A case of ChatGPT and Midjourney. Interactive Learning Environments: 1–17. DOI: https://doi.org/10.1080/10494820.2023.2253861.
15.
ChiuTKFMoorhouseBLChaiCS, et al. (2023) Teacher support and student motivation to learn with artificial intelligence (AI) based chatbot. Interactive Learning Environments: 1–17. DOI: https://doi.org/10.1080/10494820.2023.2172044.
16.
CromptonHBurkeD (2023) Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education20(1): 22.
17.
DailyMOulasvirtaARekimotoJ (2017) Technology for human augmentation. Computer50(2): 12–15.
18.
Dell’AcquaFMcFowlandEMollickER, et al. (2023) Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Technology & Operations Mgt. Unit Working Paper24(13): 1–44.
19.
DowlingMLuceyB (2023) ChatGPT for (finance) research: The Bananarama conjecture. Finance Research Letters53: 103662.
20.
DwivediYKKshetriNHughesL, et al. (2023) Opinion paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management71: 102642.
21.
EkeDO (2023) ChatGPT and the rise of generative AI: Threat to academic integrity?Journal of Responsible Technology13: 100060.
22.
FangTYangSLanK, et al. (2023) Is ChatGPT a highly fluent grammatical error correction system? A comprehensive evaluation. arXiv. DOI: 10.48550/arXiv.2304.01746.
23.
GaoJDingXQinB, et al. (2023) Is ChatGPT a good causal reasoner? A comprehensive evaluation.Working paper from the Research Center for Social Computing and Information Retrieval. Harbin Institute of Technology, China. DOI: https://doi.org/10.48550/arXiv.2305.07375.
24.
GirotraKMeinckeLTerwieschC, et al. (2023) Ideas are dimes a Dozen: Large language models for idea generation in innovation. SSRN Electronic Journal: 1–13. DOI: https://doi.org/10.2139/ssrn.4526071.
25.
Gozalo-BrizuelaRGarrido-MerchanEC (2023) ChatGPT is not all you need. A state of the art review of large generative AI models. Working paper from the Quantitative Methods Department, Universidad Pontificia Comillas, Madrid, Spain. DOI: https://doi.org/10.48550/arXiv.2301.04655.
26.
HannaELevicA (2023) Comparative analysis of language models: hallucinations in ChatGPT. Bachelor’s Thesis, Computer Science. Linnaeus University, Faculty of Technology, Department of Computer Science and Media Technology (CM).
27.
HernandezESharmaASHaklayT, et al. (2024) Linearity of relation decoding in transformer language models. arXiv. DOI: arxiv.org/abs/2308.09124.
28.
HughesA (2023) ChatGPT: Everything you need to know about OpenAI’s GPT-4 tool. BBC Science Focus. 25 September. Available from: https://www.sciencefocus.com/future-technology/gpt-3 (accessed 17 October 2023).
29.
JensenJLKarellDTanigawa-LauC, et al. (2022) Language models in sociological research: An application to classifying large administrative data and measuring religiosity. Sociological Methodology52(1): 30–52.
30.
JiZLeeNFrieskeR, et al. (2023) Survey of hallucination in natural language generation. ACM Computing Surveys55(12): 1–38.
31.
JoyceKSmith-DoerrLAlegriaS, et al. (2021) Toward a sociology of artificial intelligence: A call for research on inequalities and structural change. Socius7: 1–11.
KocońJCicheckiIKaszycaO, et al. (2023) ChatGPT: Jack of all trades, master of none. Information Fusion99: 101861.
34.
KuKYLHoIT (2010) Metacognitive strategies that enhance critical thinking. Metacognition and Learning5(3): 251–267.
35.
LångKJosefssonVLarssonA-M, et al. (2023) Artificial intelligence-supported screen reading versus standard double reading in the mammography screening with artificial intelligence trial (MASAI): A clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study. The Lancet Oncology24(8): 936–944.
36.
LiBZTamkinAGoodmanN, et al. (2023a) Eliciting human preferences with language models. arXiv pre-print. Available from: http://arxiv.org/abs/2310.11589 (accessed 24 October 2023).
37.
LiCWangJZhangY, et al. (2023b) Large language models understand and can be enhanced by emotional Stimuli. arXiv pre-print. Available from: http://arxiv.org/abs/2307.11760 (accessed 27 September 2023).
38.
LiangWZhangYCaoH, et al. (2023) Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv pre-print. Available from: http://arxiv.org/abs/2310.01783 (accessed 9 October 2023).
39.
LiuNFZhangTLiangP (2023) Evaluating verifiability in generative search engines. arXiv pre-print. Available from: http://arxiv.org/abs/2304.09848 (accessed 21 November 2023).
40.
LozićEŠtularB (2023) Fluent but not factual: A comparative analysis of ChatGPT and other AI Chatbots’ proficiency and originality in scientific writing for humanities. Future Internet15(10): 336.
41.
Microsoft Research AI4Science & Microsoft Azure Quantum (2023) The impact of large language models on scientific discovery: A preliminary study using GPT-4. Microsoft Research Paper. Available from: http://arxiv.org/abs/2311.07361 (accessed 17 November 2023).
42.
MillerEJStewardBAWitkowerZ, et al. (2023) AI hyperrealism: Why AI faces are perceived as more real than human ones. Psychological Science: 09567976231207095. DOI: 10.1177/09567976231207095.
43.
MittelstadtBWachterSRussellC (2023) To protect science, we must use LLMs as zero-shot translators. Nature Human Behaviour7(11): 1830–1832.
44.
MollickERMollickL (2023) Assigning AI: Seven approaches for students, with prompts. SSRN Scholarly Paper. DOI: https://doi.org/10.2139/ssrn.4475995.
ParreiraMTGilletSLeiteI (2023) Robot duck debugging: Can attentive listening improve problem solving?. arXiv pre-print. Available from: http://arxiv.org/abs/2301.06511 (accessed 15 November 2023).
47.
PataranutapornPLiuRFinnE, et al. (2023) Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nature Machine Intelligence5(10): 1076–1086.
48.
RaisamoRRakkolainenIMajarantaP, et al. (2019) Human augmentation: Past, present and future. International Journal of Human–Computer Studies131: 131–143.
49.
RathjeSMireaD-MSucholutskyI, et al. (2023) GPT is an effective tool for multilingual psychological text analysis. PsyArXiv pre-print. DOI: https://doi.org/10.31234/osf.io/sekf5.
50.
StinessenL (1985) The influence of verbalization on problem-solving. Scandinavian Journal of Psychology26(1): 342–347.
TuXZouJSuWJ, et al. (2023) What should data science education do with large language models? arXiv pre-print. Available from: http://arxiv.org/abs/2307.02792 (accessed 29 September 2023).
53.
WangKVariengienAConmyA, et al. (2022) Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. arXiv. DOI: 10.48550/arXiv.2211.00593.
54.
WatermeyerRPhippsLLanclosD, et al. (2023) Generative AI and the automating of academia. Postdigital Science and Education 6: 446–466. DOI: https://doi.org/10.1007/s42438-023-00440-6.
55.
ZhengSHuangJChangKC-C (2023) Why does ChatGPT fall short in providing truthful answers? arXiv pre-print. Available from: http://arxiv.org/abs/2304.10513 (accessed 23 November 2023).
