Can Language Models Represent the Past without Anachronism?
This addresses the risk of anachronism for researchers using language models in social research, but it is incremental as it builds on existing fine-tuning methods.
The paper tackled the problem of language models producing anachronistic outputs when simulating the past, finding that fine-tuning can fool automated judges but not human evaluators, suggesting pretraining on period prose may be necessary for reliable historical simulation.
Before researchers can use language models to simulate the past, they need to understand the risk of anachronism. We find that prompting a contemporary model with examples of period prose does not produce output consistent with period style. Fine-tuning produces results that are stylistically convincing enough to fool an automated judge, but human evaluators can still distinguish fine-tuned model outputs from authentic historical text. We tentatively conclude that pretraining on period prose may be required in order to reliably simulate historical perspectives for social research.