Navigating Human Language Models with Synthetic Agents
This provides a novel sociological tool for analyzing human beliefs, though it is incremental in applying existing methods to a specific domain.
The study tackled the problem of verifying if language models accurately reflect human beliefs by training GPT-2 on historical chess games and using synthetic agents to generate text, finding that the model's move percentages closely matched human patterns and it created an accurate latent representation of the chessboard.
Modern natural language models such as the GPT-2/GPT-3 contain tremendous amounts of information about human belief in a consistently testable form. If these models could be shown to accurately reflect the underlying beliefs of the human beings that produced the data used to train these models, then such models become a powerful sociological tool in ways that are distinct from traditional methods, such as interviews and surveys. In this study, We train a version of the GPT-2 on a corpora of historical chess games, and then "launch" clusters of synthetic agents into the model, using text strings to create context and orientation. We compare the trajectories contained in the text generated by the agents/model and compare that to the known ground truth of the chess board, move legality, and historical patterns of play. We find that the percentages of moves by piece using the model are substantially similar from human patterns. We further find that the model creates an accurate latent representation of the chessboard, and that it is possible to plot trajectories of legal moves across the board using this knowledge.