Simon Thorne

Harry Collins, Simon Thorne

We explore the capabilities of Large Language Models (LLMs) by comparing the way they gather data with the way humans build knowledge. Here we examine how scientific knowledge is made and compare it with LLMs. The argument is structured by reference to two figures, one representing scientific knowledge and the other LLMs. In a 2014 study, scientists explain how they choose to ignore a 'fringe science' paper in the domain of gravitational wave physics: the decisions are made largely as a result of tacit knowledge built up in social discourse, most spoken discourse, within closed groups of experts. It is argued that LLMs cannot or do not currently access such discourse, but it is typical of the early formation of scientific knowledge. LLMs 'understanding' builds on written literatures and is therefore insecure in the case of the initial stages of knowledge building. We refer to Colin Fraser's 'Dumb Monty Hall problem' where in 2023 ChatGPT failed though a year later or so later LLMs were succeeding. We argue that this is not a matter of improvement in LLMs ability to reason but in the change in the body of human written discourse on which they can draw (or changes being put in by humans 'by hand'). We then invent a new Monty Hall prompt and compare the responses of a panel of LLMs and a panel of humans: they are starkly different but we explain that the previous mechanisms will soon allow the LLMs to align themselves to humans once more. Finally, we look at 'overshadowing' where a settled body of discourse becomes so dominant that LLMs fail to respond to small variations in prompts which render the old answers nonsensical. The 'intelligence' we argue is in the humans not the LLMs

Harry Collins, Hartmut Grote, Paul Newbury et al.

This paper is under review in AI and Ethics This study examines whether large language models (LLMs) can reliably answer scientific questions and demonstrates how easily they can be influenced by fringe scientific material. The authors modified custom LLMs to prioritise knowledge in selected fringe papers on the Fine Structure Constant and Gravitational Waves, then compared their responses with those of domain experts and standard LLMs. The altered models produced fluent, convincing answers that contradicted scientific consensus and were difficult for non-experts to detect as misleading. The results show that LLMs are vulnerable to manipulation and cannot replace expert judgment, highlighting risks for public understanding of science and the potential spread of misinformation.

Khwima Mckinley Mkamanga, Simon Thorne

This paper explores the impacts of spreadsheets on business operations in a water utility parastatal in Malawi, Sub-Saharan Africa. The organisation is a typical example of a semi-government body operating in a technologically underdeveloped country. The study focused on spreadsheet scope of use and life cycle as well as organisational policy and governance. The results will help define future spreadsheet usage by influencing new approaches for managing potential risks associated with spreadsheets in the organization. Generally, findings indicate that the proliferation of spreadsheets in the organization has provided an enabling environment for business automation. The paper also highlights management, technological and human factor issues contributing to high risks associated with the pervasive spreadsheet use. The conclusions drawn from the research confirms that there is ample room for improvement in many areas such as implementation of comprehensive policies and regulations governing spreadsheet development processes and adoption.

Mahmood H. Shubbak, Simon Thorne

Heavy use of spreadsheets by organisations bears many potential risks such as errors, ambiguity, data loss, duplication, and fraud. In this paper these risks are briefly outlined along with their available mitigation methods such as: documentation, centralisation, auditing and user training. However, because of the large quantities of spreadsheets used in organisations, applying these methods on all spreadsheets is impossible. This fact is considered as a deficiency in these methods, a gap which is addressed in this paper. In this paper a new software tool for managing spreadsheets and identifying the risk levels they include is proposed, developed and tested. As an add-in for Microsoft Excel application, "Risk Calculator" can automatically collect and record spreadsheet properties in an inventory database and assign risk scores based on their importance, use and complexity. Consequently, auditing processes can be targeted to high risk spreadsheets. Such a method saves time, effort, and money.