44.1CRMay 6
Evaluating the Reliability of Multiple Large Language Models in Risk Assessment: A CIS Controls Based ApproachGustavo Roberto Pinto, Arthur do Prado Labaki, Rodrigo Sanches Miani
Proper implementation of technical and administrative controls reinforces an organization's cybersecurity posture and business resilience, reduces risks, and enhances governance, ultimately elevating business maturity. The dynamics of the technological landscape and emerging threats negatively affect the most diverse companies, regardless of their size. This, associated with a global gap in the cybersecurity workforce, imposes enormous challenges and the need for a profound change in how companies respond to threats. Generative Artificial Intelligence from large language models has become an influential tool across various companies, emerging as a viable option to help address those challenges while partially addressing the shortage of skilled labor. Although large language models can help in this scenario, there may be risks, such as generating unreliable or 'hallucinated' content, which could lead people and companies to make bad decisions. Our study proposes integrating human experts into the validation process as a crucial step toward ensuring the proper implementation of technical and administrative controls. Furthermore, we sought to identify how large language models perform in assessing cybersecurity risk scenarios compared to human experts, highlighting the importance of integrating humans and machines in the cybersecurity risk assessment process. Using a questionnaire with risk scenarios, we analyzed responses from 50 human experts. We compared their responses with those of five popular large language models to determine whether it is possible to use only large language models for cybersecurity risk assessment. The results reveal that the large language models consistently underestimated cybersecurity risks compared to human experts, reinforcing the need for human oversight and suggesting that LLMs should be used as complementary tools rather than standalone assessors.
CRNov 28, 2025
Identification of Malicious Posts on the Dark Web Using Supervised Machine LearningSebastião Alves de Jesus Filho, Gustavo Di Giovanni Bernardo, Paulo Henrique Ribeiro Gabriel et al.
Given the constant growth and increasing sophistication of cyberattacks, cybersecurity can no longer rely solely on traditional defense techniques and tools. Proactive detection of cyber threats has become essential to help security teams identify potential risks and implement effective mitigation measures. Cyber Threat Intelligence (CTI) plays a key role by providing security analysts with evidence-based knowledge about cyber threats. CTI information can be extracted using various techniques and data sources; however, machine learning has proven promising. As for data sources, social networks and online discussion forums are commonly explored. In this study, we apply text mining techniques and machine learning to data collected from Dark Web forums in Brazilian Portuguese to identify malicious posts. Our contributions include the creation of three original datasets, a novel multi-stage labeling process combining indicators of compromise (IoCs), contextual keywords, and manual analysis, and a comprehensive evaluation of text representations and classifiers. To our knowledge, this is the first study to focus specifically on Brazilian Portuguese content in this domain. The best-performing model, using LightGBM and TF-IDF, was able to detect relevant posts with high accuracy. We also applied topic modeling to validate the model's outputs on unlabeled data, confirming its robustness in real-world scenarios.
CRFeb 13, 2021
Towards reliable and transparent vaccine phase III trials with smart contractsIvan da Silva Sendin, Rodrigo Sanches Miani
Transforming a vaccine concept into a real vaccine product is a complicated process and includes finding suitable antigens and regulatory, technical, and manufacturing obstacles. A relevant issue within this scope is the clinical trial process. Monitoring and ensuring the integrity of trial data using the traditional system is not always feasible. The search for a vaccine against the coronavirus SARS-CoV-2 illustrates this situation. The scientific credibility of findings from several vaccines' clinical trials contributed to distorted perceptions concerning the benefits and risks of the drug. This scenario is ideal for applying technologies such as Blockchain and Smart Contracts in healthcare issues. This paper proposes a protocol based on Smart Contracts, named VaccSC, to enable transparency, accounting, and confidentiality to Phase III of vaccine experiments. The protocol was implemented in Solidity language, and results show that the VaccSC enables double-blindness, randomization, and the auditability of clinical data, even in the presence of dishonest participants.
CRNov 5, 2020
Evaluating the Performance of Twitter-based Exploit DetectorsDaniel Alves de Sousa, Elaine Ribeiro de Faria, Rodrigo Sanches Miani
Patch prioritization is a crucial aspect of information systems security, and knowledge of which vulnerabilities were exploited in the wild is a powerful tool to help systems administrators accomplish this task. The analysis of social media for this specific application can enhance the results and bring more agility by collecting data from online discussions and applying machine learning techniques to detect real-world exploits. In this paper, we use a technique that combines Twitter data with public database information to classify vulnerabilities as exploited or not-exploited. We analyze the behavior of different classifying algorithms, investigate the influence of different antivirus data as ground truth, and experiment with various time window sizes. Our findings suggest that using a Light Gradient Boosting Machine (LightGBM) can benefit the results, and for most cases, the statistics related to a tweet and the users who tweeted are more meaningful than the text tweeted. We also demonstrate the importance of using ground-truth data from security companies not mentioned in previous works.