Farzin Gholamrezae

Abubakar Sadiq Shittu, John Sadik, Farzin Gholamrezae et al.

Commit signing is widely promoted as a foundation of software supply-chain security, yet prior work has studied it through the lens of individual repositories or curated project samples, missing the broader picture of how developers behave across an entire platform. Grounded in replicability theory, we vary the sampling unit from repositories to individual developers, following 71,694 active GitHub users, defined as accounts that have authored at least one commit, across all their repositories and their entire commit history, spanning 16 million commits and 874,198 repositories. This platform-wide, user-centric view reveals a fundamental gap that repository sampling cannot detect. The ecosystem's apparent high signing adoption rate is an illusion. Once platform-generated signatures are excluded, fewer than 6% of developers have ever signed a commit themselves, and the vast majority of apparent signers have never signed outside a web browser. Among the minority who do sign locally, signing rarely persists over time or across repositories, and roughly one in eight developer-managed signatures fails verification because signing keys are never uploaded to GitHub. Examining the key registry, we find that expired keys are almost never revoked and more than a quarter of users carry at least one dead key. Together, these findings reveal that commit signing as practiced today cannot serve as a dependable provenance signal at ecosystem scale, and we offer concrete recommendations for closing that gap.

Ran Elgedawy, Porter Dosch, John Sadik et al.

$ $Large Language Models (LLMs) are being increasingly utilized in various applications, with code generations being a notable example. While previous research has shown that LLMs have the capability to generate both secure and insecure code, the literature does not take into account what factors help generate secure and effective code. Therefore in this paper we focus on identifying and understanding the conditions and contexts in which LLMs can be effectively and safely deployed in real-world scenarios to generate quality code. We conducted a comparative analysis of four advanced LLMs--GPT-3.5 and GPT-4 using ChatGPT and Bard and Gemini from Google--using 9 separate tasks to assess each model's code generation capabilities. We contextualized our study to represent the typical use cases of a real-life developer employing LLMs for everyday tasks as work. Additionally, we place an emphasis on security awareness which is represented through the use of two distinct versions of our developer persona. In total, we collected 61 code outputs and analyzed them across several aspects: functionality, security, performance, complexity, and reliability. These insights are crucial for understanding the models' capabilities and limitations, guiding future development and practical applications in the field of automated code generation.