Diego Elias Costa

Andre Hora, João Eduardo Montandon, Diego Elias Costa

GitHub is the largest code hosting platform, with millions of repositories spanning multiple technologies. Despite this, little is known about the actual contents of GitHub's repositories in the wild. This paper presents an initial empirical analysis to better understand the contents of real-world GitHub repositories. We analyze the files, directories, and extensions present in 10,000 GitHub repositories, as well as their evolution over ten years. Our results show major changes in GitHub over the last decade: (1) the consolidation of README.md, .gitignore, and LICENSE as standard artifacts; (2) the rise of GitHub Actions as the dominant CI/CD platform; (3) the growth of configuration formats such as TOML, YAML, and JSON, alongside a decline in XML; (4) new trends, such as the growth of Dockerfile; and (5) emerging content related to LLMs and generative AI (e.g., AGENTS.md). Based on our findings, we discuss implications, including that open source is not only evolving organically but also increasingly guided by GitHub's standards, the rise and fall of technologies, and the potential support for mining software repository studies.

Kawsar Ahmed Bhuiyan, Mohamed Bilel Besbes, Rachna Raj et al.

Modern software development relies heavily on open-source components. Reusing components accelerates innovation but increases exposure to supply-chain attacks exploiting known vulnerabilities. Software Bills of Materials (SBOMs) improve software supply chain transparency by enumerating components, their versions, and their provenance. GitHub, the largest open-source development hosting platform, now automatically generates SBOMs for repositories, providing valuable metadata for risk assessment. Yet, it is unclear whether GitHub SBOMs can serve as a reliable source for vulnerability and license analysis, and how incomplete or inconsistent metadata may affect different programming ecosystems. To address this, we conduct a large-scale analysis of 10,000 GitHub repositories across ten programming language ecosystems, evaluating GitHub SBOMs against three other popular SBOM generators: Syft, Trivy, and the Microsoft SBOM Tool. Our study finds a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present. We also find that component version and license information availability is highly dependent on the programming ecosystem. Compared with the other three tools, GitHub yields results similar to the Microsoft SBOM Tool and often outperforms Syft and Trivy in providing version and license information. Finally, we discuss potential shortcomings of the GitHub SBOM Tool, directly related to how each ecosystem manages its dependencies.

Oluwatoyosi Oyelayo, Ghada Abushaqra, Parham Asadi et al.

Ensuring web accessibility at scale remains challenging because rule-based tools provide limited coverage while manual remediation is costly and error-prone. This paper evaluates large language model based agents, specifically Kimi K2.5, for automated accessibility detection and repair compared with rule-based approaches. For detection, the LLM achieves performance comparable to rule-based tools, with F1 around 0.65, strong semantic understanding with F1 of 0.83, but lower reliability for syntactic and layout-related violations. For remediation, LLM-generated fixes are syntactically valid in over 99.7 percent of cases and improve accessibility compliance in 80.2 percent of instances, reducing violations from 3.98 to 1.7 per file. However, fewer than 26 percent of cases are fully resolved, and about 30 percent of patches introduce structural changes. We also find that iterative agent-based refinement increases computational cost by 52 percent and API usage by 1.64 times without improving remediation outcomes. These findings indicate that while LLMs are effective for partial accessibility repair, they are insufficient for complete and reliable remediation. Scalable accessibility solutions require hybrid approaches that combine LLM capabilities with rule-based validation and constraint-aware correction mechanisms.

Mohamed Bilel Besbes, Gregory Mierzwinski, Suhaib Mujahid et al.

Software performance regressions can have significant business consequences, making automated detection a critical component of modern continuous integration pipelines. At Mozilla, performance anomaly detection is handled by Perfherder, Mozilla's performance engineering management system that relies on a Student's T-test-based approach to flag regressions across hundreds of daily code changes. However, our preliminary analysis of one year of Mozilla performance data reveals that 12.5% of generated alert groups are false positives, while approximately 6.8% of them contain regressions missed by the automated system. This paper presents an empirical study evaluating 25 change-point detection (CPD) methods and 15 ensemble approaches as alternatives to Mozilla's current method. We construct a ground-truth dataset of 174 performance time series manually annotated by eleven Mozilla performance engineers, representing one of the first practitioner-annotated CPD benchmarks for performance engineering. Our results show that while offline and hybrid CPD methods improve recall over Mozilla's method, they do so at a high cost to precision. Ensemble voting strategies alleviate this trade-off and offer more consistent performance, resulting in 11% improvement in the F1-score. We validate the experimental results through a practitioner survey and report on lessons learned from integrating the best methods into Mozilla's performance engineering system.