Victoria Jackson

Victoria Jackson, Yoonha Cha, Rafael Prikladnicki et al.

While much prior work examines Large Language Models (LLMs) for solo development tasks (e.g., coding), far less is known about how LLMs shape collaborative group work in software engineering. This study focuses on one such collaborative task, namely software design. It presents the results of an exploratory laboratory study of 18 pairs of software professionals who could use an LLM however they saw fit, to design a University campus bicycle parking application. Our findings reveal that introducing an LLM leads to distinct patterns of joint use: shared-instance use facilitated shared understanding, whereas parallel use across separate instances sometimes led to ''context drift''. We also observe wide variation in reliance, from non-use to treating the LLM as an information source or producer. Across these modes, professionals scrutinized and reflected on LLM responses, often yielding design insights; however, early anchoring sometimes curtailed exploration. We provide implications for tools to aid designers while retaining the human-centricity important to design.

Fabio Calefato, Alexandra Pajonk, Victoria Jackson et al.

Generative AI tools are rapidly transforming software development practice, prompting unprecedented research interest. However, existing studies have predominantly examined initial adoption rather than sustained use. Understanding what drives developers to continue using these tools after initial adoption remains underexplored, particularly in small and medium-sized enterprises where resource constraints shape technology decisions differently than in large organisations. This study investigates factors associated with developers' intentions to continue using GenAI tools, adapting the UTAUT2 framework to post-adoption professional contexts. We employed a two-phase mixed-methods design. Phase 1 comprised a six-month longitudinal pilot study at an Italian software company combining surveys and interviews with 17 developers to explore how perceptions of GenAI evolve as experience accumulates. These insights informed a structural model tested in Phase 2 through a cross-sectional survey of 154 developers across Italian SMEs, analysed using PLS-SEM. The model explained substantial variance in continued use intention (R2 = 0.647), with individual-level perceptions, particularly around productivity, enjoyment, and ease of use, driving sustained adoption, whereas social and organisational factors played no significant role. These findings suggest that, for GenAI tools, post-adoption behaviour differs from initial adoption patterns: in voluntary professional contexts, sustained use is driven primarily by individual-level factors rather than by social and organisational support.

Norman Anderson, Tarek Alakmeh, Victoria Jackson et al.

A rapidly growing body of research is examining how LLMs influence developers when they code. To date, this research has tended to focus on productivity and code quality outcomes, rather than the underlying cognitive processes involved in programming. To address this gap, we report on the results of an exploratory laboratory study of ten advanced student developers (five with support from AI and five without) who had to make a non-trivial extension to a sizable software system. Leveraging Polya's four problem-solving phases and 25 inductively-generated codes detailing distinct problem-solving behaviors as the primary lenses, we examined: (1) how AI impacted the problem-solving approach the developers used to solve the programming task, and (2) how AI impacted their progress when they became stuck. For the analysis, we triangulated data across multiple sources (e.g., think-aloud, code changes, web searches, and LLM prompts). Unexpectedly, while developers in the AI group repeatedly turned to the AI tool to offload certain aspects of the process, all detailed problem-solving behaviors appeared in both groups. We also found that nine out of ten participants found themselves stuck in their work, but with key differences in how they became stuck and unstuck. We highlight seven distinct causes for being stuck and highlight how AI in some cases helped and in other cases hindered becoming unstuck.

Victoria Jackson, Grischa Liebel, Rafael Prikladnicki et al.

While the use of Large Language Models (LLMs) in programming has been extensively studied, there is limited understanding of how LLMs support collaborative work where creativity plays a central role. Software design, as a collaborative and creative activity, provides a valuable context for exploring the influence of LLMs on creativity. This study investigates how and where creativity naturally emerges when software designers collaborate with an LLM during a design task. In a laboratory setting simulating a workplace environment, 18 pairs of software professionals with design experience were asked to complete a design task. Each pair had 90 minutes to produce a software design based on a set of requirements, with optional access to a custom LLM interface. Pairs were not primed to be creative. We find that creativity was present in all pairs in design processes, with 13 producing design documents containing creativity. We primarily attribute creativity to the human designers, driven by traits such as prior experience, empathy, and the use of analogies. The LLM contributed by producing novel ideas and elaborating human ideas. However, in some cases, the LLM appeared to hinder creativity by suggesting complex solutions or adding to unproductive digressions. LLMs can support creativity in collaborative software design, but human insights remain central. To effectively augment human creativity, designers must be intentional in their engagement with LLMs.