Lessons Learnt in Conducting Survey Research
This work aims to improve survey quality for software engineering researchers, but it is incremental as it builds on existing experiences and general theory.
The paper tackles the problem of pitfalls in software engineering survey research, providing an overview and practical ways to address them, resulting in eight lessons covering aspects from research objectives to questionnaire design.
Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.