Adriënne M. Mendrik

Arkadiy Dushatskiy, Adriënne M. Mendrik, Peter A. N. Bosman et al.

There has recently been great progress in automatic segmentation of medical images with deep learning algorithms. In most works observer variation is acknowledged to be a problem as it makes training data heterogeneous but so far no attempts have been made to explicitly capture this variation. Here, we propose an approach capable of mimicking different styles of segmentation, which potentially can improve quality and clinical acceptance of automatic segmentation methods. In this work, instead of training one neural network on all available data, we train several neural networks on subgroups of data belonging to different segmentation variations separately. Because a priori it may be unclear what styles of segmentation exist in the data and because different styles do not necessarily map one-on-one to different observers, the subgroups should be automatically determined. We achieve this by searching for the best data partition with a genetic algorithm. Therefore, each network can learn a specific style of segmentation from grouped training data. We provide proof of principle results for open-sourced prostate segmentation MRI data with simulated observer variations. Our approach provides an improvement of up to 23% (depending on simulated variations) in terms of Dice and surface Dice coefficients compared to one network trained on all data.

Elizaveta Sivak, Paulina Pankowska, Adrienne Mendrik et al.

The social sciences have produced an impressive body of research on determinants of fertility outcomes, or whether and when people have children. However, the strength of these determinants and underlying theories are rarely evaluated on their predictive ability on new data. This prevents us from systematically comparing studies, hindering the evaluation and accumulation of knowledge. In this paper, we present two datasets which can be used to study the predictability of fertility outcomes in the Netherlands. One dataset is based on the LISS panel, a longitudinal survey which includes thousands of variables on a wide range of topics, including individual preferences and values. The other is based on the Dutch register data which lacks attitudinal data but includes detailed information about the life courses of millions of Dutch residents. We provide information about the datasets and the samples, and describe the fertility outcome of interest. We also introduce the fertility prediction data challenge PreFer which is based on these datasets and will start in Spring 2024. We outline the ways in which measuring the predictability of fertility outcomes using these datasets and combining their strengths in the data challenge can advance our understanding of fertility behaviour and computational social science. We further provide details for participants on how to take part in the data challenge.

Laura Boeschoten, Adriënne Mendrik, Emiel van der Veen et al.

We present PORT, a software platform for local data extraction and analysis of digital trace data. While digital trace data collected by private and public parties hold a huge potential for social-scientific discovery, their most useful parts have been unattainable for academic researchers due to privacy concerns and prohibitive API access. However, the EU General Data Protection Regulation (GDPR) grants all citizens the right to an electronic copy of their personal data. All major data controllers, such as social media platforms, banks, online shops, loyalty card systems and public transportation cards comply with this right by providing their clients with a `Data Download Package' (DDP). Previously, a conceptual workflow was introduced allowing citizens to donate their data to scientific- researchers. In this workflow, citizens' DDPs are processed locally on their machines before they are asked to provide informed consent to share a subset of the processed data with the researchers. In this paper, we present the newly developed software PORT that implements the local processing part of this workflow, protecting privacy by shielding sensitive data from any contact with outside observers -- including the researchers themselves. Thus, PORT enables a host of potential applications of social data science to hitherto unobtainable data.

Evelien Schat, Rens van de Schoot, Wouter M. Kouw et al.

In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is similar to new unseen data one wishes to apply it to. It is often unknown in advance how an algorithm will perform on new unseen data, being a crucial reason for not deploying an algorithm at all. Therefore, tools are needed to measure the similarity of data sets. In this paper, we propose the Data Representativeness Criterion (DRC) to determine how representative a training data set is of a new unseen data set. We present a proof of principle, to see whether the DRC can quantify the similarity of data sets and whether the DRC relates to the performance of a supervised classification algorithm. We compared a number of magnetic resonance imaging (MRI) data sets, ranging from subtle to severe difference is acquisition parameters. Results indicate that, based on the similarity of data sets, the DRC is able to give an indication as to when the performance of a supervised classifier decreases. The strictness of the DRC can be set by the user, depending on what one considers to be an acceptable underperformance.

Adrienne M. Mendrik, Stephen R. Aylward

In the medical image analysis field, organizing challenges with associated workshops at international conferences began in 2007 and has grown to include over 150 challenges. Several of these challenges have had a major impact in the field. However, whereas well-designed challenges have the potential to unite and focus the field on creating solutions to important problems, poorly designed and documented challenges can equally impede a field and lead to pursuing incremental improvements in metric scores with no theoretic or clinical significance. This is supported by a critical assessment of challenges at the international MICCAI conference. In this assessment the main observation was that small changes to the underlying challenge data can drastically change the ranking order on the leaderboard. Related to this is the practice of leaderboard climbing, which is characterized by participants focusing on incrementally improving metric results rather than advancing science or solving the driving problem of a challenge. In this abstract we look beyond the leaderboard of a challenge and instead look at the conclusions that can be drawn from a challenge with respect to the research problem that it is addressing. Research study design is well described in other research areas and can be translated to challenge design when viewing challenges as research studies on algorithm performance that address a research problem. Based on the two main types of scientific research study design, we propose two main challenge types, which we think would benefit other research areas as well: 1) an insight challenge that is based on a qualitative study design and 2) a deployment challenge that is based on a quantitative study design. In addition we briefly touch upon related considerations with respect to statistical significance versus practical significance, generalizability and data saturation.

Wouter M. Kouw, Marco Loog, Wilbert Bartels et al.

Generalization of voxelwise classifiers is hampered by differences between MRI-scanners, e.g. different acquisition protocols and field strengths. To address this limitation, we propose a Siamese neural network (MRAI-NET) that extracts acquisition-invariant feature vectors. These can consequently be used by task-specific methods, such as voxelwise classifiers for tissue segmentation. MRAI-NET is tested on both simulated and real patient data. Experiments show that MRAI-NET outperforms voxelwise classifiers trained on the source or target scanner data when a small number of labeled samples is available.

Wouter M. Kouw, Marco Loog, Lambertus W. Bartels et al.

Voxelwise classification approaches are popular and effective methods for tissue quantification in brain magnetic resonance imaging (MRI) scans. However, generalization of these approaches is hampered by large differences between sets of MRI scans such as differences in field strength, vendor or acquisition protocols. Due to this acquisition related variation, classifiers trained on data from a specific scanner fail or under-perform when applied to data that was acquired differently. In order to address this lack of generalization, we propose a Siamese neural network (MRAI-net) to learn a representation that minimizes the between-scanner variation, while maintaining the contrast between brain tissues necessary for brain tissue quantification. The proposed MRAI-net was evaluated on both simulated and real MRI data. After learning the MR acquisition invariant representation, any supervised classification model that uses feature vectors can be applied. In this paper, we provide a proof of principle, which shows that a linear classifier applied on the MRAI representation is able to outperform supervised convolutional neural network classifiers for tissue classification when little target training data is available.