Jason Lin

Adrià Recasens, Jason Lin, Joāo Carreira et al. · deepmind

Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering. The resulting representations are however fully entangled throughout the network, which may not always be desirable: in learning, contrastive audio-visual self-supervised learning requires independent audio and visual features to operate, otherwise learning collapses; in inference, evaluation of audio-visual models should be possible on benchmarks having just audio or just video. In this paper, we introduce Zorro, a technique that uses masks to control how inputs from each modality are routed inside Transformers, keeping some parts of the representation modality-pure. We apply this technique to three popular transformer-based architectures (ViT, Swin and HiP) and show that with contrastive pre-training Zorro achieves state-of-the-art results on most relevant benchmarks for multimodal tasks (AudioSet and VGGSound). Furthermore, the resulting models are able to perform unimodal inference on both video and audio benchmarks such as Kinetics-400 or ESC-50.

Stephen Casper, Jason Lin, Joe Kwon et al. · deepmind

Deploying large language models (LMs) can pose hazards from harmful outputs such as toxic or false text. Prior work has introduced automated tools that elicit harmful outputs to identify these risks. While this is a valuable step toward securing models, these approaches rely on a pre-existing way to efficiently classify undesirable outputs. Using a pre-existing classifier does not allow for red-teaming to be tailored to the target model. Furthermore, when failures can be easily classified in advance, red-teaming has limited marginal value because problems can be avoided by simply filtering training data and/or model outputs. Here, we consider red-teaming "from scratch," in which the adversary does not begin with a way to classify failures. Our framework consists of three steps: 1) Exploring the model's range of behaviors in the desired context; 2) Establishing a definition and measurement for undesired behavior (e.g., a classifier trained to reflect human evaluations); and 3) Exploiting the model's flaws using this measure to develop diverse adversarial prompts. We use this approach to red-team GPT-3 to discover classes of inputs that elicit false statements. In doing so, we construct the CommonClaim dataset of 20,000 statements labeled by humans as common-knowledge-true, common knowledge-false, or neither. We are making code and data available.

Chia-Wei Tsai, Chun-Wei Yang, Jason Lin

This study proposes a multiparty mediated quantum secret sharing (MQSS) protocol that allows n restricted quantum users to share a secret via the assistance of a dishonest third-party (TP) with full quantum capabilities. Under the premise that a restricted quantum user can only perform the Hadamard transformation and the Z-basis measurement, the proposed MQSS protocol has addressed two common challenges in the existing semi-quantum secret sharing protocols: (1) the dealer must have full quantum capability, and (2) the classical users must equip with the wavelength quantum filter and the photon number splitters (PNS) to detect the Trojan horse attacks. The security analysis has also delivered proof to show that the proposed MQSS protocol can avoid the collective attack, the collusion attack, and the Trojan horse attacks. In addition, the proposed MQSS protocol is more efficient than the existing SQSS protocols due to the restricted quantum users can only equip with two quantum operations, and the qubits are transmitted within a shorter distance.