Adam J. Lee

Sungkyung Kim, Adam Lee, Junyoung Park et al.

Recent advancements in large language models have demonstrated enhanced capabilities in visual reasoning tasks by employing additional encoders for aligning different modalities. While the Q-Former has been widely used as a general encoder for aligning several modalities including image, video, audio, and 3D with large language models, previous works on its efficient training and the analysis of its individual components have been limited. In this work, we investigate the effectiveness of parameter efficient fine-tuning (PEFT) the Q-Former using InstructBLIP with visual reasoning benchmarks ScienceQA and IconQA. We observe that applying PEFT to the Q-Former achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Additionally, we employ AdaLoRA for dynamic parameter budget reallocation to examine the relative importance of the Q-Former's sublayers with 4 different benchmarks. Our findings reveal that the self-attention layers are noticeably more important in perceptual visual-language reasoning tasks, and relative importance of FFN layers depends on the complexity of visual-language patterns involved in tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.

Mohammad Hasanzadeh Mofrad, Adam Lee

Enforcing integrity and confidentiality of users' application code and data is a challenging mission that any software developer working on an online production grade service is facing. Since cryptology is not a widely understood subject, people on the cutting edge of research and industry are always seeking for new technologies to naturally expand the security of their programs and systems. Intel Software Guard Extension (Intel SGX) is an Intel technology for developers who are looking to protect their software binaries from plausible attacks using hardware instructions. The Intel SGX puts sensitive code and data into CPU-hardened protected regions called enclaves. In this project we leverage the Intel SGX to produce a secure cryptographic library which keeps the generated keys inside an enclave restricting use and dissemination of confidential cryptographic keys. Using enclaves to store the keys we maintain a small Trusted Computing Base (TCB) where we also perform computation on temporary buffers to and from untrusted application code. As a proof of concept, we implemented hashes and symmetric encryption algorithms inside the enclave where we stored hashes, Initialization Vectors (IVs) and random keys and open sourced the code (https://github.com/hmofrad/CryptoEnclave).

Adam Lee

Appearance-based gaze estimation has been very successful with the use of deep learning. Many following works improved domain generalization for gaze estimation. However, even though there has been much progress in domain generalization for gaze estimation, most of the recent work have been focused on cross-dataset performance -- accounting for different distributions in illuminations, head pose, and lighting. Although improving gaze estimation in different distributions of RGB images is important, near-infrared image based gaze estimation is also critical for gaze estimation in dark settings. Also there are inherent limitations relying solely on supervised learning for regression tasks. This paper contributes to solving these problems and proposes GazeCWL, a novel framework for gaze estimation with near-infrared images using contrastive learning. This leverages adversarial attack techniques for data augmentation and a novel contrastive loss function specifically for regression tasks that effectively clusters the features of different samples in the latent space. Our model outperforms previous domain generalization models in infrared image based gaze estimation and outperforms the baseline by 45.6\% while improving the state-of-the-art by 8.6\%, we demonstrate the efficacy of our method.

Sung-Yeon Park, Adam Lee, Juanwu Lu et al.

Driving scene manipulation with sensor data is emerging as a promising alternative to traditional virtual driving simulators. However, existing frameworks struggle to generate realistic scenarios efficiently due to limited editing capabilities. To address these challenges, we present SIMSplat, a predictive driving scene editor with language-aligned Gaussian splatting. As a language-controlled editor, SIMSplat enables intuitive manipulation using natural language prompts. By aligning language with Gaussian-reconstructed scenes, it further supports direct querying of road objects, allowing precise and flexible editing. Our method provides detailed object-level editing, including adding new objects and modifying the trajectories of both vehicles and pedestrians, while also incorporating predictive path refinement through multi-agent motion prediction to generate realistic interactions among all agents in the scene. Experiments on the Waymo dataset demonstrate SIMSplat's extensive editing capabilities and adaptability across a wide range of scenarios. Project page: https://sungyeonparkk.github.io/simsplat/

William C. Garrison, Adam Shull, Steven Myers et al.

The ability to enforce robust and dynamic access controls on cloud-hosted data while simultaneously ensuring confidentiality with respect to the cloud itself is a clear goal for many users and organizations. To this end, there has been much cryptographic research proposing the use of (hierarchical) identity-based encryption, attribute-based encryption, predicate encryption, functional encryption, and related technologies to perform robust and private access control on untrusted cloud providers. However, the vast majority of this work studies static models in which the access control policies being enforced do not change over time. This is contrary to the needs of most practical applications, which leverage dynamic data and/or policies. In this paper, we show that the cryptographic enforcement of dynamic access controls on untrusted platforms incurs computational costs that are likely prohibitive in practice. Specifically, we develop lightweight constructions for enforcing role-based access controls (i.e., $\mathsf{RBAC}_0$) over cloud-hosted files using identity-based and traditional public-key cryptography. This is done under a threat model as close as possible to the one assumed in the cryptographic literature. We prove the correctness of these constructions, and leverage real-world $\mathsf{RBAC}$ datasets and recent techniques developed by the access control community to experimentally analyze, via simulation, their associated computational costs. This analysis shows that supporting revocation, file updates, and other state change functionality is likely to incur prohibitive overheads in even minimally-dynamic, realistic scenarios. We identify a number of bottlenecks in such systems, and fruitful areas for future work that will lead to more natural and efficient constructions for the cryptographic enforcement of dynamic access controls.

William C. Garrison, Adam J. Lee

Access control is fundamental to computer security, and has thus been the subject of extensive formal study. In particular, *relative expressiveness analysis* techniques have used formal mappings called *simulations* to explore whether one access control system is capable of emulating another, thereby comparing the expressive power of these systems. Unfortunately, the notions of expressiveness simulation that have been explored vary widely, which makes it difficult to compare results in the literature, and even leads to apparent contradictions between results. Furthermore, some notions of expressiveness simulation make use of non-determinism, and thus cannot be used to define mappings between access control systems that are useful in practical scenarios. In this work, we define the minimum set of properties for an *implementable* access control simulation; i.e., a deterministic "recipe" for using one system in place of another. We then define a wide range of properties spread across several dimensions that can be enforced on top of this minimum definition. These properties define a taxonomy that can be used to separate and compare existing notions of access control simulation, many of which were previously incomparable. We position existing notions of simulation within our properties lattice by formally proving each simulation's equivalence to a corresponding set of properties. Lastly, we take steps towards bridging the gap between theory and practice by exploring the systems implications of points within our properties lattice. This shows that relative expressive analysis is more than just a theoretical tool, and can also guide the choice of the most suitable access control system for a specific application or scenario.

William C. Garrison, Adam J. Lee, Timothy L. Hinrichs

To date, most work regarding the formal analysis of access control schemes has focused on quantifying and comparing the expressive power of a set of schemes. Although expressive power is important, it is a property that exists in an absolute sense, detached from the application-specific context within which an access control scheme will ultimately be deployed. In this paper, by contrast, we formalize the access control suitability analysis problem, which seeks to evaluate the degree to which a set of candidate access control schemes can meet the needs of an application-specific workload. This process involves both reductions to assess whether a scheme is capable of implementing a workload, as well as cost analysis using ordered measures to quantify the overheads of using each candidate scheme to service the workload. We develop a mathematical framework for analyzing instances of the suitability analysis problem, and evaluate this framework both formally (by quantifying its efficiency and accuracy properties) and practically (by exploring a group-based messaging workload from the literature). An ancillary contribution of our work is the identification of auxiliary machines, which are a useful class of modifications that can be made to enhance the expressive power of an access control scheme without negatively impacting the safety properties of the scheme.