publications

2024

1. Arxiv

   OmniParser for Pure Vision Based GUI Agent

   Lu, Yadong, Yang, Jianwei, Shen, Yelong, and Awadallah, Ahmed

   2024
2. Arxiv

   Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

   Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, and Hui, Zack

   2024
3. Arxiv

   Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

   Ren, Liliang, Liu, Yang, Lu, Yadong, Shen, Yelong, Liang, Chen, and Chen, Weizhu

   2024
4. Arxiv

   Multi-LoRA Composition for Image Generation

   Zhong, Ming, Shen, Yelong, Wang, Shuohang, Lu, Yadong, Jiao, Yizhu, Ouyang, Siru, Yu, Donghan, Han, Jiawei, and Chen, Weizhu

   2024

2023

1. Arxiv

   An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models

   Yadong, Lu, Chunyuan, Li, Haotian, Liu, Jianwei, Yang, Jianfeng, Gao, and Yelong, Shen

   Arxiv 2023

   HTML
2. Arxiv

   Efficient RLHF: Reducing the Memory Usage of PPO

   Michael, Santacroce, Yadong, Lu, Han, Yu, Yuanzhi, Li, and Yelong, Shen

   Arxiv 2023

   HTML
3. NeurIPS

   In-Context Learning Unlocked for Diffusion Models

   Wang, Zhendong, Jiang, Yifan, Lu, Yadong, Shen, Yelong, He, Pengcheng, Chen, Weizhu, Wang, Zhangyang, and Zhou, Mingyuan

   NeurIPS Spotlight 2023

   HTML

2022

1. Patent

   Progressive data compression using artificial neural networks

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, and Cohen, Taco

   US Patent 2022

   HTML
2. Patent

   Variable bit rate compression using neural network models

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, Pourreza, Reza, and Cohen, Taco

   US Patent 2022

   HTML
3. JHEP

   Resolving Extreme Jet Substructure

   Lu, Yadong, Romero, Alexis, Fenton, Michael James, Whiteson, Daniel, and Baldi, Pierre

   Journal of High Energy Physics 2022

   Abs HTML

   We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to N=8). Previous studies indicate that high-level observables are powerful, interpretable tools to probe jet substructure for N≤3 hard sub-jets, but that deep neural networks trained on low-level jet constituents match or slightly exceed their performance. We extend this work for up to N=8 hard sub-jets, using deep particle-flow networks (PFNs) and Transformer based networks to estimate a loose upper bound on the classification performance. The high-level jet observables perform well, with an overall classification accuracy of 86.90%, but fall short of the PFN and Transformer models, which reach classification accuracies of 89.19% and 91.27% respectively, suggesting that the constituent networks utilize information not captured by the set of high-level observables. We then identify additional high-level observables which are able to narrow this gap, and utilize LASSO regularization for feature selection to identify and rank the most relevant observables and provide further insights into the learning strategies used by the constituent-based neural networks. The final model contains only 31 high-level observables and is able to match the performance of the PFN and approximate the performance of the Transformer model to within 2%.

2021

1. Phys Rev.D

   Sparse autoregressive models for scalable generation of sparse images in particle physics

   Lu, Yadong, Collado, Julian, Whiteson, Daniel, and Baldi, Pierre

   Physics Review D Feb 2021

   Abs HTML

   Generation of simulated data is essential for data analysis in particle physics, but current Monte Carlo methods are very computationally expensive. Deep-learning-based generative models have successfully generated simulated data at lower cost, but struggle when the data are very sparse. We introduce a novel deep sparse autoregressive model (SARM) that explicitly learns the sparseness of the data with a tractable likelihood, making it more stable and interpretable when compared to generative adversarial networks (GANs) and other methods. In two case studies, we compare SARM to a GAN model and a nonsparse autoregressive model. As a quantitative measure of performance, we compute the Wasserstein distance ( W p ) between the distributions of physical quantities calculated on the generated images and on the training images. In the first study, featuring images of jets in which 90% of the pixels are zero valued, SARM produces images with W p scores that are 24%–52% better than the scores obtained with other state-of-the-art generative models. In the second study, on calorimeter images in the vicinity of muons where 98% of the pixels are zero valued, SARM produces images with W p scores that are 66%–68% better. Similar observations made with other metrics confirm the usefulness of SARM for sparse data in particle physics.

2. ICIP

   Progressive Neural Image Compression With Nested Quantization And Latent Ordering

   Lu, Yadong*, Zhu, Yinhao*, Yang, Yang*, Said, Amir, and Cohen, Taco S

   In IEEE International Conference on Image Processing (ICIP) Feb 2021

   Abs HTML

   We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.

3. IJCAI

   Deep Bucket Elimination

   Razeghi, Yasaman, Kask, Kalev, Lu, Yadong, Baldi, Pierre, Agarwal, Sakshi, and Dechter, Rina

   In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 Aug 2021

   Abs HTML

   Bucket Elimination (BE) is a universal inference scheme that can solve most tasks over probabilistic and deterministic graphical models exactly. However, it often requires exponentially high levels of memory (in the induced-width) preventing its execution. In the spirit of exploiting Deep Learning for inference tasks, in this paper, we will use neural networks to approximate BE. The resulting Deep Bucket Elimination (DBE) algorithm is developed for computing the partition function. We provide a proof-of-concept empirically using instances from several different benchmarks, showing that DBE can be a more accurate approximation than current state-of-the-art approaches for approximating BE (eg the mini-bucket schemes), especially when problems are sufficiently hard.

4. SCAM

   D-REX: Static Detection of Relevant Runtime Exceptions with Location Aware Transformer

   Farmahinifarahani, Farima*, Lu, Yadong*, Saini, Vaibhav, Baldi, Pierre, and Lopes, Cristina

   In 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM) Aug 2021

   Abs HTML

   Runtime exceptions are inevitable parts of software systems. While developers often write exception handling code to avoid the severe outcomes of these exceptions, such code is most effective if accompanied by accurate runtime exception types. Predicting the runtime exceptions that may occur in a program, however, is difficult as the situations that lead to these exceptions are complex. We propose D-REX (Deep Runtime EXception detector), as an approach for predicting runtime exceptions of Java methods based on the static properties of code.The core of D-REX is a machine learning model that leverages the representation learning ability of neural networks to infer a set of signals from code to predict the related runtime exception types. This model, which we call Location Aware Transformer, adapts a state-of-the-art language model, Transformer, to provide accurate predictions for the exception types, as well as interpretable recommendations for the exception prone elements of code. We curate a benchmark dataset of 200,000 Java projects from GitHub to train and evaluate D-REX. Experiments demonstrate that D-REX predicts runtime exception types with 81% of Top 1 accuracy, outperforming multiple non-Transformer baselines by a margin of at least 12%. Furthermore, it can predict the exception prone elements of code with 75% Top 1 precision.

2020

1. ECCV

   Differentiable Joint Pruning and Quantization for Hardware Efficiency

   Wang, Yin, Lu, Yadong, and Blankevoort, Tijmen

   European Conference on Computer Vision (ECCV) Aug 2020

   Abs HTML

   We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize the method for more efficient hardware inference, we extend DJPQ to integrate structured pruning with power-of-two bit-restricted quantization. We show that DJPQ significantly reduces the number of Bit-Operations (BOPs) for several networks while maintaining the top-1 accuracy of original floating-point models (e.g., 53x BOPs reduction in ResNet18 on ImageNet, 43x in MobileNetV2). Compared to the conventional two-stage approach, which optimizes pruning and quantization independently, our scheme outperforms in terms of both accuracy and BOPs. Even when considering bit-restricted quantization, DJPQ achieves larger compression ratios and better accuracy than the two-stage approach.

2. IEEE

   DCNN-Based Multi-Signal Induction Motor Fault Diagnosis

   Shao, Siyu, Yan, Ruqiang, Lu, Yadong, Wang, Peng, and Gao, Robert X.

   IEEE Transactions on Instrumentation and Measurement Aug 2020

   Abs HTML

   Deep learning (DL) architecture, which exploits multiple hidden layers to learn hierarchical representations automatically from massive input data, presents a promising tool for characterizing fault conditions. This paper proposes a DL-based multi-signal fault diagnosis method that leverages the powerful feature learning ability of a convolutional neural network (CNN) in images. The proposed deep model is able to learn from multiple types of sensor signals simultaneously so that it can achieve robust performance and finally realize accurate induction motor fault recognition. First, the acquired sensor signals are converted to time-frequency distribution (TFD) by wavelet transform. Then, a deep CNN is applied to learning discriminative representations from the TFD images. Since then, a fully connected layer in deep architecture gives the prediction of induction motor condition based on learned features. In order to verify the effectiveness of the designed deep model, experiments are carried out on a machine fault simulator where both vibration and current signals are analyzed. Experimental results indicate that the proposed method outperforms traditional fault diagnosis methods, hence, demonstrating effectiveness in induction motor application. Compared with conventional methods that rely on delicate features extracted by experienced experts, the proposed deep model is able to automatically learn and select suitable features that contribute to accurate fault diagnosis. Compared with single-signal input, the multi-signal model has more accurate and stable performance and overcomes the overfitting problem to some degree.

3. AP

   Scalable Bayesian variable selection regression models for count data

   Miao, Yinsen, Kook, Jeong Hwan, Lu, Yadong, Guindani, Michele, and Vannucci, Marina

   In Flexible Bayesian Regression Modelling, Academic Press. Aug 2020

   Abs HTML

   Variable selection, also known as feature selection in the machine learning literature, plays an indispensable role in scientific studies. In many research areas with massive data, finding a subset of representative features that best explain the outcome of interest has become a critical component in any researcher’s workflow. In this chapter, we focus on Bayesian variable selection regression models for count data, and specifically on the negative binomial linear regression model and on the Dirichlet-multinomial regression model. We address the variable selection problem via spike-and-slab priors. For posterior inference, we review standard MCMC methods and also investigate computationally more efficient variational inference approaches that use data augmentation techniques and concrete relaxation methods. We investigate performance of the methods via simulation studies and benchmark data sets.

2019

1. ICSE

   Towards Automating Precision Studies of Clone Detectors

   Saini, Vaibhav, Farmahinifarahani, Farima, Lu, Yadong, Yang, Di, Martins, Pedro, Sajnani, Hitesh, Baldi, Pierre, and Lopes, Cristina V.

   In Proceedings of the 41st International Conference on Software Engineering Aug 2019

   Abs HTML

   Current research in clone detection suffers from poor ecosystems for evaluating precision of clone detection tools. Corpora of labeled clones are scarce and incomplete, making evaluation labor intensive and idiosyncratic, and limiting inter-tool comparison. Precision-assessment tools are simply lacking.We present a semiautomated approach to facilitate precision studies of clone detection tools. The approach merges automatic mechanisms of clone classification with manual validation of clone pairs. We demonstrate that the proposed automatic approach has a very high precision and it significantly reduces the number of clone pairs that need human validation during precision experiments. Moreover, we aggregate the individual effort of multiple teams into a single evolving dataset of labeled clone pairs, creating an important asset for software clone research.

2. NeurIPS-ML4S

   Sparse Image Generation with Decoupled Generative Models

   Lu, Yadong, Collad, Julian, Bauer, Kevin, Whiteson, Daniel, and Baldi, Pierre

   In Aug 2019

   Abs HTML

   Learning to generate very sparse images with deep generative models has been a challenging problem due to issues such as mode collapse and sparse gradient signals. In this work, we propose a novel model combining a neural network generator with an explicit mixture model, which induces sparsity with a learnable Dirac delta mass at zero instead of using rectified linear units at the output. Our model decouples the sparsity level and the non-zero distribution of each pixel in the data while fitting both of them simultaneously by minimizing the model’s entropy. We demonstrate both theoretically and empirically that the model is able to learn a rich distribution of a sparse muon image dataset while maintaining desired physical properties such as isolation.

2018

1. ESEC/FSE

   Oreo: Detection of Clones in the Twilight Zone

   Saini, Vaibhav, Farmahinifarahani, Farima, Lu, Yadong, Baldi, Pierre, and Lopes, Cristina V.

   In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering Aug 2018

   Abs HTML

   Source code clones are categorized into four types of increasing difficulty of detection, ranging from purely textual (Type-1) to purely semantic (Type-4). Most clone detectors reported in the literature work well up to Type-3, which accounts for syntactic differences. In between Type-3 and Type-4, however, there lies a spectrum of clones that, although still exhibiting some syntactic similarities, are extremely hard to detect – the Twilight Zone. Most clone detectors reported in the literature fail to operate in this zone. We present Oreo, a novel approach to source code clone detection that not only detects Type-1 to Type-3 clones accurately, but is also capable of detecting harder-to-detect clones in the Twilight Zone. Oreo is built using a combination of machine learning, information retrieval, and software metrics. We evaluate the recall of Oreo on BigCloneBench, and perform manual evaluation for precision. Oreo has both high recall and precision. More importantly, it pushes the boundary in detection of clones with moderate to weak syntactic similarity in a scalable manner