Yadong (Adam) Lu

I am a senior researcher at Microsoft Research, Redmond working on vision and language models and efficient finetuning methodologies of OpenAI models. Before that I received my Ph.D. degree in Statistics at UC Irvine (advisor: Pierre Baldi).

I worked on a wide variety of scalable machine learning algorithms with applications to neural network model efficiency, image processing, and natural language processing.

Recent publications

Arxiv

An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models

Yadong, Lu, Chunyuan, Li, Haotian, Liu, Jianwei, Yang, Jianfeng, Gao, and Yelong, Shen

Arxiv 2023

HTML
Arxiv

Efficient RLHF: Reducing the Memory Usage of PPO

Michael, Santacroce, Yadong, Lu, Han, Yu, Yuanzhi, Li, and Yelong, Shen

Arxiv 2023

HTML
NeurIPS

In-Context Learning Unlocked for Diffusion Models

Wang, Zhendong, Jiang, Yifan, Lu, Yadong, Shen, Yelong, He, Pengcheng, Chen, Weizhu, Wang, Zhangyang, and Zhou, Mingyuan

NeurIPS Spotlight 2023

HTML
Patent

Progressive data compression using artificial neural networks

Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, and Cohen, Taco

US Patent 2022

HTML
Patent

Variable bit rate compression using neural network models

Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, Pourreza, Reza, and Cohen, Taco

US Patent 2022

HTML
JHEP

Resolving Extreme Jet Substructure

Lu, Yadong, Romero, Alexis, Fenton, Michael James, Whiteson, Daniel, and Baldi, Pierre

Journal of High Energy Physics 2022

Abs HTML

We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to N=8). Previous studies indicate that high-level observables are powerful, interpretable tools to probe jet substructure for N≤3 hard sub-jets, but that deep neural networks trained on low-level jet constituents match or slightly exceed their performance. We extend this work for up to N=8 hard sub-jets, using deep particle-flow networks (PFNs) and Transformer based networks to estimate a loose upper bound on the classification performance. The high-level jet observables perform well, with an overall classification accuracy of 86.90%, but fall short of the PFN and Transformer models, which reach classification accuracies of 89.19% and 91.27% respectively, suggesting that the constituent networks utilize information not captured by the set of high-level observables. We then identify additional high-level observables which are able to narrow this gap, and utilize LASSO regularization for feature selection to identify and rank the most relevant observables and provide further insights into the learning strategies used by the constituent-based neural networks. The final model contains only 31 high-level observables and is able to match the performance of the PFN and approximate the performance of the Transformer model to within 2%.
Phys Rev.D

Sparse autoregressive models for scalable generation of sparse images in particle physics

Lu, Yadong, Collado, Julian, Whiteson, Daniel, and Baldi, Pierre

Physics Review D Feb 2021

Abs HTML

Generation of simulated data is essential for data analysis in particle physics, but current Monte Carlo methods are very computationally expensive. Deep-learning-based generative models have successfully generated simulated data at lower cost, but struggle when the data are very sparse. We introduce a novel deep sparse autoregressive model (SARM) that explicitly learns the sparseness of the data with a tractable likelihood, making it more stable and interpretable when compared to generative adversarial networks (GANs) and other methods. In two case studies, we compare SARM to a GAN model and a nonsparse autoregressive model. As a quantitative measure of performance, we compute the Wasserstein distance ( W p ) between the distributions of physical quantities calculated on the generated images and on the training images. In the first study, featuring images of jets in which 90% of the pixels are zero valued, SARM produces images with W p scores that are 24%–52% better than the scores obtained with other state-of-the-art generative models. In the second study, on calorimeter images in the vicinity of muons where 98% of the pixels are zero valued, SARM produces images with W p scores that are 66%–68% better. Similar observations made with other metrics confirm the usefulness of SARM for sparse data in particle physics.
ECCV

Differentiable Joint Pruning and Quantization for Hardware Efficiency

Wang, Yin, Lu, Yadong, and Blankevoort, Tijmen

European Conference on Computer Vision (ECCV) Feb 2020

Abs HTML

We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize the method for more efficient hardware inference, we extend DJPQ to integrate structured pruning with power-of-two bit-restricted quantization. We show that DJPQ significantly reduces the number of Bit-Operations (BOPs) for several networks while maintaining the top-1 accuracy of original floating-point models (e.g., 53x BOPs reduction in ResNet18 on ImageNet, 43x in MobileNetV2). Compared to the conventional two-stage approach, which optimizes pruning and quantization independently, our scheme outperforms in terms of both accuracy and BOPs. Even when considering bit-restricted quantization, DJPQ achieves larger compression ratios and better accuracy than the two-stage approach.
ICIP

Progressive Neural Image Compression With Nested Quantization And Latent Ordering

Lu, Yadong*, Zhu, Yinhao*, Yang, Yang*, Said, Amir, and Cohen, Taco S

In IEEE International Conference on Image Processing (ICIP) Feb 2021

Abs HTML

We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.