Yadong (Adam) Lu

I am a principal researcher at Microsoft Research, Redmond working on computer use agent and efficient pre/post-training methodologies of large vision-language models. Before that I received my Ph.D. degree in Statistics at UC Irvine (advisor: Pierre Baldi). I worked on a wide variety of scalable machine learning algorithms with applications to neural network model efficiency, image processing, and natural language processing.

Check out our recent work on computer use agent OmniParser (ranked #1 Trending repo on GitHub and HuggingFace model hub, 24k+ star so far), and scaling synthetic trajectory data for web agent.

Selected Works

Arxiv

Beyond Clicking: A Step Towards Generalist GUI Grounding via Text Dragging

Liao, Zeyi, Lu, Yadong, Gou, Boyu, Sun, Huan, and Awadallah, Ahmed

2025

HTML
ACL 2025

Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Pahuja, Vardaan, Lu, Yadong, Rosset, Corby, Gou, Boyu, Mitra, Arindam, Whitehead, Spencer, Su, Yu, and Awadallah, Ahmed

ACL 2025 2025

HTML
Arxiv

OmniParser for Pure Vision Based GUI Agent

Lu, Yadong, Yang, Jianwei, Shen, Yelong, and Awadallah, Ahmed

2024

HTML
CVPR

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, and Hui, Zack

CVPR 2025

HTML
ICLR

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

Ren, Liliang, Liu, Yang, Lu, Yadong, Shen, Yelong, Liang, Chen, and Chen, Weizhu

ICLR 2024

HTML
TMLR

Multi-LoRA Composition for Image Generation

Zhong, Ming, Shen, Yelong, Wang, Shuohang, Lu, Yadong, Jiao, Yizhu, Ouyang, Siru, Yu, Donghan, Han, Jiawei, and Chen, Weizhu

TMLR 2024

HTML
Arxiv

An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models

Yadong, Lu, Chunyuan, Li, Haotian, Liu, Jianwei, Yang, Jianfeng, Gao, and Yelong, Shen

NeurIPS, Workshop on Instruction Tuning and Instruction Following NeurIPS

HTML
Arxiv

Efficient RLHF: Reducing the Memory Usage of PPO

Michael, Santacroce, Yadong, Lu, Han, Yu, Yuanzhi, Li, and Yelong, Shen

Arxiv 2023

HTML
NeurIPS

In-Context Learning Unlocked for Diffusion Models

Wang, Zhendong, Jiang, Yifan, Lu, Yadong, Shen, Yelong, He, Pengcheng, Chen, Weizhu, Wang, Zhangyang, and Zhou, Mingyuan

NeurIPS Spotlight 2023

HTML
Patent

Progressive data compression using artificial neural networks

Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, and Cohen, Taco

US Patent 2022

HTML
Patent

Variable bit rate compression using neural network models

Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, Pourreza, Reza, and Cohen, Taco

US Patent 2022

HTML
JHEP

Resolving Extreme Jet Substructure

Lu, Yadong, Romero, Alexis, Fenton, Michael James, Whiteson, Daniel, and Baldi, Pierre

Journal of High Energy Physics 2022

Abs HTML

We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to N=8). Previous studies indicate that high-level observables are powerful, interpretable tools to probe jet substructure for N≤3 hard sub-jets, but that deep neural networks trained on low-level jet constituents match or slightly exceed their performance. We extend this work for up to N=8 hard sub-jets, using deep particle-flow networks (PFNs) and Transformer based networks to estimate a loose upper bound on the classification performance. The high-level jet observables perform well, with an overall classification accuracy of 86.90%, but fall short of the PFN and Transformer models, which reach classification accuracies of 89.19% and 91.27% respectively, suggesting that the constituent networks utilize information not captured by the set of high-level observables. We then identify additional high-level observables which are able to narrow this gap, and utilize LASSO regularization for feature selection to identify and rank the most relevant observables and provide further insights into the learning strategies used by the constituent-based neural networks. The final model contains only 31 high-level observables and is able to match the performance of the PFN and approximate the performance of the Transformer model to within 2%.
ECCV

Differentiable Joint Pruning and Quantization for Hardware Efficiency

Wang, Yin, Lu, Yadong, and Blankevoort, Tijmen

European Conference on Computer Vision (ECCV) 2020

Abs HTML

We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize the method for more efficient hardware inference, we extend DJPQ to integrate structured pruning with power-of-two bit-restricted quantization. We show that DJPQ significantly reduces the number of Bit-Operations (BOPs) for several networks while maintaining the top-1 accuracy of original floating-point models (e.g., 53x BOPs reduction in ResNet18 on ImageNet, 43x in MobileNetV2). Compared to the conventional two-stage approach, which optimizes pruning and quantization independently, our scheme outperforms in terms of both accuracy and BOPs. Even when considering bit-restricted quantization, DJPQ achieves larger compression ratios and better accuracy than the two-stage approach.
ICIP

Progressive Neural Image Compression With Nested Quantization And Latent Ordering

Lu, Yadong*, Zhu, Yinhao*, Yang, Yang*, Said, Amir, and Cohen, Taco S

In IEEE International Conference on Image Processing (ICIP) 2021

Abs HTML

We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.