Yadong (Adam) Lu

Selected Works

1. Arxiv

   OmniParser for Pure Vision Based GUI Agent

   Lu, Yadong, Yang, Jianwei, Shen, Yelong, and Awadallah, Ahmed

   2024

   HTML
2. CVPR

   Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

   Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, and Hui, Zack

   CVPR 2025

   HTML
3. ICLR

   Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

   Ren, Liliang, Liu, Yang, Lu, Yadong, Shen, Yelong, Liang, Chen, and Chen, Weizhu

   ICLR 2024

   HTML
4. TMLR

   Multi-LoRA Composition for Image Generation

   Zhong, Ming, Shen, Yelong, Wang, Shuohang, Lu, Yadong, Jiao, Yizhu, Ouyang, Siru, Yu, Donghan, Han, Jiawei, and Chen, Weizhu

   TMLR 2024

   HTML
5. Arxiv

   An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models

   Yadong, Lu, Chunyuan, Li, Haotian, Liu, Jianwei, Yang, Jianfeng, Gao, and Yelong, Shen

   NeurIPS, Workshop on Instruction Tuning and Instruction Following NeurIPS

   HTML
6. Arxiv

   Efficient RLHF: Reducing the Memory Usage of PPO

   Michael, Santacroce, Yadong, Lu, Han, Yu, Yuanzhi, Li, and Yelong, Shen

   Arxiv 2023

   HTML
7. NeurIPS

   In-Context Learning Unlocked for Diffusion Models

   Wang, Zhendong, Jiang, Yifan, Lu, Yadong, Shen, Yelong, He, Pengcheng, Chen, Weizhu, Wang, Zhangyang, and Zhou, Mingyuan

   NeurIPS Spotlight 2023

   HTML
8. Patent

   Progressive data compression using artificial neural networks

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, and Cohen, Taco

   US Patent 2022

   HTML
9. Patent

   Variable bit rate compression using neural network models

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, Pourreza, Reza, and Cohen, Taco

   US Patent 2022

   HTML
10. JHEP

    Resolving Extreme Jet Substructure

    Lu, Yadong, Romero, Alexis, Fenton, Michael James, Whiteson, Daniel, and Baldi, Pierre

    Journal of High Energy Physics 2022

    Abs HTML

    We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to N=8). Previous studies indicate that high-level observables are powerful, interpretable tools to probe jet substructure for N≤3 hard sub-jets, but that deep neural networks trained on low-level jet constituents match or slightly exceed their performance. We extend this work for up to N=8 hard sub-jets, using deep particle-flow networks (PFNs) and Transformer based networks to estimate a loose upper bound on the classification performance. The high-level jet observables perform well, with an overall classification accuracy of 86.90%, but fall short of the PFN and Transformer models, which reach classification accuracies of 89.19% and 91.27% respectively, suggesting that the constituent networks utilize information not captured by the set of high-level observables. We then identify additional high-level observables which are able to narrow this gap, and utilize LASSO regularization for feature selection to identify and rank the most relevant observables and provide further insights into the learning strategies used by the constituent-based neural networks. The final model contains only 31 high-level observables and is able to match the performance of the PFN and approximate the performance of the Transformer model to within 2%.

11. ECCV

    Differentiable Joint Pruning and Quantization for Hardware Efficiency

    Wang, Yin, Lu, Yadong, and Blankevoort, Tijmen

    European Conference on Computer Vision (ECCV) 2020

    Abs HTML

    We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize the method for more efficient hardware inference, we extend DJPQ to integrate structured pruning with power-of-two bit-restricted quantization. We show that DJPQ significantly reduces the number of Bit-Operations (BOPs) for several networks while maintaining the top-1 accuracy of original floating-point models (e.g., 53x BOPs reduction in ResNet18 on ImageNet, 43x in MobileNetV2). Compared to the conventional two-stage approach, which optimizes pruning and quantization independently, our scheme outperforms in terms of both accuracy and BOPs. Even when considering bit-restricted quantization, DJPQ achieves larger compression ratios and better accuracy than the two-stage approach.

12. ICIP

    Progressive Neural Image Compression With Nested Quantization And Latent Ordering

    Lu, Yadong*, Zhu, Yinhao*, Yang, Yang*, Said, Amir, and Cohen, Taco S

    In IEEE International Conference on Image Processing (ICIP) 2021

    Abs HTML

    We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.