Yadong (Adam) Lu

Recent publications

1. Arxiv

   OmniParser for Pure Vision Based GUI Agent

   Lu, Yadong, Yang, Jianwei, Shen, Yelong, and Awadallah, Ahmed

   2024
2. Arxiv

   Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

   Ren, Liliang, Liu, Yang, Lu, Yadong, Shen, Yelong, Liang, Chen, and Chen, Weizhu

   2024
3. Arxiv

   Multi-LoRA Composition for Image Generation

   Zhong, Ming, Shen, Yelong, Wang, Shuohang, Lu, Yadong, Jiao, Yizhu, Ouyang, Siru, Yu, Donghan, Han, Jiawei, and Chen, Weizhu

   2024
4. Arxiv

   An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models

   Yadong, Lu, Chunyuan, Li, Haotian, Liu, Jianwei, Yang, Jianfeng, Gao, and Yelong, Shen

   Arxiv 2023

   HTML
5. Arxiv

   Efficient RLHF: Reducing the Memory Usage of PPO

   Michael, Santacroce, Yadong, Lu, Han, Yu, Yuanzhi, Li, and Yelong, Shen

   Arxiv 2023

   HTML
6. NeurIPS

   In-Context Learning Unlocked for Diffusion Models

   Wang, Zhendong, Jiang, Yifan, Lu, Yadong, Shen, Yelong, He, Pengcheng, Chen, Weizhu, Wang, Zhangyang, and Zhou, Mingyuan

   NeurIPS Spotlight 2023

   HTML
7. Patent

   Progressive data compression using artificial neural networks

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, and Cohen, Taco

   US Patent 2022

   HTML
8. Patent

   Variable bit rate compression using neural network models

   Lu, Yadong, Yang, Yang, Zhu, Yinhao, Said, Amir, Pourreza, Reza, and Cohen, Taco

   US Patent 2022

   HTML
9. JHEP

   Resolving Extreme Jet Substructure

   Lu, Yadong, Romero, Alexis, Fenton, Michael James, Whiteson, Daniel, and Baldi, Pierre

   Journal of High Energy Physics 2022

   Abs HTML

   We study the effectiveness of theoretically-motivated high-level jet observables in the extreme context of jets with a large number of hard sub-jets (up to N=8). Previous studies indicate that high-level observables are powerful, interpretable tools to probe jet substructure for N≤3 hard sub-jets, but that deep neural networks trained on low-level jet constituents match or slightly exceed their performance. We extend this work for up to N=8 hard sub-jets, using deep particle-flow networks (PFNs) and Transformer based networks to estimate a loose upper bound on the classification performance. The high-level jet observables perform well, with an overall classification accuracy of 86.90%, but fall short of the PFN and Transformer models, which reach classification accuracies of 89.19% and 91.27% respectively, suggesting that the constituent networks utilize information not captured by the set of high-level observables. We then identify additional high-level observables which are able to narrow this gap, and utilize LASSO regularization for feature selection to identify and rank the most relevant observables and provide further insights into the learning strategies used by the constituent-based neural networks. The final model contains only 31 high-level observables and is able to match the performance of the PFN and approximate the performance of the Transformer model to within 2%.

10. Phys Rev.D

    Sparse autoregressive models for scalable generation of sparse images in particle physics

    Lu, Yadong, Collado, Julian, Whiteson, Daniel, and Baldi, Pierre

    Physics Review D Feb 2021

    Abs HTML

    Generation of simulated data is essential for data analysis in particle physics, but current Monte Carlo methods are very computationally expensive. Deep-learning-based generative models have successfully generated simulated data at lower cost, but struggle when the data are very sparse. We introduce a novel deep sparse autoregressive model (SARM) that explicitly learns the sparseness of the data with a tractable likelihood, making it more stable and interpretable when compared to generative adversarial networks (GANs) and other methods. In two case studies, we compare SARM to a GAN model and a nonsparse autoregressive model. As a quantitative measure of performance, we compute the Wasserstein distance ( W p ) between the distributions of physical quantities calculated on the generated images and on the training images. In the first study, featuring images of jets in which 90% of the pixels are zero valued, SARM produces images with W p scores that are 24%–52% better than the scores obtained with other state-of-the-art generative models. In the second study, on calorimeter images in the vicinity of muons where 98% of the pixels are zero valued, SARM produces images with W p scores that are 66%–68% better. Similar observations made with other metrics confirm the usefulness of SARM for sparse data in particle physics.

11. ECCV

    Differentiable Joint Pruning and Quantization for Hardware Efficiency

    Wang, Yin, Lu, Yadong, and Blankevoort, Tijmen

    European Conference on Computer Vision (ECCV) Feb 2020

    Abs HTML

    We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatically for hardware efficiency. DJPQ incorporates variational information bottleneck based structured pruning and mixed-bit precision quantization into a single differentiable loss function. In contrast to previous works which consider pruning and quantization separately, our method enables users to find the optimal trade-off between both in a single training procedure. To utilize the method for more efficient hardware inference, we extend DJPQ to integrate structured pruning with power-of-two bit-restricted quantization. We show that DJPQ significantly reduces the number of Bit-Operations (BOPs) for several networks while maintaining the top-1 accuracy of original floating-point models (e.g., 53x BOPs reduction in ResNet18 on ImageNet, 43x in MobileNetV2). Compared to the conventional two-stage approach, which optimizes pruning and quantization independently, our scheme outperforms in terms of both accuracy and BOPs. Even when considering bit-restricted quantization, DJPQ achieves larger compression ratios and better accuracy than the two-stage approach.

12. ICIP

    Progressive Neural Image Compression With Nested Quantization And Latent Ordering

    Lu, Yadong*, Zhu, Yinhao*, Yang, Yang*, Said, Amir, and Cohen, Taco S

    In IEEE International Conference on Image Processing (ICIP) Feb 2021

    Abs HTML

    We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the finest quantization level. To achieve finer progressiveness in between any two quantization levels, latent elements are incrementally refined with an importance ordering defined in the rate-distortion sense. To the best of our knowledge, PLONQ is the first learning-based progressive image coding scheme and it outperforms SPIHT, a well-known wavelet-based progressive image codec.