Avatar

Lei Li

Master Student

Peking University

About

Hi, I am a final year master student at School of EECS, Peking University.

Currently, I am a member of Language Computing and Machine Learning Group (LANCO), advised by Prof. Xu Sun.

I am enthusiastic about Natural Lanuge Processing (NLP). My research focuses on applying machine learning techniques for improving the efficiency of pre-trained language models.

Interests

  • Multi-modal Pre-training
  • Efficient Methods for NLP
  • Knowledgeable NLP

Education

  • MSc in Computer Science, 2020 - 2023 (Expected)

    Peking University

  • BE in Software Engineering, 2016 - 2020

    Xidian University

Publications

Distributional Correlation–Aware Knowledge Distillation for Stock Trading Volume Prediction

Traditional knowledge distillation in classification problems transfers the knowledge via class correlations in the soft label produced …
Lei Li, Zhiyuan Zhang, Ruihan Bao, Keiko Harimoto, Xu Sun
PDF Code

Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models

As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to …
Lei Li, Yankai Lin, Xuancheng Ren, Guangxiang Zhao, Peng Li, Jie Zhou, Xu Sun
PDF

Rethinking the Openness of CLIP

Contrastive Language-Image Pre-training (CLIP) has demonstrated great potential in realizing open-vocabulary image classification in a …
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, Xu Sun
PDF

Dynamic Knowledge Distillation for Pre-trained Language Models

Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun
PDF Code

CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade

Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun
PDF Code
See all publications