← Back to Publications List

Token Pruning in Transformer Models for Efficient Natural Language Processing

Students & Supervisors

Student Authors
Hasin Almas Sifat
Bachelor of Science in Computer Science & Engineering, FST
Arizit Chaki Artha
Bachelor of Science in Computer Science & Engineering, FST
Koushik Biswas Arko
Bachelor of Science in Computer Science & Engineering, FST
Supervisors
Md. Mortuza Ahmmed
Associate Professor, Faculty, FST

Abstract

Transformer-based models have reached the state-of-the-art performance on natural language processing tasks, especially in sentiment classification. Nonetheless, they are computationally expensive and cannot be practically used in real-time and resource-constrained settings. This paper explores the concept of token pruning and how it can be used to enhance computation efficiency without compromising predictive performance. There are four pretrained transformer models; BERT, RoBERTa, ALBERT, and ELECTRA that are tested on IMDb Movie Reviews dataset in three different configurations, no pruning, attention based pruning, and gradient based pruning. The models are evaluated in terms of accuracy, Matthews Correlation Coefficient (MCC), inference latency, and floating point operations (FLOPs).These results indicate that RoBERTa-base has the highest classification accuracy and ELECTRA small has the best computational efficiency. The cost of computation is decreased at a small cost to accuracy with token pruning, and this suggests redundancy in the input sequences. Gradient-based pruning is a more stable and reliable pruning technique than attention-based pruning. In general, the results indicate that a proper choice of model architecture and pruning strategy can result in a desirable trade off between performance and efficiency, and transformer models have more realistic chances of being deployed in the real world.

Keywords

Transformer Models Token Pruning· Sentiment Analysis· Model Efficiency· Computational Cost· Gradient-Based Pruning· Attention Mechanism

Publication Details

  • Type of Publication:
  • Conference Name: 4th International Conference on Data Analytics and Insights (ICDAI-2026)
  • Date of Conference: 30/07/2026 - 30/07/2026
  • Venue: Techno International New Town, Block- DG 1/1, Action Area 1, New Town, Kolkata-700156
  • Organizer: Techno International New Town
View on Publisher