The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
This addresses inefficiencies in deploying NLP models for practitioners, highlighting a critical but often overlooked bottleneck.
The paper identifies that improvements in NLP model efficiency and hardware do not translate to faster inference due to bottlenecks from deep learning frameworks, termed the 'framework tax', and shows this disparity grows with hardware advancements.
Increased focus on the computational efficiency of NLP systems has motivated the design of efficient model architectures and improvements to underlying hardware accelerators. However, the resulting increases in computational throughput and reductions in floating point operations have not directly translated to improvements in wall-clock inference latency. We demonstrate that these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomenon as the \textit{framework tax}, and observe that the disparity is growing as hardware speed increases over time. In this work, we examine this phenomenon through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency. Code is available at https://github.com/JaredFern/Framework-Tax.