Knowledge Distillation for Real-Time Classification of Early Media in Voice Communications

Classifying early media audio streams in real-time is a critical challenge for communication platforms relying on VoIP. Neural audio tagging models like CNN14 have set the standard for accuracy on large datasets, but their resource demands hinder practical deployment at scale. In this work, we directly tackled this tradeoff by comparing heavyweight neural approaches against a lightweight gradient-boosted tree (GBT) model, engineered specifically for the industrial realities of voice communications.

Everyone assumes that convolutional networks are necessary for robust audio classification, given their dominance on tasks like AudioSet tagging. But our experiments show a more pragmatic path: by distilling knowledge from a state-of-the-art neural network (CNN14) into a simple GBT model—combined with smart class aggregation—we can replicate nearly all of the accuracy at a fraction of the computational cost. On our proprietary early media dataset, the GBT classifier achieves a 99.3% alignment with CNN14’s predictions. The only recurring confusion is between music and announcement classes, which is expected due to overlap in acoustic features (think acapella or rap tracks versus spoken announcements).

Crucially, speed gains are massive. The median inference time for the GBT model is about 1.74 ms per second of audio, compared to 68 ms for CNN14—representing a 39× improvement. This translates to serving 3× more concurrent calls on the same hardware, validated in production with over 17 million real segments processed at our India data center. The operational savings are real and immediate; we saw average per-file processing drop from 77.81 ms to 25.05 ms while keeping misclassification risk minimal.

If there’s a catch, it’s that distillation does reduce the granularity—you’re moving from 527 classes to just four main ones (announcement, music, ringing, silence). This works because only announcements trigger further costly analysis, and so ‘good enough’ really is good enough for the purpose. On public AudioSet data, the GBT matches CNN14’s mAP almost exactly (0.193 vs 0.2), but struggles—just like CNN14—with silence and ringing classes. That’s an area for future improvement.

The bigger message: with domain-focused aggregation and well-executed knowledge distillation, simple models can outperform resource-hungry deep networks for targeted real-time tasks. The work proves that practical engineering decisions—reducing classes, moving to GBTs—pay off at production scale, with no significant loss of utility for the target application.

@misc{altwlkany2024knowledge,
  title = {Knowledge Distillation for Real-Time Classification of Early Media in Voice Communications},
  author = {Altwlkany, K. and Had\v{z}i\'{c}, H. and Kuri\'{c}, A. and Laci\'{c}, E.},
  year = {2024},
  howpublished = {2024 32nd International Conference on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS)},
  pages = {1-4},
  publisher = {IEEE},
  doi = {10.1109/mascots64422.2024.10786538},
  url = {https://doi.org/10.1109/mascots64422.2024.10786538},
}