Comparative Analysis of Deep Learning Architectures for Student Engagement Detection: A Study on FER-2013 and DAiSEE Datasets

Abstract

Automated student engagement detection presents a critical challenge in modern educational technology: existing deep learning systems trained on the DAiSEE dataset achieve deceptively high benchmark accuracy by defaulting to the majority “Engagement” class, leaving pedagogically vital states such as Boredom, Confusion, and Frustration effectively undetected. This paper presents a comparative experimental study of machine learning and deep learning architectures applied across two bench mark datasets FER-2013 and DAiSEE with a specific focus on resolving class imbalance through aggressive class- weighting and surgical transfer learning. On FER-2013, we first establish a traditional ML baseline and subsequently train an enhanced custom CNN with data augmentation, batch normalization, and adaptive learning rate scheduling, achieving a best validation accuracy of 61.9% over 50 epochs. On the DAiSEE dataset, sampled at three frames per video, we evaluate four classical classifiers and two deep learning pipelines. A Random Forest baseline achieves 89.91%, while a fine-tuned MobileNetV2 with standard class-weighting reaches 74.76%, improving to 80.34% under aggressive re-weighting (Frustration weight ≈78.7×). A custom CNN trained end-to-end on DAiSEE frames achieves the highest reported accuracy of 93.83%. All inference is designed to run locally on standard consumer hardware without GPU dependency, satisfying privacy-first edge-computing constraints for real classroom deployment. Our results demonstrate that targeted class-weighting combined with custom CNN architectures substantially outperforms generic transfer learning on engagement detection tasks, and we identify temporal smoothing as the primary open challenge for real- time inference.