LightDep

• Paper accepted at IEEE ANTS 2026 (IIT Roorkee). I'm joint first author.
• The deployed model is a 3.96 MB INT8 ONNX file, 28× smaller than the teacher it learned from. It runs in 96 ms on one CPU thread, and quantization cost no accuracy.
• 75.9 binary F1 on D-Vlog, within 1.8 points of the best server-side models. The tiny 0.16M version still gets 75.4.
• Distillation lifted depression recall by +10.4 points (p = 0.045). A 10-seed rerun on a second GPU confirmed it on all four metrics (p ≤ 0.003 after Holm correction).
• The same file runs in a browser tab (217 ms), on an iPhone (223 ms) and on a mid-range Android phone (664 ms). Inference needs no server.

Most depression-detection models from papers are huge and need server-side tools like OpenFace to even produce their inputs, so they can't run on a phone. At IIT BHU I worked on shrinking that down: LightDep is a small two-stream transformer that learns from bigger teacher models (knowledge distillation) and then gets quantized for on-device use. It comes in four sizes, from 0.16M to 6.36M parameters. It's a screening-risk indicator for research, not a diagnostic tool.

Two open questions. First, does distilling a big model into a tiny one actually help, or is it just a nice idea? Second, even with a tiny model, can a phone produce the same input features the model was trained on?

I trained the student from scratch and with distillation on two datasets (LMVD and D-Vlog), using the same seeds for both and paired significance tests, so a lucky seed couldn't fake a result. Then I exported the best model to ONNX, quantized it to INT8, and built a bridge that turns MediaPipe face landmarks (which run on phones) into the OpenFace format the model expects, to measure how much the predictions drift.

• Two streams, visual (136-d landmarks) and audio (25-d acoustic features), each with its own small pre-norm transformer, then a fusion transformer and masked mean pooling.
• No positional encodings on purpose, so the model doesn't care about frame rate or clip length. Those are exactly what change between a research video and a phone recording.
• The distillation loss is cross-entropy plus a temperature-scaled KL term. Soft targets from the 5-model teacher ensemble are precomputed once.
• INT8 through ONNX dynamic quantization. PyTorch's eager INT8 breaks nn.TransformerEncoder's fast path, so ONNX was the only clean route. torch↔ONNX outputs match to 3e-7.
• Modality-dropout training (randomly zeroing one stream) so the model still works when audio is missing. Visual-only F1 went from 0.0 to 69.7.
• OpenFace ran in an amd64 Docker image I built myself, because the public arm64 one wouldn't work on x86.
• Compared against logistic regression, an MLP, a GRU and a 1-D CNN under the same training. LightDep-M had the best accuracy and F1. The MLP came close on F1 (the gap isn't significant), but LightDep kept an accuracy edge when 30–50% of frames were dropped.

The main finding is that distillation only helps when the teacher is a stronger and well-calibrated screener. On D-Vlog it was, and distillation raised recall by +10.4 and F1 by +4.3. It also made training more stable across seeds, and on the best checkpoint it halved calibration error (ECE 0.076 → 0.038). On LMVD the teachers were poorly calibrated, and none of the 13 distillation variants I tried improved accuracy or F1 significantly. Distilling from one of them actually made recall worse. Two things I caught along the way. While reproducing the MMFformer teacher I found its weight_decay default was 1e-3 instead of the paper's 0.1. And an early "the model collapses without video" result turned out to be a NaN bug from fully masking a modality, not real model behaviour. What's still open: the visual bridge works (prediction drift 0.041 on a 2-clip proof of concept), but there's no on-device replacement for the audio features yet, and audio is the stream the model relies on most.