Method and accuracy
Developed by Divyansh Srivastava. Every number below comes from the evaluation run that produced the deployed model. Nothing here is estimated or rounded in the model's favour.
Pipeline
- The image is resized to at most 1280 px on its longest side.
- MTCNN finds the largest face. If the face is under 48 px or its sharpness is under 40, the system makes no decision.
- The image is passed through one H.264 compression step at CRF 40, because the training videos were compressed the same way.
- The face is cropped with a 30% margin and resized to 299 x 299 px.
- An Xception network pretrained on ImageNet and fine-tuned for this task scores the crop and its mirror image. The two scores are averaged.
- The result is called fake only at a probability of 98.2% or higher, uncertain from 49.1%, and otherwise no manipulation detected.
Training data
17,680 face crops: FaceForensics++ (real videos and Deepfakes-method fakes, c40 compression), FFHQ real faces and StyleGAN fake faces (2,500 each), UTKFace Indian-labelled real faces (1,500), and LFW real faces (400). Videos and images were split 70, 15 and 15 percent into train, validation and test before any scoring. The decision threshold was chosen on the validation split so that at most 2% of real faces from each real source are flagged fake.
Test results
| Set | Level | Real | Fake | AUC | Accuracy | Recall | Precision |
|---|---|---|---|---|---|---|---|
| All test items | Video | 678 | 525 | 0.946 | 75.7% | 46.3% | 96.1% |
| All test items | Frame | 1380 | 1274 | 0.966 | 83.9% | 68.5% | 97.1% |
| FF++ video faces | Video | 18 | 150 | 0.987 | 75.0% | 72.0% | 100% |
| FFHQ vs StyleGAN | Frame | 375 | 375 | 0.882 | 66.7% | 36.0% | 93.1% |
Overall video-level AUC 95% bootstrap interval: 0.934 to 0.956. Fake is the positive class. Recall is low because the threshold is deliberately strict.
Per source, at the fixed threshold
| Source | Items | Rate | Meaning |
|---|---|---|---|
| FFHQ (real) | 375 | 2.7% | false-fake rate |
| FF++ video faces (real) | 18 | 0.0% | false-fake rate |
| LFW (real) | 60 | 0.0% | false-fake rate |
| UTKFace, Indian-labelled (real) | 225 | 0.0% | false-fake rate |
| FF++ Deepfakes (fake) | 150 | 72.0% | detected |
| StyleGAN (fake) | 375 | 36.0% | detected |
Across all 678 real test items, 10 were wrongly flagged fake (1.5%). Only 18 real items come from FF++ video faces, so that row is not reliable on its own.
Robustness to small faces
The same test frames were downscaled and re-compressed (JPEG quality 50) before scoring. AUC on all test frames, by face size:
| Face size | AUC |
|---|---|
| 299 px | 0.954 |
| 128 px | 0.950 |
| 64 px | 0.940 |
| 48 px | 0.922 |
Limits you should know
- These tests use the same data sources as training, with the splits kept apart. They are not a generalisation result. A cross-dataset test on Celeb-DF v2 was not run.
- The author also tried 29 personal photos (10 AI-generated, 19 real). Of the AI images, 2 were called fake, 6 uncertain, 1 real and 1 had no face found. Of the real photos, 1 was wrongly called fake, 9 uncertain, 8 real and 1 had no face found. This is an informal check, not a benchmark, and it shows the model does not yet cope well with modern phone photos or recent AI generators.
- Only one face-swap method (Deepfakes) and one GAN (StyleGAN) were used for fakes. Other methods, diffusion models and video are not covered.
- The Indian-face data is a labelled subset of UTKFace and covers real faces only. Demographic fairness has not been measured.
- Not a security guarantee, and not legal advice. A probability, not proof.
References
- Rossler et al., FaceForensics++: Learning to Detect Manipulated Facial Images, ICCV 2019.
- Chollet, Xception: Deep Learning with Depthwise Separable Convolutions, CVPR 2017.
- Zhang et al., Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks (MTCNN), 2016.
- Karras et al., A Style-Based Generator Architecture for GANs (FFHQ and StyleGAN), CVPR 2019.
- Huang et al., Labeled Faces in the Wild, 2007. Zhang et al., UTKFace, CVPR 2017.