Physical changes to the face

The most direct evidence for gender guessing comes from a 2014 VISAPP paper by Cunjian Chen, Antitza Dantcheva and Arun Ross. They tested automated gender and age estimation against two kinds of makeup changes, men made up to appear more feminine and women made up to look older or younger, and concluded that cosmetics can potentially confound these estimators. From a modeling point of view that's unsurprising. The features the model relies on, such as local contrast, texture and the geometry of the eyes and lips, are exactly what makeup alters. There's no notion of identity anywhere in the pipeline.

Adversarial accessories

The 2016 paper "Accessorize to a Crime," by researchers from Carnegie Mellon University and the University of North Carolina at Chapel Hill, presented at ACM CCS, is the classic physical adversarial example for faces. The team optimized a pattern, printed it onto eyeglass frames, and showed that wearers could evade commercial face recognition or impersonate another person, with a reported success rate of about 90 percent. It targeted recognition rather than gender classification, but the mechanism generalizes. A small, structured perturbation in a region the model weights heavily can move the output far more than a large, random one.

Occlusion

Masks gave the field an unplanned occlusion benchmark. In July 2020, NIST evaluated 89 commercial face recognition algorithms on images with digitally applied masks and found that even the best had error rates between 5 and 50 percent. Wider masks with more nose coverage degraded performance further, and black masks hurt more than light blue ones. Remove a large region of signal and the model's confidence has much less to stand on.

Attacks age, and so do defenses

Robustness is a moving target. Adam Harvey's CV Dazzle, created in 2010, used makeup and asymmetric hair to defeat the Viola-Jones detector. His project page now states that the original looks no longer work on modern CNN-based systems and that he has advised against them since 2016. Once a technique is known, it tends to end up in training data, and the next model learns to ignore it.

The failure mode nobody has to engineer

The most important finding isn't an attack at all. In a 2019 University of Colorado Boulder study of four commercial facial analysis services, accuracy was about 98 percent for cisgender women and about 97.6 percent for cisgender men. The same services misclassified trans men as women up to 38 percent of the time and got agender and nonbinary people wrong in every case. That's a label-space problem, not an adversarial one. A binary classifier trained on appearance can't represent people it was never designed to represent. If you need a primer to share with non-technical teammates, this guide to age and gender prediction covers the basics well.

What this means if you build with face models

Test robustness explicitly. Evaluate on occlusions, accessories, makeup, lighting and pose, not only on clean benchmark images. Report results per subgroup. Surface confidence scores and treat low-confidence outputs as unknown rather than forcing a label. Prefer on-device inference where you can, so user photos never leave the browser. And never use an appearance-based guess as a stand-in for identity or as the basis for decisions about people.

Try a quick robustness test

You can see these effects in a minute. Run a free gender guesser on a clean, front-facing photo, then add glasses, a cap, a hand partly covering the face, or poor lighting, and compare the confidence values. It runs fully in the browser, which makes it a convenient sandbox. More small utilities for experiments like this are on AltFTool.

Aggregate metrics hide subgroup failures

That result comes from the 2019 follow-up to MIT Media Lab's Gender Shades project, which audited Amazon and Kairos, two vendors not included in the original study. Overall error rates were 8.66 percent and 6.60 percent. For darker-skinned women, the error rates were 31.37 percent and 22.50 percent. The original 2018 study had already reported error rates of up to 34.7 percent for that group across three commercial systems, against a maximum of 0.8 percent for lighter-skinned men. In other words, the aggregate metric and the subgroup metric told opposite stories, and both were accurate. Any evaluation that reports only one of them is incomplete.

Model choice matters more than you'd expect

NIST, the US National Institute of Standards and Technology, has benchmarked face technology at scale for years. Its 2015 evaluation of gender classification reported roughly a threefold difference between the most and least accurate algorithms. A separate 2019 NIST study of face recognition, not gender guessing specifically, tested 189 algorithms from 99 developers and found that most showed performance differences across sex, age, and race or country of birth. NIST also warned that it's usually incorrect to make statements across algorithms, because they behave so differently. For anyone choosing a model or an API, that's a strong hint to test the specific system on data that looks like your real users, rather than trusting a generic accuracy claim.

A ceiling you can't engineer past

Good systems do get close under good conditions. A 2015 research paper reported about 96.9 percent accuracy on a widely used benchmark of everyday photos, with a model trained on four million images. Humans land around the same level in studies where hair is hidden, with one report putting them at 96 percent. Even so, that leaves a few percent of errors before real-world noise like blur, odd angles and underrepresented faces shows up. There's also a labeling problem no architecture solves. The model maps appearance to labels and has no access to how a person identifies. A 2019 study reported in the press found commercial systems misclassified transgender people more than a third of the time while scoring about 98 percent on cisgender men and women.

Age estimation is noisier still

Age is a regression problem with even more spread. NIST's age estimation evaluation, first reported in 2024, found that none of the six algorithms tested was clearly better than the others, and accuracy depended on image quality, gender, region of birth and age. Biometric Update's analysis of those results concluded that no single number captures how a system performs, and that the best performer changes considerably by age band. A 2022 study of 21 AI age tools found them less accurate than human observers in its tests. If you want a non-technical primer to share with teammates, this guide to face-based age and gender prediction covers the basics well.

If you build with these models

A few habits follow directly from the research. Report metrics per subgroup, not just overall, and make sure your test set actually contains enough examples of each group to be meaningful. Show users a confidence score rather than a bare label, and treat low-confidence outputs differently. Test across lighting, pose and image quality, since these shift confidence noticeably. Most importantly, never present an appearance-based guess as someone's identity, and keep these models away from decisions that affect real people.

Why 100 percent is the wrong target

Faces vary enormously and no training set covers all of them equally. Photos vary just as much, and capture conditions change how confident systems are. The task itself measures appearance, which is not identity. Better data and engineering can improve the first two, but the third is a hard limit. The right goal is a model that's honest about its uncertainty and performs reasonably well for as many people as possible.

Try it yourself

The quickest way to feel this is to experiment. Run a free gender guesser on a well-lit, front-facing photo and then on a dim or angled one, and compare the confidence values. The tool runs in the browser, so it's an easy sandbox for seeing how inputs move outputs. You'll find more utilities for quick tests like this on AltFTool.