Facial matching is performed within the Secure Enclave using neural networks trained specifically for that purpose. Apple developed the facial matching neural networks using over a billion images, including IR and depth images collected in studies conducted with the participants’ informed consent. Apple then worked with participants from around the world to include a representative group of people accounting for gender, age, ethnicity, and other factors. The studies were augmented as needed to provide a high degree of accuracy for a diverse range of users. Face ID is designed to work with hats, scarves, eyeglasses, contact lenses, and many sunglasses. Furthermore, it’s designed to work indoors, outdoors, and even in total darkness. An additional neural network that’s trained to spot and resist spoofing defends against attempts to unlock the device with photos or masks. Face ID data, including mathematical representations of a user’s face, is encrypted and available only to the Secure Enclave. This data never leaves the device. It’s not sent to Apple, nor is it included in device backups. The following Face ID data is saved, encrypted only for use by the Secure Enclave, during normal operation:
The mathematical representations of a user’s face calculated during enrollment
The mathematical representations of a user’s face calculated during some unlock attempts if Face ID deems them useful to augment future matching
Face images captured during normal operation aren’t saved, but are instead immediately discarded after the mathematical representation is calculated for either enrollment or comparison to the enrolled Face ID data.
Improving Face ID matches
To improve match performance and keep pace with the natural changes of a face and look, Face ID augments its stored mathematical representation over time. Upon successful match, Face ID may use the newly calculated mathematical representation—if its quality is sufficient—for a finite number of additional matches before that data is discarded. Conversely, if Face ID fails to recognize a face, but the match quality is higher than a certain threshold and a user immediately follows the failure by entering their passcode, Face ID takes another capture and augments its enrolled Face ID data with the newly calculated mathematical representation. This new Face ID data is discarded if the user stops matching against it and after a finite number of matches. These augmentation processes allow Face ID to keep up with dramatic changes in a user’s facial hair or makeup use, while minimizing false acceptance.