By Dan Grimm, GM and VP Computer Vision, RealNetworks
July 19, 2019
The United States National Institute of Standards and Technology (NIST) conducts an on-going battery of tests, known as the Face Recognition Vendor Test (FRVT), to measure the key characteristics of facial recognition algorithms, including accuracy, performance, and bias. Companies and academic institutions are given the opportunity to submit one or more algorithms that NIST then applies to a set of tests. The two most recent FRVT test results of are each over 370 pages in length and were published in April 2019 and July 2019.
In the cohort of algorithms that exceeded 96% accuracy, SAFR is both the fastest and lightest model.
The July 2019 NIST results found that SAFR was the fastest and most compact among facial recognition algorithms for camera-unaware faces (wild images) with top-tier accuracy – less than 0.0335 False Non-Match Rate. Compared to the other high-accuracy algorithms, SAFR is 2x faster than average, and 35 percent smaller than the second-smallest algorithm. SAFR was the fastest algorithm tested in the April 2019 NIST results and its speed improved by 30% in the new July results. SAFR has 99.87 percent proven accuracy in the University of Massachusetts Labeled Faces in the Wild test.
NIST not only measures specific characteristics of facial algorithms, such as performance, accuracy, and bias, the standards and measurements body also reports on those attributes by image type, such as visa photos, mugshots, webcam, or “wild” images. Wild images are camera unaware faces captured on video. They’re complex because the faces may be tilted, with wide yaw and pitch pose variation. Not to mention there may be many faces within a video frame. Wild images are comprised of challenging facial qualities that are precisely the type of real-world conditions for which SAFR was designed. NIST conducts its tests of facial recognition using still photographs. Facial recognition in live video requires concerted optimization in acquisition, accuracy, and speed.
SAFR from RealNetworks is the most accurate high-performance facial recognition algorithm for live video as tested by NIST.
Vendors, researchers, and academic institutions can optimize their submissions for the Face Recognition Vendor Test (FRVT) and are not required to submit their actual commercial facial recognition algorithm, which can lead to misrepresentative results. A vendor or academic can submit an algorithm that performs remarkably well in accuracy but in commercial real-world conditions may be so computationally expensive as to be impractical. For example, in the July 2019 FRVT results several algorithms achieved high accuracy marks for wild faces but performed so slowly – 1.5x to 55x slower than the SAFR algorithm — that they would be impractical in many real-world conditions, requiring unrealistically expensive computing power, excessive time to recognize a face, and be overwhelmed by many faces in the video.
Some of the algorithms tested by NIST are a bit like Formula One race cars: built to perform well on a particular circuit but not reflective of the commercial world where a thoughtful balance of driving conditions, such as noise, braking, range capacity, safety, and comfort are required.
So how does SAFR perform against the top-rated algorithm for accuracy in the July report? The top algorithm has a wild face score of 0.0271 but would take 3.6x times longer to run and is 7x the size of the SAFR algorithm, which has a score of 0.0334. A number of algorithms perform well on accuracy, but by comparison to SAFR in a large-scale, real-world commercial deployment one would need to have significantly more hardware on-site to achieve results that would be 3-4 times slower, as illustrated in the graph below. The increase in wild image accuracy by 0.0064 has a debilitating effect on performance and cost.
High performance makes a material difference, since it increases the number of opportunities to attempt recognition in a computationally constrained system. In the cohort of algorithms that exceeded 96% accuracy, SAFR is both the fastest and lightest model. This means SAFR is able to sample a face multiple times during the same period of time of other algorithms, subsequently compounding SAFR’s accuracy.
SAFR is highly competitive in accuracy for still photos: as tested by NIST it ranked in the top 25 commercially available products worldwide and in the top 6 from a United States company. However, as noted, the NIST score of accuracy doesn’t convey the entire picture. NIST measures the match of a single image to a single image for wild faces, while in real life the people moving within a video frame are in constant motion. SAFR uses edge intelligence to select the right image to match from hundreds of frames of video. What that means is that SAFR’s accuracy is actually higher than what is measured by NIST since SAFR continuously monitors and resamples the video to capture and submit the best frame for recognition. NIST doesn’t use video in its tests.
SAFR stands apart from other facial recognition algorithms because it achieves results with a fraction of the compute power required by most of the algorithms in its class. A lot of companies that submit to NIST are tuned to score high on accuracy but fail to strike a balance between performance and accuracy.
SAFR from RealNetworks is committed to providing the best accuracy and performance with the least bias, using readily available hardware to recognize people in real-world conditions.
SAFR is the premier platform for facial recognition in the real world.
– – – – –
Results shown from NIST do not constitute an endorsement of any particular system, product, service, or company by NIST.