A few days after Christmas I emailed some pictures of myself to a stranger I met online. The recipient wanted to see my face in each photo, preferably at different angles. I was also welcome to include shots featuring a "beard, glasses, etc." I chose four photos that I wasn't too embarrassed about, and hit send. A few days later, the facial recognition firm Animetrics—my pen pal—sent me several photos of their own. You can see one of them above. On the left is a photo I sent; on the right is that same image after Animetrics rendered it into 3D.
Today's surveillance video cameras can't capture a still image anywhere near as clean as the one you see above, which matters quite a bit in determining someone's identity. But a few years from now? Surveillance cameras, like smartphone cameras, will continue to become more powerful and their pictures more detailed. Computers will be faster. Algorithms will be more fine-tuned. Data storage will be even cheaper. In the unlikely event that future Mike Riggs commits a serious crime or participates in an act of terror, the technology that created the image above could, in theory, help law enforcement identify him and track him down.
The age of practical facial recognition software has begun, and it's only going to get more precise.
Police already use facial recognition, but it doesn't work like in the movies.
Ask a stranger what they think of facial recognition, and odds are good they'll bring up the movie Minority Report, in which billboards identify and speak directly to individual consumers as they pass by. So when I speak to Animetrics, a firm that works with the NSA and the Department of Defense, one of my first questions is: How far are we from the facial recognition you see in the movies?
"To expect a computer to automatically recognize you like in Minority Report, it's far-fetched," says Animetrics president and CEO Paul Schuepp. "And on TV you see people do this all the time. That scares the wits out of people. If it really worked that great, I would be concerned about my privacy." That's not to say such computing power isn't the end goal of today's research. "The holy grail of face recognition," Schuepp adds, "is to be able to pick out a terrorist's face in a stadium of 100,000 people using a high-power camera."
The facial recognition tech that police are using right now is a far cry from that. The Chula Vista Police Department in San Diego County, for example, is piloting the Tactical Identification System (TIS), which allows officers in the field to snap a picture of an arrestee and then compare that image to those in San Diego County's mugshot database, returning personal information and criminal history. Pennsylvania's JNET Facial Recognition System has access to a similar statewide mugshot database, as well as the Pennsylvania DOT's driver's license database.
Schuepp says that JNET, which licenses an Animetrics program called ForensicaGPS, demonstrates the accomplishments as well as the limitations of current facial recognition tech. Unlike TIS, where an officer has control over the picture he's taking of an arrestee, JNET uses pictures from crime scene surveillance video and social media. The officer uploads those photos to JNET, which quickly compares them to the millions of images in JNET's database.
"Nine times out of ten," Schuepp says, "not only does he not get a match, but [JNET] says the photo doesn't work." That's because aging, pose, illumination, and expression—known in the industry as A-PIE—are all central to comparing faces. In the ideal photo, the subject is well lit, facing forward, and has a neutral expression on her face. (These are, not coincidentally, the conditions under which both arrest booking and driver's license photos are taken.) But surveillance camera footage is seldom "ideal." That's where Animetrics' ForensicaGPS program comes in. If officers using JNET don't get a match with the raw photos they have, ForensicaGPS can create a 3D model of the subject's face and correct the subject's pose. That's what Animetrics did with two of the images I sent them:
My face, turned into a 3D model. 
Schuepp: "If you're looking to the left, and you straighten it out, the camera didn't see the part of your face that was hidden, so when we correct it, we have these blanks. We fill out the blanks using symmetric textures."
"In many, many cases," says Schuepp, "we're getting high scores just by pose correction. It's what we have our patents on." The advantage of the 3D conversion, meanwhile, allows investigators to rotate the head, getting a better sense of what a suspect looks like, even if they can't nail down his identity. "Facial analysis experts can better visually compare two faces against each other in 3D with 3D overlay tools," Schuepp says.
While ForensicaGPS is advanced, it's not certain. No facial recognition technology currently is.
"Everybody is already talking about a match in face recognition," Schuepp says. "There is no match. The match is actually a score, which is a statistical measurement of one face to another. So the score is going to be a percent on a scale between zero and one. It could be 78 percent, it could be 99 percent. It's always a statistical score." He says that you could compare one photo to the same photo, and the result still wouldn't be 100 percent. "It'll be 98 or 97 percent because it isn't about matching pixel for pixel. It's about matching the nose and the eyes, the hair color." In other words, facial recognition software doesn't try to match photo subjects the way a human eye would. Instead, it attempts to discover the unique topography of a person's face, and then seek out that same topography in other pictures.
"The biggest issue for face recognition, from a biometric perspective, is that your face is always changing," Schuepp says. "The environment, the lighting, expressions, age. It's almost unfair compared to iris or fingerprint identification."
If it's dark out, if I'm wearing glasses, if only half my face is showing, a lot of facial recognition systems won't be able to identify me. The artist Adam Harvey demonstrated the shortcomings of facial recognition tech with his CV Dazzle project, in which he altered models' haircuts and makeup to completely stump a facial recognition program.
Where law enforcement and government agencies are currently having the most success with facial recognition is in catching people who try to get multiple driver's licenses or government-issued ID under assumed names. Last year, New Jersey indicted 38 people for driver's license fraud. Many of the suspects had acquired new licenses under assumed names after having their own licenses suspended. Operation Facial Scrub used facial recognition software to root out the duplicates.
There is a plan to radically transform facial recognition technology.
Three months from now, a group within the Office of the Director of National Intelligence will begin a four-year project to revolutionize facial recognition software. The Janus program, named for the Roman deity whose two faces allow him to look both forward and backwards in time, is one of several projects hosted by ODNI’s Intelligence Advanced Research Projects Activity, launched in 2006 as an intelligence companion to the Pentagon's DARPA.
"This is high-risk, high-payoff research," says Bojan Cukic, the director of West Virginia University's Center for Identification Technology Research. "The low-hanging fruit has been taken care of." While a nondisclosure agreement keeps Cukic from divulging too much about the Janus program, he did give me a sense of what the program hopes to accomplish.
"The emphasis is on the seamless integration of video and still imagery from various sources (think crowd sourcing situations) where one cannot centrally control acquisition angles, image resolution, weather conditions, image sharpness, illumination etc.," Cukic writes in an email. "Someone’s pose could be, for example, a profile with only an ear visible. So face recognition would now include any head poses (head recognition), with various degrees of accuracy and match confidence."
Cukic thinks that researchers in machine vision and pattern recognition "have made great strides towards making many of the face recognition goals of the JANUS program feasible." He also sees a "significant challenge in the scale of the problem (doing this for situations similar to Boston bombing, for example) in which the sheer size and diversity of digital media surpass what we have seen so far."
Are we moving too fast?
Last October, at the annual meeting of the International Association of Chiefs of Police, Philadelphia Police Chief Charles Ramsey delivered an address in which he evoked facial recognition technology in order to warn against it.
"We must drive technological solutions to our problems and not be driven by the technologists," he said. "We have to remind ourselves, just because we can do something doesn’t mean we should do it."
Privacy and civil liberties advocates like the ACLU's Jay Stanly and Kade Crockford, and the Center for Investigative Reporting's Ali Winston, have made the same arguments. In addition to proving that new forms of technology are prone to abuse, they've also revealed that law enforcement agencies almost always adopt new technology without first establishing internal rules, oversight, or protections for citizens (see Oakland's Domain Awareness Center, Seattle's mesh network, and the Brennan Center for Justice's recent report on fusion centers).
The abuse potential is there for facial recognition as well. Last year, an audit found that police officers across Minnesota had illegally accessed the state driver's license database to sneak a glimpse at one particularly attractive woman. Nearly 70 officers abused their database privileges. The argument goes that it's hard to believe we won't eventually see similar abuses with facial recognition software. Perhaps an officer or other government employee snaps a picture of someone at a bar, then runs that image through a driver's license database. Next thing you know, the bar patron is getting unwanted Facebook messages from a civil servant.
But what's interesting about facial recognition, as opposed to the myriad other ways that our government surveils us, is that even with perfect oversight, transparency, and accountability, it's still unnerving. I can't, personally, read my own DNA or tell my fingerprints from yours. In fact most biometric identifiers—the two I mentioned, and iris scans—are way over my head (and probably yours). But the face is so basic to how we communicate and recognize each other, even a toddler knows to be skeptical upon meeting her dad's identical twin for the first time.
I had my own toddler moment when I started looking at the images Animetrics had produced. Originally, I'd asked them to demo ForensicaGPS because it would be fun for the story. But I'll confess that after I saw it, I felt...different. 
To have a computer look at me and see, well, me, was odd. It's not like I'm a totally private person. I've been on a few cable news shows, and I use a picture of my own face on Facebook and Twitter. What got me, I think, is that despite all the time I spend being a public person, when I'm actually out in public—at a restaurant or the gym—I'm nobody. That's going to change. We've surrounded ourselves with computers that can see, and we're slowly but surely teaching them to see us the way we see each other.
I'm not sure that even the smartest people in the room know what that world is going to look like.