Mark, a stay-at-home dad from San Francisco, found himself locked out of all his Google accounts in February last year after taking pictures of his son's genitals to send to a doctor amid the pandemic. He noticed that his son's penis looked swollen and was causing him pain, so he decided to take photos to document the progression of the issue.
Mark's wife called an advice nurse at their health care provider to schedule an emergency consultation for the next morning, which had to be conducted via video due to the pandemic. The nurse asked for photos so that the doctor could review them in advance, and Mark uploaded them to the healthcare provider's messaging system from his Android phone.
However, the pictures were flagged as child sexual abuse material by an AI system designed to detect and prevent criminal behavior. This triggered a police investigation and led to Mark being locked out of all his Google accounts, including his email, photos, and contacts.
The dad was initially confused by the account lockout, but then he remembered his son's infection. He thought, "Oh God, Google probably thinks that was child porn." He tried to appeal the decision but was denied by Google, leaving him unable to access any of his data. Even his Google Fi account was shut down, which meant that he had to get a new phone number from another service provider.
Months later, Mark was informed that the San Francisco Police Department had opened a case against him, simply because he had lost access to his phone number. In December 2022, he received an envelope from the police containing documents informing him that he had been investigated. Copies of the search warrants served on Google and his internet service provider were also included.
The investigator said that the case was closed as "no crime had occurred," but Mark was left with the permanent loss of all his data on Google. He requested the police to help him get his account back and even appealed to Google with the police documents, but he was again denied. He was informed that his account was going to be permanently deleted.
Unfortunately, Mark is not the only parent to have faced such a situation. Another parent in Texas, known as Cassio, had a similar experience after taking pictures of his son's "intimate parts" to send to a doctor. The pictures were automatically backed up to Google Photos, triggering an investigation by police in 2021. However, Houston Police cleared him of any wrongdoing after he showed them communications with the doctor.
These cases have raised concerns about the use of AI systems by technology companies to detect and prevent criminal behavior. While cooperation from tech giants is essential to combat the rampant online spread of sexual abuse imagery, the intrusion into private archives such as digital photo albums has cast innocent behavior in a sinister light. As a result, innocent people like Mark and Cassio have suffered severe consequences, including the loss of all their data.
Check out the original article By Kashmir Hill New York Times Aug. 21, 2022
Comments