Technologist Anil Dash recently told Pew: “The best parts of algorithmic influence will make life better for many people, but the worst excesses will truly harm the most marginalized in unpredictable ways. We’ll need both industry reform within the technology companies creating these systems and far more savvy regulatory regimes to handle the complex challenges that arise.”

Data scientists generally agree that those most threatened by algorithms are those traditionally disenfranchised, including Black people, Native people, people of color, women, people with disabilities, older people, and people in the criminal justice system.

Algorithms can reproduce bias even in attempts to measure our individual physical or behavioral characteristics — our “biometrics.” For example, Apple has provided refunds to iPhone X customers who have repeatedly discovered that the phone’s face recognition software can have trouble accurately recognizing people of color, raising questions about how Apple tests its software and how — or based on whom — it determines “effectiveness.”

A New York Times article, “Facial Recognition Is Accurate, if You’re a White Guy”, reviews a study by M.I.T. researcher Joy Buolamwini measuring the performance of face recognition systems created by Microsoft, IBM and Chinese company Megvii. Buolamwini found that the algorithms delivered 99% accuracy rates when scanning the faces of white men, with disparate results for people of color: “the darker the skin, the more errors arise — up to nearly 35 percent for images of darker skinned women.”

Yet, companies are using face recognition algorithms to verify the identity of consumers in online payment systems and banking; for photo identification and marketing on social media; in lending processes; and even to assess behavior of job candidates. Some companies are experimenting with the possibility to personalize an advertisement based on the perceived “mood” of the face in front of it.

As government agencies deploy new forms of technology under the guise of national security, airlines and other companies face new risks. Over four years, U.S. Customs and Border Protection (CBP) seeks to carry out a $1 billion “biometric exit” face scanning program; already, face scanning technology is in place at 11 airports across the nation, for passengers traveling on some flights with American Airlines, British Airways, Delta, Emirates, JetBlue and United Airlines.

In recent testimony to a Congressional subcommittee, Laura Moy, Deputy Director of the Center on Privacy & Technology at Georgetown Law, noted that “hiring algorithms have been accused of unfairly discriminating against people with mental illness. Sentencing algorithms — intended to make sentencing fairer by diminishing the role of potentially biased human judges — may actually discriminate against Black people. Search algorithms may be more likely to surface advertisements for arrest records — regardless of whether such records exist — when presented with characteristically Black names.”

In a powerful new book, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor, Virginia Eubanks investigates the impact of algorithms and predictive risk models on poor and working class Americans, presenting three case studies: an attempt to automate eligibility processes for the State of Indiana’s welfare system; an electronic registry of the unhoused in Los Angeles; and a risk model that promised to predict which children will be future victims of abuse or neglect in Allegheny County, Pennsylvania.

When algorithms were employed in Indiana’s public assistance program, Eubanks reports, “The assumption that automated decision-making tools were infallible meant that computerized decisions trumped procedures intended to provide applicants with procedural fairness. The result was a million benefit denials.”

Eubanks argues that algorithmic decision-making creates a “digital poorhouse” which “uses integrated databases to target, track, and punish.” She says engineers and data scientists concerned about the economic and social implications of their designs should ask themselves two questions: “Does the tool increase the self-determination and agency of the poor? Would the tool be tolerated if it was targeted at non-poor people?”

Data researcher Mimi Onuoha warns about the danger of what she calls “algorithmic violence” brought about by “the availability of huge datasets, advances in computational power, leaps in fields like artificial intelligence and machine learning, and the subsequent incorporation and leveraging of all these things into a hierarchical and unequal society.”

Some new AI products seem born from sci-fi film scenarios. Faception, an Israeli company, advertises “facial personality analytics” employing computer-vision and machine learning which “automatically reveals personalities in real-time.” The technology purportedly allows “security companies and agencies to be more effective in detecting anonymous persons of interest.”

For example, predictive policing software — which analyzes data on past arrests to gain insight into where and when future crime is most likely to occur — is being increasingly deployed by major police departments in the U.S., including New York City, Los Angeles, Atlanta and Chicago. Leading marketers of predictive policing software include IBM, Hitachi, Motorola (a division of Lenovo), LexisNexis (a division of the RELX Group) and the privately-owned Palantir.

While researchers such as Jennifer Bachner, a government expert at Johns Hopkins University, contend that “predictive policing offers an opportunity to make significant advances toward a safer and more just society,” critics such as Jennifer Lynch, an attorney with the Electronic Frontier Foundation, say predictive policing will “only further focus police surveillance on communities that already are overpoliced and could threaten our constitutional protections and fundamental human rights.”

The RAND Corporation studied the Chicago Police Department’s use of predictive software to decrease that city’s notoriously high murder rate. In 2016, RAND concluded: “To make a long story short: It didn’t work.”

A major focus is pre-trial risk assessment: developing tools to help judges decide where defendants will await trial — at home or in jail. By law, this decision hinges on the judge’s prediction of what the defendant would do if released. In many cities and states, new “risk assessment algorithms” use data about a defendant’s age, location, family history, and a host of other factors to make a recommendation to a judge regarding the defendant’s “likelihood” of re-arrest or failure to appear at the next court date.

Civil rights and racial justice groups have questioned the effectiveness of risk assessment tools because they have been found to replicate racism. In 2016, ProPublica found that a software used across the country to assess risk — COMPAS — was biased against Black people; in Florida, the researchers found that the algorithm was more likely to falsely flag Black defendants as future criminals, “wrongly labeling them this way at almost twice the rate as white defendants” while white defendants were also more likely to be mislabeled as low risk. (A Dartmouth College computer science study recently found that COMPAS is no better at predicting an individual’s risk of recidivism than choices made by random volunteers recruited from the internet.)

And a January 2018 paper by data science researchers John Logan Koepke and David Robinson concludes that today’s risk assessment tools make “zombie predictions,” with predictive models that are trained on data from older bail regimes, and “are blind to the risk-reducing benefits of recent bail reforms.” Koepke and Robinson argue that these new tools “risk giving an imprimatur of scientific objectivity to ill-defined concepts of ‘dangerousness’” and “pave the way for a possible increase in preventive detention.”

Some experts believe that such pre-trial risk assessment tools could offer great benefit, if the algorithm focuses on “predicting judges’ decisions, rather than defendant behavior.” A recent paper by a team led by Cornell University’s Jon Kleinberg created a simulation with an algorithm, trained through machine learning, “to improve and understand” judges’ decision-making about defendants — “a concrete prediction task for which there is a large volume of data available”. The study found that, by integrating algorithms into an economic framework (in this case, using different econometric strategies like quasi-random assignment of cases to judges) and “being clear about the link between predictions and decisions”, pre-trial risk assessment tools could result in “reductions in all categories of crime, including violent ones. Importantly, such gains can be had while also significantly reducing the percentage of African-Americans and Hispanics in jail.” Importantly, for the algorithm to effectively reduce crime and incarceration rates, it focused on human decision-making by judges, rather than attempting to predict defendants’ future behavior.

As these algorithmic tools are increasingly deployed in the criminal justice system, “we should have the right not just to see and understand what the risk assessment tools say, but to independently audit the results that come from their introduction in the system where they are used,” says Hannah Sassaman, a policy director at the Philadelphia-based Media Mobilizing Project and a current Soros Justice Fellow focusing on community oversight of risk assessment algorithms. Sassaman adds: “This could mean including independent data scientists, focused on the rights of communities, on robust community advisory boards governing child welfare, criminal justice, or other contexts using algorithmic risk assessment. It could also mean halting the use of these algorithms if they are not producing the results we want.”

A 2017 report by the AI Now Institute at New York University, concluded that “core” public agencies responsible for areas such as criminal justice, healthcare, welfare, and education should no longer use “black box” AI and algorithmic systems, including both AI systems licensed from third party vendors and algorithmic processes created in-house. “The use of such systems by public agencies raises serious due process concerns, and at a minimum they should be available for public auditing, testing, and review, and subject to accountability standards,” the report said.

Given widespread concern and growing evidence about the discriminatory nature of AI tools, the major companies producing these products and selling them to government agencies should be held to demonstrate the fairness of these tools before — not after — they are deployed. Investors can be key players in engaging companies to audit algorithmic impact at the outset.