Automation and the Quantified Society

Artificial intelligence is just one technique for automating decisions. Other, older software tools are also being deployed in new ways that touch vulnerable groups.

THROUGHOUT THE WORLD, POWERFUL INSTITUTIONS are increasingly using computers to automate decisions that matter to people’s lives. Artificial intelligence and other advances in software development, pervasive data collection, and vanishingly cheap computing power are working together to drive considerable change, creating what some have called the “quantified society.”

NetGain funders and the social sector at large are deeply committed to protecting civil and human rights, defending the vulnerable, and building a more humane and equitable world. Meanwhile, technological change is sweeping the landscape, creating both new opportunities and new needs. The current moment demands careful thought and sustained attention, informed by both deeply held values and technical insight.

This framing paper is designed to spark and structure an urgently needed conversation about what automation in a quantified society means for core human values, across the full spectrum of issues that motivate NetGain funders and their allies—and how we can work to amplify its benefits while mitigating its harms. This report is concerned with decisions made by software, that shape the lives of vulnerable people and groups. It reflects extensive research, interviews with civil society stakeholders across the world, and our own sustained work over years of consulting projects, coalition advocacy, and scholarship and teaching on these issues.

We begin by briefly describing what’s new in computing that is driving so much social, economic, and political change. Reading those few pages, you will learn what “machine learning” is all about. This technique, the dominant form of artificial intelligence in use today, harnesses computing power to find patterns in historical data—patterns that can then be used as the basis for predictions and decisions. At the same time, AI is just one technique for automating decisions, and other, older software tools are also being deployed in new ways that touch vulnerable groups.

In the second section, we share a few key insights about how social and political systems tend to change at times like this, when computers take on new responsibilities. The benefits of these new technologies are substantial and widespread. At the same time, research on human-computer interaction bears out what many people feel intuitively: Giving computers more power inside a complex, important institution doesn’t just make things faster or cheaper—the institution itself, and the people inside it, often redefine their goals and change the way they act. When people place too much trust in computers, they risk deferring to automated judgments that may be wrong or ill-informed. This trust, coupled with problematic training data, algorithms based on incorrect assumptions, and embedded social bias, can often leave minority populations as what engineers call “edge cases”—people and situations the computer does not handle well.

Finally, the heart of our report offers five core themes at the intersection of social change, automation, and the quantified society. With each theme, we explain what is happening and why, anchoring a high-level discussion with real-world examples. We then describe trends to watch for, which are important pathways of benefit and risk that we see inside each area, and potential approaches to addressing these developments. We do not propose or assess specific investments, but instead suggest directions that may warrant further exploration. Each theme is here because it brings a cluster of related developments into focus.

Core Themes: Automation and the Quantified Society

  1. Corporate Power, Information, and the Attention Economy: A handful of major companies, most based in the U.S., operate globally dominant Internet platforms. These firms are constantly linked to their users via smartphones and other technology. They have direct, intimate, and immediate knowledge of each of their users, and have a historically unprecedented role in informing, influencing, and motivating the behavior of billions of people. Voting, civic mobilization, and intellectual exchange, around the world, are among the many facets of life these firms increasingly enable, mediate, and shape through increasingly automated processes. Beyond the direct impact of platforms themselves, philanthropy by the major platforms and their founders plays a growing role in the social sector, where it could shape public debates by highlighting the best impacts of AI and big data while downplaying structural risks. Understanding and responding to the impact of major online platforms is a crucial challenge for the social sector in the years ahead.

  2. Patterns, Discrimination, and Justice: When automated decisions are based on historical data, they risk entrenching unjust social patterns and projecting such patterns into the future. Much of the data that exists today about crime, health, commerce and other vital domains was gathered to suit the needs of well-resourced organizations, sometimes to the detriment of the marginalized. From a computer’s perspective, minorities are often “edge cases,” exceptions that a system may not be well designed to handle. Systems that reinforce existing patterns often work against the goals of human rights and civil rights advocacy, insofar as advocates seek to change longstanding social patterns. Emergent efforts aim to develop more inclusive data, systematically monitor bias and risk, and rectify discriminatory patterns.

  3. Jobs, Work, and Meaning in an Era of Automated Decisions: Automated hiring, work- place monitoring, employee assessment, and supply chain and logistics systems are shifting the landscape of non-discrimination law, workplace protection, and labor organizing. Other forms of automation threaten to knock out or redefine entire categories of work. Whether technology displaces workers or simply changes the nature of their work, this shift risks deepening the economic inequality and social instability that is already emerging across the world. The specifics of changing labor markets will be different in developed and developing economies, in countries with and without strong social safety nets, and in geographies with relatively robust or relatively weak worker protections—but across the board, civil society will need to be attentive to shifting currents to determine appropriate interventions.

  4. Automated Decisions in the Public Sector: Governments around the world increasingly use automation to make important decisions about people’s lives, often without broad public consultation or careful assessment of new systems’ impact. When attempting to adapt these technologies for public ends, governments have struggled to access needed expertise and to navigate normative and legal concerns related to equal treatment, privacy, and other ethical challenges. To leverage new technologies in ways that prioritize the public good, technical expertise, and knowledge is vital. But in their efforts to catch up with private sector innovation, governments often rely on exogenous sources of expertise—such as partnerships with leading technology platforms—that may interpose different values into the exercise of public authority. It remains uncertain to what extent the progress of digital technology will be reflected in public services—and where the technology does arrive, how it may reshape how those services operate—for example, by changing the way that judges perceive and interact with defendants in the courtroom.

  5. Freedom, Transparency, and State Power: Automated profiling and predictive analytics make mass surveillance both less expensive and more powerful. That puts more people at risk of being swept up in government scrutiny, targeted by preemptive enforcement, subjected to state prejudice, or manipulated by disinformation. State and state-supported exploitation of automated tools to manipulate public opinion is developing rapidly, while efforts to map and constrain such activities lag behind. Prompt and coordinated philanthropic activity may be able to shift this dynamic, including by building or strengthening walls between government and corporate surveillance. Approaches in this area align with broader privacy and digital rights goals.

    Automation and the quantified society are transforming areas of vital concern to NetGain and its allies. We hope this paper provides a clear frame and a useful starting point for the high-impact discussions that lie ahead. 

Eric Sears