The concept of social license to operate, where a firm openly works with the communities that will be affected by its actions to gain their trust and acceptance, offers an approach for crafting AI solutions that are acceptable to stakeholders.
AI And Ethics: A Question Of Social License?
“My company spends US$7 million per year on community programs. We still face work interruptions from the communities we help. Obviously, the money does not buy us the goodwill we need, but I have no idea where we are missing the point.” – Managing director of an oil company
Three friends were having morning tea on a farm in the Northern Rivers region in New South Wales (NSW), Australia, when they noticed a drilling rig setting up in a neighbor’s property on the opposite side of the valley. They had never heard of the coal seam gas (CSG) industry, nor had they previously considered activism. That drilling rig, however, was enough to push them into action. The group soon became instrumental in establishing the anti-CSG movement, a movement whose activism resulted in the NSW government suspending gas exploration licenses in the area in 2014. By 2015, the government had bought back a petroleum exploration license covering 500,000 hectares across the region.
Mining companies, like companies in many industries, have been struggling with the difference between having a legal license to operate and a moral one. The colloquial version of this is the distinction between what one could do and what one should do – just because something is technically possible and economically feasible doesn’t mean that the people it affects will find it morally acceptable. Without the acceptance of the community, firms find themselves dealing with “never-ending demands” from “local troublemakers” hearing that “the company has done nothing for us” – all resulting in costs, financial and nonfinancial, that weigh projects down. A company can have the best intentions, investing in (what it thought were) all the right things, and still experience opposition from within the community. It may work to understand local mores and invest in the community’s social infrastructure – improving access to health care and education, upgrading roads and electricity services, and fostering economic activity in the region resulting in bustling local businesses and a healthy employment market – to no avail.
Without the community’s acceptance, without a moral license, the mining companies in NSW found themselves struggling. This moral license is commonly called a social license, a phrase coined in the ’90s, and represents the ongoing acceptance and approval of a mining development by a local community. Since then, it has become increasingly recognized within the mining industry that firms must work with local communities to obtain, and then maintain, a social license to operate (SLO). The concept of a social license to operate has developed over time and been adopted by a range of industries that affect the physical environment they operate in, such as logging or pulp and paper mills.
What has any of this to do with artificial intelligence (AI)? While AI may seem a long way from mining, logging, and paper production, organizations working with AI (which, these days, seems to be most firms) are finding that the technology’s use raises similar challenges around its acceptance by, and impact on, society. No matter how carefully an AI solution is designed, or how extensive user group testing has been, unveiling a solution to the public results in a wide range of reactions. A Bluetooth-enabled tampon can be greeted with both acclaim and condemnation, with some seeing the solution as a boon that will help them avoid embarrassment and health problems while others see privacy and safety concerns or worry about the device being hacked, leaking personal information. Higher-stakes solutions result in more impassioned reactions, as has been the case with COMPAS, a tool for estimating a defendant’s risk of recidivism (or reoffending) in a criminal trial, and MiDAS, a solution intended to detect fraud and then automatically charge people with misrepresentation and demand repayment. These solutions are considered biased against less privileged groups, exacerbating structural inequalities in society and institutionalizing this disadvantage. Just as with building an oil rig, the fact that an AI solution is legally and economically feasible doesn’t imply that the community will find it morally or ethically acceptable, even if they stand to personally benefit.
AI, like all technology, can benefit as well as harm both individuals and society as a whole. How we use technology – how we transform it from an idea into a solution – determines whether potential benefits outweigh harms. “Technology is neither good nor bad; nor is it neutral.” It is how we use technology that matters, for what ends, and by what means is it employed, as both require contemplation. There are choices to be made and compromises to be struck to ensure that the benefits are realized while minimizing, or suitably managing, the problems. Forgoing a technology due to potential problems might not be the most desirable option, though, as a “good enough” solution in an (already) imperfect world might, on balance, be preferable to the imperfect world on its own. The question is, however, what is “good enough”?
The challenge, then, is to discover what we should do. How do we identify these opportunities? What processes might be used to make compromises? And how can we ensure that the diverse voices in the community have their concerns listened to and accounted for?
Framing The Challenge
AI enables solutions as diverse as machine translation, self-driving cars, voice assistants, character and handwriting recognition, ad targeting, product recommendations, music recognition, and facial recognition. AI is being used to instruct, advise, report measurements, provide information and analysis, report on work performed, report on its own state, run simulations, and render virtual environments. Solutions that seemed impossible a few years ago are now embedded in products and services we use every day.
Over this time, our view of AI has also changed. Hopes that AI-powered solutions would counter some of our human weaknesses have given way to fears that AI might be an existential threat. At first, it was thought that regulation could control how AI is used – open letters were sent to government with long lists of signatories attached, asking for regulation to be enacted. This has failed to bear fruit. More recently, the focus has been on developing ethical principles to guide the development of AI-enabled solutions. These principles are useful distillations of what we want from AI (and what we’d like to avoid), but they are not enough, as they fall short of describing how particular solutions should adhere to them. The latest hope is that design (and design methodologies) will enable us to apply these principles, but it’s not clear that design will be enough either.
- Our efforts to grapple with the challenge of realizing AI’s value while minimizing problems have been complicated by three challenges:
- The definitional challenge of understanding what exactly AI is, and therefore, what the problems are
- The challenge of aligning technical (AI) solutions with social norms
- The challenge of bridging different social worlds – the different cultural segments of society that shape how their members understand and think about the world
We’ll deal with each of these in turn.
The Definitional Challenge: What Is AI, And What Are The Problems?
There is no widely agreed-upon and precise definition of what AI is and what it isn’t. This is in part because AI is a broad church, home to a range of otherwise unrelated technologies. A useful working definition is:
“Artificial intelligence is that activity devoted to making machines intelligent, and intelligence is that quality that enables an entity to function appropriately and with foresight in its environment.” – Nils J. Nilsson
While imprecise, this definition does capture the huge scope and ambition of what we might call the AI project. The lack of a precise definition might also have helped the field grow, as it has enabled AI to be something of a bowerbird, with its practitioners “borrowing” ideas and techniques from other fields in pursuit of their goals. A more cynical approach might be to define AI as “things that don’t quite work yet,” as many technologies stop being seen as AI once they are broadly adopted. Roboticist Rodney Brooks once complained: “Every time we figure out a piece of it, it stops being magical; we say, ‘Oh, that’s just a computation.’” There is a sense that AI is a label for the (currently) impossible.
More pragmatic would be to consider AI as an area of practice, a community working to replicate human cognitive (rather than just physical) achievements. AI technology is simply whatever technology the AI community uses to solve problems that they find interesting. AI can progress by applying old techniques to solve new problems just as much as it can by discovering new techniques to solve old problems. Indeed, a significant driver for the current wave of investment we’re seeing in AI is a confluence of cloud services, easy access to data, and low-cost, ubiquitous compute and networks enabling new solutions to be built from old technologies, rather than the development of new disruptive technologies per se. After several decades of steady progress, it seems that discovery of new AI techniques might be stalling.
Regardless of where one draws the line between “intelligent” technologies and others, the growing concern for ethical AI is not due to new technology – such as, for instance, the development of CRISPR or genetically modified organisms (GMOs) – that enables us to do new and unprecedented things. The concern is due to dramatic reductions in price-performance that enable existing technologies to be applied in a broad range of new contexts. The ethical challenges presented by AI are not due to some unique capability of the technology, but to the ability to easily and cheaply deploy the technology at scale. It is the scale of this deployment that is disruptive. As technology historian Melvin Kranzberg puts it:
Many of our technology-related problems arise because of the unforeseen consequences when apparently benign technologies are employed on a massive scale. Hence, many technical applications that seemed a boon to mankind when first introduced became threats when their use became widespread.
Thanks to the growing scale of AI deployment, society seems to be at a tipping point: a transition from a world containing some automated decisions to a world dominated by automated decisions. Society is formalizing decisions in algorithms, cementing them in software to automate them, and then connecting these decisions to each other and the operational solutions surrounding them. Where previously the digital landscape consisted of the isolated islands of enterprise applications and personal computing, the landscape today is one of always online, available, and interconnected cloud solutions and smartphones.
The technology used to automate decisions is less important than the volume of decisions being automated and the impact of connecting these automated decisions so that they affect each other. We’re also integrating these automated decisions with hardware that can affect the real world. And we’re doing this at scale, creating a landscape dominated by overlapping decisioning networks. It’s not that the individual decisions being automated are necessarily problematic on their own (though they may be, and we need guardrails to help ensure that this isn’t the case). Rather, problematic behavior often emerges when automated decisions are integrated and affect each other directly, something we might consider distributed stupidity – situations where emergent unintended consequences and clashes between automated decisions result in “smart” systems going bad.
A car rental firm, for example, might integrate the end-to-end rental process, from payments through to provisioning, reaching all the way into individual rental cars by using Internet of Things (IoT) sensors and effectors. This could enable the firm to track car location and provide more tailored rental plans and support renters on the road, while also reducing theft by immobilizing (stationary) cars should they be stolen. However, these systems might lead the firm to inadvertently immobilize a long-term rental car while the renters are camping in a remote location with intermittent (at best) mobile phone coverage, believing the car to be stolen due to a temporary fault with a payment gateway that was progressively escalated by a series of automated decisions when the firm was unable to contact the renters via SMS or an outbound call center. The renters in this case would be left without a functioning vehicle in an isolated location and with limited resources, unable to walk out or contact help.
The point is that a bad (automated) decision can now have a cascading series of knock-on effects, triggering further bad decisions that escalate the problem. The unforeseen consequences Kransberg warns of might well, in such instances, be the result of unintended interactions between previously manual decisions that have been automated and then integrated. These interactions could be highly contingent, as with the rental car example. They can also be prosaic, such as mistakenly adding a name to the list of redundancies after a merger, which could force a firm to terminate and then rehire an employee. Integrating payroll with operational and access control systems streamlines internal processes, but it also creates a network of automated decisions that, once started, the firm no longer controls.
This is a difference in degree, not type, with the low (and dropping) cost of technology shifting the question from can we to should we. We need to consider the four “ares”: Are we doing the right things? Are we doing them the right way? Are we getting them done well? and Are we getting the benefits? The dual edge here is that because the cost to deploy and integrate these automated decisions is low and dropping, governance and oversight are also lowered, while issues concerning privacy, persuasion, and consent come to the fore.
We need to focus on the system, rather than the technology, as it’s systems in use that concern us, not technology as imagined.
Aligning Technical Solutions With Social Norms
Our second challenge – the problem of aligning technical (AI) solutions with social norms – is one of not seeing the wood for the trees. The technical community, by nature of its analytical approach, focuses on details. The problem of creating an autonomous car becomes the problem of defining how the car should behave in different contexts: what to do when approaching a red light, when a pedestrian stumbles in front of the car, and so on. Designing “correct” car behavior is a question of identifying enough different contexts – different behavioral scenarios – and then crafting appropriate responses for each situation. Similarly, creating an unbiased facial recognition algorithm is seen as a question of ensuring that the set of behavioral scenarios (and responses) used to design the algorithm is suitably unbiased, trained on a demographically balanced set of images rather than relying on historical (and potentially biased) data sets.
This reductionist approach is rightly seen as problematic, as whether or not a particular response is ethical (or not) is often an “it depends” problem. For autonomous cars, this manifests in the trolley problem, a thought experiment first posed in its modern form by Phillipa Foot in 1967. The trolley problem proposes a dilemma where a human operator must choose whether or not to pull a lever that will change the track that a trolley is running down. The dilemma is that a group of people is standing on the first track, while a separate individual is on the second, so the operator is forced to choose between the group dying due to their inaction, or the individual dying due to their action. The point here is that there is no single “correct” choice; any choice made will be based on subjective values applied to particular circumstances one finds oneself in, nor can one refuse to choose. Many of the scenarios identified for our autonomous car will not have obvious responses, and reasonable individuals may disagree on what the most appropriate response is for a particular scenario. Similarly, attempting to align the training set for a facial recognition system with demographics leads to the question of which group of people will determine the demographic profile to be used.
The diverse and complex real world makes slicing any problem into a sufficient number of scenarios to ensure ethical behavior a Sisyphean task. There will always be another, sometimes unforeseen scenario to consider; newly defined scenarios may well be in conflict with existing ones, largely because these systems are working with human-defined (socially determined) categories and types that are, by their nature, fluid and imprecise. Changing the operating context of a solution can also undo all the hard work put into considering scenarios, as assumptions about demographics or nature of the environment – and therefore, the applicable scenarios – might no longer hold. Autonomous cars designed in Europe, for example, can be confused by Australian wildlife. Or a medical diagnosis solution might succeed in the lab but fail in the real world.
The natural bias of practitioners leads them to think that “fair” or “ethical” can be defined algorithmically. This is not possible – a blind spot, generally, for the technologists.
Bridging Social Worlds
The third and final challenge is bridging different social worlds. All of us have our own unique lived experience, an individual history that has shaped who we are and how we approach the world and society. The generation that came of age in the Great Depression during the 1930s is a case in point: Failing banks during that time took countless individuals’ life savings with them, generating a lifelong distrust of banks among many people.
Disagreements in society are typically framed as differences in values or principles, differences in how we evaluate what we see around us. However, some of society’s deepest and most intractable disputes are not primarily about values and principles. Indeed, we can often agree on principles. The differences lie in the social worlds to which we apply these values and principles: the way we interpret what we see around us. We might agree with the principle that “it’s wrong to [unjustly] kill people,” for example, while disagreeing on what constitutes a person.
Progress on these most intractable disputes is difficult, as it’s common to assume that there is a single secular society (a fully normalized social world) against which to measure principles such as fairness. The assumption is that everyone sees the same world as we do ourselves but just approach it with different values, when this is not necessarily the case – a blind spot for many social commentators.
We can see these differences in social worlds come to the fore in some more recent and more controversial AI solutions. COMPAS, the recidivism-predicting tool mentioned earlier, is a good example. The team developing COMPAS took a utilitarian approach, creating a solution for a world where all individuals are treated equally and where harms (roughly, the proportion of incorrect predictions) are minimized for the greatest number of people. If we use a different measure and judge COMPAS according to the norms of a different world, one focused on equity where all individuals experience similar outcomes in life no matter what circumstances they start under, then COMPAS is lacking, as the unintended harms it causes fall disproportionately on disadvantaged groups. This is the “fairness paradox,” as improving COMPAS’s performance in one world results in the solution performing worse in others (and vice versa).
While we agree that our AI solutions should be ethical – that they should adhere to principles such as fairness (promoting fair treatment and outcomes) and avoiding harm, we can also disagree on which trade-offs are required to translate these principles into practice – how the principles are enacted. Applying the same clearly defined principle in different social worlds can result in very different outcomes, and so it’s quite possible, in our open and diverse society, for different teams working from the same set of principles to create very different solutions. These differences can easily be enough for one group to consider a solution from another to be unethical.
It’s common at conferences to pose the (rhetorical) question: Who decides what is ethical? Any design decision is likely to disenfranchise or otherwise affect some demographic group or fail to address existing inequalities or disadvantages, so it’s implied that care must be taken to ensure that decisions are made by a suitably sensitive decision-maker. This is likely to be the wrong question, though, as focusing on who makes the decision means that we’re ignoring how this individual’s particular social world (which will be used to frame what is or is not ethical) was selected. A better question is: How can one build a bridge between the different social worlds that a particular solution touches? There are trade-offs to be made, but without such a bridge, one cannot begin to determine how to make them.
We might summarize the challenges of developing ethical AI solutions (moral decisioning networks) as being similar to thermodynamics in that you can’t win, you can’t break even, and you can’t leave the game. We can’t win, because if we choose to frame “ethical” in terms of a single social world – an assumed secular society – then we must privilege that social world over others. We can’t break even, because even if we can find a middle ground, a bridge between social worlds, our technical solution will be rife with exceptions, corner cases, and problems that we might consider unethical. Nor can we leave the game, banning or regulating undesirable technologies, because what we’re experiencing is a shift from a world containing isolated automated decisions to one largely defined by the networks of interacting automated decisions it contains.
If we’re to move beyond the current stalemate, we need to find a way to address all of these challenges: a method that enables us to address the concerns of all involved social worlds (rather than privileging one over others), that enables us to consider both the (proposed) system and the community it touches (rather than just the technology), and one that also provides us with a mechanism for managing the conflicts and uncertainty, the ethical lapses, that are inherent in any automated decisioning system. We need an inclusive dialogue.
Trust And Acceptance
A successful AI solution – a successful automated decisioning network – is one that not only effectively performs its intended function, but one that is accepted, approved, and ultimately trusted by the people it touches. While there will be challenges, managements shouldn’t find themselves dealing with “never-ending demands” from “local troublemakers,” hearing that “the company has done nothing for us” while incurring costs that weigh the project down. The relationship between management and community should be collaborative rather than adversarial, working together to understand when AI should be used. Unfortunately, we’re a long way from such a state of affairs.
The concept of a social license to operate for AI has the potential to address all three challenges – definitional, aligning a solution with social norms, and bridging social worlds – discussed above. An SLO puts the focus on the overall solution and the social and physical environment into which it is deployed rather than on the technology, avoiding the problem of centering our method on particular AI technologies. It also addresses the challenge of bridging social worlds by acknowledging that the solution can never be considered ethical per se. While a firm might have the legal right to operate, it must also obtain, with the consent of the community, a moral license to operate, and this license must be maintained and renewed as both the solution and community evolve and circumstances change. The ongoing process of developing and maintaining an SLO enables a firm to build a bridge between the social world of the firm and the social world of the affected community – which itself may contain multiple social worlds that also need to be bridged. The SLO process does this by providing a framework within which the firm can work with the community to understand each other, the proposed solution, and each party’s goals, norms, and principles. They work together to develop a shared understanding of the proposed solution (focusing on the decisioning network rather than on the particular technologies) and then to determine how shared principles are enacted in real life – addressing the problem of aligning a solution with social norms – by identifying problems and opportunities and finding solutions. Being open to this type of dialogue means being vulnerable, because honesty is required in order for the dialogue to be open and inclusive. Products and services must be fairly represented. Stakeholders need to be willing to trust that technology has not been misrepresented or, in the event of informed consent, that data will be stored and used as promised.
A Case Study: An Intelligent Hospital
Consider a case where a firm is developing a “smart” hospital. This hospital will have all the usual accouterments of a smart building: IoT sensor networks to track how inhabitants use the building – identifying patterns of room use and individual preferences – and automation to both optimize the building’s operation and tailor it to individuals, minimizing maintenance costs, reducing the building’s environmental footprint, and improving convenience and comfort for its users. Floor-by-floor and zone-by-zone air quality and staff presence data will enable air conditioning and heating to be optimized, reducing power and water use while improving comfort. Data on ambient light levels and staff activity can be used to minimize lighting. Plant equipment, such as backup generators and oxygen supply lines, can be instrumented to enable just-in-time maintenance. Smartphone apps will enable inhabitants to interact with these systems and personalize their experience. And so on.
AI will be used to string these systems together, transforming our smart hospital into an “intelligent” one. Voice assistants will be ubiquitous – installed in registration (including for the emergency room), patient and treatment rooms, surgery, and so on – providing staff, patients, and their guests with a more convenient way of interacting with hospital processes, calling for help, and bridging any language barriers. Staff, patients, and visitors are tracked from when they first approach the building and associated with records maintained in operational systems – patients should never go missing again, visitors will be directed to whomever they’re visiting via wayfinding, and staff can always find the nearest specialist in an emergency. Decision support tools speed diagnosis, highlighting potential problems on medical images and suggesting what a patient’s particular collection of symptoms might imply. All this information is fed into AI-powered situational awareness and planning systems that identify problems (possibly before they crystallize into emergencies) and present decision-makers with both potential problems and possible solutions. A patient’s mutterings, for instance, are correlated with unusual readings from bedside monitors and interpreted as advanced heart disease, resulting in the situational awareness and planning system dispatching a drone crash cart while alerting support staff and the nearest specialist, and suggesting a change to the operating room schedule to accommodate a potential emergency.
While a boon, our intelligent hospital will likely suffer from many of the problems associated with a large-scale AI deployment. Voice assistants, for example, must support a range of languages, but which dialects within each language should be supported to avoid biasing the solution, and how should the hospital support those who (for whatever reason) can’t talk? A tool that “reads” X-ray images and highlights lung damage or other signs of pneumonia, a tool that worked well in the hospital where it was developed, might be biased against one of the demographic groups that our intelligent hospital serves, providing an undesirably high level of false negatives or positives. What should the situational awareness and planning system prioritize when confronted with conflicting needs for a scarce resource, such as a particular specialist or machine: Which patient gets priority, and should the system be empowered to make these decisions on its own? There is also the possibility of unexpected interactions between these systems causing problems via emergent distributed stupidity: A voice assistant in a patient room might consistently misrecognize a patient with an uncommon dialect and, exacerbated by biases in diagnosis recommendation solutions, cause situation analysis to create many erroneous low-level requests that the staff soon dismiss, leading the staff to turn off decision support and so miss the patient’s underlying problem before it becomes critical.
Our intelligent hospital can also amplify existing discrimination, disadvantage, and privacy concerns. Flawed AI behavioral profiling derived from social media and smartphone data could, for example, influence medical risk profiles determining which treatments are offered. Data from medical devices pieced together by situation analysis – blood oxygen, heart rate, and so on – might provide accurate prognoses that are implicitly treated by staff as do not resuscitate (DNR) decisions, decisions that might not be in a patient’s best interest but represent the most efficient use of hospital resources.
AI enables the hospital to take data generated by the sensor network (security cameras, for example), identify individual people, profile them, and then to discriminate between them, either individually or as groups, and treat them differently. This discrimination can be a boon – allowing the firm to adjust the building or medical treatment to their needs and preferences while smoothing their journey through the day. The discrimination could also be harmful – creating undue stress by enabling the firm to track toilet breaks, generating maps of who is talking to whom and use them to identify groups unrelated to work for union-busting purposes, determining what treatments are offered to a patient, or even determining which patient is treated when resources are scarce. This discrimination relies on a wealth of personal data (both captured and inferred) stored in operational systems, elevating the risks and consequences of our intelligent hospital’s operational systems being hacked or leaking personal data.
From Acceptance Through Approval To Trust
To understand how a firm might go about gaining a social license, it’s important to consider the major role that trust plays in this effort. The benefits of a social license to operate are the result of the community’s acceptance and approval of a solution – the intelligent hospital in our example – and this acceptance and approval stems from the community’s trust in the firm. If the firm is to realize the anticipated benefits of the intelligent hospital, it needs to ensure the acceptance and approval of the community that will be using it. Failure to do this is likely to result in disruptions that drive up cost and prevent the benefits from being realized. These can range from the minor (small disobediences such as sabotaging the sensors on a floor or using patterned clothing to hinder AI profiling and location tracking) to the major (attempts to hack the system and render it inoperable, or protests). Unanticipated bias in voice assistants, for example, could lead to protests by affected community groups unable to engage with hospital systems. Prioritization decisions by the planning system that are not aligned with community norms, or simply surprising to many in the community, could result in the entire project being questioned.
The firm has a great deal of freedom in how AI is used to realize the intelligent hospital. While voice assistants will require some form of voice recognition technology, a range of audio and video techniques can be used to track inhabitants to similar effect. A number of different approaches – many configurations of sensors, decisions (potentially made by AI technologies), and (consequential) actions – are possible, though only some of them will be acceptable, and even fewer may be desirable, to both the people working in and using the hospital and the firm commissioning it.
What is important is what decisions are made, which of these decisions are automated and which are not, how these decisions affect the quality of the working and private lives of the people using the building, the effect of the decisions on the human dignity of the people they touch, and how the decisions align with community expectations. The firm needs a social license for the intelligent hospital. The community needs to trust the firm if it is to grant the license: trust that the firm will do (and is doing) what it says it will, and trust the firm’s ability to execute and deliver on its commitments.
Ultimately, trust is a relationship of reliance. It’s the belief that a counterpart will behave in certain ways, as well as the belief that the counterpart is dependable and competent, that they can be relied on. A firm that works collaboratively with the community, demonstrating integrity and competence in how it shapes the solution and manages operational risk, will likely be seen in a positive light. A firm that takes advantage of a community’s vulnerabilities, is seen as cynical or incompetent, or shows poor stewardship of its own vulnerabilities, will be viewed poorly.
Trust-building enables members of the groups associated with the initiative to accept being vulnerable to one another (something many businesses may need to learn), and it also helps deescalate conflicts. Failure by the firm to meet community expectations, either for reasons beyond the firm’s control or because the results of the firm’s labors don’t align with community expectations, erodes trust. When trust breaks down, it is often replaced by suspicion – suspicion that results in “never-ending demands” from “local troublemakers.”
Within the context of social license to operate, trust relies on four factors: a firm’s (or its solution’s) impact on the community; the quantity of contact between the firm and the community; the quality of that contact; and the procedural fairness of decisions made regarding the solution (figure 1). A firm can take action in all four of these areas to build trust with the community and so increase the community’s acceptance and approval of its actions.
Understanding a solution’s impact on the community entails recognizing that all solutions bring with them problems as well as benefits. Our intelligent hospital potentially has a smaller environmental footprint though more efficient energy use. It may facilitate more inclusive operations by enabling staff to support a broader range of languages. And diagnoses might be more accurate and swifter. However, the building also has the potential to increase work stress; introduce the privacy risks of sensitive personal data being leaked or otherwise misused; or institutionalize undesirable biases, inequalities, or disadvantage; as well as being subject to emergent distributed stupidity. But many of these benefits and problems can be anticipated by firms, enabling them to bolster benefits while mitigating problems.
It is also important to consider how the community experiences a solution, and how individuals experience it personally. For instance, integrating Bluetooth-enabled medical devices directly into the intelligent hospital’s IoT network might be met with a similar response to the Bluetooth-enabled tampon discussed earlier. Or a desire to streamline operations by simplifying how staff can collaborate around an image recognition solution might not adequately address concerns about privacy and human dignity. It’s quite possible for different stakeholders within the community to have different expectations for a solution’s benefits and problems. Similarly, an unanticipated dialect could result in frustration or even exclusion of an individual unless the speech recognition failure is dealt with gracefully. This mismatch between the firm’s intention and community expectations of a solution’s impact and benefits can be a significant source of the unanticipated consequences for the firm.
The distinction between the impact of a smart hospital and an intelligent one, between a hospital without and with AI, is one of degree rather than kind. AI increases the potential benefits, but it also elevates the risks.
This brings us to the next two factors supporting trust: the quantity and quality of the firm’s contact with the community. Trust is the result of frequent positive contact between the firm and the community. The firm that builds our hospital needs to present a human face to the community (the firm, after all, is also a community), a face that the community can learn to trust and work with.
Contact should be frequent (quantity) and meaningful (quality). Practically, contact can range from formal impact studies attempting to gauge how a solution will affect a community and their disposition toward it, to day-to-day contact in the field via community groups or between individuals and representatives of the firm, as well as contact with stakeholders who are not directly affected by the solution but who have an interest in influencing the outcome. Some of this contact might also be mandated via regulations such as the General Data Protection Regulation (GDPR) or those associated with the industry in which the firm is operating. Frequent, meaningful contact enables the community and the firm to learn about each other, reducing the unknowns (and the unexpected) by minimizing misinterpretation and avoiding the projection of one’s own belief systems onto the other.
The fourth factor influencing trust, procedural fairness, is the decision-making and dispute resolution processes that govern a solution’s development and operation. Individuals must perceive that they have a reasonable voice in the decision-making process, that the decision-makers have treated them respectfully, and the procedure is one they regard as fair. They must also feel that there is equal power between parties – community and firm – so that the solution is truthful.
For the community to accept our intelligent hospital and trust the firm behind it, they need to feel that their opinions are valued, that their point of view has been accounted for, that they are being treated respectfully and with dignity, and that their view is being integrated into the solution. End-of-life care or intensive care treatment augmented by AI, for instance, needs to support patients and treat them with dignity and respect, rather than be based on an economic calculus. It should be practical for individuals and groups, for example, to respond to the proposal to use voice assistants throughout the hospital, pointing out problems and suggesting alternatives. Both decision-making and dispute processes need to be understandable and navigable by individuals so that they can see their views being accommodated and weighed against not only those of others in the community, but with technical and financial constraints and the firm’s own interests.
The Missing Parts
The concept of social license to operate can provide us with a solid foundation for a moral license for AI, but work needs to be done to adapt it to the needs of firms developing AI solutions. There are three questions that we’ve been skirting in this article so far that we need to address if we’re to move forward. These questions are:
- How do we describe the (proposed) solution without unnecessary (and confusing, for many stakeholders) technical details or reverting to overly abstract concepts?
- What constitutes “community” for our solution – that is, how do we identify our stakeholders?
- How do we evolve the solution, working from a proposed solution to one that the stakeholders consider ethical, identifying where the trade-offs are to be made and how to make them?
We can deal with these in order.
Describing The Solution
The first hurdle to overcome is to find a way to describe our solution, such as the smart hospital in our example. While our familiarity with voice assistants makes them easy to understand, it is more challenging to understand a situational awareness and planning solution due to its more nebulous nature, as it requires data to be sourced from around the hospital to drive a network of interconnected decisions that provides recommendations and triggers actions for a diverse range of (potential) patient problems. We need a language that the community and the people proposing it can use to discuss the shape the AI solution will take – how inhabitant location will be tracked and what the tracking data will be used for, how it will interact with situational awareness, what actions and processes situational awareness can drive, and so on – as well as the relative problems and benefits of alternative approaches to realizing this functionality. It’s AI’s ability to integrate this broad range of sensors and effectors, to transform our smart hospital containing isolated automated decisions into an intelligent hospital that contains an integrated automated decisioning network, that highlights this need.
Describing our solution involves solving what we might call the brewing problem. Brewing required the development of microbiology – a language integrating biology and chemistry – before it could transition from craft to engineering. This made it possible to fine-tune the brewing process and obtain more consistent results. Similarly, if we’re to fine-tune our AI solution, then we need to be able to describe and discuss it in a language that is accessible to both the community and the people proposing it, a language that encompasses both ethics and implementation, but without including too many technical details. To be both comprehensible and useful, this language needs to be more specific than our high-level ethical principles, but more general than implementation details. It should also avoid technical jargon, using straightforward and accessible terms to support a common understanding that contributes to building trust. We need to be able to describe the interconnected and aggregated set of decisions (the decisions and their relationships) in our proposed solution; which actor (human or machine) enacts each decision; what information drives the decision; the consequences (and information) resulting from a decision; and the impact of these actions (and changing information) on humans.
It can be important to distinguish between decisions made by a human and those made by AI, as humans and machines think (and decide) differently. As humans, we use our senses and lived experience when we make a decision, even if we’re making it unconsciously. We notice the unusual and unexpected and factor it into our deliberations. Machines, on the other hand, only consider that data that they’re designed to consider. If a decision is consequential – such as the decision to fire a missile, withdraw an individual’s social benefits, or to move a lifesaving machine to a different patient – then it is common to prefer that the decision is made by a human, as only a human will consider an unusual factor, something unexpected but important enough to sway a decision. In some cases, regulation might require particular decisions to be made by a human (or even by a group) rather than algorithmically. However, while we want to distinguish between human and machine decisions, we might be less interested in how the machine decision is implemented.
Our intelligent hospital might be described in terms of what information is captured, the decisions that are informed by this information, the entities that make the decisions (human or machine), and the information and actions that spring from each decision. For instance, the description may specify that a temporary identification badge issued to a visitor (information) will be associated with video images and a voice print (information) to identify the visitor (via a machine decision) so that the hospital can track them as they move through the building. (The technology used to associate the two is less important than the fact that the association is made.) If the building determines that the visitor wanders into a prohibited area, then it notifies (a machine decision) security staff on the floor who will determine what to do (a human decision). A complete description of a solution could contain many of these information-decision-action threads covering our intelligent hospital’s operations (“Man is an animal suspended in webs of significance he himself has spun”), which will be evolved and refined in collaboration with the community.
Defining The Community
Before we begin any work, we need to delineate the social boundary of our system. We must establish who the stakeholders are, understand their dispositions, discover the social worlds at play, and identify our “experts,” gatekeepers, and informants.
“Community” may well be too narrow a term to capture the diverse set of stakeholders that a complex solution such as our intelligent hospital touches and whose lives it affects. It’s easy to assume a social license to be a single license granted by a well-defined community. This is not true in complex environments, where the community is composed of a diverse collection of subgroups drawn from other geographic areas and communities. In these cases, it’s more productive to think of a social license to operate as a continuum of multiple licenses across these subgroups, across multiple overlapping and interrelated communities.
An anthropologist might start by listing the different behaviors, thoughts, and attitudes that should be considered, along with demographic attributes such as employment status, income, gender, primary language, and so on – factors that describe differences in the community. These factors are mapped to a set of community factors, with each factor capturing a tension or difference in preference that might exist in the community. Obvious examples from our intelligent hospital are a worker’s attitude to gender (whether gender is considered strictly binary or if a broader definition is accommodated), the nature of their work (analytical and bureaucratic or manual), their educational attainment, their religious or belief system, socioeconomic (dis)advantage, or whether they work in the hospital regularly or only visit occasionally. A complete set of factors provides us with a mud map of the landscape our community might cover.
Firms can use a range of formal and informal methods to investigate community members’ behaviors, thoughts, and attitudes, such as observation, structured and semi-structured interviews, group discussions, diary studies, or workshops with members of the group being studied. The important thing is to establish an open dialogue where information flows back and forth between researchers and subjects. Participants can be selected from the community to ensure that all known factors are covered, with particular attention given to edge cases. The goal is to learn as much as possible about the community’s history and the individuals within it to develop a full understanding of the social worlds the community contains and how it functions.
What the firm learns can be captured in an actor network – a web of human and nonhuman “actants,” their relationships, conflicts and alliances, and the processes that bind them together – which can be used to identify a set of representative community member profiles (and representative community members) and how they might relate to the proposed solution.
Refining The Solution
Our last challenge is to work with our community to refine our solution. In an approach inspired by the technique of general morphological analysis (GMA), we can break this into four phases.
First, we take an idea, such as our intelligent hospital, and create a description of it. The building might use this data to drive these decisions, with this decision resulting in these actions. This is the language discussed earlier in the article, the information-decision-action threads that describe how the building will monitor visitors while in the building, support diagnosis, identify and help manage emergencies, and so on. The description can be kept general at this point by, for example, not concerning ourselves with whether a decision is made by a machine or a human.
Next, we refine our solution over two phases: eliminating the impossible, and then discovering what is allowable (and acceptable) to the community (as regulation lags behind ever-evolving social norms).
Eliminating the impossible involves enumerating all possible solution configurations – combinations of which information might feed which decision to trigger which action – and then eliminating the ones that are clearly impossible, such as those configurations that are either technically impossible or that are prevented by regulation. Regulation might require that a particular decision must be made, or supervised, by a human, leading us to add “this decision is performed by a human” to our solution description. Our intelligent hospital, for example, might require that any decision to transfer a lifesaving machine to a higher-priority patient is made by a human. In cases where we want the benefits of both human and machine decision-making, we might split the decision in two: a machine suggestion that can be considered as part of a human decision. The situation analysis and planning solution could be restricted to providing recommendations to a human manager who is responsible for determining the course of action. Or a decision might be required to be made by a suitably qualified person, or one with a particular level of seniority, such as a medical specialist – a requirement that is noted in the description of the decision. We might also require that a machine decision is also understandable by a human, noting in the machine decision’s specifications that whatever technique used must provide a rationale for the decisions it makes. Our planning engine, for example, might be better implemented via rule-based constraint satisfaction rather than machine learning, as this may simplify users interacting with and tweaking the solution’s reasoning.
This first phase of analysis will also determine when a piece of data represents personal data (such as gender) that can only be used as an input to a few specific decisions. Based on this analysis, our description of the solution can be evolved, either by changing elements – information, decisions, and actions and their relationships – or by annotating them to restrict how each element might be used or implemented.
The next step, removing the unacceptable, is a similar process, but must be done in consultation with the community. Working with community representatives (aligned with the representative community member profiles identified earlier), a firm can identify what outcomes and processes are more or less acceptable to the community.
This phase can also explore the solution’s benefit (to the community) and maturity, using a tool such as a Wardley map to expose assumptions, permit challenges, and create consensus. For example, if a particular decision is required to be “fair” – such as the choice in COMPAS between equality and equity, or the prioritization of patient needs in an emergency – then how fairness is to be enacted could be determined in collaboration with the community representatives and noted in the decision’s description. Groups of related components – such as an automated registration process that integrates voice and touch interfaces with image recognition – can be reviewed to ensure the ensemble as a whole will not disadvantage or otherwise negatively affect individuals even though particular AI components are not perfect. The community’s attitude to (potentially) controversial technologies can also be considered: The community may be uncomfortable with ubiquitous video surveillance, prompting our intelligent hospital’s owners to find a more acceptable way to track inhabitants as they move through the building. The role of situation analysis and planning might be questioned due to concerns (mentioned earlier) that accurate prognoses will be treated as implicit DNR recommendations that are not in a patient’s best interests. With challenging questions such as this, the firm may need to consult with many diverse groups in the community to develop a coherent approach that is acceptable to the community as a whole.
At the conclusion of eliminating the impossible and discovering what is allowable, we have a detailed outline of our solution – though not a complete solution, as it won’t have details that the firm and community do not consider pertinent. The algorithm used to maintain the temperature in a building zone will likely, for example, remain unspecified. Other details, on the other hand, might be quite tightly specified, such as the allowable uses for the video streams emanating from security cameras, how consequential recommendations from AI solutions (such as accurate prognoses) should be treated, the extent to which behavioral profiles can influence decision-making, which machine decisions are required to be understandable by a human, or how “fair” should be interpreted when dealing with conflicting patient priorities.
The processes of eliminating the impossible and discovering what is allowable enable the firm, in collaboration with the community, to determine how ethical principles (such as fairness or preventing harm) are enacted, documenting this in a shared description of the solution. We have what might be called an “ethical requirements architecture.”
The final, fourth phase is the technical challenge of taking the refined solution description and determining how it should be realized. It’s in this phase that the wealth of work on methodologies and techniques to create unbiased and ethical algorithms – “trustworthy AI” – is leveraged.
Developing An Ethical Requirements Architecture
If we integrate these frameworks – social license to operate, a language for our ethical requirements, understanding the community via social science approaches, and developing and refining the solution via GMA – then we might have something similar to what is shown in figure 2. (Though we would like to note that this article has only been sufficient to develop an outline or description for this process, and not the process itself.)
The left half of the circle in figure 2 depicts the GMA process, from initial solution proposal through refinement, to create an ethical solution architecture and an eventual system. The circle’s right half depicts the social sciences flow, where the firm develops a mud map of the community by identifying community factors before mapping the community terrain via an impact analysis and then engaging with the community to refine the solution.
There are two touch points between these streams. At the first, the firm develops the initial solution outline and its understanding of community factors side by side; the second comes later, when the firm works with community stakeholders to shape what an acceptable solution might look like. Cocreating the ethical solution architecture in this way addresses the possibility of the community not wanting the solution in the first place – such as our intelligent hospital’s users resisting the very concept due to fears of invasive monitoring – and enables a firm to establish trust in the community by being transparent about the firm’s motivations and goals. The entire process also loops back since, as we pointed out much earlier in this article, an SLO must be revisited and maintained as both the solution and the community evolve over time.
What we deem to be “good” evolves in a way that enables humans to live together in groups and ultimately create harmonious civilizations. Some morals (“thou shalt not kill,” “thou shalt not steal,” and so on) have been with us for millennia, while others (inclusive voting and marriage rights are good examples) have emerged relatively recently. Some morals are unique to particular communities. It is this quality of morals, their emergence as communities come together and interact, that we need to be sympathetic to. For AI solutions to be accepted and trusted by a community, one must engage with the communities they are being applied to and their emergent morals. The engagement must be meaningful and ongoing, and it must ensure that what the community believes is moral and “right” is incorporated into the solution.
It’s important to note that this journey should be undertaken with an open mindset. The process we’ve outlined could be followed with a closed and unempathetic, or even merely disinterested, mindset, which would provide quite a different outcome. Without the right skills and mindset, the process may not yield the desired result. Interactions between firm and community members are touchpoints to establish and build trusted, respectful relationships to work toward a mutually acceptable outcome. As discussed earlier, the challenge is to bridge social worlds, rather than privileging one over the other.
We also need to allow for the possibility that firms may choose not to engage with a community in good faith, or for situations when there is a power asymmetry between the firm developing the AI and the community it will affect – or, at worst, situations where the firm’s intentions are nefarious. In this case, a framework such as a moral license for AI could provide regulators with the leverage required to enact reporting requirements that ensure that nefarious firms have at least gone through the motions, using known methods while documenting their interactions with the community and the outcomes. Developing an “ethical requirements architecture” could well become the regulatory equivalent of an environmental impact study for AI.
The Need For A Moral License For AI
Work on ethical AI has focused on developing the principles, requirements, technical standards, and best practices needed to realize ethical AI. However, while there is a clear consensus that AI should be ethical, and a global convergence around principles for ethical AI, there remain substantive differences on how these principles should be realized, on what “ethical AI” means in practice.
While this article is notionally about “ethical AI,” it never addresses the question of ethics and AI directly, taking a different tack. Rather than attempt to define what AI uses are and aren’t ethical, it proposes that firms need to work with the communities they touch, and obtain and maintain a moral license for the AI-enabled solutions they want to operate. Moreover, firms should consider doing this for any solution that automates decisions and integrates them with other operational systems to create decisioning networks – not just solutions that contain what is currently considered AI technology.
This difference in approach is due to three observations:
- That AI solutions cannot be made ethical though the development of “fair” or “ethical” algorithms or development methodologies
- That there is no single secular society (a fully normalized social world, an objective standard) against which we can determine if a solution is ethical or good
- That the importance of ethical AI is not due to the development of disruptive AI technology or an existential threat from isolated, self-aware, AI solutions, but rather due to the widespread emergence of automated decisioning networks
Ethical AI – the development of regulation, techniques, and methodologies to manage the bias and failings of particular technologies and solutions – isn’t enough on its own. Ethics are the rules, actions, or behaviors that we’ll use to get there. Our goal should be moral AI. We must keep a clear view of our ends as well as our means. In a diverse, open society, the only way to determine if we should do something is to work openly with the community that will be affected by our actions to gain their trust and then acceptance for our proposal.
originally posted on deloitte.com by Peter Evans-Greenwood, Rob Hanson, Sophie Goodman, Dennis Gentilin
About Authors:
Peter Evans-Greenwood is a fellow at the Deloitte Centre for the Edge Australia, helping organizations embrace the digital revolution through understanding and applying what is happening on the edge of business and society. Evans-Greenwood has spent 20 years working at the intersection between business and technology. These days, he works as a consultant and strategic advisor on both the business and technology sides of the fence.
Rob Hanson is a senior research consultant at the Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia’s national science agency. Hanson is a transdisciplinary researcher who works at the intersection of emerging technology and their policy implications. He has a professional background in technology, security, risk management, and strategic foresight. Hanson’s current research focus includes data privacy, consumer data, and trust. As a PhD candidate, his thesis is titled “Fables for future technology.”
Sophie Goodman is an applied anthropologist who previously worked in customer-led strategy and service design projects as part of Deloitte Digital. She uses her experience and training in ethnographic research to help organizations better understand and act on the needs and experiences of their customers. Goodman has worked in research roles for an Australian university and a global workplace culture consulting firm as well as on customer and user experience projects.
Dennis Gentilin helps organizations build the infrastructure required to promote ethical behavior and drive sustainable performance. A unique career experience catalyzed his strong interest in ethics. His book, The Origin of Ethical Failures, won the textbook prize in the 2017 UK Chartered Management Institute Book of the Year awards. He is an adjunct fellow at Macquarie University and an honorary fellow at the Centre for Ethical Leadership.
