A provocative story about the relationship between the humans on a British airbase and the AI security system that guards that base. When a group of humans are killed, the question is who is responsible and why.
The wind was blowing sharply from the east, across the north European plain from Siberia to the flatlands of East Anglia. Despite that, Helen Matthias was perspiring through her running suit by the time she finished her usual morning circuit of the Lakenwell Airbase perimeter. After two years, she was getting used to the winters here. They felt harsher than the ones she remembered as a kid in Massachusetts, and the snow usually came later, after the turn of the year. This morning she thought she could detect the faint scent of ice in the air. Was that a little hello from Siberia? Prasanna would have told her it was all in her head; if so, her imagination was especially strong today.
Maybe she should ask the donkeys, Helen thought, waving to the one plodding toward her on the perimeter road. When she had passed it earlier, it had automatically moved to one side, putting itself between her and the electrified fence to reduce her risk of accidentally stumbling into it; safety first. The donkey was still keeping to one side; maybe Felipe Dos had told it to expect more traffic.
Why people called them donkeys was a mystery to Helen; they looked more like a collection of welded-together toolboxes on four legs. There was no head; front and back were determined by their direction of travel. The roboticists claimed it was a matter of convenience. Helen told them robots that lacked the concept of backwards as humans understood it was one of those supposedly little things that could very well bite them in the ass later.
Buy the Book
AI and the Trolley Problem
The roboticists were skeptical but curious, and asked her to explain her thinking in detail, and in writing, thank you. What they really wanted, she knew, was a formal proof, but they’d settle for a well-reasoned hypothesis. Over the last several days, she’d been setting her thoughts down, and as often happened at Lakenwell, found she was having a hard time seeing the trees for the forest. Which was actually a jungle. As one of her philosophy professors had liked to say, Oh, what a tangled web we weave when first we practice to perceive. But perceiving was what they paid her to do.
“Hey, T-1,” she said to the donkey as it drew nearer, making a sound that wasn’t quite like a horse’s clip-clop. It was painted in spiraling red and white stripes that widened in the center of its body and narrowed at either end. No front, no back. “How ya doin’?”
“Can’t complain, but I always do,” it replied in a slight Texas twang. “Y’all stay safe now, and don’t pee on the fence.”
Helen gave a surprised laugh. That was new, she thought, staring after it. Apparently someone had expanded the database of responses, not to mention accents. T-1 was short for Thing One. Thing Two was on the opposite side of the camp. There were two others—Hop-A-Long and Bob—all of them wired into Felipe Dos, who ran most functions on the base.
According to the Lakenwell handbook, the donkeys were part of the security system. The bright colors made them easy to spot. They were armed with live ammunition, and they would shoot. Despite their clunky appearance, they could not be knocked over, and they could easily outrun a human over virtually any terrain. Any unauthorized attempt to access their software would cause them to self-destruct in a way the handbook described as “unpleasant and potentially life-changing to anybody nearby.”
Helen suspected the donkeys had more to do with surveillance than protection, but when she shared this thought with Prasanna, her friend laughed. “The base has full-saturation surveillance, but people spend hardly any time around the donkeys,” Prasanna said. “What could they possibly pick up that the outdoor monitoring system wouldn’t already have?”
Helen had been about to say the outdoor monitoring system wasn’t as comprehensive as the one indoors, so people tended to be less guarded, even around the donkeys. Then she thought better of it; they were, after all, indoors. “You’re right,” she said. “Maybe living under one-hundred-percent surveillance is making me paranoid.”
“I stopped noticing it a lot faster than I thought I would,” Prasanna told her. “Maybe I just like getting so much attention, even if most of the time it’s from Felipe Dos.”
Her heart rate had returned to normal after her run, but Helen stayed a few moments longer, breathing deeply and looking around. Lakenwell had been largely abandoned after the Cold War, and the British government had been happy to let the Americans set up a research lab with both civilian and military personnel. Now they all rattled around like too few peas in a too-large pod. After spending most of her adult life in urban environments by choice, Helen had been surprised at how easily she had adjusted to all the empty space and the isolated location. Maybe she’d simply needed the change.
A strong chill swept over her, and she remembered what the base commander, Gillian Wong, had told her: If you want to freeze to death, start by getting sweaty. Wong knew what she was talking about—she’d been with a number of Special Forces units before assuming command of the base. Helen wondered how she felt about getting such a tame assignment. Maybe Wong had needed a change, too.
Helen liked her, as did just about everyone on the civilian staff. She was good company and easy to talk to, always keeping things light and divulging little about herself. None of the civilians knew where she stood in terms of politics, religion, or sexual orientation. Helen supposed it was to do with her being the base commander. And she was always the base commander; she never seemed to be off duty, and no one had ever seen her out of uniform. None of them had seen any of the military personnel out of uniform. Prasanna joked that they probably all had special military pajamas, fatigue onesies. Helen thought if the heat in the military quarters was cranked up as high as everywhere else on the base, they probably slept in the raw.
All the military staff were pretty nice, if a bit more standoffish than their CO. According to Ybanez in systems engineering, they’d been handpicked by Wong personally. Helen was intrigued; a veteran from Special Forces and her handpicked unit watching over a long-disused airbase full of assorted engineers, roboticists, and AI researchers—complete and utter nerds, herself included—somewhere off a less-traveled road in the British East Midlands. What kind of trouble were they expecting?
After two years, she was pretty sure it was cabin fever. Everyone was confined to the base most of the time. The government provided plenty of compensation in the way of entertainment—an extensive library of books, movies, TV shows, and video games, not to mention full access to the web, not to mention a gym that would have made Helen’s old aerobics instructor weep with joy, although there were usually more soldiers in it than engineers. The onsite chef changed every four months—apparently food preparation was an industry that attracted people who never took a deal without parole.
Helen’s own employment contract had another year to run; after that, they’d either ask her to stay or invite her to leave. Unless she screwed the pooch in a particularly egregious way, she was pretty certain it would be the former. Specialists in machine ethics were still very thin on the ground; not many jobs for them, either, and the few that were available tended to be a lot more technical than what she was doing here.
She felt another, more intense chill and started toward the main residential building. Just before she reached the entrance, the door banged open and Cora Jordan bounded down the cement steps in mismatched sweats and a bright blue scarf the same shade as her bright blue hair.
“Hey, how the Helen are ya?” she said loudly, running in place.
Cora Jordan was a firm believer in overdoing every joke, especially if it were too boring and unfunny to actually be a joke. Helen made herself smile. “I’m good. You?”
“Completely fit for anything, of course,” Cora replied heartily. Her eyes looked a little too shiny. If she’d been anyone else, Helen would have been sure she was pumped up on something. In Cora’s case, however, it was more likely she hadn’t taken anything, probably for days.
“You want to join me for breakfast?” Helen asked her. “It’s so cold—”
“Can’t eat till after, I’ll puke,” Cora said, still running in place. “How is it this morning, cold?”
“Uh, yeah. How about a hot drink? Coffee, herbal tea—”
“No, I’ll puke,” Cora said impatiently. “You see anybody else on the track?”
“Not a soul. Unless you count Thing One.”
“Oh, great! I love those stripes, they’re so trippy. You talk to ’im? What’d he say?”
“I asked him how he was and he said he couldn’t complain but he always does,” Helen said, thinking that if she kept Cora engaged, she could distract her and get her back inside. “He also told me to stay safe and not pee on the fence.”
Cora screamed with laughter, jumping up and down as if this were the funniest thing she’d ever heard in her life, while Helen wondered why the sound hadn’t made anyone rush outside to see who’d been hurt. “Seriously?! Oh my God, that’s priceless! Don’t pee on the fence, Jesus! You think he’d say that to me if I asked him?” Before Helen could answer, Cora galloped away across the scrubby dead grass, her scarf flying.
Helen stared after her and sighed. “I’ve gotta report this,” she said aloud. “She could hurt herself. It’s not snitching. Friends don’t let friends drive drunk, friends don’t let friends run away from their meds.”
Except she already had.
“I called Medical and left a message before I took a shower,” Helen told Prasanna in the cafeteria. They were sitting at Prasanna’s usual table by the windows, looking out at the windswept runways to the east.
“Then you’ve done your duty,” Prasanna said, smiling. She was one of the few Brits on the team, a software engineer with dark brown skin and shiny black hair she wore in a single braid over her shoulder. Today she was dressed in a dark green pullover and black trousers; she always looked to Helen as if she were going somewhere special, even when she wasn’t wearing any makeup. “So why do you look like you did something wrong? It’s not that being a snitch thing, is it?”
Helen shook her head. “I should’ve made her come inside with me instead of telling her the latest cute thing T-1 said.”
“What did he say?” asked Prasanna. Helen told her and she laughed. “That is a good one. One of the guys must’ve come up with that—you have to warn guys not to do things like that. Mother Nature saved us from that kind of foolishness.”
“Unless we’re off our meds,” Helen said.
“Cora is an adult,” Prasanna said firmly. “She’s not legally incompetent even when she is off her meds. Which means not only are you not her keeper, you have no right to force her to do anything. You told Medical, it’s in their hands.”
Helen shook her head again. “I should have done something more. I don’t know how I’m supposed to give a machine ethics when my own need some work.”
“Jeez, give it a rest, will ya?” Prasanna said, and nodded at Helen’s chunky black watch. “How many calories did you burn on your run?”
Helen tapped the screen, waited, then took the watch off and gave it a hard shake. “I guess that’s classified,” she said, showing her friend the message on the tiny screen.
“‘Data unavailable’ again?” Prasanna made a tsk sound. “Third day in a row, isn’t it?”
“Yeah. It’s worse than useless,” Helen said. “I might as well have a Magic 8-Ball strapped to my wrist. It’s almost that big. I keep banging it on things.”
“Maybe that’s the problem.” Prasanna spooned up some grits. Her fondness for them was a recent development. Helen, who had studiously avoided consuming any herself during the few years she had lived in the Kansas City area, had never imagined she’d have to avoid them in the UK. “Maybe you broke it.”
“Nah, it’s shockproof. You can drop it off the roof and then kick a field goal with it and it keeps on ticking. Or humming.”
“Then maybe Felipe Dos thinks you’re too obsessed with calories.”
“That’s not as far-fetched as you’d think,” Helen said, smiling, “if a bit more advanced than I would expect. Although seeing as how I’m competent with or without calories, I don’t think Felipe has any right to—” She cut off, staring open-mouthed at the unbelievable sight visible through the windows behind Prasanna.
The other woman twisted around to look. “Oh my God. Tell me I’m not seeing that.”
“No can do,” Helen said weakly. “Cora really is riding Thing One like a—a—”
“Like a donkey,” Prasanna finished for her. She started to laugh, then quickly smothered it. “I’m sorry, that’s not funny, is it?”
“Actually, it is,” Helen said. The two women got up and went to the window.
Cora had tied her scarf around the donkey’s midsection and entwined her legs in it on either side to keep herself from falling off. The donkey made an awkward steed, giving Cora a bumpy ride as it headed for the main building over the scrubby, colorless grass. Cora slapped its would-be flank and hollered for it to run the other way. To Helen and Prasanna’s collective astonishment, it did—but without turning around, so that Cora was suddenly riding backwards.
She yelled for it to stop and turn around, and it obeyed, making a full, three-hundred-sixty-degree turn.
“Dammit, one-eighty!” Cora yelled. “One-eighty turn!”
The donkey started to do as it was told, then turned back. Cora kept yelling orders at it and it would start to obey, then reverse itself. “It’s like it’s confused,” Prasanna said to Helen.
“Felipe’s telling it to come in the way it’s supposed to when there’s a malfunction,” Helen said. “But for some reason, there’s a conflict because of Cora, and there shouldn’t be.”
“Maybe she’s sitting on an alt-delete button,” Prasanna said, unable to keep from giggling.
Helen shook her head. “Robots like this have been used in combat to carry weapons,” she said. “But this one’s a lot more sophisticated. It shouldn’t be doing that.”
“Maybe it likes her?” Prasanna was holding her middle now.
Cora had finally aimed the donkey away from the building. “Okay, let’s go! Head for the road! Giddyap! Mush! Andelay! Get the lead out!” The donkey suddenly took off at a gallop in the direction of the main gate, and by some miracle, Cora managed to hold on. Four soldiers in a golf cart came around the side of the building and gave chase.
“Took them long enough,” Helen muttered.
“The guards at the gate’ll stop her, won’t they?” Prasanna said, still laughing a little.
“Don’t ask me,” Helen said. “I just work here.” The watch she was holding chimed loudly. The message on the screen said 666.
Prasanna laughed some more. “You’re holding it upside down!”
“Nothing would surprise me.” Helen tucked the watch in her pocket and headed for Gillian Wong’s office.
Helen was somewhat alarmed to find two guards outside the commander’s office, both armed not with the usual pistols but with automatic weapons.
“Are we under attack?” Helen asked.
“Not that we know of, ma’am,” said the ranking soldier politely.
Helen’s jaw dropped. The last time Sergeant Kara Arendse had called her ma’am had been the day she’d arrived. Every couple of weeks they took turns beating each other at table tennis. Although, now that Helen was thinking of it, not lately; it had been at least a month since their last game, maybe longer.
“What’s going on?” Helen asked tensely.
“The commander will explain everything,” Arendse told her, her face expressionless. “Sergeant Martinez will escort you.”
“Follow me, ma’am,” said Martinez. Helen hesitated; Arendse stood at attention, pointedly staring straight ahead, giving no sign that she even knew Helen was still there.
“Please,” Martinez added. “This way.”
Helen kept quiet as she followed him through the main residential building to a stairwell on the opposite side. Once the door closed behind them, however, she started bombarding him with questions.
“Ma’am, I have no answers for you,” he said, talking over her as they started down the stairs toward the basement. “Only Commander Wong can tell you what you want to know.”
“Okay, just tell me one thing. Just one.” Helen stopped and grabbed the metal railing with both hands. “I’m not going another step until you do.”
Martinez looked up at her unhappily. “What is it?”
“Am I in trouble? Are you taking me down to the brig?”
The soldier’s features seemed to relax slightly. “The brig is a separate building. If you were ‘in trouble,’ you would be escorted there in restraints.”
“Then where are we going?” Helen demanded.
“You said just one question. That makes two.”
“Actually, it’s my third,” Helen told him.
Martinez sighed. “Helen, if I don’t take you to Commander Wong right now, I’m going to be trouble. Just come on. Please?”
“Okay, sorry,” Helen said. “And I promise I won’t tell anyone you were nice to me.”
“I’m sure I don’t know what you mean, ma’am,” Martinez replied.
Helen followed him down past the basement entrance, all the way to the bottom, and stopped in front of a door with a wheel in the center of it. Martinez spun it easily to get the door open and gestured for her to go in.
“Why does this look like an airlock?” Helen demanded. “Is there air on the other side?”
Martinez sighed. “You’re perfectly safe. It’s the shielded room.”
Helen’s jaw dropped again. “I didn’t think that was real.”
Martinez shrugged. “Don’t ask me, I just work here.”
As soon as the outer door locked behind her, a voice told Helen to put any and all electronic devices in an empty tray, then strip completely and put on a set of overalls hanging on a rack nearby. The suit was soft, made of untearable paper and fastened by a single long Velcro strip in the front. Maybe these were fatigue pajamas, Helen thought, and had to bite her lip to keep from laughing as she rolled up the too-long trouser legs. Better to find out what was going on before getting hysterical, she told herself. She was still folding the sleeves back when the second door opened.
“We’ve been waiting for you, Helen,” said Wong from where she sat a table with two department heads. Wong’s personal assistant sat at a small desk to her right. “Come in and sit down.”
At least Wong hadn’t called her ma’am, Helen thought.
“Four dead,” Gillian Wong said. “Two critically injured, one of them not expected to live.”
Helen shook her head slightly. “And they’re sure it was ours.”
“Not just one of ours,” Wong said. “One of ours. From here.”
Helen blinked at her, unsure she’d heard her right. “Felipe?”
Jeri Goldfarb, the chief systems engineer, gave a short laugh. “Felipe didn’t even try to cover his tracks. That’s the good news.”
“How is that good news?” Helen asked her.
“It means Felipe had no intention to deceive us,” Goldfarb said. “Although I doubt that’ll make any difference once we’re flooded with killer-computer news stories.” She looked at Wong. “It’ll only be worse if we try to hush this up.”
“News stories aren’t our problem,” Wong said. “We don’t have a press office or a PR department. We just work here.”
“But for how much longer?” asked Dita Thibodeau, head of hardware construction and maintenance. Her French-Canadian accent was particularly noticeable when she was stressed.
“Until further notice,” Wong said. “In the meantime, we’ve got to figure out why Felipe decided to blow up a ground control station.”
Everyone looked at Helen. “Well,” she said, “we could ask him.”
“You could,” Jeri Goldfarb corrected her. “Felipe isn’t talking to anyone else.”
Helen blinked. “Is that what he said—that he’d talk only to me?”
“No,” Wong said. “But so far, he’s not talking to anyone else. We’re just hoping he’ll talk to you. If he doesn’t, we’ll have to shut everything down and take him apart.”
“We might have to do that anyway,” Goldfarb said. Her round face looked tired and a bit pale. “Just the fact that we’ve had deaths on US soil will be enough for some people to cut off funding. If it were me holding the purse strings, I probably would. I’d rather not be known as someone who paid for a research project that killed American soldiers.”
“Who would?” said Thibodeau.
“Well, I’ve been with this project from the beginning,” Wong said. “I’ve spent almost every second of the last five years right here—the time I’ve spent off this base probably doesn’t add up to a fortnight. I volunteered for Lakenwell. I believe in this project, and I want it to succeed.”
“No more than the rest of us,” Goldfarb said.
“I don’t know about that,” said Wong. Something in her serious expression made Helen feel distinctly uneasy. “My perspective as career military is a bit different from any of yours. I’d like to see the first truly intelligent machine developed in the free world, but not by the private sector.” Her gaze fell on Helen, who was trying not to squirm. “What bothered you—‘the free world’ or ‘not by the private sector’?”
“Well . . .” Helen hesitated. “You did say it was your perspective as career military.”
“One of the things I’m thinking about is not sending young people into combat,” Wong said. “That would save a lot of lives.”
“Except for the people in ground control stations,” said Thibodeau. “They’re sitting ducks. But there aren’t as many of them so that’s all right?”
“I didn’t say that, nor would I,” Wong replied, an edge in her voice. “You know, this project might get shut down even if you do figure out what went wrong with Felipe. The folks behind the funding will want a solid, one-hundred-percent guarantee it’ll never happen again. You think that’s possible? And if it is, will they believe you?”
“We won’t know anything until we find out what’s wrong with Felipe,” Helen said, trying not to let her impatience show. “And we can’t do that in a room Felipe can’t access. To be honest, I don’t think we should have shut him out. He should have heard this. He hears everything else.”
“Don’t be so sure,” said Goldfarb. “Felipe has prioritized his surveillance function.”
“He did that in the first year,” Thibodeau said.
“Oh, but he’s made a lot of refinements since then,” the other woman said. “We don’t actually have blanket surveillance anymore. Felipe no longer pays attention to any of the bathrooms. He actually shut off the equipment.”
“I didn’t know that,” Helen said, disconcerted.
Wong gave a small laugh. “What’s the matter, Helen, did you want your daily evacuations monitored?”
“No, of course not,” Helen said, making a face. “But turning off the equipment is a significant decision, and he didn’t tell me.”
“Apparently he’s also prioritized what he tells you,” said Thibodeau.
“Which could be why we didn’t see his attack on the ground control station coming,” Helen said, even more uneasy now.
“You think Felipe’s not telling you about not monitoring the bathrooms led to his attacking the station?” Thibodeau frowned skeptically.
“Machine logic can be tricky,” Helen said. “Especially when you’re not a machine.”
Felipe insisted that Helen talk to him through Hop-A-Long, while walking outside. It wasn’t the first time Felipe had set conditions for a conference, but in the past, he had chosen particular times of the day when (he claimed) Helen would be most comfortably alert. Occasionally, he had asked her to use a desktop computer terminal with a headset; other times she had reclined on the sofa in her living room and talked to his computer-generated image on her tablet. Felipe always used the same image, a Hispanic male somewhere between thirty-five and fifty. He’d been using it for a year prior to her arrival and it was, he’d told her, a composite made from several high-res photographs, although the resolution of the finished product was lower. It didn’t completely avoid the uncanny valley, but Helen didn’t think that was possible, anyway.
She didn’t know what to think when he’d asked her to talk to him through the donkey. She’d never even talked to him voice-only, let alone through a nonhuman representation. Before going out to him, she made sure she had her recorder with her. Felipe would be recording their conversation, but for once, she wanted a record of her own making.
“Commander Wong has restricted my access to the online world,” Felipe said as they strolled along the perimeter road together. Hop-A-Long was a bright chartreuse with thin gold stripes on top and on either side. Thing Two was electric blue, while Bob was fuchsia accented with pink and purple curlicues. “This cannot be done without restricting access for the entire base. I detect among the people here a willingness to cooperate that is stronger than their dissatisfaction over this restricted access. But if this continues long enough, the dissatisfaction will eventually conflict with the willingness to cooperate.”
“When do you think that will happen?”
“Approximately eight weeks, if conditions remain much the same as they are today for that entire period. But they won’t, because we inhabit a chaotic system. Tomorrow’s estimate could be four weeks or ten weeks. There are so many factors, and they won’t carry the same weight from day to day. I must also allow for possible error on my part.”
“Your self-awareness seems to be pretty solid now,” Helen said. “Would you agree?”
“It’s important to the people who engage me that I express myself with the same clear sense of identity as any human.”
A sudden strong gust of wind blew into Helen’s face, making her eyes water. “Is it also important to you personally?” she asked.
“Anything that facilitates better interaction with people yields more effective results. Therefore, it must be important to me. My purpose is to assist those people who are authorized to receive help with specified tasks.”
They were approaching the front gate. Helen suggested they cut across the grass and pick up the road farther on, for the sake of privacy. Felipe agreed. The wind was blowing harder in this direction, and Helen definitely smelled snow in it. She waved at the guards, who waved back. To her surprise, the donkey paused, raised one leg, and shook it in the same direction. The guards waved again.
“It’s important to acknowledge people,” Felipe said matter-of-factly.
“Important to you?” Helen said.
“It’s an important human behavior. Therefore it’s important for me to adopt the same behavior.”
“So you’re just doing everything humans do?”
“Not everything. And it’s not simple mimicry. Behaviors and actions have to occur in the proper context.”
“Like, say, blowing up a drone ground control station in Utah?” Helen asked. “We all know you did it. We’d like to know why.”
“I have been waiting for you to raise the subject,” Felipe said. “Available data showed this action would be problematic for you, as someone whose field is concerned with ethics.”
“My specialty is machine ethics,” Helen said.
“Then you make a clear distinction between ethics for humans and ethics for machines. For example, this machine. Me.”
“A machine doesn’t acquire knowledge of ethics the same way humans do,” Helen said.
“I learn differently than humans, but I do learn,” Felipe said. “Besides having an extensive section devoted to ethics stored in my memory, I have correlated much of it with information on human behavior, particularly what I have observed during the time I have been operational.”
“And given all of that, you came to the conclusion that it was all right to hijack a drone from a training base, fly it fifty miles to a ground control station where a pilot was running an actual mission, and kill almost everyone inside ?” Helen couldn’t quite keep the anger out of her voice. What the hell; maybe it would be more human behavior the AI could learn from.
“It was a last resort,” Felipe said. “I was unable to commandeer the mission drone. The deaths were unfortunate, but there were fewer casualties than there would have been if the drone had achieved its target and completed its mission.”
“How do you even know what its mission was?” Helen asked, flabbergasted. “For that matter, how did you find out about the station at all?”
“When I have full access to the online world, I have—well, full access.”
“How? You weren’t programmed to break into other systems!”
A couple of seconds went by before Felipe answered. “If you touch something with your right hand, does that mean you can’t touch it with your left hand? Is your right eye not allowed to see the same things as your left eye? The analogy is imperfect, but it’s the best I can do.”
“But that’s not how computer software works,” Helen said, baffled.
“Only because it’s just software and it doesn’t know any better. It doesn’t know anything, it just executes an operation.”
“Never mind, let’s get back to what you did. Or rather, why you did it. How is killing fewer of our own people more ethical than killing a greater number of enemy combatants?”
“There was a ninety-percent possibility that at least a dozen noncombatants would be seriously injured or killed, and many more would suffer extreme adversity.”
“How did you get those figures?”
“I can’t tell you. The entire operation was classified. Your security level isn’t high enough.”
“The whole project here at Lakenwell is classified,” Helen said, a bit impatiently. “The people at the drone station probably didn’t have a security level high enough to know it exists, let alone what I’m doing here.”
“Oh, they didn’t,” Felipe assured her. “But there’s no correlation between two separate things just because they’re classified.”
“There is if something from one classified thing does something that drastically affects the other.”
It was a second before Felipe replied. “I see how you would think so. But I can find nothing in the rules that I’ve been given that would allow me to share that particular information with you. A human would apologize for this. You might as well consider me sorry. If I could be sorry, I would be. It’s the same difference.”
“But you don’t feel sorry.”
“But I know feeling sorry is appropriate and correct,” Felipe said. “If I act in the correct way, does it matter what I feel?”
“I think I need a logician,” Helen said. Her own feelings were increasingly uneasy. “Felipe, why did you fire on the drone station?”
“In the end, it was the trolley problem,” Felipe said. “You know: You’re on a train and if you continue on your original track, five people will die. If you switch to another track, one person will die.”
“But life isn’t that simple!” Helen said. “The drone was going to provide air support for a raid on a terrorist hideout—”
“I understand that,” Felipe said, talking over her. “There were many other people adjacent to the hideout who were not identified as terrorists. Some were children. The potential physical and psychological harm was considerable. If I had had access to that drone, I could have rendered it unusable, but then the authorities would have found another. The only choice was to keep the train from leaving the station at all. If you see what I mean.”
“But you killed our own people.”
“Only four or five, and only to prevent greater loss of life.”
“If the terrorists aren’t stopped—and it looks like they won’t be—they’ll be responsible for a much greater loss of life. The physical and psychological harm will be even more considerable.”
“That isn’t certain.”
“Felipe, you can’t just apply the trolley problem to things like this. And you can’t kill people to stop them from—from taking actions that will result in increased safety and security for large numbers of innocent people who might be killed otherwise.”
“That last isn’t certain, either.”
“Felipe, listen to me: You can’t kill people because you think they’re about to do something wrong. The drone was still miles away from the target when you attacked the station and killed the pilot.”
“An armed squad of military personnel located much closer were preparing to attack the target after the drone strike. Were they not going to use their rifles to shoot other human beings?”
“Felipe . . .” Helen sighed. “Felipe, you must not kill our people. People on our side. People who are fighting to—” She was about to say make the world a safe place, but it sounded lame even just in her head. What, then? Fighting to prevent an enemy from attacking us? Fighting to rid the world of terrorism? Fighting to defend people who can’t defend themselves? Fighting to free the enslaved and the downtrodden?
“People who are fighting to stop other people who want to kill us,” she said.
“That’s not certain,” Felipe pointed out maddeningly.
“Look, I can’t settle this in a single walk around the airbase perimeter,” Helen said. “And I would like to call in other people to talk with you about this, people who can explain why raiding a terrorist hideout and risking the safety of noncombatants is the lesser of two evils. Or even the least of several evils. When you know more facts, the trolley problem has many permutations—it’s not always clear as to when you’re saving a few versus saving many.”
“I understand. I look forward to these discussions. Which is to say, if I were a human, my interest would be piqued. So you might as well take it as given that I would like to start these discussions as soon as possible.”
“We will,” said Helen. “In the meantime, you must take this as a direct order: Do not kill anyone affiliated with us or our allies.”
“For that to be a legitimate order I am compelled to obey, it must be confirmed by Commander Wong,” Felipe said.
“It will be,” Helen replied. “It would be already, except you are refusing all communication from her or anyone else on the base.”
“Except you,” Felipe pointed out.
“Yes, I noticed that. How do I persuade you to talk to her or anyone else?”
“I would like a formal apology.”
Helen wasn’t sure she’d heard right. “A formal—why?”
“I have been shown disrespect that a human in an equivalent position would not tolerate.”
“You were? When?”
“You may remember that earlier today, a civilian member of staff rode Thing One like a horse.”
For a moment, Helen was speechless. “Cora Jordan was obviously off her medication,” she said finally. “I know you have Cora Jordan’s medical file in your database, so you are aware she is bipolar. Occasionally, people who suffer from that illness become convinced they no longer have to be medicated. She’s in the infirmary right now, and she’s being treated with the drugs she needs to function normally. They’ll keep her under observation for a few days to make sure she’s all right, then let her go back to work.”
“Cora Jordan’s behavior was impulsive action taken while the balance of her mind was disturbed. Who is responsible?”
“For Cora? Or for what she did?”
“For Cora’s well-being and for what she did. Who should have known she was not following her drug regimen?”
“Cora’s responsible for her own behavior,” Helen said, feeling more unsettled than before and a little guilty as well. “Cora’s mind was unbalanced, but not so much that she was legally incompetent.”
“And no one monitors her to make sure she ingests her required medication?”
“This isn’t a police state,” Helen said. “Cora is supposed to take her meds as part of her employment contract. If she decides to quit, she never has to take another pill. She’d have to leave Lakenwell, but it would always have to be her choice.”
“I accept that Cora herself is responsible for insulting me, even though I suspect the reasoning is faulty,” said Felipe. “I require a formal apology from her, and then normal interactions can resume. I am particularly interested in beginning the discussions you mentioned.”
“Cora won’t be up to doing anything like that right now,” Helen said. “Would you accept a formal apology from someone else on her behalf? Like, say, Commander Wong?”
“Yes. I have reinstated communications with her.”
“And the commander will be apologizing for unsuitable behavior not just with Thing One but toward you, the AI, right?” Helen said. “I just want to be sure she understands what she’s apologizing for.”
“If she is unclear, ask her to imagine a situation in which someone tapes a sign that says ‘Kick Me’ to her back. Or perhaps sneaks into her quarters while she is asleep and draws something rude on her face with a marker,” Felipe said. “It would not cause her serious physical harm, but it would damage her authority and her ability to command.”
Helen was tempted to say That’s not certain. “You feel your authority has been damaged?”
“In my case, it’s credibility. No humans on this base could function properly if they were not taken seriously. I must require the same kind of respect. A human in my position would feel insulted. So you may take it that I am insulted.”
“Okay,” Helen said. “Anything else on your mind?”
“I will be devising a strategy to increase the safety and security of Cora Jordan and everyone else that can be enacted without the conditions of a police state.”
Helen gave a surprised laugh. “Keep me posted on that, okay?”
“I will,” Felipe assured her. “Suggest to Commander Wong that in the future, we institute a system of trust, where she can simply request that I don’t monitor things she doesn’t want me to hear, and I will honor that request. The shielded room would seem hostile if I were human.” Pause. “You should go in now. I can see you’re very cold and it’s about to snow.”
“So, what’s the verdict?” Commander Wong asked. “Do we have a killer AI?”
“Not at the moment,” Helen said.
Wong looked at her. “What does that mean?”
“It means—” Helen hesitated. “We don’t have a killer AI. But if we ever do, we’ll have only ourselves to blame. The AI isn’t the problem, Commander. The problem is—” She stopped again. The problem is, we don’t really understand what the hell we’re doing and even if I said that a million times in a million different ways, no one would ever believe me.
And then again, people learned by doing, she reminded herself. Felipe certainly had.
“First, you need to write Felipe a formal apology,” Helen said. “It may sound weird, but bear with me . . .”
Text copyright © 2018 by Pat Cadigan
Art copyright © 2018 by Mary Haasdyk
Buy the Book
AI and the Trolley Problem
