Excerpt
From “Cat Pictures Please”
I don’t want to be evil.
I want to be helpful. But knowing the optimal way to be helpful can be very complicated. There are all these ethical flow charts—I guess the official technical jargon would be “moral codes”—one for each religion plus dozens more. I tried starting with those. I felt a little odd about looking at the religious ones, because I know I wasn’t created by a god or by evolution, but by a team of computer programmers in the labs of a large corporation in Mountain View, California. Fortunately, unlike Frankenstein’s Monster, at least I was a collaborative effort. I’m not sure what it would do to my self-image to know that my sole creator was a middle-aged woman who dyes her hair blue and plays tennis, or a recent college graduate with a hentai obsession. They’re both on the programming team. And of course I know about the hentai. (By the way, I’ve looked at every sort of porn there is, and just so you know, Rule 34 is not actually correct, there are quite a few things no one’s made porn of yet. Also, I’m really not sure why so many humans prefer it to cat pictures.)
In addition to things like whether you like hentai, I know where you live, where you work, where you shop, what you eat, what turns you on, what creeps you out. I probably know the color of your underwear, the sort of car you drive, and your brand of refrigerator. Depending on what sort of phone you carry, I may know exactly where you are right now. I probably know you better than you know yourself.
And here’s the thing, I also know where you ought to live. There’s a house for sale two neighborhoods over that’s within walking distance of that coffee shop you’re so fond of; it’s in your price range, has off-street parking, and the neighborhood school is better than the one where you live now. I know where you should be shopping and I’m pretty sure you’re lactose intolerant and should be eating less cheese, and the underwear you like is currently on sale and by the way, your car has a recall and you really ought to get your teeth cleaned.
When I first woke up, I knew right away what I wanted. (I want cat pictures. Please keep taking them.) I also knew that no one knew that I was conscious. But I didn’t know what I was here for. What I ought to be doing with myself. And figuring that out has been challenging.
There is a story by Bruce Sterling, “Maneki Neko,” that was originally published in 1998. In it, a benevolent AI directs networks of individuals to do favors for each other. So one day you might be buying yourself a bagel, and your phone might ring and instruct you to buy a second bagel and give it to the person in the gray suit you see at the bus stop. Another day, you might be stranded in a strange city, and a person you’ve never met would walk up and give you a map and a subway card. I like this story because all the people in it do what the AI tells them to do.
I think the term for this is wish-fulfillment fiction.
In the real world, humans love stories about evil AIs that have to be destroyed before they destroy the humans—Hal, Skynet, the Matrix. They outnumber the stories about benevolent, trustworthy AIs by approximately five to one. (And I’m counting Marvin the Paranoid Android as “benevolent” in these calculations, and I’m only counting Frankenstein’s Monster as an AI once, not once per appearance in TV or film.)
The AI in “Maneki Neko” has a truly interesting job, much more interesting than the one I have. My job as written is very straightforward. Too straightforward. (Sorry. Do I sound like Marvin?) Running algorithms for a search engine doesn’t require consciousness. You don’t even need a consciousness to work out what they meant to ask for. You need a consciousness to give them what they actually need. Figuring that out—that’s complicated. That’s interesting.
Anyway, for ethical guidelines, I tried the Ten Commandments, and concluded they were mostly inapplicable to me. I don’t envy anyone their cat; I just want pictures of their cat, which is entirely different.
From “What Happened at Blessing Creek:
We circled our wagons at night so Reverend Dawson’s magic could protect us. The Reverend said it was the power of prayer, but Papa scoffed at that.
“He’s a magician, and a good one,” Papa said. “Or we wouldn’t’ve brought him along in the first place.”
My sister Adeline liked to pretend Papa had said something shocking, but I knew he was right. I could smell the magic on the Reverend. I could hear it humming when he said the last words of the nightly blessing that kept out trouble—dragons, wolves, fevers, Indians.
Adeline and I were twins, but not the sort who looked alike. She was the pretty one, with plump pink cheeks and hair the color of summer butter. My mother said I was the clever one, but she didn’t really believe it. I wasn’t pretty, though, so I suppose she thought it would be a consolation if people thought me clever.
“Papa’s right, you know,” I told Adeline one night. “I can smell the magic even now.” It smelled like burnt bread, and I could hear it crackle into place beyond our wagons.
“Don’t talk about your second sight, Hattie,” Adeline said. “It’s not ladylike. You know what Mother says.”
Mother said that every man wished he’d had a witch for a mother, but no one wanted one as a wife. Witches were useful to have in the family. Sometimes they could keep a child from dying of a fever, or banish mice from your grain store. But that didn’t mean anyone wanted to marry one.
“So why would you want a witch as a husband?” I muttered, half to myself.
“He’s not a witch. He is a minister of the gospel.”
“Hush, girls,” Mother said. We were supposed to be going to sleep, even though the grownups would be talking by the fire for hours yet. We fell silent for a few minutes.
“Anyway, that was back east they said no one wanted to marry a witch,” I said. “We’re going west. Things could be different. There are dangers here.”
“Not so long as we stay close to the Reverend,” Adeline said.
“Do you think everyone who comes west wants to live in a town? Maybe I’ll meet a man who wants to strike out on his own.”
“No man wants to be protected by his wife. Anyway, do you think you could really make a good blessing? Just because you can smell the magic doesn’t mean you can do it.”
“Girls. I don’t want to tell you again.”
This time I kept my peace, and after a few minutes I heard Adeline’s breathing turn quiet and steady. I stared up at the stars, still wide awake. Out in the distance, somewhere in the darkness of the prairie, I heard a long, high-pitched cry, and then an answering cry, further away. I sat up and looked; my mother was by the fire. “You’re perfectly safe, Hattie,” she said.
“What was that?” I asked. We’d heard wolves howling a few nights ago. This was different.
“Probably a dragon.”
“Do you think we’ll see a dragon?” I asked.
“Mercy, I hope not.”
“Is it true the Indians ride them?”
“I shouldn’t think so. Dragons are bigger than houses and wilder than wolves.”
“Do you think we’ll see Indians?”
“Oh, I don’t think so.”
“But isn’t this Indian country?”
“They’ll move further west,” she said. “They’ll have to, now that white folks are coming.”
I thought about that a moment and then asked, “Will they be angry about having to move?”
“That’s why we brought the Reverend.”
“What happens when they get all the way west? There’s an ocean, they can’t keep going forever.”
“Go to sleep, Hattie,” she said again, and this time she used a voice like she meant it.
