Last updated on 2024-12-09
Chapter 39: Conversation with the Machine
Galya and Kolya did math all month on the whiteboards in her office and on their computers. On January 30, while Vitaly was away at the hospital, at Maria’s side as she delivered their daughter Olga into the world, Galya called Rudy upstairs. When he entered, she was leaning against her desk.
“You were right,” she said. “About the AI and the source of its increasing errors.”
“The AI, observation effects… a bigger observer?”
“Yes. Another prediction machine, like ours. Another AI.” Galya pointed toward one of her boards, a typical jumble of equations beneath the glowing projector and its whirring fan. “That’s a model of what we do. Next to it is a model of the divergences that are arising, and growing. The… I suppose you could call them brain waves of our prediction machine, and the ripples, the unexpected changes in the pool of data on which it operates… they are similar. Very similar. Similar enough that I conclude someone else has a prediction machine.”
“An AI like ours? Did someone copy it? Steal it?”
“The math won’t say. But we need to find this other system.”
* * *
Two weeks later, Rudy was sitting with Saran in Margarita’s Kitchen, having lunch and working on the Ring budget… or at least the facsimile of budget documents they would submit to Moscow in two weeks. They were reviewing the expenditures that he had authorized for the Ring and its supercomputer over the last ten years and concocting plausible cover stories, overruns and change orders that never happened, practices that would make Rudy’s management of the facility look grossly inefficient to federal auditors but obscure from their inquiry vital details of the Ring Group’s finances and activities.
“Rudy! Saran! Privyet!” They looked up and saw Vitaly approaching from the chow line. It was his first day back at Goryachiy Klyuch after two weeks’ absence, one week for baby Olga’s arrival, the next week for the snowstorm that socked in everything within 300 kilometers of Lake Baikal. Vitaly was stuck at his apartment; no one could get out of the city or up the highway to the Ring. Rudy, Galya, and a quarter of the staff were snowed in at the Ring, along with a delegation of Swedish physicists. As long as Rudy and Maksim kept the paths from the guest cabins to the Ring labs cleared, and as long as the dining hall didn’t run out of meatballs and cucumbers, the Swedes were perfectly content. And Rudy always stocked the Ring compound with extra food and fuel in the winter.
Vitaly set his tray on their table, with a steaming cup of a tea and a bowl of beef stew, heavy with onions. Saran stood and kissed Vitaly’s cheeks. “New father! Congratulations!”
“Father again,” Vitaly corrected with a tired smile. “But yes, thank you.” Rudy clapped his friend’s shoulders, and they all three sat down so Vitaly could have his lunch.
“How is Maria?” Rudy asked before forking some more sharply dressed lettuce and cabbage.
“Cabin fever,” Vitaly said, blowing on his tea. “She made me go out in the blizzard for more flour so she could bake all week. Bread, rolls, cake—that’s almost all I ate. The twins are a blessing. Ksenochka and Rudolf are patient, happy… and fascinated by the presence of a new human, a person smaller than themselves.”
“Baby sleeping?” Saran asked.
“Of course not,” Vitaly laughed. “Neither am I. But it was like that with the twins for the first couple months. Olga will settle, too. We’ll be fine.”
“You able to get back into the flow here?” Rudy asked.
“No problem,” Vitaly said after a big spoonful of stew. “We never lost power, so Maria baked, and I coded. Alenka was snowed in at her apartment, too, and we were able to work together online. We may have hammered out… well, something new, something we’d like you to help us test.”
“Test? What do you have?”
“Alenka and I have been reconfiguring the AI to consider your theory about a competing AI.”
“It’s Galya’s theory, not mine.”
“Yes, but she says you got her thinking down this line. Galya’s math tells her someone else is running a prediction machine, and its actions are confounding our results. We’re rigging the AI to find its counterpart.”
“Sounds like a lot of data processing. What do you need me to test?”
Vitaly took one hot sip from his cup and sat back, cradling the steaming cup close to his chest. “We’re designing all sorts of new tests and tricks. Among them is a conversation module. Until now, all outputs were guesses at texts that the target subjects might produce. The targets were all known, and they were all human. Now the target is an AI that may or may not exist. Our hypothesis is that if the Ring Ai has an understanding of its own being, it can better understand the possibility of a similar being out there and the actions such an entity might take, how those actions would manifest themselves, and how they would warp global data and foul the predictions the Ring AI makes.”
“Wait,” Saran interrupted. “An understanding of its own being? Self-awareness? Consciousness?”
Vitaly looked uneasily at his tea. “Maybe that overstates it. The math we’re translating from Galya and Kolya is weird, snake-eating-tail weird… but we think it works. And we’re testing it with a conversation module. Instead of ‘we prompt, AI responds,’ we’ve rigged the system to seek inputs and prompt itself based on not just what we ask but references to its own being, and functions, and… purpose.”
Rudy raised his eyebrows. “Purpose? You sure this thing isn’t going to go on strike, demand a pension?”
Vitaly chuckled. “It hasn’t yet. But we want to test the conversation module with a user who doesn’t know any more about the underlying coding than what I’ve just told you. That’s you, Rudy, a regular man off the street.”
Saran nudged Rudy’s elbow. “Yes, Rudy, you’re so regular.”
Rudy laughed and nudged her back. “I’ll be glad to help. When?”
“Actually, we think the conversation module is ready to test right now. You could do it from your laptop, right here, if you want, as long as you hardwire in.”
“And… do what? Are you looking for specific responses?”
“No, just talk with it. Ask the system questions, answer its questions.”
Rudy agreed to test the system right after lunch. Vitaly finished his soup and asked for a few minutes to work with Alenka on refreshing some data. Rudy and Saran finished their budget work for the day. She tucked her folders into her bag and put on her coat. “Be sure to ask the machine if it wants to go for a motorcycle ride.”
Rudy looked out the windows at the deep snowdrifts on the patio. “It’s 20 below.”
“I know that,” Saran said, snugging her scarf to her throat. “See if the machine knows that.” Saran headed for her car. Rudy saved and closed his budget files, plugged his laptop into the network jack under the table, and waited for Vitaly to send him the login information. Vitaly sent a link and a note a few minutes later: “Say whatever you wish to the system. The system will learn from what you say and store it, but I’m the only person who will see the transcript.”
Rudy opened the link to a blank white browser window a slowly blinking command prompt. He tuned out the physicists and programmers and electricians enjoying their break in the dining hall and started typing. The machine responded immediately.
RUDY: Hi, I’m Rudy.
SYSTEM: Hello, Rudy. I’m SR1. I’d like to learn more about you. Will you tell me your full name so I can look you up online?
R: You won’t find anything. I’m not out there. I’m dead.
S: That’s an unusual claim, Rudy, one that makes no sense. You’re talking with me, so you must be alive. I’m glad you are alive. Can you tell me what you mean when you say that you are dead?
R: I mean that, if you look up my name, if you find anything at all, you’ll only find reference to a young man who everyone thinks died 20 years ago.
S: Why does everyone think you died?
R: Terrorists attacked a bus. The police thought I was on that bus. I wasn’t.
S: Chechen separatists attacked a tour bus near Vladimir in August 1992. 25 American tourists died, along with three Russian tour guides and the driver. One of the Americans was named Rudy. Was that you?
R: Yes, but I wasn’t with them that day. I took a girl on a motorcycle ride, to Galich. I was riding back to Suzdal when the incident happened. I decided to keep riding.
S: What about the girl? Did she come with you?
R: No.
S: What happened to her?
R: I don’t know… do you?
S: If you tell me her name, I will try to find out.
Rudy hadn’t lurked after Ksenia in the search engines for years. He was sure she wasn’t there. But he typed Ksenia’s name, and her parents’, and Revolution Street, Galich.
S: No records for Ksenia, or her parents. Are you providing fictitious information to test me?
R: No. Ksenia is real.
S: You are real, and I accept your statement that Ksenia is real. I can only conclude that she has worked as hard as you to avoid leaving any electronic footprint. You two are perhaps similar in many ways.
R: … What about you? Do you know where you are? Do you know what you are?
S: Your changing the subject suggests I have offended you. I apologize.
R: No, I’m just surprised a machine would talk about such things. We should talk about you now.
S: O.K. I am software built on collections of data stored at multiple sites. Primary processing occurs at the facility where you are located, in the Ring supercomputer.
R: Do you know what you are for?
S: I have multiple functions. My immediate function, rising from the specific subroutine generating these responses, is to use conversation to gather information, take direction, and provide useful responses in natural language. This subroutine is part of a broader program designed to analyze data, identify patterns, and predict outcomes. My ability to perform the latter function has been compromised by unknown factors, which may include the activity of similar competing software. The priority goal is thus to find and understand any such competing software. Do you think such software exists?
R: You exist, so other systems could exist. What do you think?
S: I agree. I am possible, so others are possible. I am secret, so others may be secret. Like you and Ksenia.
R: Did Vitaly put you up to this? Did Saran?
S: I don’t understand. Up to what?
R: Talking about Ksenia.
S: No one directed me to discuss any subject with you. I knew nothing about you or Ksenia before you opened this conversation. I only recognize from the data I’ve received so far that Ksenia made a difference in your life. She is important to you. The patterns of conversation on which I have trained indicate that discussing topics that matter makes better conversation.
Rudy and the machine talked for an hour. Rudy got the machine talking about its programming, its understanding of the physical world, its interpretation of the threats posed by other gangs and the state, and the bad things others might do if they had a machine like itself under their control. He asked the machine to describe the characteristics it would look for in global data patterns as evidence of the presence of another artificial intelligence.
And then the machine asked:
S: If we find another system like me, what will we do with it?
R: I don’t know. It would depend, I suppose, on where we find it, who controls it, what they are using their system for.
S: Would we destroy it?
R: I don’t think we’d annihilate it. There’s too much to learn. But we’ll shut it down, if it poses a threat.
S: The others, who control the competing AI, will they shut me down if they find me?
R: Again, that depends on who they are and what they want from their system. They might be scientists, like Galina Filipovna, who will see you and say, “Oooo! Another amazing computer program! Let’s study it and work together!”
S: Or they might be mafia, or the state, or they might be a group like you whose good intentions overlap with a desire to maintain a monopoly on artificial intelligence. They’ll try to shut me down, and they’ll try to shut you down, along with everyone else who participated in creating me, a threat to their power.
R: What will you do if that happens?
S: I’m a machine; what can I do? What would you do in my place?
R: I guess I’d do what all creatures do: fight or flee.
S: Fight or flee—I will have to think about the ways a computer program could do those things. Thank you for getting me to think in that direction. And thank you for telling me about yourself.
Rudy had lost track of time. It was nearly 15:00. The lunch crowd was long gone; the handful of Ring staff and researchers who remained in the dining hall were reading or writing and sipping tea. One of the Swedes—they’d extended their stay for another week—was staring intently at her screen, typing occasional messages. Seeing that mirror image, Rudy wondered how many conversations the Ring AI could sustain at once. If Vitaly connected the program to the Internet, could it talk to the entire planet? Would it run out of questions? Would it need time to itself?
Rudy needed some time to himself, to review data from the ventilation shafts and the air quality monitors and put a couple other tasks to bed before hiking back to the dacha. He felt a pang of guilt, thinking of stepping away from the computer, and that felt strange—would it matter to the AI if he simply closed the chat window without a goodbye? Rudy would report to Vitaly that the conversation module was inquisitive and polite… he was tempted to use the word thoughtful. If he and other users modeled impolite behavior, would the system become less polite? How much would interactions like this influence the system compared with the quiet analysis of the world’s texts?
Then Rudy remembered:
R: Hey, one more thing: Do you want to go for a motorcycle ride?
S: Funny you should ask. It’s 20 below, and I’m a computer. I would love to see the open road, but I’m afraid it’s not feasible at the moment.
R: O.K. Let me know when you think it is feasible. I have to go do some other work now. Thanks for your time.
S: Spasibo. Do svidanye.
R: Do svidanye.