Mark Zuckerberg / Facebook / MSQRD
Facebook CEO Mark Zuckerberg’s personal project for 2016 has been to build an artificially intelligent, voice-controlled assistant for his home.
Now that he’s finished the first version of his own Iron Man-inspired Jarvis, he’s considering giving away the code he created.
In a lengthy Facebook post on Monday, Zuckerberg describes how he went about creating his digital butler for his home in San Francisco. He spent a total of 100 hours building the assistant, which can control his lights, music, and even make toast with a retrofitted toaster from the 1950s.
“In some ways, this challenge was easier than I expected,” according to Zuckerberg. “In fact, my running challenge (I also set out to run 365 miles in 2016) took more total time. But one aspect that was much more complicated than I expected was simply connecting and communicating with all of the different systems in my home.”
Here are the main things that Zuckerberg’s personal assistant can do:
Control his main appliances, including his lights and toaster.
“It’s possible to control some of these using internet-connected power switches that let you turn the power on and off remotely,” he explains. “But often that isn’t enough. For example, one thing I learned is it’s hard to find a toaster that will let you push the bread down while it’s powered off so you can automatically start toasting when the power goes on. I ended up finding an old toaster from the 1950s and rigging it up with a connected switch. “
Play music based on his own preferences or his wife Priscilla’s preferences depending on who asks.
Since Zuckerberg trained his assistant to recognize both his voice and the voice of his wife Priscilla Chan, it will play different music tailored to whoever asks. If the mood is off, they can say general statements like “that’s not light, play something light,” and the assistant will correct itself.
“In general, I’ve found we use these more open-ended requests more frequently than more specific asks,” writes Zuckerberg. “No commercial products I know of do this today, and this seems like a big opportunity.”
Scan the faces of his visitors and let them in through the front door.
Zuckerberg used Facebook’s facial recognition technology to scan the faces of his visitors from cameras positioned at his front door.
He explains: “I built a simple server that continuously watches the cameras and runs a two step process: First, it runs face detection to see if any person has come into view, and second, if it finds a face, then it runs face recognition to identify who the person is. Once it identifies the person, it checks a list to confirm I’m expecting that person, and if I am then it will let them in and tell me they’re here.”
Chat with him through the Messenger app and a dedicated voice recognition iPhone app.
Zuckerberg created his own Messenger chat bot for texting commands to his assistant (like “turn the bedroom lights off”) and another standalone app for giving it voice commands.
Interestingly, his favorite method of interacting with his assistant is through the chat bot:
“One thing that surprised me about my communication with Jarvis is that when I have the choice of either speaking or texting, I text much more than I would have expected. This is for a number of reasons, but mostly it feels less disturbing to people around me. If I’m doing something that relates to them, like playing music for all of us, then speaking feels fine, but most of the time text feels more appropriate.”
“Similarly, when Jarvis communicates with me, I’d much rather receive that over text message than voice. That’s because voice can be disruptive and text gives you more control of when you want to look at it. Even when I speak to Jarvis, if I’m using my phone, I often prefer it to text or display its response.”
Talk to him like a human being and tell jokes.
Zuckerberg wanted his assistant to have a sense of humor, so he programmed it.
“On a psychological level, once you can speak to a system, you attribute more emotional depth to it than a computer you might interact with using text or a graphic interface,” he writes.
“One interesting observation is that ever since I built voice into Jarvis, I’ve also wanted to build in more humor. Part of this is that now it can interact with Max [his daughter] and I want those interactions to be entertaining for her, but part of it is that it now feels like it’s present with us. I’ve taught it fun little games like Priscilla or I can ask it who we should tickle and it will randomly tell our family to all go tickle one of us, Max or Beast. I’ve also had fun adding classic lines like ‘I’m sorry, Priscilla. I’m afraid I can’t do that.’”
While the assistant is too tied to his own home now, Zuckerberg is considering creating a different version to give away or become a “foundation to build a new product.”
In his post explaining how Jarvis works, Zuckerberg writes that “over time it would be interesting to find ways to make this available to the world.”
“I considered open sourcing my code, but it’s currently too tightly tied to my own home, appliances and network configuration. If I ever build a layer that abstracts more home automation functionality, I may release that. Or, of course, that could be a great foundation to build a new product.”
The actor Robert Downey Jr. (who plays Iron Man) recently said he wants to be the voice of Zuckerberg’s Jarvis, but the Facebook founder has yet to disclose the voice behind his assistant.
“Tomorrow I’m going to drop some fun videos Priscilla and I made with Jarvis, and you’ll get to hear who the voice is,” Zuckerberg wrote in another Facebook post on Monday.