If you missed Google CEO Sundar Pichai’s keynote presentation at the company’s annual I/O Conference in San Francisco earlier this week, you missed a pretty special first-look at just how much our wired-world is about to change. Even though Pichai started his talk off by recognizing that the tech industry must always be responsible for the tools and services it creates, he then went on to show us some of the coolest — and potentially most creepy — new set of Artificial Intelligence (AI) tricks to date.

Imagine typing an email and having your computer not just autocomplete words you use a lot, but entire sentences. (Now, imagine that those sentences are always right. I mean, it’s mind-blowing enough not to shout “damn you autocorrect” just using it for words. But watching it dictate your thoughts correctly— well that’s just crazy-town.)

And that’s not even the best part. Google Photos can now colorize a black-and-white photo with a single click. And in the most mind-melting demonstration, Google Assistant booked a haircut appointment by placing a phone call to a human receptionist at the salon and having a conversation on your behalf, sounding indistinguishable from a real person. The receptionist probably never knew she’d been talking to AI.

Check it out for yourself. Go ahead, we’ll discuss when you’re done.

That voice! (Google calls it Duplex.) It sounds so real, right? Especially those tiny little nuances of human speech like the “Mmm-hmm” to let the other person know “you” understood. It’s incredibly cool, but also incredibly creepy. Just like many a sci-fi movie has hinted in the past, I can’t help but wonder … do we really want this?

I mean, what if it gets hacked? Can Google impersonate me? What if this falls into the wrong hands? These are just the start of several questions a whole lot of us have now. We don’t know any of the answers quite yet, but here’s what’s worth thinking long and hard about.

Invasion Of Privacy Issues

Is asking your AI to make a phone call for you is an invasion of privacy — not your privacy — the person on the other end?

One of the biggest questions in the wake of that wild demonstration was whether Google Assistant (or any other AI for that matter) be required to identify itself when it reaches a live person. Some thought it should definitely identify itself, and others thought it might just cause confusion. It wasn’t long — like, less than 48 hours later — before Google released its own statement, saying that Google Assistant will “properly identify” itself when it makes calls.

We still don’t know exactly what that means — will Google Assistant identify itself as an AI or just say that it’s making a call on behalf of its owner? —  but the immediate buzz surrounding the demo was obviously enough to put Google on notice.

Google Assistant obviously can’t impersonate you now — accurately mimicking a human voice in real time takes some sci-fi tech that simply doesn’t exist yet — but if Google launches a feature that lets it make phone calls for you, that’s could still cause some headaches. What if things go wrong? What if it says something it’s not supposed to or, heaven forbid, offends someone who thinks they’re talking to you?

In a blog post about the “Duplex” software that powers these new natural language abilities, Google says that the feature has plenty of fail-safes.

“The Google Duplex system is capable of carrying out sophisticated conversations and it completes the majority of its tasks fully autonomously, without human involvement,” Google says. “The system has a self-monitoring capability, which allows it to recognize the tasks it cannot complete autonomously.”

Basically, the AI is smart enough to know if it can’t handle the conversation and, if that happens, it would have to hand the phone call off to you. That’s good news because the last thing I want is an AI bumbling around with an important appointment, but we still don’t know exactly how careful the AI will be. Will it repeat a question a half dozen times before it decides to have you take over, or will it nudge you at the first sign of trouble? We’ve only heard a handful of example calls so far and they’ve all gone pretty well, but that’s such a small sample size it’s impossible to know for sure.

Mistrust Of Technology

It’s harder and harder for people to trust technology and tech companies these days. With incidents like the recent Facebook privacy scandal and security breaches that leak personal info from credit agencies and retail stores, that’s more than understandable. It also makes things even more complicated for a company like Google to launch an autonomous calling feature like the one they just showed off.

You know that feeling you get when you call your cable or wireless company and hear the robotic voice on the other end? Those “natural language” bots are supposed to be able to handle most issues without having to forward you to a real person in a call center, but they hardly ever work like they’re supposed to. If you’re anything like me, you just mash “0” until you hear a human voice to save time.

In a future where Google Assistant can call to schedule your haircut, will the person on the other end feel the same way? If the phone call starts with something like “Hi, this is Google Assistant and I’m calling to make a haircut appointment for Jennifer,” will that cause more problems than it solves?

In the demo calls, the AI never identifies itself as a computer and the calls go pretty smoothly, but they might not have gone quite so well if the people on the other end knew they were talking to a computer.

Google is kind of stuck between a rock and a hard place here: On the one hand, a natural-sounding AI can make calls faster and easier than a robotic one but, on the other hand, Google knows it has to make its AI identify itself or risk outrage from people who think it’s creepy to get a call from a computer.

A Long Way To Go 

Google may have shown off Assistant’s ability to make personal calls, but it quickly noted that the feature isn’t ready to be rolled out to everyone yet.  Google says the Duplex tech that manages AI phone calls is still in an experimental stage, so we’re still a long way from seeing the feature on our phones if it ever actually comes out at all.

Here’s where I turn to you to ask: What do you want out of an AI secretary? Is full disclosure more important than convenience? If you’re just using it to book dinner reservations and haircuts, is privacy even on your mind? I’d love to hear from you on Twitter or Facebook!