Have you ever had a panic attack in mid-flight? Those that have will tell you it’s not fun. And there are plenty of reasons that make people panic. Some of them – irrationally so – fixate on the state of the pilot. Are they tired? Stressed? Paying enough attention?

Would these people feel any better if we could get rid of pilots altogether?

The technology already seems to be there – drones are not a particularly new invention. Armies use drones the size of aeroplanes in war zones, where they are controlled remotely or loiter flying a pattern. Even helicopters can be pilotless, like the K-MAX, which is as big as a standard helicopter, and delivers aid supplies with stunning precision to dangerous locations.

Aeroplane accidents are rare today, but when they happen, they are getting harder and harder to solve, says Tim Robinson, editor-in-chief of the Royal Aeronautical Society’s magazine Aerospace. That’s why investigations often focus on ‘human factors’, identifying psychological and physiological issues as a probable cause.

“So with pilots relying on autopilots for 95% of today's flights, the argument goes, why not make the final 5% – take-off and landing – automated?” says Robinson. “Computers fly ultra-precise, repeatable trajectories, do not fly drunk, do not get tired, do not get distracted and so the thinking goes could be safer than human pilots in the future.”

He says that in a debate at the society earlier this year, with pilots, engineers, scientists and airline representatives debating autonomous planes, the motion ‘there will be no need for pilots in 40 years’ was carried by approximately 60 votes to 40.

With its propellers folded, the 184 takes up as much room as a small car

At the Consumer Electronics Show (CES) in Las Vegas in January, Chinese company Ehang unveiled the first-ever passenger drone, the electric-powered Ehang 184. The quadrocopter can fit one person with a small backpack, and even has air conditioning and a light. To fly, the passenger needs to set up a flight plan, click ‘take off’ and ‘land’ on a tablet, and the computer does the rest. With its propellers folded, the 184 takes up as much room as a small car.

There are other similar efforts to develop personal air transport systems. In the US, a twin-propeller experimental plane with two passenger seats and two cockpit seats was flight-tested last year. Made by Aurora Flight Sciences Corp and called the Centaur, it can be operated by pilots from the cockpit or from the ground – and during the test, it successfully flew with no one on board.

Airbus Group is working on Vahana, an autonomous ‘flying car’ for passengers or cargo, while in Germany the Volocopter project hopes to build a ‘scaled-up’ drone that can carry one or two people. Another European endeavour, myCopter, looked into the kind of technologies that would be needed to bring personal transportation into the air. Researchers who took part in the project, from the Max Planck Institute for Biological Cybernetics in Tübingen, are still trying to figure out how to make it easier to control, says Heinrich Bülthoff, the managing director of the institute. “We try to make flying a helicopter as easy as driving a car with very little training,” he says.

In the UK, similar research was undertaken by a consortium of public and private companies called Astraea; in 2013 they received £62m to investigate civilian pilotless aircraft concepts. The aim was to “investigate the kind of technologies you would need, such as sense and avoid, secure communications and so on, to integrate UAVs into manned airspace,” says Robinson.

But be it a plane that’s controlled remotely, or by an artificial intelligence system, whether it’s just a passenger pressing a button, or a ground crew pulling the wireless strings, one fundamental question hasn’t been answered yet: Would we feel at ease flying if we can’t see two smartly dressed pilots at the helm, calmly providing us with weather updates during a storm?

These days nobody thinks twice before getting into a lift – even though decades ago their ‘safe’ operation required an operator

“It appears that people are more comfortable with a pilot directly at the controls, and on the plane,” says aeronautics expert Stephen Rice, formerly at the Florida Institute of Technology and now at Embry-Riddle Aeronautical University, who co-authored a 2014 report on public perception of robotic flights. "People are of the opinion that being in the vehicle while controlling it is easier than controlling it remotely. Most people have driven remote control cars or planes and find it difficult.”

Still, these days nobody thinks twice before getting into a lift – even though decades ago their ‘safe’ operation required an operator called a ‘liftboy’. The same goes with unmanned trains and self-driving cars. It shows that “society is now undergoing a mental shift over autonomous vehicles,” says Robinson. “If, in the future you grow up living in a society where cars drive themselves and UAVs reliably deliver packages – would passenger aircraft that fly themselves be a big surprise?”

Today, however, Rice thinks that it might be trickier for people to accept unmanned planes because when trains or cars lose their autopilot, they don’t necessarily crash. “When a plane stops, it falls out of the sky,” he says. And of course, many people are already nervous about flying, while not many of us are nervous about riding in a car or on a train.

But computers also don’t suffer the kind of mental stress which leads to incidents such as co-pilot Andreas Lubitz, who in 2015 flew a Germanwings passenger jet into a mountain in the French Alps. Unmanned planes could potentially prevent such incidents, thinks Rice. “This is not to malign human pilots… No autopilot today could have landed the plane on the Hudson River. But a machine doesn’t get tired, doesn’t get emotional, doesn’t let fear or fatigue affect its performance, and doesn’t make poor decisions. It simply does what it’s programmed to do. So, in the long run, I believe they will be safer overall.”

Having a remote control pilot or autopilot available to take over the cockpit could potentially have prevented the Germanwings suicide accident, he adds. It took eight-to-10 minutes for the plane to descend from 38,000ft and crash; “if protocols had been in place, this should have been enough time for ATC to authorise a lockout of the human pilot and have the remotely controlled pilot or autopilot divert to the nearest airport,” says Rice.

Some complex problems may be difficult to diagnose from the ground, without direct access to all the plane’s systems – Michael Clamann, Duke University

But not everyone agrees. Michael Clamann, research scientist at the Humans and Autonomy Lab at Duke University in Durham, North Carolina, thinks the level of safety will be about the same. “While it’s true that there have been rare cases of pilots purposefully crashing planes (or making grave mistakes), there are also examples of pilots preventing disasters through creative problem solving. For example, using throttles for flight control. Some complex problems may be difficult to diagnose from the ground, without direct access to all the plane's systems.”

And controlling the plane from the ground will be far from easy, he warns. “Sitting in a cockpit taking in all the information through the window, from displays and controls, from the ‘feel’ of the aircraft is very different to experiencing flying through software. While replicating the controls of an aircraft is possible with current technology, ensuring a feeling of ‘presence’ and giving a ground controller access to 100% of the information in the cockpit is a more difficult problem.”

Robinson believes that the first step to remove pilots on commercial airliners could be to do it gradually. For instance, a European research project called Across (Advanced Cockpit for Reduction Of Stress and Workload) on future passenger aircraft flight decks is looking into the ‘reduced crew’ operations with just one pilot.  

Two decades ago, it was standard to have three pilots in the cockpit; now the standard is to have two, says Scott Winter, assistant professor of aeronautics at Florida Institute of Technology. “I think over time participants will become more trusting of this technology.” Both he and Rice recommend conducting trials in cargo aircraft as a good starting point to deploy unmanned planes, make them totally reliable and in the process make the public trust the idea of a plane flying without a pilot.

Even if it works, there will still be one nagging worry: the risk of the unmanned plane getting hacked

Nasa is looking into the same thing. Its Single-Pilot Operations (SPO) concept features one seat in the cockpit for a captain and one seat on the ground, for an operator who would be at times dispatcher and at times first officer. The ground operator would wear the dispatcher hat when the flight is progressing smoothly, and oversee not one but 12 planes – effectively being what Nasa calls a ‘super dispatcher’. And if there is a problem or difficulty on one of the aircraft, he or she would become first officer dedicated to that specific flight.

Even if it works, there will still be one nagging worry: the risk of the unmanned plane getting hacked. Ground systems would have to have highly secure datalinks and potentially military-level cybersecurity, thinks Robinson. Clamann adds that it would be necessary to have not only preventive but also recovery measures – to stop an attack from happening and ‘rescue’ the aircraft during a hack. As the final backstop, says Robinson, a UAV airliner “might also have a higher-level AI or knowledge that if it received ‘odd’ instructions (for example, to accelerate and dive towards the centre of a city) it would not comply until it had double-checked with its airline or another human supervisor.”

So maybe in a few decades, you’ll be sipping your wine in the lounge where the cockpit used to be. Until then, be prepared to listen to your captain’s updates, and know that they’re seeing the same view out of the window as you are – but they’re just in the best seat.

Join 700,000+ Future fans by liking us on Facebook, or follow us on Twitter, Google+, LinkedIn and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.