At 47, Thomas Sommer sensed it was time to change the way he worked. He had held multiple, challenging roles at Credit Suisse, but after 20 years at the firm, he felt exhausted, fenced in and rudderless.

Sommer felt he couldn't make a difference anymore. "The values at Credit Suisse no longer matched my own values. My heart said change something. You aren't able to have an impact in your work,” he said. “But my head didn't want to hear that."

Despite those nagging feelings, Sommer continued in the job for three more years. His 14-hour workdays, including a long commute via train to Zurich, Switzerland, from his home in Germany, meant he was too busy for deep self-reflection. Then, right before the most important meeting of the year in 2012, Sommer blacked-out, sweating and panicked. He burned out: Sommer didn't return to work for nine months.

Why had Sommer failed to make the changes he knew he needed to? It was a case of self sabotage, he says with hindsight."I kept telling myself that no other employer would want me at 50, I hadn't gone to many professional development courses lately and, anyway, I had a family to support," he said. "In the end, my body made the choice for me. It said, 'I'm taking you out of the game.'" 

Sommer’s not alone. According to experts, self-sabotage can affect almost anyone, since it's an expression of basic human self-doubt and may be a symptom of disengagement. A Gallup poll showed that 51% of employees in the US were "not engaged" in their work in 2014, and 17.5% were "actively disengaged".

Gay Hendricks, the author of The Big Leap, a book about taking your life to the next level, says self-sabotaging behaviour is a response to hitting your upper limit, whether that's your upper limit of career success, creative expression or even relationship harmony.

He identifies four fears behind a subconscious unwillingness to enjoy the "positive energy" you've created for yourself: feeling fundamentally flawed, worrying that you might be disloyal to your roots or past if you attain your goal, believing that more success brings a bigger burden, and fearing that you'll outshine others.

"As you begin to open up to what your unique abilities are — your inner genius — you begin to try on a bigger version of yourself. When you do that, you bump up into what I call the upper limit problem because it awakens fears in you," Hendricks said.

Very few of us can really know what success feels like until we step into it and then stabilise there for a while without sabotaging ourselves.

He added, "Everybody knows how to be a loser because we all learned to walk and have fallen on our bottoms many times. But very few of us can really know what success feels like until we step into it and then stabilise there for a while without sabotaging ourselves."

Recognising the symptoms

If you want to stop shooting yourself in the foot, first learn to recognise self-sabotage in yourself and others.

It could be as simple as negative thoughts, or small actions like overeating right before an important presentation. Instead of sending blood to your brain, you send it to your stomach because you are nervous, worried or subconsciously want to prove to yourself that you're not, in fact, good at public speaking, say those who have studied the problem.

Fundamentally, you may see yourself as somehow inadequate or incompetent, and a mediocre presentation is just the evidence you were looking for to prove it yourself.

Self-sabotage can cross boundaries, too, for instance from the office to the home. Maybe you get promoted, but you go home the same night and start a fight with your partner, thereby making strides in one area but taking a step back in another. Hendricks says the basic problem is hitting the upper limit of how much positive feeling you can handle. He wrote, "Having a willingness to feel good and have life go well all the time is a genuinely radical act."

Continually raising the bar

There's more. Self-sabotage may come in the form of overcommitting yourself or habitually setting the bar higher, with the irony that your drive to reach the next level then stops you from ever reaching your ultimate goal.

Erwin Oberender, a consultant and coach who wrote a book about self-sabotage, describes this as "not being able to take your foot off the gas."

Oberender, who worked with Sommer, also helped a man in his mid 40s who was a manager for an automobile industry supplier. The client came to Oberender complaining of never feeling really satisfied, though he had achieved success by many measures. "He was never his own cheerleader. And as soon as he was close to achieving something, he had another goal on the horizon," Oberender said. The man didn't pause to enjoy the success he had just attained and immediately set another Herculean goal.

He was never his own cheerleader. And as soon as he was close to achieving something, he had another goal on the horizon.

Oberender uses a variety of methods to help people, including a combination of kinesiology, bioenergy, consciousness, and cognition. He also encourages people to repeat key phrases to themselves, such as, "I'm allowed to be imperfect. I am allowed to withhold some of my energy and conserve it for other tasks."

Eventually, Oberender's client began to see and acknowledge his self-sabotage. "He found the brakes and decided consciously which projects would get gas or the brakes."

Up isn't the only direction

Blocking yourself isn't always self-sabotage, and up isn't the only direction to move.

If a person believes he or she cannot rise to a challenge, it may be a healthy reaction based on an accurate self-assessment.

An act of self-sabotage may be an act of self-preservation that helps you avoid challenges that you are not prepared to meet.

According to Ilse Schmidt-Zimmermann, a psychologist in Frankfurt, an act of self-sabotage may be an act of self-preservation that helps you avoid challenges that you are not prepared to meet. Or, you may actually be creating circumstances that support your true values, such as finding work-life balance or having time to volunteer, even if that means turning down the next management level.

"People try to readjust to homeostasis," said Schmidt-Zimmermann. And sometimes that's not a bad thing.

Believe in yourself

Preston Ni, the author of How to Let Go of Negative Thoughts and Emotions, defines self-sabotage as unrealistic negative thoughts that keep you from performing at your best.

For instance, a colleague declined a promotion for a job at a high-tech animation company in Silicon Valley because he didn’t think he was completely ready for the job. What should the colleague have done instead that would have given him the missing confidence?

Ni recommends resetting your mindset through positive self-talk. "Change 'I can't,' which is a self-defeating thought, into 'I'm going to learn as I go.' Or change 'I'm not ready' to 'No one else applying for the job is completely ready either, so I'm going to go in there and do my best.'"

Another idea is to envision your goal on a spectrum, instead of always having to meet the ultimate goal. Ni said, "Look at success as a work in progress rather than yes or no, right or wrong. A work in progress is always positive, as long as you're moving in the right direction."

Look at success as a work in progress rather than yes or no, right or wrong. A work in progress is always positive, as long as you're moving in the right direction.

Sommer is now helping others move in the right direction. He is telling his story of how his "perfectionist” ideas and inability to admit weakness kept him going in a job he dreaded.

Now he advises people through an outplacement service. "I am the coach and sparring partner for people who suddenly lose their jobs. I'm there when they are asking 'Who am I? What can I do? And what do I want?'"

Perhaps these are the best questions to ask yourself when setting your sights on your next level.

To comment on this story or anything else you have seen on BBC Capital, head over to our Facebook page or message us on Twitter.