Bullying at work, which can involve anything from shouting at colleagues, to public criticism or threatening to get them fired, exacts an enormous toll on everyone.

And workplace abuse doesn’t just damage employees it hits company profits hard too. Bullied employees are more likely to miss work, for instance, because of sickness.

So, with a wealth of evidence driving it, you might hope work environments had evolved to be kinder, gentler places. But you would be wrong. Both anecdotal and more systematic evidence suggests that workplace bullying persists. Stanford Management Science and Engineering professor Bob Sutton, in response to the numerous e-mails he received from people asking for advice on how to cope with abusive managers, published The Asshole Survival Guide in 2017. 

When your own rewards depend on what others do, you evaluate and judge the people you work with more strongly on their competence rather than sociability and warmth

So, this begs the question: if the data on the effects of workplace abuse are so clear, why do so many companies hire and promote manipulative and selfish people?  University of Virginia business school professor Peter Belmi and I have given this question a lot of thought.

We carried out a series of experiments involving more than 900 participants. Some mimicked real world circumstances (for example, picking someone to work with on a task) and we found that if people’s rewards (what they expected to earn) depended solely on their own performance, they very strongly preferred to work with (or hire) someone who was described as being sociable (i.e., friendly, warm, nice) even if not that competent.  But when their rewards depended at least partly on the other person’s performance, the importance of sociability went down and the emphasis on competence increased.

Simply put, when your own rewards depend on what others do, you evaluate and judge the people you work with more strongly on their competence rather than sociability and warmth.

This research suggests that one reason people downplay interpersonal warmth and interpersonal skills at work is because they instead choose to analyse the ability of others to do the job.

And there’s a further twist. Princeton University psychology professor Susan Fiske and Amy Cuddy, who teaches leadership at Harvard University, have identified warmth and competence as the two fundamental dimensions along which people describe others.  They and other researchers have found that people often presume warmth and competence are negatively correlated—in part because if people are nice, it is presumed it is because they have to be.  The feeling that if they were any good, they wouldn’t need to be as kind and supportive to others. 

And now you see the problem. It’s not just that people downplay the importance of warmth and good behaviour at work when their earnings depend on the performance of others, they often equate “bad behaviour” with higher levels of ability.

Descriptions of CEOs not renowned for their friendly or warm leadership styles - leaders such as Steve Jobs, Jeff Bezos, or Elon Musk, to take just three - often emphasise their need to be demanding and difficult in order to disrupt an industry.  And such descriptions invariably excuse or rationalise their tough methods by pointing to the great results these, and others like them, have achieved. People put up with and make excuses for bad behaviour if they think the bad actor is going to create a lot of economic value—particularly if they are going to benefit from that economic value creation personally.

 

Here’s the problem, the research proves behaving badly does have a financial cost.  Numerous talented people, including his co-founders, reportedly left Apple because of Steve Jobs’ management style.  In the months before Uber’s Travis Kalanick was fired, the senior vice president of engineering, the head of policy and communications, the vice president of product and growth, the vice-president of maps, and the head of finance quit. At Hewlett-Packard, former CEO Carly Fiorina was criticised over thousands of layoffs and departure of executive talent.  So, if talent is important for business success, building a more humane culture would have reduced turnover, increased the ability to attract even more talent, and produced much more productive dynamics within the management teams.

It’s not just that people downplay the importance of warmth and good behaviour at work when their earnings depend on the performance of others, they often equate “bad behaviour” with higher levels of ability

Therefore, I propose one solution to implement cultures like those advocated by writers Christine Porath and Sutton. We remind people hiring that, yes, competence is crucial but that this competence entails interpersonal competence as well as the technical kind i.e. something sometimes called emotional intelligence.

It is possible that by reminding people of the importance of emotional intelligence that they will place greater emphasis on warmth and social skills, and create more humane, less toxic workplaces. And that profits will soar as a result.

--

Jeffrey Pfeffer is the Thomas D. Dee II Professor of Organizational Behavior at the Graduate School of Business, Stanford University.  His latest book is Dying for a Paycheck:  How Modern Management Harms Employee Health and Company Performance—and What We Can Do About It.

To comment on this story or anything else you have seen on BBC Capital, please head over to our Facebook  page or message us on Twitter.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.