More than half of people say they would cycle more if roads were in a better condition, a survey for the charity Cycling UK suggests.
The YouGov poll of 2,024 people found that potholes and drivers overtaking too closely were the joint second most common reasons for avoiding cycling.
The most common concern was having to share the road with large lorries.
Cycling UK wants "fundamental changes" to road design and the Highway Code to give confidence to potential cyclists.
Between 2007 and 2016, potholes and other faults in the road were a factor in the deaths of 22 cyclists, according to the Department for Transport.
More than 350 others were seriously injured.
The poll showed that 57% of people were put off cycling by having to share the road with lorries and other large vehicles, while 56% of people would cycle more if roads had fewer potholes and other faults.
One in five local roads in England and Wales is in poor condition as councils lack the funding to repair them, the Asphalt Industry Alliance says.
Andrew's story: 'I hit a pothole and ended up in hospital'
Andrew Slorance caught the front wheel of his bike on the corner of a manhole cover which had been exposed by a pothole, while travelling home in Edinburgh in 2013.
He lost control, narrowly avoiding the traffic around him, and hit the ground. His teeth went through his lip and he broke his elbow.
He was taken to hospital but was soon discharged and had to return that same day to have pins put in his arm.
After weeks off work and a round of physiotherapy, Mr Slorance remains unable to fully extend his injured arm.
The father-of-five could not ride his bike for six months and says it took another year for him to regain the confidence to commute to work on it again.
Duncan Dollimore, Cycling UK's head of campaigns, said: "Cycling is still a minority activity in the UK with only 2% of all journeys made by bike.
"Those who do cycle put up with the potholes and dangerous traffic conditions daily and still continue.
"However, it's not always pleasant and it's no surprise most people do not consider cycling for their short everyday journeys."
The survey found other concerns that put people off cycling included threatening behaviour from drivers (for 43% of the respondents), car doors being opened in front of them (40%) and speeding motorists (37%).
Mr Dollimore said: "The government wants more and safer cycling, but as Cycling UK's research shows, people who don't currently cycle need change if they're going to choose to cycle short distances rather than drive."
Martin Tett, the Local Government Association's transport spokesman, said: "Councils are fixing a pothole every 21 seconds and keeping roads safe for cyclists is one of the most important jobs we do.
"However, only long-term, consistent and fairer government investment in local road maintenance can allow councils to embark on the widespread improvement of our roads that is desperately needed."