Very fit men in their late 40s are less likely to get lung cancer and colorectal cancer than unfit men, a study in JAMA Oncology suggests.
Their high fitness levels also appear to increase their chances of surviving cancer if they are diagnosed later on.
University of Vermont researchers said even small improvements in fitness could help to reduce cancer risk.
Cancer Research UK said investigating links between men's fitness levels and cancer risk was a new approach.
Being physically active and eating a healthy, balanced diet are already known to be important factors in reducing people's risk of developing cancer and other diseases.
But study author Dr Susan Lakoski said it would be more beneficial to tell people how much they needed to improve their fitness in order to reduce their risk of cancer to acceptable levels.
This could come in the form of a personalised plan, which should start with measuring their cardio-respiratory fitness.
This study of 14,000 men aged between 46 and 50, in Texas, tested their cardio-respiratory fitness levels at the outset by making them run on a treadmill to the point of exhaustion.
After that, their fitness levels were regularly tested over an average of six and a half years between 1971 and 2009.
Between 1999 and 2009, 1,310 of the men had been diagnosed with prostate cancer, 200 with lung cancer and 181 with colorectal cancer.
The study found that the men with high levels of fitness in middle-age reduced their risk of lung cancer by 55% and their risk of colorectal cancer by 44%, compared with the men with low levels of fitness - those who took more than 12 minutes to run or walk a mile.
However, the study found that the fit men in middle age did not appear to reduce their risk of prostate cancer.
The authors said the exact reasons for this were unknown but men with high cardio-respiratory fitness may be better at looking after their health and therefore more likely to undergo screening for prostate cancer, making them more likely to be diagnosed.
Dr Lakoski said fitness prior to a cancer diagnosis was important.
"This preventative message starts earlier than you think, way before you develop cancer.
"Your health behaviours and your fitness earlier in life has an impact 20 or 30 years later - and that's what people don't realise."
In the study, she said as long as people were above the low-fitness category, they already had a lot of advantages.
Tom Stansfeld, health information officer at Cancer Research UK, said: "Investigating links between men's fitness levels and cancer risk, rather than just the amount of physical activity they do, is a new approach.
"The results reconfirm the benefit of physical activity in decreasing men's risk of bowel cancer.
"Interestingly, the study also found a positive effect of fitness on reducing lung cancer risk, but more research is needed to understand this potential link better."
He said other research in women had shown that increased levels of exercise could reduce the risk of breast and womb cancers.
He added: "Being regularly physically active is great for your overall health and, as this study demonstrates, has benefits far beyond the health of your heart."