Fishermen and charter skippers in Dorset are worried the 2012 Olympic sailing races will seriously damage their livelihoods.
There will be an exclusion zone during the Olympic Games covering almost all of Weymouth Bay.
It means fishermen will be banned from their normal fishing grounds for 16 days during what they say is the most lucrative time of the year.
The Olympic organisers said it was too early to discuss possible compensation.
The London Organising Committee of the Olympic Games and Paralympic Games (LOCOG) said it was listening to concerns raised by the fishermen and other commercial vessels and were working on "number of solutions".
Charter skippers had asked for a designated viewing area close to the racing but that was turned down.
Nineteen of the dozens of fishing and leisure boats based in Weymouth and Portland have been signed up to act as marshals during the event, giving them three weeks guaranteed work, LOCOG added.
There will also be restrictions in place for more than two months before the Olympic sailing events begin.
Bob Summerhayes, one of eight fishermen working out of Weymouth, estimates he will lose up to £2,000.
He said most fishing grounds outside the zone are either already taken or too far for his boat to reach.
"That time of year is important for us because the demand and prices are high for the spider crab which we're fishing for at that time," he said.
"And the weather is fine so we don't tend to miss many days that time of year either."
Councillor John Birtwhistle, Weymouth and Portland Borough Council, is calling for compensation to be paid to the fishermen and skippers.
He said: "Because unfortunately they're going to lose their livelihoods otherwise, and that's not on.
"I want the Olympics to be welcomed to this town. I don't want people saying 'we don't want it here'."
LOCOG said: "We have only just finished a lengthy consultation process with fishermen and other boatmen at Weymouth and Portland.
"We have listened to their concerns and have a number of great solutions to put on the table.
"It is too early to talk about any compensation."