More than one million older people could downsize if there was no levy, found YouGov research for homebuilder McCarthy & Stone.
Dubbed a tax on moving, stamp duty is contributing to the housing crisis.
Supply is now at record low levels, raising house prices and pushing ownership out of reach for younger people.
At the same time, it could help fund retirement for older generations who have inadequate savings.
Steven Cameron, pensions director at Aegon, said: “Removing stamp duty would encourage pensioners to downsize, freeing up family homes.
“For those pensioners who are property rich but cash poor, this would also offer a new means of funding their retirement.
“This could become increasingly important as future generations of retirees will no longer benefit from the generous defined benefit pensions of today’s retirees.”
Calls are now growing for chancellor Philip Hammond to reform the tax in the Autumn Statement later this year.
The Taxpayers Alliance said the levy should be completely abolished.
Chief executive John O’Connell said:”This pernicious tax prevents younger people from getting on the housing ladder, and stops older people downsizing to more manageable homes.
“This hurts people at every stage of life and has slowed the housing market in the UK, greatly damaging the economy.
“Brits are struggling under the heaviest tax burden in 30 years, and this is in part because the UK has the highest property taxes in the developed world.
“If the government wanted to help people to buy their own homes and get on in life, they should scrap this relic of a tax completely.”