SOUTHAMPTON, N.Y. (CBSNewYork) — Allegations were flying Thursday that a Southampton, Long Island couple demolished American history for profit.

As CBS2’s Jennifer McLogan reported, the onetime home of a slave has been torn down, but the new owners now say they no longer even want the site to build their dream house.

One couple, the Cinallis, brought their granddaughter to see a plaque and historical markers devoted to Southampton slave Pyrrhus Concer (1814-1897). But when they looked across the street from his homestead, there was nothing left but a hole in the ground.

“This history is lost as a result of this owner that bullied and bulldozed over the village and the people,” said Georgette Grier-Key of the Eastville Historical Society.

Grier-Key is among dozens who fought a losing two-year battle to save the structure.

Concer was born into slavery and gained his freedom at the age of 18. He was later a whaler in Japan, operated a ferry across Lake Agawam, and became a prominent philanthropist.

But a Brooklyn couple threatened to sue and won the right to demolish the 19th century home of Concer and his family because it lacked landmark status.

The new owners planned to use the historic site at edge of Lake Agawam – 51 Pond Lane – to build a brand new house of their own. But they do not anymore.

“This is a slap in the face,” Grier-Key said.

The village administrator received a shocking letter from owners David and Sylvia Hermer. The letter read, “Due to a change in plans, no longer planning a building project; request refund of $18,000 building fee.”

“I am astonished to learn that they’re not even going forward — the owners — with what they set out to do,” historical preservationist Robert Strada.

Strada, who is also a designer, dismantled the home and was able to save the frame and beams that dated back to the early 19th century – with a plan to reconstruct it.

He noted that the owners paid $2.75 million for the Pyrrhus Concer home, and are now selling the vacant site for $5 million – a hefty profit.

“The battle that we have out here is with tear-down mania,” Strada said.

Sottom Edmonds, the director of the Southampton Historical Museum, said the Concer home debacle should be a rallying cry for tear-down laws.

“We tried to work with the owners of the property, and to have this result is a big disappointment around for a lot of people around here,” Edmonds said. “It just really hurt.”

The Hermers’ Water Mill attorney said his clients had no comment.

Village officials have not settled on a site to build a reproduction of the Pyrrhus Concer house.

Check Out These Other Stories From CBSNewYork.com:

More From CBS New York

CELEBRATING 50 YEARS
Get Our Morning Briefs

Watch & Listen LIVE