SOUTHAMPTON, N.Y. (CBSNewYork) — His obituary described him as one of the most respected residents of the Hamptons.

But the important legacy of former slave Pyrrhus Concer was lost in the last century. As CBS2’s Jennifer McLogan reported Thursday, it took a scandal to ignite a community.

A historic home overlooking Agawam Lake was demolished by a wealthy couple after they bought the property.

It is now an empty field, sparking community outrage. As curator Dr. Georgette Grier-Key explained, the structure at 51 Pond Lane in Southampton had been the homestead of Concer.

“Time has been lost and his story was sealed away,” Grier-Key said.

CONTINUING COVERAGE: Black History Month

Concer was born indentured in 1814 and sold into slavery in the Hamptons.

“Most people don’t even think there was slaves even in the North,” said Brenda Simmons of the Southampton African American Museum. “This man was a former slave. He was sold at 5 years old, worked on a farm. He became a whaler.”

Concer gained his freedom at age 18 and then ran a ferry service from village land to the ocean.

“He was beloved by this community. The grandfather to all the community children. He would take them on his ferry boat across to the beach,” Grier-Key said.

He also sailed to Japan. He is believed to be the first Black man to do so.

“The Japanese were actually trying to rub his skin because they had never seen a person of color before,” Simmons said.

READ MOREBlack History Is Our History: Divine Nine Fraternities, Sororities Offer Lifetime Of Brotherhood, Sisterhood For African Americans

Concer eventually returned to his beloved Southampton and became a wealthy businessman and philanthropist.

“Our local Black history has been something that has been ignored,” said state Assemblyman Fred Thiele, a Democrat who represents the East End.

And now, overdue honors have been bestowed upon Concer.

“Being the grant administrator tasked with this project, I have taken it to my heart professionally and personally,” Nicole Jean Christian said.

READ MOREBlack History Is Our History: MLB Working To Incorporate Negro League Stats Into Official Record

The property was returned to the town and a grass roots movement helped salvage the home’s original beams and framing.

The Architectural Review Board voted unanimously to designate the site a historic landmark. Now, village, town and private donors are helping fund the Pyrrhus Concer House restoration.

“To be able to restore his fairly humble house here in Southampton Village, one of the wealthiest places in the world. A lot of the wealth here is old money, really was on the backs of exploitation and slavery,” Southampton Town Supervisor Jay Schneiderman said.

“When we rebuild his home, his history and achievements will be at forefront of New York and the country’s history,” added Mayor Jesse Warren.

Concer died 125 years ago, but now there is a new chance to pay homage to his legacy.

During his lifetime, Concer began a widow and children’s educational fund through the First Presbyterian Church that still exists today in Southampton.

Jennifer McLogan