AMAGANSETT, N.Y. (CBSNewYork) — Residents are choosing sides in a Long Island enclave where the average house sells for $2.8 million. Some worry that building affordable housing will be paving paradise.

As CBS2’s Jennifer McLogan reported, in the exclusive Hamptons hamlet of Amagansett, a multimillion-dollar beachfront home is being built steps from the ocean — it is a far cry from affordable housing. But just down Montauk Highway, in the swanky ZIP code filled with movie and music stars, there is a push to build 46 apartments for struggling local families. The 4½-acre plot is designated for low-income housing.

READ MORE: Coroner Confirms Remains Found Are Gabby Petito's, Says Manner Of Death Is A Homicide As Search Resumes For Brian Laundrie

“Everyone is in favor of affordable housing in theory,” said Catherine Casey of the East Hampton Housing Authority. “But as soon as you put it on a map, someone is going to object. That’s just the way it is.”

Leaders of the town Housing Authority say the development would be a walkable, transit-oriented community of clustered cottages with green touches, such as rain gardens and a common green area.

But the plan is dividing Amagansett.

“It’s important for people who have lived here for generations to be able to stay in the area and not have to uproot their families to go somewhere more affordable to them,” said resident Ryan Delacruz.

“Out of respect for the community and the town and the certain culture that Amagansett has, I’m not quite sure it would be the best idea,” said Lily Nathanson.

READ MORE: Luchiano Lewis Pleads Guilty In Deadly Stabbing Of Barnard College Freshman Tessa Majors

Things became so contentious that hundreds are sigining online petitions opposing the project, saying it will have a negative impact on water quality, traffic, taxes and emergency services. Others fear a drain on the school system.

But Town Supervisor Larry Cantwell calls it the right thing to do.

“It’s about providing an opportunity for local, working families to have a place to live and rent,” he said. “Not everyone can afford to have a house here.”

A family of four with a total annual household income of $106,000 or less would qualify.

The application for the project is expected to be formally submitted to the town within two months.

MORE NEWS: David Hardy Sentenced In Deadly Stray Bullet Shooting Of 13-Year-Old Shamoya McKenzie In Mount Vernon

If approved, the low-income housing would be ready in 2019.