This is the only trade shop built in the 1600s in America that has been in continuous use and in its original location. Originally a dry goods store, it was purchased in 1717 by Francis Pelletreau who immigrated to Southampton from New York City to develop his business producing oil and candles from whales that drifted to Southampton beaches. His grandson Elias Pelletreau occupied the building between 1750 and 1810 making jewelry, shoe buckles, tankards, silverware, etc.

In 2012 the Rogers Mansion Museum Complex, with 12 historic buildings, was awarded a unique designation on the National Register of Historic Places. William Rogers purchased the property in 1648 which was owned by a Rogers descendent until 1880. In 1899 the dwelling was purchased by Samuel Longstreth Parrish; an attorney from New York City, summer colonist, and founder of the Parrish Art Museum.

The Southampton Colonial Society (now doing business as the Southampton Historical Museum) was organized in 1898 and incorporated in 1910. In its early days, the Society collected and exhibited historical objects and documents relating to the history and development of Southampton, organized pageants, and gave lectures on antiques.