Stephanus van Cortlandt was first granted a water lot on November 19, 1686, which was filled in with landfill.[12] He then retired in 1700 to his manor on the Hudson River and gave the property to his son-in-law, Étienne "Stephen" DeLancey, a French Huguenot who had married van Cortlandt's daughter, Anne. The DeLancey family contended with the Livingston family for leadership of the Province of New York.
DeLancey built the current building as a house in 1719. The small yellow bricks used in its construction were imported from the Dutch Republic and the sizable mansion ranked highly in the province for its quality.[13] The firm Delancey, Robinson & Co., composed of Stephen De Lancey's son Oliver De Lancey, Beverley Robinson, and James Parker, sold the building in 1762 to Samuel Fraunces, who converted the home into a popular tavern, originally named for Queen Charlotte (Queen's Head Tavern).[14]