Skip to Content

The Magazine /  Travel

The Avenue that 

Kept Moving North

​

The story of New York told by following its wealthiest families, from New Amsterdam and the Astors to Fifth Avenue and the Upper East Side.

By The Magazine

You can tell the story of New York by following its wealthy families, who spent more than a century and a half moving north. They started at the southern tip of Manhattan, where the port and the money were, and kept climbing: Washington Square, Union Square, Murray Hill, Fifth Avenue and, finally, the Upper East Side. Each generation of fortune tried to put a little distance between itself and the last, and every time it managed to, the city caught up and the march began again.

At first it wasn't even called New York. It was New Amsterdam, a Dutch trading post founded in the 1620s and devoted almost entirely to beaver pelts, which fetched a fortune in Europe for making hats. In 1664 the English sailed in, demanded surrender, and the Dutch handed over the town without firing a single shot. It was renamed New York after the Duke of York, and it stayed English until 1783, when the last British troops left. The Dutch part never really went away. It stayed in the surnames, in the streets and in a very mercantile way of thinking. Wall Street, for instance, was the wall that marked the northern edge of that little town.

Out of that world came the name that would define an entire era. John Jacob Astor arrived from Germany with almost nothing around 1784 and made a fortune in furs, but his real move was understanding before anyone else that New York's wealth would be in land. He bought lots up north while they were still farmland and waited for the city to grow toward them. When he died in 1848, historians describe him as the richest man in the country. People called him "the landlord of New York."

The second half of the nineteenth century was even more excessive. It's the era Mark Twain named the Gilded Age: golden on the outside, a lot more complicated underneath. Carnegie with steel, Rockefeller with oil, Vanderbilt with railroads and Morgan with banking were inventing entire industries, and there was no federal income tax yet; that only arrived in 1913. While millions of immigrants came through the port looking for work, a handful of families built European-style palaces along Fifth Avenue, which back then was a residential street, not a shopping one.


At the top of that world sat Caroline Astor, known to everyone simply as "Mrs. Astor." In an age without social media, she was the algorithm: she decided who belonged and who didn't. With Ward McAllister she put together the most famous list in the city, "the Four Hundred," which supposedly was about the number of people who fit comfortably in her ballroom. Being on the list meant you had arrived, and not being on it meant exactly the opposite.

At the top of that world sat Caroline Astor, known to everyone simply as "Mrs. Astor." In an age without social media, she was the algorithm: she decided who belonged and who didn't. With Ward McAllister she put together the most famous list in the city, "the Four Hundred," which supposedly was about the number of people who fit comfortably in her ballroom. Being on the list meant you had arrived, and not being on it meant exactly the opposite.

That world was broken by a party. To Mrs. Astor, the Vanderbilts were far too new to be let in, so in 1883 Alva Vanderbilt built herself a French château–style mansion on Fifth and threw a costume ball for more than a thousand guests, leaving out Mrs. Astor's daughter. As the story goes, for her daughter to be able to attend, Mrs. Astor had to send her calling card to the Vanderbilt house. It was old society finally acknowledging new money, and from that night on, New York's high society no longer fit in a single ballroom.

And the city moved north again. In the 1890s, Mrs. Astor left her house on 34th Street for a new mansion at Fifth and 65th, facing Central Park, and with that she made the Upper East Side the new seat of power. What happened next to those two houses says a lot about New York. On the 34th Street lot came the first Waldorf-Astoria and, once the hotel went out of fashion, the Empire State Building. Where the 65th Street mansion stood is now Temple Emanu-El, one of the largest synagogues in the world, whose congregation began decades earlier on the Lower East Side and moved north just like everyone else.

Almost none of the Fifth Avenue mansions facing the park survived. Most were torn down in the 1920s, when taxes arrived and an apartment building suddenly made more sense than a palace for one family. But the prestige stuck to the place, and that's why the Upper East Side still feels different: the history there is still visible. Walk up Fifth from the Frick Collection to the Met and you'll pass buildings that have watched more than a century of fortunes and generations go by, on the same blocks that are still the city's center of gravity.

— The Magazine