The Richest Women in New York Exposed Their One Weakness, and Warhol Built An Empire On It
They had the trust funds, the family names, and the invitations... but none of it could protect them from the one thing they actually wanted
Content warning: This piece includes references to substance abuse, mental illness, and s****de, which some readers may find distressing; please take care while reading.
The first thing you noticed about Andy Warhol’s Factory at 231 East 47th Street was that the walls were covered entirely in aluminum foil and silver spray paint.
Billy Name (photographer, lighting designer, and one of the Factory's most devoted fixtures) had done it himself.
Surfaces, pipes, exposed bricks… he arranged all of them until the entire loft shimmered like the inside of a mirror ball.
The effect was deliberate. When you walked in, you saw yourself reflected in every direction. You felt, for one intoxicating moment, like a star.
And that was the trick.
By 1965, the Factory had become the most exclusive open door in Manhattan. Drag queens shared the freight elevator with Ivy League dropouts. Underground filmmakers argued with fashion editors.
And scattered among them, drawn by Warhol’s gravitational pull on anyone who mattered, were the daughters of some of the oldest and wealthiest families in America.
They thought they were choosing him. Every one of them was wrong.
The Heiress and the $80,000 That Vanished
Edie Sedgwick arrived in New York in September 1964 carrying an $80,000 trust fund received on her twenty-first birthday from her maternal grandmother.
Her mother was the daughter of Henry Wheeler de Forest, chairman of the board of the Southern Pacific Railroad.Her maternal great-grandfather, Reverend Endicott Peabody, founded the Groton School. She was named after her father's aunt, Edith Minturn Stokes, the philanthropist and society figure whose portrait with her husband (painted by John Singer Sargent) still hangs in the Metropolitan Museum.
Five generations of American aristocracy. Not the self-made kind.
It’s the kind that arrives fully formed and expects to be recognized on sight.
Warhol met her at a dinner party in early 1965 and saw immediately what she was: a beautiful girl from an old family with substantial money and enormous psychological need.
She was precisely what his operation required.
Within weeks, Edie was at the center of the Factory. She appeared in more than a dozen of Warhol's films between 1965 and 1966 — all unpaid. Her trust fund financed the lifestyle the Factory demanded: the Hotel Carlyle, the taxis, the dinners, the drugs. The Factory’s social economy required its stars to spend lavishly while receiving nothing in return but the promise of fame.
By the end of 1966, the $80,000 was gone. Fourteen months of Manhattan living combined with a growing amphetamine dependency had burned through every cent. She had no contract. No salary. No ownership of anything she had helped create.
Warhol’s response, when people remarked on her deterioration, was to point a camera at it.
Poor Little Rich Girl, the film he made about her in 1965, follows Sedgwick through a day of getting dressed, using drugs, and lying on her bed in conversation with a friend.
He named the film. He chose the angle. And he never once turned the camera off.
Inside the Silver Room
The Factory distributed Obetrol (pharmaceutical-grade amphetamines) the way a normal office distributes coffee. Warhol himself took the small orange tablets to paint through the night, but he maintained a controlled, almost clinical relationship with the drug. The people around him did not.
“Everybody was so bi*tchy and paranoid,”recalled Jayne County, one of the Factory regulars. “And Andy loved that and everybody wanted his attention. It was awful. You’d turn your back and you knew they were talking about you.”
Ronnie Cutrone, another regular, remembered a sign on the wall: “Absolutely no dr*gs allowed.” Everyone shot up on the staircase instead. The only person permitted to take drugs inside the Factory was Warhol himself.
This was the operating principle. He created a space that was intoxicating, addictive, and structurally designed to produce dependency, then positioned himself as its sober architect, watching from behind the lens while everyone else unraveled on film.
The Ones Who Got Out and the Ones Who Didn’t
Baby Jane Holzer was the first.
Born Jane Brukenfeld, the daughter of Florida real estate developer Carl Brukenfeld, she had married into money on Park Avenue and found herself deeply bored by it. British interior designer Nicky Haslam brought Warhol to a dinner party at Holzer’s apartment in 1964, and Warhol saw what she represented immediately: high society with edge, old money that wanted to play rough.
He made her his first "Girl of the Year." She starred in his films, appeared in Vogue with David Bailey, and took Warhol to the Rolling Stones' first New York performance at the Academy of Music on East 14th Street in October 1964.
But Holzer had something the others lacked: a complete life outside the silver room.
She was already wealthy through marriage, already connected, already somebody before Warhol handed her a title. When the scene stopped being fun, she walked away without looking back.
She is now a prominent art collector and real estate investor. She got out because she never needed what Warhol was offering.
Brigid Berlin did not get out.
Her father, Richard E. Berlin, served as president of the Hearst publishing empire for thirty years. Her mother was a fixture of Upper East Side society. Guests at the Berlin family home included Richard Nixon, Joan Crawford, and the Duke and Duchess of Windsor.
Brigid rejected all of it. She disappeared into New York's underground art scene and connected with Warhol in 1964, becoming one of his closest collaborators and, eventually, his closest friend, described by Vincent Fremont, a founding director of the Andy Warhol Foundation for the Visual Arts, as "the only person who could yell at him."
But the Factory’s amphetamine culture swallowed her whole. As the speed epidemic of the early years gave way to the cocaine of the disco era, Brigid’s usefulness faded. The dr*gs outlasted the people who took them. Warhol observed it, recorded it, and turned his attention to fresher faces.
Cornelia Guest entered Warhol's orbit a generation later. Her godparents were the Duke and Duchess of Windsor. Her father, Winston Guest, was one of the country's top polo champions.
In 1982, she was named Debutante of the Year, and in 1986, Debutante of the Decade. Warhol, a close friend of her parents, was photographed at her side at galas throughout the early 1980s.
She came through it. But the dynamic never changed. Warhol moved toward old money, absorbed whatever social credibility he could extract, and the women who stayed too long paid a price he never shared.
“Do you think Edie will let us film her when she commits s****de?”
That quote is real. Multiple sources attribute it to Warhol. He reportedly said it while standing on a sidewalk, staring up at the fifth-floor window from which Factory dancer Freddy Herko had leapt to his death while high on LSD the year before.
Edie Sedgwick’s final years were a long unraveling. After leaving the Factory in late 1966, she had no money, a deepening addiction, and no one left to call. She was further destabilized when she learned that Bob Dylan had secretly married Sara Lownds in November 1965 (a marriage Warhol reportedly passed along through his lawyer, according to Paul Morrissey, who was present). Sedgwick had believed Dylan might pull her out of the wreckage.
She cycled through hospitals and psychiatric wards. Her family, determined to maintain appearances, quietly covered the bills while refusing to admit publicly that anything was wrong.
On November 16, 1971, Edie Sedgwick died of barbiturate intoxication in Santa Barbara, California. The coroner's ruling was "undetermined… accident/s****de." She was twenty-eight.
In The Philosophy of Andy Warhol, published four years after her death, Warhol wrote about a character he called "Taxi," widely understood by critics and biographers as a thinly veiled portrait of Sedgwick. He wrote about her selfishness, her drug hoarding, her recklessness. The closest he came to grief was five words: “I missed having her around.”
She was twenty-eight, four years passed, and still just content for his next book.
The $25,000 Difference
While the Factory burned through its superstars, Warhol built a parallel business that exposed the entire operation’s arithmetic. Wealthy clients, many from the exact same social register as the women he had elevated and discarded, paid $25,000 per silkscreen portrait set, with additional canvases in other colors for $5,000 each. He completed roughly fifty commissions per year.
The women inside the Factory gave their money, their time, their youth, and their stability for free. The women outside the Factory paid $25,000 for a painting and went home with their lives intact.
The difference between a muse and a client was whether you understood that Warhol was always charging. The clients knew. The muses found out too late.
Why the Women Who Needed Him Paid the Most
The Factory was thus not primarily dangerous because of the dr*gs, though though those substances killed people. It was not primarily dangerous because of Warhol’s detachment, though the detachment was bone-deep.
It was dangerous because it sold something that “old money” women, for all their trust funds and family names, could rarely find elsewhere: the experience of being wanted for who they were rather than what they represented.
Not as a Sedgwick or a Berlin or a Guest… but as a person, interesting on her own terms, freed from the weight of a surname.
Warhol understood this hunger because he had lived its opposite. He grew up as a sickly, odd-looking boy from a working-class Carpatho-Rusyn immigrant family in Pittsburgh, watching glamour from the far side of a window. He understood what it meant to stand outside and want in.
And he constructed an entire machine around that understanding: a silver room that made everyone feel visible just long enough for him to get what he came for.
The women who made it through the Factory intact were the ones who already had a clear sense of themselves before they walked in.
Baby Jane Holzer had a life on Park Avenue that existed with or without Warhol’s attention.
Bianca Jagger carried her own fame entirely independent of his, and when she felt Warhol's published diaries misrepresented her, she pursued legal action, ultimately settling a libel suit against publisher Simon & Schuster for an undisclosed amount.
The women who did not make it through were the ones who arrived carrying a vacancy.
They had some need for recognition, or escape, or reinvention that their families and their money had never been able to fill.
That vacancy was the real currency of the Factory. And Warhol spent every last cent of it.