The colorful history of America's oldest family estate - the VERY private New York island where Captain Kidd buried his booty and Jackie O was accused of stealing a solid gold lighter during a dinner party for the crème de la crème of society

• e-mail

The Hamptons have long been the playground for the uber rich and famous. Just 90 minutes by car from New York City, the postcard-perfect Valhalla of immaculate beaches and manicured hedges has become a place for millionaires to park their money in mansions that offer complete privacy and isolation from the hoi polloi.

But in a world where no expense is spared, there is still a one thing that money can’t buy: Gardiners Island.

The six mile long private paradise sits at the center of a windswept bay between Long Island’s North and South Fork, making it only accessible by boat. It boasts 27 miles of untouched coastline, jagged cliffs, pristine beaches, rolling meadows and a rich family history that pre-dates the founding of America itself.

Passed down through the Gardiner family for 380 years, it is America’s oldest family estate that has fascinated people with its shroud of mystery. Very few have ever been allowed to see it and anyone boating too close to its shores would instantly be reproached by the Gardiner family’s private security force.

Its colorful past is as long as it is varied – with stories that sound more like fairy tales than textbook histories: in 1699 Captain Kidd buried treasure in its densely packed woods, First Lady Julia Gardiner Tyler was born in 1820 and in 1966, Jacqueline Kennedy Onassis was accused of theft at a glamorous dinner party hosted by Robert Gardiner.

Gardiners Island sits three miles off the coast off the Hamptons in Long Island, New York. The 3,318 acre private atoll has been owned by the same family for nearly 400 years

Only an American aristocrat like Robert Gardiner, the flamboyant last ‘Lord of the Manor,’ with a lineage going back four centuries could have told New York Magazine in a 1989 profile: ‘The Fords, the du Ponts, the Rockefellers, they are nouveaux riche!’

Today the property is permanently occupied by a caretaker and has been passed down to Robert Gardiner’s fiercely private niece, Alexandra Creel Goelet who has negotiated a conservation easement to ensure the preservation of the island’s delicate terrain, natural wildlife and historic structures.  

The land was originally purchased by Lion Gardiner in 1639 from the Montaukett Indians for the sum of ‘one large dog, one gun, some powder and shot, some rum and several blankets, worth in all about Five Pounds sterling.’ To further ensure his right to ownership, Gardiner secured a deed from the Earl of Stirling, who, at the time owned most of Long Island and Nova Scotia by royal charter from King Charles I. This gave him the ‘right to possess the land forever.’

Lion Gardiner became friendly with the Montaukett Indians for his support of the tribe during the Pequot War and in exchange for his loyalty, the Chief rewarded him with over 78,000 acres of land in Long Island. These land holdings were enough to sustain a very lavish lifestyle for generations of Gardiners to come. 

Robert David Lion Gardiner, the '16th Lord of the Manor' poses in front of his home on his private island, Gardiners Island. 'Sometimes I feel very feudal,' he said to the New York Times.  The eccentric millionaire was a local legend in East Hampton,  known for his unapologetic snobbery, proclaiming that the Fords, du Ponts and Rockefellers were 'nouveaux riche' in comparison to his family 

Access to Gardiners Island is completely restricted and by invitation only. Over the years, Robert Gardiner would throw extravagant parties and elegant picnics catered by famous French chefs. His wife, British socialite Eunice Bailey Oakes Gardiner stands in the background wearing a printed dress

Gardiners Island can only be reached by boat. It sits in the center of Gardiners Bay and is guarded by the family's private security force that patrols the island by boat

His descendants have successfully passed down his proprietary colony for 13 generations, yet none have spurred more copy than the late Robert David Lion Gardiner; the self-titled ‘16th Lord of the Manor’ who was the island’s most illustrious steward before his death in 2004. The larger-than-life, bon-vivant became something of an institution in the sleepy East Hampton community over the years, relishing in his unique family antiquity. With a flashy diamond pinky ring that he claimed was from ‘Kidd’s collection,’ Gardiner would make the three mile trek across the water on his boat he called the Jolly Roger, flying a yellow flag emblazoned with a skull and crossbones.

‘He was a mad, eccentric, romantic historian,’ said Richard Barons, senior curator of the East Hampton Historical Society to DailyMail.com. ‘The Lord of the Manor was really an important core to his existence, he played the role and he fed off the legend.’

Perhaps better suited for the conversation in Gilded Age parlor rooms than 20th century life, Gardiner was known to be a lovable but unapologetic snob. ‘He was a font of historical information,’ said Karl Grossman, a veteran Long Island journalist and acquaintance of Robert Gardiner to DailyMail.com. He mixed with actors and socialites, threw elegant parties for his friends, married late in life and was famous for speaking in a unique accent that combined New York’s sledgehammer consonants with a tony British lift. As Barons said: ‘He was the John Gielgud of the Hamptons.’

Over the years, Robert Gardiner gave a select group of journalists a tour of the island that has been passed down in his family since 1639. Flamboyantly dressed, and larger-than-life, 'he was the John Gielgud of the Hamptons,’ said Richard Barons of the East Hampton Historical Society

Sarah Diodati Gardiner (pictured left) purchased the island for $400,000 from a distant cousin when the property was threatened for auction in 1937. She was forced to rebuild the 1774 manor home in 1947 to the current 28-room Georgian style house (pictured background) after one of her tenants burned it to the ground while 'smoking in bed with a whore' said Robert Gardiner

Eunice Bailey Oakes Gardiner (left) poses wearing a tiara for LIFE Magazine with her husband Robert David Lion Gardiner in their ornate living room. The couple never had children, which meant left his niece, Alexandra Creel Goelet to inherit his property

Eunice Bailey Oakes Gardiner sits with husband Robert David Lion Gardiner for a 'casual dinner'  at their manor home on the island. Before being married to Gardiner, Eunice was rumored to have turned down a proposal from Orson Welles

Much of what little information the public has of the mysterious island today is because of the iconic Hamptons resident who invited a select handful of journalists over the years to peek behind the curtain into a bygone era of wealth.

Explaining in 2003 how his family maintained ownership of the enviable property, Gardiner said: ‘We have always married into wealth. We covered all our bets. We were on both sides of the Revolution, and both sides of the Civil War. The Gardiner family always came out on top.’ 

Captain Kidd's buried treasure on Gardiners Island

In June 1699, Privateer William Kidd was en route to Boston where he was called to court for charges of piracy. Looking for a safe place to store his treasure chests, Kidd sailed his ship into one of Gardiners Island’s natural harbors and asked Jonathon Gardiner for permission to bury his loot that contained ‘104 Golconda diamonds, 68 large and small rubies, 400 sapphires, over 600 uncut emeralds from Spanish mines and many ecclesiastical pieces,’ Robert Gardiner told a troupe of Boy Scouts on a tour of his island in 1971.

After Kidd's arrest, Gardiner was forced to return his loot to 'the Crown,' it was eventually auctioned off in 1701 and used to fund the Greenwich Hospital in London

In return for the favor, Captain Kidd gave Mrs. Gardiner a swatch of gold fabric (currently on display at the East Hampton Library) a sack of sugar and a chilling warning: ‘If I call for it and it is gone, I will take your head or your son’s head.’

Kidd never returned for his treasure. He was sentenced to death and hung twice in London after the rope broke on a first attempt. More than 300 years later, Robert Gardiner wore a ring comprised of three glistening diamonds on his pinky finger from what he called ‘Kidd’s collection’.

Advertisement

Karl Grossman, a veteran Long Island journalist and acquaintance of Robert Gardiner told DailyMail.com that: ‘Gardiners Island is a time-capsule of what all of Long Island was.’ It is the site of America’s first witch hunt, predating Salem, Massachusetts by 35 years, in 1658. In 1699, Captain Kidd, looking for a safe place to store his treasure chest, sailed into one of the island’s natural harbors and buried his treasure in the woods. Easily accessible from the Atlantic Ocean, Gardiners Island became a prime target for countless pirate plunders over the years after Kidd made his first landing. The family survived every single one.

During the American Revolution and the War of 1812, British fleets looted the island for livestock and other provisions. Subjugating the Manor House into barracks while the superintendent’s home was turned into a military hospital. The Gardiners ‘played nice’ with the red coats while Nathaniel Gardiner worked as a surgeon in the Continental Army, attending to Major John Andre before he was executed as an accomplice to the spy, Benedict Arnold.

‘My family was perfectly horrid. They straddled; one brother was for the Revolution, the other for the Loyalists,’ said Gardiner in a 1971 meeting of The Order or Colonial Lords of Manors, an exclusive club of members with royal land titles.

For many years, Gardiner’s Island thrived as a working farm. Indentured servants tilled the 1,000 acres of fertile land that produced beef, dairy, wool and wheat; all of which earned a profitable return at the local markets in Boston. In rare historical footage owned by The Suffolk County Historical Society, Robert Gardiner takes guests on a tour of his island. Pointing to a humble red painted wooden structure he says: ‘There! The boys lived there! The boys that were sold into service to us. Not black slaves but indentured servants, we gave the child’s parents some money and the child worked off the debt, they lived there.’

In a long line of celebrated family members, none can forget First Lady Julia Gardiner who was born on the island in 1820 and grew up in a world of Gilded Age opulence with a mother as an heiress to a large fortune. She was known as ‘the Rose of Long Island’ for her legendary good looks. As a young girl, Julia was the toast of debutante circles in New York, Boston and Washington D.C. It was only a matter of time before she caught the eye of President John Tyler, who was 30 years her senior and recently widowed. Julia quickly made the most of her short eight month stint as First Lady. She threw lavish parties, introduced dancing to the White House, insisted that Hail to the Chief be played every time the president entered the room and intensely lobbied for the annexation of Texas. ‘Sam Houston was in love with Julia Gardiner Tyler, which is why Texas is in the Union,’ said David Gardiner to the New York Times in 1964.

Robert David Lion Gardiner displays a gold chalice which he says was buried on the island by Captain Kidd in 1699. His diamond pinky ring, which was also passed down through the family for years was also part of Kidd's loot

In 1966, Gardiner threw Jacqueline Kennedy a lavish dinner party but things spiraled out of control when he accused her of stealing his wife's gold cigarette lighter. Eunice Gardiner sits at the head of the table in front of a portrait of herself wearing a tiara and emerald green gown.  Gardiner threw less parties over the years, telling The New York Times, 'We don’t have any servants on the island, so we can’t do anything very elaborate'

Robert Gardiner reenacts the time his  ancestor, John Gardiner was left for dead and trussed up against a tree by invading pirates in September 1728. 80 buccaneers plundered the island, slashing feather beds, slaughtering animals and breaking furniture while looking for coin. The Gardiners had hid the majority of their valuables in a nearby water well that still exists today

The Gardiner family started to split in the subsequent years after the Civil War. Richard Barons explains to DailyMail.com: ‘As time went on, part of the Gardiner family became what people call the ‘New York Gardiners’ of which Robert David Lion Gardiner was part of while the Gardiners out here became more agrarian.’ He continues: ‘As you can expect the ones who went to New York became bankers and boards of directors and they became very, very wealthy and they also married very well.’

In a 1999 interview with The East Hampton Star, Robert Gardiner confirms the sentiment: ‘We were always marrying money.’ After taxes began to exhaust the Gardiner fortunes, it was, ‘a necessary fact of life. We are survivors.’

Doomed by mounting debts and heavy inheritance taxes, Gardiners Island became difficult to sustain its expensive upkeep, in comparison to their New York City relatives, ‘They were what some people might call, ‘land poor,’’ said Barons. By 1937, the island was put up for auction. It was the first time in 300 years that the family became perilously close to losing their 3,318 acre heirloom. But in the eleventh hour, an affluent New York cousin named Sarah Diodati Gardiner came to the rescue and purchased the island for $400,000.

Sarah began leasing the property as a hunting reserve to wealthy patrons, most notably – Winston Guest. The aristocratic, polo-playing cousin of British Prime Minister, Winston Churchill turned the island into an adult playground. He built an airstrip and flew in guests for legendary shooting parties. Ernest Hemingway and Edward VIII the Prince of Wales, to name a few - who on a single day in killed 4,500 birds. Robert Gardiner later told Chip Dayton of Avenue Magazine how Hemingway ‘would get drunk at night, stagger outside and shoot guns toward Connecticut.’ Local boys were hired to flush pheasant while the men would have lunch in the field, brought to them by costumed servants.

At night, local girls from East Hampton were boated in for entertainment. Though married to New York socialite, C.Z. Guest; Winston was a notorious playboy. It was during one of these wild parties in 1947 that misfortune struck the island. The Manor Home built in 1774 was reduced to ashes after one of the visitors fell asleep smoking a cigarette. Referring to Winston Guest, Robert told a New York Times reporter on tour of the island that he: ‘burned the place down smoking in bed with a whore.’ Old habits die hard.

Gardiners Island played host to another significant guest in Mach 1966 -First Lady Jacqueline Kennedy. In what was intended to be a refined dinner party, unravelled into a farcical event that led to Robert Gardiner accusing Mrs. Kennedy of stealing. After eating, guests retreated to the wood-panelled den for coffee, cognac and conversation. According to C. David Heyman’s book titled, ‘Bobby and Jackie: A Love Story,’ Gardiner watched Kennedy ignite her cigarette with a gold lighter that belonged to his wife Eunice before she inexplicably slipped it into her purse. Stunned, Gardiner decided to reach for a cigar and ask the room: ‘Have any of you seen my wife’s gold cigarette lighter?’ Before directly inquiring his distinguished guest, ‘Did you, Mrs. Kennedy? I believe you were the last one to use it.’ Kennedy denied it and the lighter was never returned. Gardiner dined out on the gossip which spread like wildfire up Park Avenue, only to be hushed two years later by a threatening legal letter and a $5,000 check in the mail from Aristotle Onassis.

Born on the island in 1820, Julia Gardiner Tyler, known as the Rose of Long Island married President John Tyler, 30 years her senior. She threw lavish parties, introduced dancing to The White House and created a tradition that 'Hail to the Chief' be played every time her husband, the President walked in a room. In 1839, she 'scandalously' posed for a New York City department store advertisement which led to her being sent to Europe by her parents.  Later she became a champion of the Confederate cause during the Civil War

Eunice Gardiner was a British socialite and rumored lover of Stephen Ward the infamous Profumo Affair fixer. 'She knew Aly Khan. She stayed at Cliveden all the time,' said Gardiner to New York Magazine

For years, Robert Gardiner remained single, choosing to live with his mother in a 42-room mansion in Bay Shore, Long Island. Though he was quick to tell Dinita Smith in his New York Magazine profile, that there were ‘quite a number of love affairs.’ Explaining that his problem was that every girl he fell in love with was Roman Catholic: ‘My aunt was still alive, and she would have disinherited me.’

Gardiner finally married in 1958 to Eunice Bailey Oakes when he was 48-years-old. Oakes was a British, redheaded bombshell that became a muse for Christian Dior at an early age. As a teenager, she was romantically involved with Stephen Ward, the infamous Profumo Affair fixer that was found guilty of peddling prostitution. Before marrying William Pitt Oakes in 1952, Eunice reportedly turned down a marriage proposal from Orson Welles. ‘I was not the nouveau riche Oakes who lacked culture. I could sustain my collections. I was like Lorenzo de’ Medici. I could tell a fake French commode like a dealer,’ said Gardiner to New York Magazine.

Robert Gardiner was the last of his family’s namesake to own the island. His marriage failed to produce an heir to this wealth; something that only became a problem years later when he wanted to jilt his sister’s children out of their inheritance.

Gardiner commissioned Salvador Dali to paint a portrait of his wife. It was auctioned by Sothebys after his death in 2004 for an undisclosed amount

‘I was tested on my sperm, it was wiggling. My wife didn’t have children with Pitt Oakes. I was so in love with my wife I didn’t care.’ He implied to Smith of New York Magazine that he may have fathered a son while serving WWII in the South Pacific: ‘Unfortunately, there’s a bit of Tahitian blood in there. It’s married now.’ Getting frustrated at Smith’s suggestion that ‘it’ could be the potential heir, he said: ‘You never think of anything practical!’

Robert and his sister, Alexandra Gardiner Creel were left the island and a number of valuable properties around East Hampton by their wealthy aunt, Sarah Diodati Gardiner, who herself was unmarried and without children. The relationship between siblings quickly soured; and before Creel’s death in 1990, she told New York Magazine: ‘The island has always been a troublemaker, down through the generations. There is something about the 3,300 acres of pristine beaches, rugged cliffs and white oak forests that just seem to make people greedy.’

Today the current owner of Gardiners Island is Robert’s niece, Alexandra Goelet. She became the inevitable sole inheritor of the estate after her mother and uncle passed away - something Robert tried desperately to prevent before his death in 2004. In a last ditch effort to derail Goelet out of her inheritance, Robert attempted to legally adopt a distant Gardiner relative who was living in Mississippi at the time. He was unsuccessful.

Robert and his niece were estranged for nearly 20 years before his death, a relationship plagued by very public legal disputes. At one time, he accused the Goelets of trying to murder him. Things got catty. In an interview with New York Magazine, Gardiner recalls running into his niece at soiree held at the French consulate in New York City. He remarked: ‘My niece was wearing the most awful little light sapphires, badly set. My wife absolutely outshone her.’

It was these type of statements delivered with the same unequivocal confidence and elocution of a Shakespearian actor that lent to Gardiner’s larger-than-life legacy. He was an unflinching snob, prone to outlandish statements that betrayed an out-of-date sensibility. But regardless, he is remembered fondly. Karl Grossman tells DailyMail.com: ‘I am what you call a stereotypical New York Jew and I can smell anti-Semitism from a mile away and Gardiner was so open and friendly and warm, he was a good guy and that doesn’t do it justice.’

‘Like so many people who are amateur historians he was interested in what made things unique,’ said Barons to DailyMail.com.

Gardiners Island is nothing, if not unique. The sliver of an island has remained a unique time-capsule of unspoiled paradise – a benefit of long standing private ownership. ‘It’s unlike any place I’ve ever been to before, it’s truly magical,’ said Grossman to DailyMail.com. Freshwater ponds, rushing creeks, swamps and lush meadowlands unmarred by pesticides have become a sanctuary for wild turkey, deer and osprey. The estate stands as a unique time-capsule of nature completely unmarred.

Today, the famously private Alexandra Goelet has pledged to preserve the island in exchange for a 20-year easement from the East Hampton Township. Unlike her uncle, Goelet has chosen to stay out of the limelight. Leaving the Gardiner-curious public to satiate their interest by looking to the past. Speaking about Robert Gardiner, Barons said to DailyMail.com: ‘There was a sense of glorifying the past because there wasn’t much to cling to except the past.’ None cherished family history more than Robert Gardiner, and perhaps he said it best in an interview given to The East Hampton Star before his death: ‘The island is not only the trees, the birds, the black snakes, and the salamanders, and the hunting. It's the people too.’

Gardiners Island boasts 27 miles of untouched coastline, jagged cliffs, sandy beaches, 1,000 acres of farm land and the oldest white oak forest in all the North East. Robert David Lion Gardiner was the last of his family's namesake to own the property. Richard Barons of the East Hampton Historical Society said, 'he was very Gardiner-centric' 

The windmill was rebuilt in 1795 after the original one was destroyed during the Revolutionary War. It served as an important warning signal for East Hampton residents of potential pirate invasions by the position of its smock sails. The windmill along with the carpenters shed from 1639 (which is oldest existing structure on Long Island) still stand on the island today

Robert Gardiner guides a tour of his island on the back of a truck in 1974. Karl Grossman, veteran Long Island journalist and friend of Robert Gardiner said, 'He never ever exaggerated, frankly he has nothing to exaggerate, his story was already so fantastic'