Barry Diller Reveals ‘the Truth’ About His 50-Year Romance with Wife Diane von Fürstenberg in Candid New Memoir.

The pair were together for decades before officially marrying in 2001.

Barry Diller is opening up about his marriage to legendary designer Diane von Fürstenberg.

In his new memoir, Who Knew (out May 20 from Simon & Schuster) the Chairperson of IAC (the parent company of Dotdash Meredith, which publishes PEOPLE) writes candidly about his romantic life, sharing with New York magazine in an excerpt, “While there have been a good many men in my life, there has only ever been one woman, and she didn’t come into my life until I was 33 years old.”

Diller, 83, addresses longtime scrutiny of his sexuality, writing, “I’m well aware that this part of my life has caused confusion and lots of speculation.”

“I never discussed my personal life, lowlight as it was, with anyone,” Diller writes. “Even though as the years went on I began to be realistic and understood that ‘everyone knows,’ I never wanted to make any declarations. So many of us at that time were in this exiled state, so stunted in the way we lived … Of course I could have declared my sexuality, come out as some others were doing, but I was among the many at that time who were too scared to do so.”

In the book, he also recounts his first impression of von Fürstenberg, 78, whom he met alongside her first husband, Prince Egon von Fürstenberg, in 1974. When they crossed paths later that year at a party for a mutual friend, theirs was an instant connection that “exploded into a romance as natural to us as breathing.” Calling it the “miracle of my life,” Diller admits that his relationship with von Fürstenberg “surprised us and everyone else.”

“Much of the speculation subsided when it was clear to all we couldn’t keep our hands off each other,” Diller writes. “We weren’t particularly conventional characters, and were so demonstrably in love both inside and outside the house.”

However, Diller says the pair split in 1981 after von Fürstenberg had a “momentary affair” with actor Richard Gere while he was working on the 1980 film American Gigolo. Diller, who at the time was heading up Paramount — the company behind the film — said that the romantic tryst made him “feel too much the fool.”

The pair went their separate ways and stayed apart for a decade before reuniting in 1991. They married in 2001 and “have spent 50 years intertwined with each other in a unique and complete love.”

In the memoir, Diller also writes about the family life he and von Fürstenberg built with her children from her first marriage, Alexander and Tatiana. Last year, both now-grown children participated in a documentary about von Furstenberg’s remarkable life and career, Diane von Fürstenberg: Woman in Charge.

“We weren’t just friends. We aren’t just friends,” he writes of their enduring romance.