It looks like things are going to get worse for Prince Andrew before they get better. According to the Times of London, the disgraced member of the royal family was “kicked out” of his offices at Buckingham Palace in the wake of the controversial BBC interview he gave recently, in which he attempted to explain his relationship with the late Jeffrey Epstein.

Per the report, Andrew and his staff were notified that they would no longer be welcome to use the palace as a headquarters, and would have to find another place to “do business.” The bold move comes just days after Andrew announced that he would be officially stepping away from his duties as a royal.

Andrew came under fire after it was revealed that Andrew was closer with Epstein than he originally let on. He reportedly stayed at Epstein’s New York mansion on a number of occasions, which is where some of Epstein’s most severe crimes are said to have taken place. He was also a guest on Epstein’s infamous private island, as well as at his Palm Beach residence. To make matters worse, Andrew reportedly continued his friendship with Epstein even after the billionaire was charged with sexual assault of a minor.

“It has become clear to me over the last few days that the circumstances relating to my former association with Jeffrey Epstein has become a major disruption to my family’s work and the valuable work going on in the many organizations and charities that I am proud to support,” Andrew said when announcing his departure from royal life for what he called “the foreseeable future.” And based on today’s report, that departure might be more permanent than originally thought.

Related: What Prince Andrew Stepping Back Means for Future of the Royal Family