There are plenty of celebrities with famous doppelgängers. There's Mila Kunis and Sarah Hyland, Leighton Meester and Minka Kelly, Nina Dobrev and Victoria Justice…the list of similar-looking brunettes goes on and on. As it turns out, Hilary Swank and Jennifer Garner fit the bill, as well. The Oscar winner and star of FX's new show Trust stopped by Conan yesterday, where she revealed to the host that she gets mistaken for Garner all the time.

Apparently, this can get uncomfortable for both Swank and Garner, because zealous fans don't take "no" for an answer. “People get angry at me because they think I’m just trying not to sign an autograph or take a picture," Swank told Conan O'Brien. "They’ll be like, ‘Oh, God, I’m such a big fan, I loved you in Alias.'"

Sometimes it's just easier for Swank to play along than correct Garner fans who are so obsessive they have no sense of boundaries yet can't actually recognize their idol. “They actually said, ‘Don’t be a b**ch,’" Swank said of some fans. "And I didn’t want to give her a bad name, so I took a photo with them and maybe someone will say some day, ‘That’s not Jennifer Garner.’”

The case of mistaken identity isn't an issue for just Swank, however. Garner, too, is often confused for Swank. “I told [Jennifer] and she said, ‘Oh, my God, I get that all the time too!’" Swank recalled. "I said, ‘As Jennifer Garner?’” Swank quipped. “And she said, ‘No, as you!’”

If there is any defense for these fans, it may be that Swank has been out of the spotlight for the past three years and, presumably, harder to recognize than ever (though she seemingly hasn't aged at all in that time). In that time, Swank has been tending to her father, who successfully underwent a lung transplant. "I took three years off to be with him and help him through that life journey," she told O'Brien, adding that now he's "alive and thriving and well." "It’s the hardest surgery a person can undergo, and I thank all the people that make themselves a donor—it saves lives."