The Australian star previously declared Spacey - whose career went into freefall after he was accused of sexual misconduct by a number of men - was "handsy" when they worked together on 1997's L.A. Confidential.
Explaining his reluctance to describe himself as a victim, Pearce told the Hollywood Reporter's Awards Chatter podcast: "Even though I probably was a victim to a degree; I was certainly not a victim by any means to the extent that other people have been to sexual predators."
He recalled initially brushing off Spacey's alleged advances as "nothing".
The 57-year-old actor was with then-wife Kate Mestitz filming the movie in Los Angeles and recalled telling her he only felt "safe" if a "prettier" co-star was on set too.
Pearce told her: "The only days I feel safe are the days when (Simon Baker) is on set because I'm dumped like a hot potato, and (Kevin) focuses on (Simon) because he was 10 times prettier than I am."
In 2017 when Anthony Rapp came forward to accuse Spacey of making sexual advances towards him when he was just 14, Pearce "broke down and sobbed" when he realised how much he had been affected.
"I was in London working on something, and I heard (the reports) and I broke down and sobbed, and I couldn't stop," he said.
"I think it really dawned on me the impact that had occurred and how I sort of brushed it off and how I had either shelved it or blocked it out or whatever. That was a really incredible wake up call."
The former Neighbours actor admitted he has since "had a couple of confrontations" with Spacey that "got ugly" but didn't share details.
Spacey has denied allegations of misconduct and acquitted of all of the nine sexual assault charges he faced in the UK in 2023.
