Al Fayed, who was close friends with Jackson, danced with children after unveiling the statue of the "King of Pop" before Fulham's home match against Blackpool.
The statue outside Craven Cottage has divided opinion, but Al Fayed launched a passionate defence of his move to commemorate his friendship with Jackson, who died in 2009.
"Why is it bizarre?" he said. "Football fans love it. If some stupid fans don't understand and appreciate such a gift, they can go to hell.
"I don't want them to be fans. If they don't understand and don't believe in things I believe in, they can go to Chelsea, they can go to anywhere else."
Chelsea, the defending English champion, is the nearest Premier League team to Fulham, with both clubs located in southwest London.
"People will queue to come and visit it from all over Britain and it is something that I and everybody else should be proud of," Al Fayed said.
The statue had been slated to be erected outside Harrods department store, but those plans were shelved when Al Fayed sold the business last year.
The singer attended a Fulham match in 1999.
The game on Monday was not a thriller, Fulham beating Blackpool 3-0.
