Stephens, who returned to competition at Wimbledon after nearly a year off because of a foot injury, recovered from a second-set meltdown to end ninth seed Williams's hopes of reaching a third major final this year.
She will meet Madison Keys, who demolished CoCo Vandeweghe 6-1 6-2, in an all American final.
"I have no words to describe what I'm feeling what it took to get here. The journey I took. I have no words," said Stephens, who underwent foot surgery last February.
"If someone told me I'd make two semis and a grand slam final this year I would have passed out, which is what I feel like doing now.
"I don't know how I got here. Hard work, that's it."
The 24-year-old Stephens, 13 years younger than Williams, paid tribute to the seven-time grand slam champion.
"I'm honestly just honoured to play at the same time as her. One of the greatest to ever play our game. She's one of the greatest competitors and I'm honoured to share the court with her and glad to play at the same time as her.
"For American tennis there's no question marks. The proof is in the pudding. American tennis, here we are."
Despite her lack of match play this year, Stephens began confidently and broke for 3-1 when Williams netted a routine forehand and then went on to win the last three games of the set as the 37-year-old strangely appeared to struggle with serve.
Williams was in danger again on her first service game in the second set but she managed to hold and broke in the next game thanks to a Stephens double fault.
She then held to love to stamp her domination on the second set and moved 4-0 up with a lovely service return followed by a routine volley as Stephens showed she was clearly struggling and lost the set to love when she hit yet another unforced error.
The 24-year-old, however, got back on track in the decider, taking a 1-0 lead on Williams serve when she netted an easy volley.
She saved two break points in the following game but Williams levelled for 2-2 when Stephens's forehand sailed wide.
But at the end of a seven-minute game, Williams netted a routine volley again to give her opponent a break for 4-3.
Williams broke straight back with a service return winner and she saved a break point to hold for 5-4.
But as Williams struggled with her volley, Stephens broke for love in the 11th game and followed up on serve to close it out as Williams netted a backhand.
"It was definitely a contrast of play," Williams said of the roller-coaster match.
"You know, I continued to play aggressive and continued to play the kind of match that it takes to win. Just made too many errors there at the end."
