The South Americans, who fell to Cameroon at the last-16 stage in Italy in 1990 with a team featuring the likes of Rene Higuita and Carlos Valderrama, had not qualified for soccer's global showpiece since 1998.
They were deserved winners in an entertaining encounter in Brasilia and have now won two games at a finals for the first time in five appearances.
After a tight first half the game exploded into life with three goals in nine minutes midway through the second, starting when Colombia's James Rodriguez beat substitute Didier Drogba to the ball and thumped in a header from a 64th-minute Juan Cuadrado corner.
Juan Quintero ran clear to make it 2-0 after 70 minutes after Rodriguez had robbed Serey Die in midfield but Gervinho cut in through three defenders to score an excellent individual goal three minutes later to halve the deficit.
Colombia held out reasonably comfortably, though, to the delight of thousands of their yellow-clad fans thronging the national stadium, and top the standings with six points ahead of Ivory Coast on three.
Greece and Japan, who meet in Natal later on Thursday, are pointless after their opening game defeats.
"We have had two wins with players who are playing their first World Cup," Colombia coach Jose Pekerman said at a news conference.
"It's very positive for us but we can't get ahead of ourselves," added the Argentine, who was in charge of his native country when they beat Ivory Coast 2-1 in the group stage at the 2006 World Cup in Germany.
"I'm very happy, this is a united group. We never lack that courage and valour.
"We're happy to have defeated such a good rival. We have a lot of young players, this is going to help them grow."
RAPTUROUS RECEPTION
Hordes of travelling Colombia fans transformed the giant arena in the Brazilian capital into a sea of yellow, with only a handful of patches of the orange of Ivory Coast, and gave a rousing rendition of the national anthem before kickoff.
After a cagey opening few minutes Colombia started to push forward, with midfielder Juan Cuadrado at the heart of their attacks.
It was Cuadrado's ball out to the left that released Rodriguez in the 28th minute but his fine cross was scuffed wide by striker Teofilo Gutierrez unmarked in front of goal.
Ivory Coast had their first genuine chance three minutes later when lively fullback Serge Aurier cut inside his marker and his low shot forced a good save from Colombia goalkeeper David Ospina.
Cuadrado again looked the most dangerous player on the pitch in the second half and he skipped into the Ivory Coast penalty area in the 59th minute before blasting an angled shot that clipped the gloves of keeper Boubacar Barry and shaved a post on its way to safety.
Ivory Coast talisman Drogba replaced the ineffectual Wilfried Bony a minute later to a rapturous reception from both sets of fans but was at fault for the opening goal and made little impact during the rest of the game.
After Colombia's quickfire double and Gervinho's response, Ivory Coast pressed hard for an equaliser, Salomon Kalou, another substitute, shooting straight at Ospina when well placed five minutes from time.
"It was very tough, but the team showed class and personality, and we scored at the right time," Quintero said.
"At the end, we gave them the ball, but that's normal because you can't overrun a team like Ivory Coast. We ended tight and we stuck to our tactics."
Ivory Coast coach Sabri Lamouchi said his team had not deserved to lose and a draw would have been a fair result.
"We need to recover and be proud of ourselves," he told a news conference.
"I feel we lost unfairly today and we need to fight for a victory against Greece," added the Frenchman.
"A very experienced Colombia team waited for us to make mistakes and we did make mistakes."
Ivory Coast have never made it to the knockout round, finishing third in their group on their debut in 2006 and again in South Africa four years ago.
Colombia play Japan on Tuesday in Cuiaba, when Ivory Coast take on Greece in Fortaleza.
