After recovering from Nishikori's strong start on the Caja's Magica's new red clay courts, Federer looked as if he'd found his stride as he roared back in the second set.
But Nishikori reasserted his ground game in the decider after breaking the defending champion for a 3-1 advantage and held on for the surprising victory to reach the quarterfinals.
"He was my idol and to beat him was one of the goals for my tennis career," the 16th-ranked Nishikori said. "Beating Roger is, yeah ... I need a couple of days to celebrate."
Federer's early exit leaves Nadal as the apparent front-runner after world No. 1 Novak Djokovic bowed out on Tuesday. Earlier, Nadal cruised into the quarterfinals with a 6-2, 6-3 victory over Mikhail Youzhny.
Serena Williams and Maria Sharapova also reached the quarters in straight sets.
Federer came to Madrid after a seven-week layoff. The Swiss star said the gusting winds hurt his game, but he gave all the credit to Nishikori.
"He was the better player today for sure," said Federer, who besides winning here last year had also won in 2006 and 2009. "I was lacking control from the baseline and that carried through from start to finish. Overall I'm disappointed with my play."
Federer said he will not change his plans, as he will now train for Rome and then the French Open.
This was Nishikori's fourth win over a top-five opponent. He beat world No. 1 Djokovic in 2011 and has beaten David Ferrer twice. Nishikori will play Pablo Andujar after he advanced when fellow Spaniard Daniel Gimeno-Traver retired.
Nadal's commanding win over Youzhny featured an entire array of winning shots.
Having already established control by stealing Youzhny's second service game, Nadal kept a point alive with a deft no-look backhand flick before nailing a winner down the line to hold his serve and delight his fans.
The fifth-ranked Spaniard then broke Youzhny again by dinking the ball over the net while on the run, giving it so much backspin it twisted away from the Russian.
Nadal, who is trying to reach his seventh straight final since returning from injury, improved to 28 wins and two losses on the year. He will next play either Ferrer or Tommy Haas.
Nadal said he was still feeling pain in his right knee that had sidelined him with a nagging case of tendinitis.
"Sometimes I have pain," Nadal said. "I try to avoid talking about it as much as possible, whisper it. With respect to my rivals it's not the moment to analyze how I am."
Also, Jo-Wilfried Tsonga rallied back to oust Fernando Verdasco 4-6, 6-3, 6-2, while Tomas Berdych defeated Kevin Anderson 7-6 (5), 7-5, to make the final eight.
In the women's event, the top-seeded Williams dispatched Maria Kirilenko of Russia 6-3, 6-1, after Sharapova put away Sabine Lisicki of Germany 6-2, 7-5.
Despite slowing her usually blistering serves, Williams' dominant performance made Kirilenko wither after she dropped the first set with one of several unforced errors.
"I just felt really relaxed, like I was taking my time," said Williams, adding that she went for precision over speed on serve.
The defending champion battered Kirilenko with her strong return game, working the 12th-ranked Russian back and forth on the baseline with her precise forehand drives.
"I had been missing my serve when I was hitting it harder, so I took some pace off," Williams said. "It was easy because when you take off pace you don't hit as hard and don't exhaust as much energy. And hitting a serve or a winner are both fun."
Williams will take on local favorite Anabel Medina Garrigues, who reached the quarterfinals after Yaroslava Shvedova withdrew with a right arm injury.
Sharapova will face Kaia Kanepi next after she defeated Daniela Hantuchova 6-3, 6-4.
Ekaterina Makarova of Russia beat Marion Bartoli 6-3, 6-2 and will play either Sara Errani or Varvara Lepchenko. Angelique Kerber of Germany meets former No. 1 Ana Ivanovic also in the quarterfinals.