The last time he'd failed to advance beyond the second round at a major was at Wimbledon in 2008, when he was seeded third and lost to then No. 75-ranked Marat Safin, a two-time Grand Slam champion.
Rod Laver Arena wasn't at capacity as the match extended into the scheduled night session, and Djokovic's usually vocal Serbian fans were also not a big presence. Much of the crowd, sensing an upset, was behind Istomin in the fifth set, yelling "Come on Denis!"
When Istomin made a backhand winner down the line to break Djokovic to go up 3-2, he let out a roar and the audience stood and roared, too.
When Djokovic's last service return floated long, the crowd leapt up and cheered.
"First of all, I feel sorry for Novak," Istomin said, thanking his mother and coach, Klaudiya Istomina, in the crowd. "I was playing so good today. I surprised myself, as well."
He next plays No. 30 Pablo Carreno Busta.
Third-seeded Milos Raonic shrugged off a cold and a sore throat to extend his streak of reaching the third round at all seven Australian Opens he's contest with a 6-3, 6-4, 7-6 (4) over Gilles Muller, the Sydney International champion last week. He next faces No. 25 Gilles Simon.
Brisbane International winner Grigor Dimitrov, U.S. Open semifinalist Gael Monfils, No. 11 David Goffin, No. 13 Roberto Bautista Agut, No. 18 Richard Gasquet and David Ferrer all advanced.
Ivo Karolovic followed up his tournament record 84-game first-round win with a straight sets win over wild card Andrew Whittington.
In other women's matches, U.S. Open finalist Karolina Pliskova and No. 9 Johanna Konta carried their winning form from warmup tournaments into the third round.
Brisbane champion Pliskova beat Anna Blinkova 6-0, 6-2 and Sydney International winner Konta, a semifinalist here last year, beat Naomi Osaka to set up a third-round match against former No. 1-ranked Caroline Wozniacki.
Dominika Cibulkova beat Hsieh Su-wei and No. 14 Elena Vesnina advanced to a third-round encounter against U.S. qualifier Jennifer Brady, who saved five match points before beating Heather Watson.
Murray, a five-time Australian Open runner-up, returned to the practice court to allay concern over his ankle 14 hours tumbling to the court during his second-round win.