Second-seeded Ivanovic looked headed for a routine victory in the matchup between former world No. 1s, winning the first set in 35 minutes and holding match point at 5-3 in the second. But Williams fought back to break Ivanovic twice to capture the second set and force a decider.
Williams opened the third set with two double faults to drop serve, however, and played under pressure the remainder of the match. She held two break points in the final game before Ivanovic won four straight points to close out the match.
Williams was playing in her first final since winning her 44th career title at Luxembourg in October 2012. The seven-time Grand Slam champion's ranking has fallen to 47th as she has battled injuries and Sjogren's syndrome, a fatigue-causing autoimmune disease, over the past two years.
Williams had an 8-1 record against Ivanovic heading into the match, with the Serb's only win coming at the 2008 Australian Open.
"She played so well. We've played a lot of great matches, and this is another one," Williams said. "I just kept fighting despite the odds. She was playing so well."
Ivanovic, a former French Open champion, captured her 12th WTA singles title and her first since Bali in 2011.
"It was very tough," Ivanovic said. "Venus showed again what a great sport she is in what she's done today and over all these years."
Williams' sister Serena is playing later in the day against Victoria Azarenka in the final of the Brisbane International, another Australian Open tuneup event. Venus said the sisters made a pact before the finals that they would both bring home the trophies Saturday.
"I'll just be rooting for Serena tonight," she said. "I couldn't keep up my end of the bargain. I just played too tough of a player but hopefully she can make it at least one win today."