The third set stayed on serve but in the tie-breaker, Nadal prevailed 7-4.
Nadal would go break point down in the next service game but went with a classic tactic to save it.
Nadal, who 12 months ago lost an epic final against Roger Federer, will next face Damir Dzumhur after the Bosnian ended the tournament of Australian Millman.
Things will get harder for her now, against fellow Ukrainian and No. 4-seeded Elina Svitolina, who had a 4-6, 6-2, 6-1 win over Katerina Siniakova.
Ostapenko is playing her first Grand Slam with her new coach, Australian David Taylor.
Another intriguing match-up pairs Denis Shapovalov with French 15th seed Jo-Wilfried Tsonga, a losing finalist at Melbourne Park a decade ago when the Canadian was eight.
Another of Tsonga's array of shots during the final set included one from between his legs for a victor.
Troicki saved one match point at 1-6 in the tiebreak with a deft volley but Kyrgios punched a backhand into the corner a point later to complete a good night's work and supply further evidence that he is harnessing his unique talent. Cilic overpowered Joao Sousa 6-1, 7-5, 6-2 to advance to the third round while Kyrgios had it a little tougher against veteran Troicki but still prevailed 7-5, 6-4, 7-6.
Fett is ranked 119th and was making her first appearance in the main singles draw of a Grand Slam tournament. She served a great serve down the T - it was just slightly out.
Two days after beating Venus Williams in the first round, Belinda Bencic is out of the Australian Open.
Tsonga, popular Down Under after his final appearance against Novak Djokovic 10 years ago, found some extra zip on his shots to level the match but the fearless Shapovalov continued to let rip at every opportunity to race through the third set in 26 minutes.
Former victor Jo-Wilfried Tsonga fought to the bitter end to overhaul rising Canadian youngster Denis Shapovalov in a five-set thriller. The man from Mallorca reached the third round with a 6-4, 6-4, 7-6 win. Top-seed Rafael Nadal and World No 3 Grigor Dimitrov progressed to the second round with easy victories while top American contenders were sent packing after shock defeats in the first round. She beat two top-10 players to take the trophy.
Vandeweghe went out next as things really started to go insane for the United States women, who eventually lost nine out of 10 first-round matches on a bleak opening day of the Australian Open.
For a place in the round of 16, the Croatian is set to face the United States' Ryan Harrison, who eliminated Uruguay's Pablo Cuevas 6-4, 7-6 (7-5), 6-4.
Frenchman Tsonga roared back for a 3-6, 6-3, 1-6, 7-6, 7-5 victory, the match lasting three hours and 37 minutes in soaring temperatures.