"It is inconceivable that Moon would visit North Korea when Pyongyang has shown no signs of specific action toward denuclearization", said a high-ranking Japanese Foreign Ministry official. If Moon accepts, it would be the third round of such talks since their division in 1945.
A big one could be a demand to cease the US-South Korean war games that North Korea claims are preparation for invasion.
Kim announced his nuclear program, which the communist state claims is needed for self-defense, was complete even as he extended an olive branch to the South in his New Year's address by agreeing to participate in the Olympics.
Despite Pence's remarks, the current administrations in Washington and Seoul have at times adopted contrasting tones when it comes to North Korea.
Perhaps the most alarming exchange came in August when Trump said North Korea "will be met with fire, fury, and, frankly, power, the likes of which the world has never seen".
The North Koreans were to fly back to Pyongyang later Sunday on a private jet, wrapping up a three-day trip.
As they shook hands, the North and South Koreans spoke briefly.
Also at that game: the famed cheer squad from North Korea - more than 100 identically dressed women who sing, perform choreographed moves and cheer on their team.
But the prospect could sow division between the dovish leader, who has long argued for engagement with the nuclear-armed North to bring it to the negotiating table, and US President Donald Trump, who past year traded personal insults and threats of war with Kim.
On Friday, Pence said the US will be working to come up with a peaceful solution, but added that it will oppose talks between South and North Korea that don't involve the North dismantling its nuclear program.
You could cut the tension with a ginsu knife as Mike Pence and his wife Karen sat observing the Winter Olympic festivities while Kim Yo Jong appeared to be every bit as bitter cold as the Pyeongchang clime they overlooked.
Though South Korea is hosting the Winter Olympic Games for the first time, the nation has some experience in the world of global athletics, having hosted the 1988 Summer Games in Seoul.