The formal meeting between Moon and the delegation was held the day after the two nations' athletes marched under a unified flag at the opening ceremony of the Olympics on Friday.
"Make no mistake about it, the United States of America has viable military options to deal with a nuclear threat from North Korea but, that being said, we hope for a better path", he said.
Already there have been concerns that North Korea's overture of the sports diplomacy is an indirect attempt by the regime to wedge a rift between the US and South Korea.
Pence and his wife attended the ceremony with the father of Otto Warmbier, an OH native who died after being held captive in North Korea.
"State Affairs Commission Chairman Kim Jong-un invited President Moon Jae-in to visit the North through a personal letter delivered by special envoy Kim Yo-jong", Lee said during Sunday's lunch, according to a text of his remarks provided by his office.
But if the Trump administration objects, it "will put the U.S.in a very hard position in terms of public relations in South Korea", Mr. Han said.
Vice President Mike Pence is facing backlash for his staunch efforts to ignore North Korean officials at the Winter Olympic Games, even as the two Koreas continued their temporary truce, marching and competing as one team.
But Moon has also said that he would be willing to meet Kim only if he received assurances from the North that it would help resolve the crisis over the North's nuclear weapons program.
North Korean leader Kim Jong Un invited South Korean President Moon Jae-in to Pyongyang "at an early date". On Friday, Moon pointed to the joint Korean women's hockey team - an Olympic first - as a starting point.
Moon responded by saying he hopes the right environment will be created so the meeting between the two leaders could occur, an apparent reference to demands by the USA and other allies that the North abandon its nuclear weapons program.
A White House official said although Moon did not discuss the invitation to Pyongyang with Mr Pence, the South Korean president made it very clear that only when North Korea actually starts to take steps to denuclearise would anyone even consider beginning to take the pressure off.
In its efforts to bring North Korea to the Pyeongchang Games, South Korea paused joint military exercises with the US and relaxed a series of sanctions, including on individuals like Ms. Kim.
If Moon accepts Kim's offer to come to the North, it will be the third summit meeting between the two Koreas.
Pyongyang conducted its largest nuclear test previous year and said it had developed a missile capable of carrying a warhead to the United States. North Korea has said its nuclear weapons are not for bargaining away.
They were on both sides of the ice, chanting back and forth.
The most recent was in 2007 when then South Korean President Roh Moo-hyun traveled to Pyongyang to meet with Kim Jong Il, the current leader's father.