Lord Frost warned the EU will demand a price from Labor in return for changes to the Brexit deal.
The former chief negotiator said Brussels is a “tough” negotiator and that Sir Keir Starmer's party did not appear to appreciate this.
It comes as the Prime Minister has pledged to strike a better trade deal than Boris Johnson's but insisted there will be no return to the single market, the customs union or freedom of movement.
The Tory peer told GB News: “I am fearful that they want to take us back closer.
“I don't think they can rejoin imminently, though I'm sure that the leadership of the Labor Party would like it if they got the chance.
“But I do wonder whether Labor really thought this through. They talk very glibly about smoothing the edges off the deal, getting closer to the EU; you always have to pay for doing that.
“It's very easy to wander around Europe getting warm words from the member states. It's the Commission that represents the EU's interests.
“They're tough negotiators, and if you want to change things, you're going to have to accept EU court, EU law, and subordination to EU foreign policy. I don't know whether Labor really gets that yet.
“I'm sure many of them wouldn't mind that much if they got [free movement]. I don't think they really believe in controlling migration, they can just see that it's a really difficult political issue for them and a lot of their voting base.
“I don't think the EU is willing to divide up their single market. They've never really shown themselves willing to and I don't think Labor have really understood the task they've taken on. I think they'll find it extremely difficult.”
It comes as Sir Keir will use the Nato summit to help “reset” the UK's relationship with its European neighbours.
As is usual for Nato summits, the Prime Minister will be joined on the US trip by the Foreign Secretary and Defense Secretary, but he is also taking Nick Thomas-Symonds, who has been given the newly-created job of minister for European relations.