As he arrived in Brussels Mr Davis said it was “time to get down to work and make this a successful negotiation”.
But despite this latest round of talks lasting for four days, Mr Davis returned to the UK after a few hours, leaving officials to press ahead with negotiations until he returns on Thursday.
A picture of Mr Davis sitting opposite Mr Barnier, the EU’s chief negotiator, also caused a stir – with the Brexit Secretary having no notes or papers in front of him while Mr Barnier had a thick folder of papers.
Sky News has been told Mr Davis did have notes, they were just in his bag.
Jonathan Haslam, Number 10 director of communications under former prime minister John Major, told Sky News he was “surprised” Mr Davis chose to head back so soon.
He said: “It sends out the wrong signals to the EU and also opens up an avenue where people can say ‘why is he rushing back?’
“Is this to carry on manoeuvres? Is he worried about his position back home? We know divided governments, divided parties, don’t ring through in the long term.”
Liberal Democrat Brexit spokesman Tom Brake said: “We have less than 20 months of Brexit talks left, yet David Davis has skulked back to the UK after just half a day.
“He didn’t have any position papers with him because this Government has no agreed Brexit position.
“This is a Government with no papers, no plan and no time for the most important negotiations of a lifetime.”
But the Department for Exiting the EU said there was nothing unusual in Mr Davis leaving when he did.
A spokesperson said: “As with any other international negotiation, those talks have now moved to technical working level discussions where over 90 UK officials will meet with their counterparts in the Commission enabling multiple conversations to take place at once.
“David Davis and Michel Barnier will be updated throughout the talks, before returning to negotiating table later this week. This approach is entirely consistent with all international negotiations, including the G7 and G20.”
Mr Barnier offered Mr Davis a “warm welcome” and said the latest round of discussions would “delve into the heart of the matter”.
Mr Davis said his priority for this week’s talks is citizens’ rights, with a new push to lift uncertainty for the three million EU citizens living in the UK and one million Britons living in the EU.
“We made a good start last month, and this week we’ll be getting into the real substance,” he said.
Also on the agenda this week are the contentious issues of the financial settlement for the UK leaving the EU and the arrangements for the border between Northern Ireland and the Irish Republic.
Meanwhile, Theresa May will remind her ministers that Cabinet discussions should be kept private following a series of damaging leaks.
The Prime Minister will tell members of her top team they should keep quiet about the content of meetings and focus on getting on with the job of governing.
The bid to instil discipline comes after alleged comments by Chancellor Philip Hammond were leaked to the press, with the disclosures culminating in an unnamed minister telling one newspaper Mr Hammond “is trying to “f*** up” Brexit.