People threw punches, hurled water bottles and used pepper spray during the violence as riot police were deployed to try to disperse the crowds. At least eight people were hurt.
Among the three dead were two police officers who were the pilot and passenger on a helicopter that crashed near the scene.
The other death was a 32-year-old woman who had been crossing the street when she was struck by a car that hit a group of anti-Nazi campaigners also leaving 26 people injured.
The male driver has been arrested.
The rally by far-right activists was organised to protest against Charlottesville’s decision to remove a statue of Confederate General Robert E Lee from the city.
Speaking about the disorder, US President Donald Trump said he condemned “in the strongest possible terms” what he called an “egregious display of hatred, bigotry and violence on many sides”.
He added: “What is vital now is a swift restoration of law and order and the protection of innocent lives.”
Some criticised Mr Trump for his seeming reluctance to specifically blame the white supremacists.
Republican senator Cory Gardner said: “We must call evil by its name. These were white supremacists and this was domestic terrorism.”
Democratic senator Adam Schiff said: “(The President) needs to speak out against the poisonous resurgence of white supremacy – there are not ‘many sides’ here, just right and wrong.”
Charlottesville mayor Michael Signer had said he was disgusted the white nationalists had come to his town and blamed the US leader for inflaming racial prejudices during his presidential campaign last year.
Mr Signer said: “I’m not going to make any bones about it. I place the blame for a lot of what you’re seeing in America today right at the doorstep of the White House and the people around the president.”
The fighting prompted Charlottesville city manager Maurice Jones and interim county executive Doug Walker to simultaneously declare local states of emergency for the two jurisdictions.
Virginia governor Terry McAuliffe declared an emergency as police struggled to manage the two groups.
Up to 6,000 white nationalists had been expected to attend the “Unite the Right” rally.
TV pictures showed one of the fights erupting when the anti-Nazi protesters approached the white supremacists with what looked like a wooden banner.
As they did so, they were met by a wall of men dressed in body armour and helmets.
Other footage showed rivals on either side using hand-made shields as they fought each other.
Police said some people in the crowd used pepper spray.
Initially it appeared as though police were not getting involved but the the Virginia State Police have since posted pictures of heavily armed officers carrying out arrests.
The violence prompted First Lady Melania Trump to call for an end to the clashes, tweeting: “Our country encourages freedom of speech, but let’s communicate w/o hate in our hearts. No good comes from violence. #Charlottesville”
The clashes came after a federal judge ordered Charlottesville authorities to allow a weekend rally of white nationalists and other extremists to take place.
The rally was organised by right-wing blogger Jason Kessler, who said that he was saddened people had been hurt and that the driver of the car “did the wrong thing”.
The Southern Poverty Law Center had said it was expecting a range of far-right groups to attend, including the National Socialist Movement, the League of the South, as well as “various anti-government, Patriot, militia, Klan, III Percenters and anti-immigration groups”.