Freedom of speech is a right of American citizens guaranteed against government interference by the First Amendment. While the protection does not directly apply to actions against speech by individuals or non-governemntal organizations, the laws against violence apply. Clearly, the right of free speech is not absolute: incitement to violence and the like are not protected, for example.
Nevertheless, over the past year we have seen a number of examples of scheduled right wing speakers being disinvited by universities because of threats from the left wing. Dennis Prager’s appearance as guest conductor at several has been opposed because of his right-wing opinions. In Berkeley, the anti-right protestors actually resorted to rioting.
Whether or not Donald Trump had these examples or left wing intolerance, protests, and violence against right wing people (not all of whom are Nazis, KKK, or advocates of violence) in mind when he spoke after the violence at Charlottesville, it is clear that there has been violence advocated and perpetrated by “both sides” in our recent history. A “Free Speech” rally was scheduled to take place in Boston a week after Charlottesville. Even before Charlottesville there was much handwringing by the mayor, others, and the press about not wanting the event. The anonymous organizers went public, claimed that they were only interested in upholding the right of unpopular speakers to speak, and disinvited some scheduled alt-right speakers from their event. Counter-protests were planned; and the mayor, governor, police commissioner, and newspapers called for both sides to refrain from violence. The event took place without serious incident. The police decided that they needed to protect the rally organizers and speakers from potential violence by driving them away in police cars.
It seems to me that there is hypocrisy when Trump is excoriated for calling attention to the presence of violence-prone and peaceful people on both left and right, while others who acknowledge the fact receive no rebuke (since they are merely acknowledging what is common knowledge).
Clearly, Nazis and neo-Nazis profess allegiance to an ideology hostile to the United States. The KKK and other white supremacists set themselves in opposition to the Constitution and the Declaration of Independence. There is no justification for them to resort to violence, but they have not forfeited their First Amendment rights. Nobody is obligated to provide a forum for them or to listen to them, but nobody has a right to prevent them from speaking peaceably where they are invited. Similarly, the opponents of Trump and the opponents of the alt-right have no right to resort to violence, but they have the right to present their position peaceably.
The threat of violence, and worse still its use, strikes me as a serious challenge to our system of self-government. The legitimate way for citizens to achieve their goals is by petition and through elections, not by violence.