Free Speech Doesn’t Protect Nazis. It Protects Us From Nazis
It is probably true that the value of some speech is less than the cost of the harm it imposes. But free speech advocates don't defend the speech rights of Nazis because they believe that Nazis have anything valuable to contribute to a marketplace of ideas. They defend the rights of Nazis because Nazis with the freedom to speak can cause less harm than Nazis with the power to regulate speech.
Narrowing the scope of free speech protections to accommodate limitations on hate speech, or to ban Nazis, or to shut Milo Yiannopoulos up, means reducing the scope of the individual right and expanding the power of the state to regulate speech. In order to favor expanding the power of the state to regulate speech, you have to trust the state to wield that power judiciously, and not to abuse it or use it vindictively or excessively.
Before you empower government to police speech that is hateful or offensive, or speech that is deemed violent or harmful, then you have to consider the possibility that it will not be your sensibilities that determine which speech is beyond the pale.