...because it was the banner of the armies that fought to keep slavery as an institution. I cannot IMAGINE why such an idea might develop.
It was not used for thousands of years as a symbol of peace for Eastern faiths. Therefore, it has little in common with the Swastika misconception