I have a loved one with a severe intellectual disability. When I use that word, it's not directed toward him or people like him. So, I think thou dost protest too much. You can make a personal choice about whether you want to use a certain word, but don't immediately assume an ill motive or a character flaw with someone else because they don't feel victimized by such words.
Words aren't violence, no matter how much you bleeding hearts want them to be.
I don't doubt that you don't have malice when you say it. Like I said, give it some thought.