Interesting points and that's the difficulty - I wouldn't want to define a criteria, especially since I myself am not in the position to personally comment on the feelings that these evoke. I have seen a suggestion that one framework which we could use to examine these monuments is basically "What are they best known for?". This doesn't provide a clear-cut answer of course, but provides a guide to the debate.

I think that perhaps a better way of approaching the examples you give might be to ask not if these monuments should be torn down, but if we (or, specifically, Americans) are taught and know enough about these figures so as to think about why they might be problematic.

I don't mean to sound like I'm copping out of giving a proper answer, but truthfully the point I am making is not to define what is acceptable and what is not, but to counter the argument that historical statues should be untouchable. If we accept that, then we can hopefully have honest and reasonable discussions and decisions, involving the view of those who traditionally have been sidelined, about our public spaces and how we use them to commemorate and remember.