It doesn't sound stupid at all.
Do what you feel is right, if you really want to be with your ex again, just make sure you're prepared to deal with him cheating on you again, because it may happen, despite the words he may say to make you think otherwise.
As for your new boyfriend, don't just break up with him because of his being an outcast, if you really like him, just ignore the fact that your friends hate you for it. I think its terrible for them to think that way.