I don't know about other people, but I live by principle - not appetite. It has taken me a lifetime to realize that most people live by appetite, i.e. doing what makes them happy.
I never really lived in a world where I was allowed to be happy or do things that brought enjoyment. I also faced extreme abuse for about 2/3 of my life, so got accustomed to it. So it doesn't really worry me being ostracised. There is nothing that anyone can do or say to me that hurts me or scares me anymore. I am completely on my own. I have nothing. There's nothing that anyone can take away from me.
Nobody would notice whether I did something moral or immoral. That's because I have very little contact with people in real life. I don't have a job other than writing on the web. I have very little connection with others, so technically, if I wanted to be immoral, there would be nothing to lose. Nobody would notice.
I'm not, though, because I care what I think of myself. I care how I live my life. If nothing else, when I die, although in complete anonymity, I can think to myself that even though it doesn't matter, I lived by principle, and I never did the things that were destructive to others.