I think this is doable, but because realism is being asked for in the portrayal of morality then it also must be asked for in the portrayal of psychology. Antagonists who are evil just for evil's sake do not account for the types of evil that exist in our world. Really, "evil" people are either mad - in which case are they morally responsible?- have some awful biological/social deterministic element that causes them to act a certain way - in which case they are tragic - or their values are alien to the values of our cultures - or other cultures - in which case they are simply different.
On this last point - alien values - a moral realist might argue I'm simply pleading cultural relativism, and I think that is the point I'm making, but I strongly doubt our ability to question our values in an objective manner and then assert their goodness over alien values that are our opposites. This is not meant to defeat moral realism, but merely provide practical skepticism to it.
I worry I'm rambling and not making myself clear. In a standard Heroic game I would allow evil antagonists to reach cartoonish levels of evil. They'll twist their mustaches and tie women to train tracks just for the sake of doing this activity.
In a morally ambiguous game I expect an antagonist to have a more relatable goal or something that is self-serving, but in contrast to what the players' characters want. If the antagonist kidnaps one of the PC's girlfriends he doesn't tie her to the train tracks - he does so to get a ransom OR if he does tie her to the train tracks, it's only because the PC killed the antagonist's lover and so he's claiming vengeance. This doesn't make the antagonist's action good, but we can understand their motivation and maybe even empathize with what they feel.
In future games I'd like to explore morally ambiguous settings with characters who are Heroic. My worry with this though is that Heroic characters would be one dimensional compared to morally ambiguous villains. If villains are relatable in morally ambiguous settings, than heroes need to be too. So Heroes in this setting will need strong self-motivation on why they go out of their way to fight evil. Good for good's sake doesn't seem to hold any more water in this type of setting than evil for evil's sake. In a way, this makes the Heroes selfish in the weak sense, but deeper for the purpose of this kind of setting.
What are your thoughts on morally ambiguous settings v. Heroic characters?
postscript: I'll be in Chicago for a few days (probably no update till Wednesday). I have an interview Tuesday, and I'm hoping if that goes well I'll be moving up there soon.