I actually think ZUN's designs improved massively after EoSD and PCB. He went from making every character a girl in a mob cap to trying to have their designs correspond in some way to their mythological source (apart from Kaguya unfortunately).
The other big change is that the newer characters have more clearly defined motivations and personalities than the ones hailing from the early Windows games.
Take Remilia, for example. She's a haughty vampire who is a new arrival to Gensokyo, who claims to be a descendant of Vlap Tepes, and who was willing to screw over everyone else in order to be able to go out during the day. If you wanted to get more than that about what she's like, you would need to look at one of the published sourcebooks.
Compare this to, say, Byakuren. Just from the game script + manual, we know that she's a former nun who started practicing black magic in order to become eternally youthful, that her reasons for doing so were because her brother's death made her feel her own mortality, and that, while she was originally just faking it at first, she eventually came to legitimately embrace her role as a messiah for monsters. Despite having endured 1000 years of imprisonment for all of this, she has not renounced her beliefs and intends to act upon them if she is freed.
Regardless of whether you hate her or like her, Byakuren obviously has both more substance and more potential for interpretation to her character.