There's nothing quite like a good villain. Since the earliest days of storytelling, one of the most common ways of expressing a conflict (or, more broadly, telling a tale) has been cast through one person against another person—the basic building-blocks from which we draw our hero and our villain, to be sure. Yet perspective accounts for perhaps more than we would like, and the hero of one story is often the villain of another. What is it about villains that continues to compel and impress us even as we reel from their actions? What twists and turns of psychology wedded to narrative are responsible for our long-lasting attachment to the figures who, by all rights, ought to repel us? Let's take a look.
First, a note about terminology. As I am primarily concerned with tales of the fantastic, I usually default to talk of heroes and villains, but it seems to me that the same principle applies with protagonists and antagonists. At any rate, when you reduce them—protagonists, antagonists, heroes, villains—to their base substance, break their psychologies apart and poke about with all their bits and pieces, they're all people in the end.
The most basic conception of a hero and a villain—in other words, the one people will most typically use as a default—is, in many ways, one of the newest. The Iliad shows us snippets of one of the most famous wars in literature, and yet it treats both sides with astonishing clarity and a kind of sensitivity to perspective that might surprise some; the kinds of things that Greek tragic heroes do would cause most normal people to recoil. This absence of black-and-white heroism and villainy continues in Beowulf, where—much to the dismay of many a student misled by a poor translation—Grendel and company are viewed as fierce combatants worthy of a fair fight, rather than purely monstrous beasts.
This inclination to flesh out the villains—where the villains are the opponents, the worthy foe, rather than the zombie slowly lumbering through the hallway—is likely rooted in an understanding of the way in which heroes and villains are complementary. It seems to me that a hero is only as good as their villain, because they represent two differing aspects (or perspectives) of the same story. This duality of perspective is further emphasized if the hero and villain literally share character traits and a backstory, of course, but it need not be essential to the hero/villain dynamic.
I would argue that our attachment to villains, though, is also rooted in an awareness of human interaction as well. We can well acknowledge that heroes and villains have interchangeable roles insofar as they represent two halves of a whole story. A shift of perspective is all it takes for us to realize that the villain sees himself or herself as the hero of the story, because even the most abhorrent of individuals can have convictions, resolutions, hope. This perspective-shift, ultimately, is simple enough.
We're all the heroes of our own stories.
The problem is that we're always the heroes of our own stories, and so we become blind to it. When we're the villain of someone else's story, though—that's when we're conscious of our role, when we know the kind of story we've wandered into. We are always more conscious of our role within a story, even if it's the story of a life, when it's someone else's. This hyperawareness of how we stand in opposition gives us greater understanding on an emotional level of a villain, even if we would never condone that villain's actions. We forget when we're the hero, but we forever remember when we're the villain.
Beyond our instinctual understanding of villains and why they do what they do, however, villains provide us with the opportunity to explore a number of situations and possible realities through the filter of everything that drew us to such a villain in the first place. They allow us to comprehend the emotions and situations that would create such a person while remaining distant enough to not, for one moment, condone their actions. Villains, and the dual balancing act they create with other characters, cause us an exercise in compassion and stimulate our intellect by trying to puzzle out how they work—yet, in doing so, we are redirected to puzzle over ourselves in an attempt to understand not simply how that character acted, but how we might have acted, and how any given human might respond to such circumstances. The semi-standard tragic backstory now provided for a number of villains is a vehicle to facilitate this examination, certainly, but it can serve as a worthwhile exercise—even if subconscious—with less sympathetic characters as well.