It's obvious that madness meters and charts won't make a player scared. Generally it isn't possible to scare the life out of someone with a book, a movie or a game. The goal of the GM when running a horror game is trying to recreate for the players an experience alike to watching a good horror movie or reading a good horror book. Which is to say, a mix of fear (10%? 20%?) and entertainment (the rest). The last thing a GM wants to do when GMing a horror a game, is *really* scaring his players.

Involvement
You can't just go "The monster is trying to eat your mother who's never been mentioned in the game before!" and expect a big reaction. For maximum impact, the PCs need time (possibly multiple sessions) to interact with each other and the important NPCs before real fear for the safety of others will set in.

The Unknown
Don't describe things by standard names. Leave mysteries and uncertainties. Don't say "you see a ghoul" if you can say that "You see a slavering, hunched over humanoid figure, hairless and mottled and obviously very, very hungry."

Here's the bottom line for what I've found in reading horror and really studying it as a genre. Only one thing is actually scary: the unknown. As soon as you show the bad guy or as soon as you describe him clearly, he's freaky and horrific, yes, but he's no longer scary.

Think about the best horror movies and you'll realize this is true. For example, isn't Jaws much more frightening BEFORE the shark model starts lurching up onto the boat and trying to kill people? Freddy Kruger is frighting not because of his messed up face and claws, but because he's got a new way to attack people in every movie and everytime they don't understand it until later in the movie. Lovecraft's monsters are frightening because they never get described in detail. Even when most horror movies reveal the bad guy, they won't usually tell you were he is. The only time you both know the bad guy's nature and location is the big final battle... and that isn't generally scary, it's just necessary to conclude the story.

That's important to remember. Players will never be scared because their characters are at risk or any other in-game circumstance. What will scare them is the unexpected and the unexplainable. This is why I suggest never telling your players that you run a horror campaign. I would run an ordinary, boring everyday campaign until suddenly the inexplicable starts killing characters. Seriously, the players should have a moment where they say "But [ghouls, ghosts, demons, mutant beasts] don't even exist in this game!"

Build tension
You build up to the big reveal... you want enough foreshadowing, spooky atmosphere and general hints about it that everyone's excited and nervous when the creature finally shows (if it does) but not so much that they become convinced that it'll never actually do anything to them.

Spooky Atmosphere
You can raise the tension level with spooky atmospheric effects. If their normally reliable flashlights start to flicker and buzz in a darkened area... if the normal noises of the night are replaced with strange buzzing that doesn't quite sound like regular insects... if the light in the room grows unnaturally dim as something big moves nearby... think of things that would make you nervous in real life like unexplained noises and introduce them into the game.

Isolation
You need to figure out ways to isolate the characters and make it credible. Also, the protagonist should realize that there is no way to escape except back where they came. Fights against bosses in games are most likely in very tight, isolated quarters. Therefore, if really nasty creature comes through the door, the characters need to make a last stand and fight against the overwhelming odds. Players need to realize that if they screw up a die roll, there is no 'reload'. That's it.

You also need to keep the players on their toes at all times. When there is a minute to breathe, then you should hit them again and again and again... they need to feel like if they don't find more ammo soon, they are toast (even if there is a large group that night).

Ever more Horrific Revelations
Like building tension, except that you do it with clues. The missing priest was embezzling from his church... and it looks like he was using the money to buy these ancient texts... and he was excavating the old ruins behind his church... and here's a bunch of documents he'd stolen about an ancestor of his that was supposedly buried back there... You build up to the full revelation, when the PCs find the body of the priest who vanished last week and he's obviously been dead for months... stabbed with the knife that his ancestor was supposedly buried with!

So much of horror is a balancing act. The GM has to judge when it's best to have the big bad show up (if it ever does... that's actually not necessary for a horror game). Build up to it for too long, and folks become jaded with the idea... too short of a time and no tension has built up.