Psychological thrillers are fast-paced books that focus on the unstable emotional states of characters. The suspense in psychological thrillers often comes from two or more characters preying upon one another's minds, either by playing deceptive games with the other or by merely trying to demolish the other's mental state. 

Some examples of psychological thrillers are:

Asylum by Patrick McGrath

Before I Go To Sleep by S.J. Watson

The Collector by John Fowles

Gone Girl by Gillian Flynn

Shutter Island by Dennis Lehane

If you like this genre, a Readers' Advisor can help you find more books like these.  If this genre is new to you, you might want to give it a try.  Some of the best literature can be catergorized as psychological thrillers; Dostoevsky's Crime and Punishment is an example.