I am about 40 pages away from finishing Victor LaValle's novel Big Machine. I don't know how well this book sold, but it did well enough to be reprinted as trade paperback after its run in cloth*, so it must have done respectably enough, and it's won awards and been reviewed well here and there. Which comes as no surprise, because it's pretty good, so go read it.
What does surprise me, however, is that it's been marketed as a work of literary fiction. I don't deny that it is, mind you: the language is wonderful and the book deals with such deep subjects as identity, doubt, trust and the existence of God. But it also veers sharply into "X-Files" territory, with some Edgar Allen Poe thrown in for good measure. So this is like a paranormal literary fiction.
It's not sold as paranormal, and on the jacket copy the suspense is played up and the theology/weirdness is played down, but this is a strange novel about strange events, the real world's edges and underbelly being home to supernatural activity and beings. But you won't find Big Machine in the "horror" or whatever section of Barnes & Noble, you'll find it in the general fiction section, and the book is being discussed as literature.
I also can't help noticing that in the YA section of book shops, there is a lot of zombie/vampire/supernatural activity going on, and it's all called "YA" and not "science fiction" or "horror" or anything. And it further seems to me that there are few people who read only zombie novels, or only police procedurals, or only vampire romances, or only literary fiction about estrangement from parents or whatever. I do not believe in the idea that I hear often from agents and editors that people don't stray from "their" section of book stores, and so I begin to strongly doubt the notion--again, one we hear often from agents--that your book had better fit into a neatly-defined genre.
Because readers don't read like that. If you ask readers what their genre tastes are, I think most of them are going to look at you as if you're speaking a foreign language. Readers, I think, don't think about "genre," they think about books and authors that they like. And bookstore workers are generally well- and widely-read enough that they aren't afraid of a book if it doesn't fit neatly into one of their previously-defined genre slots. They do, after all, have that big "general fiction" area.
Where, really, am I going with this? I think that what I realized while reading Big Machine is that whenever someone tells you that you are ill-advised to mix genres in your work, that you really need to focus on one style or another, they are full of it. Writers lead the way; agents and editors all follow, always. They follow writers and they follow buying trends begun by...writers, yes, that's right. As my very own agent says about agents, "We're all sheep." If you are writing a really cool, really good book, you should follow your instincts and just write that really cool, really good book.
Yes, at some point you will need to market it to an agent who will have to sell it to a publisher, but you do NOT have to say what genre it is in your query, as long as you send it to an agent who you think will be interested in your book. My agent and I have never once discussed the genre of my book, and I didn't say "literary fiction" in my queries. I just talked about how really cool my book is.
And now, kids, I have to work. Which sucks, because I really want to go finish Big Machine.
* Yo, Victor LaValle: I bought your book in cloth to help you earn out your advance. Just saying.