Ashley asked: "What's the difference between literary, commercial, and mainstream fiction? Mainstream isn't touched upon very often (and that's what I consider my WIP), so I wonder, is it not as prominent as the other two? And how is it very much different from commercial?"
Ashley, I don’t think anyone will say that the lines between literary, commercial, and mainstream fiction are at all clear. To answer your question, I decided to consult the sources that actually rely on this sort of categorization, and those sources aren’t the writers.
The two main reasons why you will need to fit your book into a category at all is because 1. you want to find an agent to represent your work and 2. you want to place your book in the corner of the store where interested readers will find it. So, let’s see how those two groups are breaking it down.
Agents
I checked out the preferences of a few literary agents to find out how they are using the terms. The website QueryTracker.net, which serves as an agent database, has two of the three genres in question, commercial and literary fiction, among a host of other genres. Emma Sweeney from the Emma Sweeney Agency uses only the term literary fiction. Paige Wheeler from Folio Literary Management prefers both commercial and “upscale” fiction. Laney Katz Becker, also from Folio, prefers literary, commercial, and mainstream. Blogging agent Nathan Bransford considers commercial fiction to be a blanket term for all genre fiction. So, he would say that literary fiction is the un-genre, un-commercial stuff, which has nothing to do with the actual commercial value of the book. Sandra Dijkstra, from what I can interpret, also seems to go by this definition. She prefers both commercial and literary fiction, which I’m assuming covers everything—maybe I’m wrong. Interestingly, another agent from the Dijkstra agency, Elise Capron, says she prefers “character driven fiction,” “offbeat fiction,” and “debut fiction.”
So, my take home message from this short look into the agent side of things is that different agents interpret the terms differently, and you should do your homework for each specific agent before you query them. Decide if your book falls into the categories they prefer, and decide what you should call your book based on those categories. Don’t feel committed to any one category. You might end up calling your book mainstream fiction for one agency and literary fiction for another. For a third you could end up calling it offbeat debut fiction.
Book Stores
Genre categories are probably most important in that they help your readers find you. When going into a bookstore or shopping online, genres help to orient the customers so that they don’t look for Russian Classics next to Romance novels. This is especially important if they’ve never heard of you as a writer. I’ve discovered new writers in the literary fiction shelves of stores, but never in the sci-fi shelves, simply because I rarely go there. It would take forever to find anything if all of the books were just alphabetized.
Looking at the big books stores, Borders has only the literary fiction category, as does Barnes & Noble. Amazon, on the other hand, has both literary fiction and general fiction. None of these places have mainstream fiction or commercial fiction. In general, I would guess that the stores (and libraries) lump all three of these categories together.
So, Ashley, for you, I’d say to call it whatever the agent calls it when you are querying each of them. You probably won’t need to choose once it’s ready for publication, but I’m guessing—based on what little I know about your writing—that it will fall into the literary fiction shelves.