Self-publishing is the act of publishing your work independently of an established publishing house. In the past writers unable to publish their work through larger presses have gone through vanity presses, but with print-on-demand services, there are more options today for writers interested in self-publishing than before.
Likewise, the internet provides authors the opportunity to promote books themselves, making a publicist less necessary. (And, in fact, these days many writers do their own book promotion even when they are published by a house.) In recent years, some writers -- shut out of the publishing world for being too experimental or too literary -- have self-published their work and not only created an audience, but gained critical support for their work.
Kelly Link (Magic for Beginners) is an example of an author who self-published her work and gained this kind of acceptance. (Link actually created her own press, Small Beer Press.) Elizabeth Merrick (Girly) is another writer who has created a buzz, despite self-publishing.
Writers who have not found support from mainstream publishers should also consider independent and small presses, especially if they don't have the know-how, money, or time to self-publish.