Oliver's horror fiction first appeared in Christopher Barker's journal
Weirdly Supernatural; Barker then published Oliver's first two collections,
The Dreams of Cardinal Vittorini and
The Complete Symphonies of Adolf Hitler under the Haunted River imprint. The former was nominated for an International Horror Guild Award and the latter short-listed by the Dracula Society for a Children of the Night award. Both books received favourable notices from reviewers in such small press magazines as
Weird Tales and
All Hallows.
In
All Hallows 34, Jim Rockhill praised Oliver's
The Dreams of Cardinal Vittorini, noting:
Oliver’s ability to create a sense of time and place in every one of these stories is exemplary.... As a work of spiritual terror it has few peers.... Thomas Ligotti and Matt Cardin are the only authors writing today who equal the assurance demonstrated by the author of this tale in ripping away the veil separating mundane reality from the shrieking abyss it conceals.
Ramsey Campbell has also written positively about the same work: "Oliver’s sharp eye for character and ear for dialogue never desert him."
His experiences in the worlds of academe, the Church of England, and the arts have all provided inspiration for his work. A number of his stories are set within the rather seedy end of show business, drawing on his background as a playwright, director and actor. Douglas Campbell wrote of one such story, "The Skins", "I find it hard to believe that there wasn’t some kind of a dare involved when Oliver set out to write a tale about a haunted pantomime horse, but the story itself is an unforgettable piece, drawing to a grotesque and pathetic climax in a horribly plausible world of down-at-heel theatre folk."
He has pastiched a number of styles and authors, from Restoration comedy and sixteenth-century mystical texts to Oscar Wilde and M. R. James. A story in Arthur Machen's style resulted in his winning the Friends of Arthur Machen short story competition in 2005.
Oliver's work has appeared in a number of anthologies, including
Acquainted with the Night and
Year’s Best Fantasy and Horror 19. He has acted as consultant on a project which will see all of M. R. James's ghost stories released on CD.
In April 2010, Centipede Press published Oliver's collected short stories in a volume which features many new illustrations.