[Warning: this entry contains mild spoilers for A Song of Ice and Fire and The Book of the New Sun.]
When reading an intricately-plotted novel, it is easy to fall into the trap of feeling intimidated by the author's great intellect and technical skill. How can this author have planned this all out in advance?
The truth of the matter is, it is almost certainly the case that the story was largely made up as the writer went along. (If memory serves, Bernard Cornwell and Kurt Vonnegut are notable exceptions.) They have followed Humpty Dumpty's advice and begun at the beginning and kept going until they have got to the end, and then stopped. The intricate plotting really just comes from the author using what he has already written as seeds for future plot developments ("I know, wouldn't it be interesting if the lady in the red hat from way back in Chapter Two turned out to be the sister of the dwarf who I've just written into Chapter Nine?"), and then going back and ret-conning things later to keep things tidy. It's a product of being constantly attentive to what one has written and how it might tie into what one is writing, or imagining, now - coming up with interesting premises, situations and characters and then making connections between them as one goes.
Think, for example, of Dr Talos and Baldanders in The Book of the New Sun. I don't care how much has been read into that book, or Gene Wolfe's writing process, but you will have a hell of a job convincing me that those two didn't just appear as interesting figments of the imagination to begin with ("Wouldn't it be cool if Severian ended up in cahoots with a fox-like playwright and his giant companion?"), who Wolfe later on decided to bestow with much greater significance during the course of the writing of what became The Sword of the Lictor. He didn't plot out the entirety of their role before sitting down to begin the first sentence of The Shadow of the Torturer. He wove it into the telling of the story as it unfolded in the course of the writing. Once it was clear who Baldanders really was, and he had completed his first run-through of what would become the final version of the book, Wolfe then presumably went back and made all the necessary amendments in the rewrite so you couldn't see the "join".
Think also of Tyrion killing Tywin Lannister towards the end of A Storm of Swords. Did George RR Martin have it in mind that would happen at the start of writing A Game of Thrones? Almost certainly not. The idea that Tyrion might end up killing Tywin may have crept up on him gradually over the course of writing the first drafts of the books, or occurred to him in a flash of insight, but it would have been something that emerged from the story - and how the relationship between those two characters, and the characters themselves, had developed in Martin's mind - during the initial writing process.
What seems like very clever plotting, in other words, is clever, but it is really better described as clever rereading and rewriting. As readers, it is very easy to forget we are not reading the story as the writer wrote it. We are reading the final product of a long process of rewriting and editing: the final version that is presented to the world, not the first draft that was 100,000 words too long and will forever remain locked in the author's attic.
It helps to bear this in mind as a DM. The idea that one could plot out "an adventure" in advance (except for a very simple and boring railroad), or could fill in all the details of a campaign setting before play commences, is a pie-in-the-sky. As with writing a novel, DMing is really an iterative process - it's just that while an author merely riffs on his own ideas, a DM can also riff on those of the players. Events transpire not because they were carefully detailed back before the start of the campaign itself. They transpire because the interesting locations and NPCs the DM has come up with, and the PCs' interactions with them, bring about connections in his mind. It's not that Steffi the Orc was intended to be the PCs' arch enemy all along. It's that Steffi the Orc was captured by the PCs after the rest of her companions were killed in a random encounter, and she was then released, and the DM thought, "Wouldn't it be interesting if she decided to get her revenge? And wouldn't it be nice if she teamed up with Cedric, the hireling the PCs kicked out of their party for stealing, after finding him wandering in the wilderness? And wouldn't it be good if the two of them decided to burgle the PCs' treasure stash while they were away...?"
The exercise comes, in other words, not from pre-plotting, but plotting as a continuous process or flow.
[I am running a Kickstarter throughout April. You can read more about it, and back it, here.]
[I also did a nice interview with the ever-interesting Solomon VK of Worldbuilding & Woolgathering here. You will find it relevant to your interests.]