Zak S recently wrote a post about theory. It is worth reading. I have nothing in particular to add to it, except insofar as I am somebody who, as an academic, does theory for a living. I think about it and write about it and teach it. I am the kind of person who has actually read stuff by Luhmann, Foucault, HLA Hart, Adam Smith and Ronald Dworkin, and thinks about what it all means. Some would call me a despicable overeducated bookish geek. Others would call me a despicable overeducated bookish and pernicious nerd. In any event, I can speak with a little qualification about theory.
There are theories and there are theories, and some are more persuasive than others. Ultimately I think the ones that stand the test of time are the ones which are grounded in actual empirical knowledge and practice. (This may seem obvious to somebody who doesn't work in academia. Let me tell you - amongst some people that observation would be very controversial.) Adam Smith clearly spent a heck of a lot of time simply watching people and how they behaved - and was also employed as a customs officer. HLA Hart was a practising barrister. David Ricardo was a successful businessman. Foucault's work is based on an extremely detailed, intimate and careful familiarity with the history of his subjects. And so forth.
Now, let me qualify this. I am not making an anti-intellectual statement here: being a "good" theorist, if there is such a thing, does not mean being a philistine who is only interested in practice and practical things. HLA Hart's work was steeped in a classical education. Ronald Dworkin may have had the practical experience of being law clerk working at the US Supreme Court, but he was also one of the most gifted scholars of his generation. And nor am I making a blanket statement. Some theory has no grounding in practice at all, and you could even say it flies in the face of what seems to be empirical knowledge, yet remains (to my mind) successful and persuasive: I am thinking of people like Arendt and Sartre. But in the main, theory tends to be most successful when it is founded on practical experience with the subject matter at hand.
Ultimately this goes back to Francis Bacon, obviously. In The Novum Organon Bacon comes up with an extended metaphor about scientists, and divides them into three categories - the ants, spiders, and bees. Ants are those who only work at the ground level - "they only collect and use" - meaning they do not think about general principles or how to apply the knowledge they glean in a general way. They are all about the practicalities and nothing else. Spiders are the opposite - they only "make cobwebs out of their own substance" - meaning they spend all their time coming up with rules and principles but are divorced from actual real world practice. But then you have the real scientists, the bees, who gather pollen from flowers but then convert it into something useful: honey. People interested in producing useful knowledge have to be like bees, gathering information from the real world and converting it into principles which can be applied generally.
Bacon's metaphor is quite powerful and can be used in a variety of contexts. It works also with RPG theory. Who are the ants, spiders and bees here?
The ants are those who simply go around picking useful things out from the internet or books and putting them into use, but don't really think about how the useful things they are getting can be the basis for general principles. I am thinking here about DMs and players who trawl through forums for stuff they can use in their game, buy lots of modules and adventure paths, and so on, but never actually use what they are getting as the basis for their own creativity. It never really occurs to them that, as well as using things that other people create, they can do their own imagining too. They don't look at the latest Pathfinder adventure path and go, "Hang on, I can do that just as well if not better." They don't look at Monster Manual IV and go, "Hang on, I'll create my own monsters." They "only collect and use".
The spiders are those who think and discuss but never play. They spend all their time on the internet arguing with other people about principles, ideas, and rules, but they never actually do anything. They merely "make cobwebs out of their own substance", which ultimately is neither use nor ornament. (Never mind that cobwebs are actually useful for spiders - Bacon's metaphor isn't perfect, but everybody gets what he means.) We all know where these sorts of people can be found, and who they are. Their principles and theories are worthless because they are untested and simply emerging from the aether within the spider's mind.
The bees are those who read RPG books, trawl forums and books for stuff to use in their games, read about RPGs, and also, crucially, create lots of materials for RPGs and play RPGs. This gives them a sense of what works and what doesn't - a genuine understanding of how things work in practice - and as a result, they are able to come up with general theoretical principles that can be applied universally. How to construct a sandbox that thrives. How to come up with interesting monsters, traps or tricks. How to make a megadungeon. How to adjudicate fairly. How to draw up a good random encounter table. And so on.
This is of course just a roundabout way of saying that good theory is contingent on good practice. But I am after all a despicable overeducated bookish and pernicious nerd.