While broadly I agree that system doesn't matter nearly as much as who is at the table, I still find that I like some system a lot more than others. Viz:
Games I Like
D&D prior to 3rd edition
Warhammer Fantasy Roleplay 2nd edition (I have never actually played 1st edition)
Games I Don't Like
White Wolf's oWoD "storyteller" system
D&D from 3rd edition onwards
Basic Role Playing
Mutants & Masterminds
Call of Cthulhu
The 'special cases' are games were I am not necessarily a massive fan of the system, but I just get the setting and genre, and everything else to do with the game clicks with me so it doesn't matter.
What are the differences between the Games I Like and the Games I Don't Like? Clearly, it isn't a matter of rules-lite versus rules-heavy. WFRP 2nd edition is fairly rules-heavy, as is MERP, but so is D&D 3rd edition and d20 modern. Nor is it a matter of genre: that cuts across both camps. It isn't to do with the lengthiness of character generation: that was a huge turn-off for me with Hackmaster, but it takes ages in MERP too. And nor is it about traditional versus story-games either, clearly.
As I look at the list of Games I Don't Like and consider the reasons, I think it largely comes down to a combination of systemic blandness mixed with alienating nebulousness.
We'll start off with systemic blandness, because this is easier to explain. Systemic blandness is, simply, a system that is neither elegant in itself, nor interesting despite not being elegant, nor charmingly inelegant. To explain: Risus is elegant in itself. It is incredibly easy to pick up, has surprising depth, and is very well thought-out. Cyberpunk 2020 is interesting despite not being elegant. It is somewhat complicated and you have to keep quite a few things in your mind as you play, but there are things in there that are genuinely interesting (the combat system, the life events system, etc.). Pre-3rd edition D&D is charmingly inelegant. In the abstract it is stupidly overcomplicated and messy, but it doesn't matter because it could not be more charming. It is lovable.
The bland systems are the ones I am sure you can just pick out from the list from a once-over: Savage Worlds and the storyteller system are chief among them. I would include D&D 3rd edition in that too. It ironed out all the charm from TSR-era D&D but did nothing to make it more interesting or elegant.
Alienating nebulousness is harder to define, but it can be described as follows: I have to be able to picture in my mind how a scene is playing out during the game. If a mechanic is too disassociated from anything I can easily see in my mind's eye, I dislike the system. Diaspora suffers in this respect because of the way fate points and tagging work. You have a vision in your mind about how a scene is playing out, or how a character is, and suddenly that has to shift because a fate point gets played. I dislike that. It alienates me. d20 Modern suffers in this respect too, because of hit points. Getting shot with a rifle and losing a certain number of hit points but suffering no other ill effects makes no sense to me. I can't picture how that works in reality. I can picture it in a D&D fight because of Gary Gygax's extended metaphor about the sword fight between Robin Hood and Guy of Gisbourne in the Errol Flynn film. I can't picture it when people are shooting each other. The nebulousness of it alienates me.
One or two of the games I don't like don't fit this model. I don't like Hackmaster for the simple reason that character generation is ridiculously fiddly for the light-hearted, humourous fantasy game it purports to be. It is the point at which charmingly inelegant becomes plain annoying. And Mutants & Masterminds is one of those games which just don't make it easy to DM: the system is interesting and everything but do I really have to spend five fucking hours creating a single villain? Who has time for games like that?
No model is perfect. But it's a safe bet that if you want to win my heart, game designers, avoid bland nebulousness like the plague. Okay?