Friday 21 June 2019

Bad Reviews, Bad Moods and Bad Games

There is a certain school of thought, of which many RPG bloggers and online commenters are members, which holds that the superior way to review a game or gaming product is through play. "Ah," these people will pipe up whenever a review is posted, "But how does it work in play? How is it at the table?"

I know what they mean. What they want to make clear is that it's all very well for a book or product to read well and look good. But these things are made to be played, and in the final analysis it is how they work at the table, under fire as it were, that really matters.

I respectfully disagree with this view. Partly this is because I have never run anything I have bought as is, and can never really imagine how anybody would; I can only really imagine somebody buying an adventure or module and pulling out bits here, removing bits there, switching X around with Y and Z with A, or perhaps just going away inspired to do their own vague pastiche of the contents. Given that every reader is going to use the product differently, then, I'm never sure what value a review of how something works in play has beyond the individual reviewer.

Partly, though, it is also because of a related problem, which is that of the players. No two groups of players are the same, and no combination of players is the same as any other - we all know this, of course. This means that, much as with DMs, you have to wonder whether a review of the experience of a particular group of players can be generalised to others. But that's not the important point, for me, which is that while no two groups of players are the same, it is just as true that no group of players is the same from one session to the next. Sometimes somebody is in a bad mood. Sometimes people have had a few too many drinks. Sometimes people at the next table are noisy. Sometimes the DM has just had a row with his girlfriend. Sometimes people are just in the mood for one type of session and not another. Circumstances differ. The play experience is so subjective, and so variable, that it is very hard to trust an "actual play" review as being of widespread usefulness.

That's not to say that some products are not better than others. But I suspect this is something that can be discerned easily enough in the abstract merely through reading by an experienced eye. I don't need to play my way through a module to know if it's useful, mostly because I'm not going to play my way through it anyway, in that sense, but also because I have run some gaming sessions in my life and have a feel for what's good and what isn't.

And nor is to suggest there is no value in people going online to post about how they've just run a brilliant campaign set in Yoon-Suin* and how wonderful/terrible it was, because, let's face it, discussion and feedback are interesting in their own right. It's only to suggest that I would never not buy a product just because somebody posted a review in which they felt it hadn't worked as well as they would have liked in play.

*Other RPG products are available. [Do people outside the UK get this joke?]

21 comments:

  1. "I respectfully disagree with this view. Partly this is because I have never run anything I have bought as is, and can never really imagine how anybody would;"

    Respectfully, I think this is a bit of a straw man. I don't think the "how is it in actual play?" question suggests at all that the product must be run straight out of the box.

    It's true that any review is colored by specific circumstances, but I think it's fair to break down RPG modules into broad categories:

    (1) Modules that, while they may be great fun to read or leaf through, as a practical matter mainly sit on the shelf because some quality about them makes them not easily or effectively used in a game.

    (2) Modules that, as a practical matter, are often run to good effect in an actual game, or contribute to some substantial degree to an actual campaign.

    I think it's fair to say that any one review of a module used in "actual play" is not determinative of much, but asking the question of a lot of people you start to get a pretty good sense of whether the product falls into category (1) or category (2).

    I'm not as sure as you that it is always obvious from a casual look at a product (especially without already having bought it) which category it falls into.

    ReplyDelete
    Replies
    1. You are probably right about that - I wouldn't disagree. What I meant was that reading a single review which is about actual play experience individually probably has no bearing on my decision to buy a product or not.

      Delete
  2. "Do people outside the UK get this joke?"

    Well, us Anglophiles do! Others will get the meaning but not the reference.

    I think that published campaign settings are in an interesting and somewhat awkward position. In some ways, they are better off providing less "crunch" material and more "fluff" because, like you say, anything I run is going to get filtered through my own system, house rules, and personal style, and the best thing a campaign setting can do is provide an evocative setting that I can read and immerse my imagination into.

    Maybe the worst sin of "how does it play at the table?" is when a publisher tries to create detailed game mechanics that don't work well at all in actual play. Then you have neither useful mechanics nor imaginative setting material.

    ReplyDelete
    Replies
    1. Yeah, perhaps I should have caveated all of this by saying the actual core rules/mechanics are a bit of an exception. Even then, though, I think it can depend a lot on the group. I HATED Fate, but others seem to adore it.

      Delete
  3. I think modules can be interpreted in three ways:

    1) This is an entertainment experience in itself. In such case, it should probably be judged on the merits of the writing, "entertainment value", art, etc.

    2) This is a resource to influence my worldbuilding or gaming. It should probably be judged based on the merits of its ideas and usability on a piece-by-piece level.

    3) This is a thing I should be able to run mostly as-is with my group. It should probably be judged on the merits of its precision, coherence, and usability.

    I don't think it necessarily matters what the creators intend, I think it just matters which of these concerns the reviewer is addressing, and how clear it is that they are addressing them.

    There are plenty of books I've read that I'd never want to play as-is, or wouldn't even really want to use any part of per se, but still enjoyed the experience of reading. As long as I have this expectation when I read the book, I'm not going to be too disappointed if it fails in the latter two points.

    There are plenty of books I've read that had interesting parts I borrowed from or would consider borrowing from, but would not necessarily use as-is. Again, as long as that was my expectation, I'm comfortable with this.

    There are few books I've read that really feel like they could be run as-is, but personally I have no interest in doing that anyway, so a review focused on that is going to have less utility for me.

    For the latter two of the three points above, I do think that actual play experience is important, but of course that's only going to go so far. If the reviewer is someone I've played with, or whose preferences and dispositions I feel reflect my own, I'll put more weight in their review. I may take an aggregate; if several reviewers, even reviewers with different preferences and dispositions, all agree on certain aspects of the product (not necessarily depend on whether they consider those aspects good or bad), I'm also willing to put more weight in it. Even if I disagree with the reviewers dispositions or preferences, if I at least understand them, their review may be useful for me.

    At the end of the day, reviews of anything in the ballpark of "art" are going to have a degree of subjectivity, but that doesn't mean you can't take a systematic approach to how you review art, or how you interpret others' reviews of art.

    ReplyDelete
    Replies
    1. I feel like we need to have a definition of "as is" here - or maybe I need to make a follow-up post, because I think it's a bit of a floating signifier in the sense that probably no two DMs would agree on what it means, and nor would their experience of trying to play the same module "as is" be the same.

      I agree with what you say about art. Nothing wrong with reviewing art - as long as the review is taken with a pinch of salt. I think the best reviews are those by people who you can reliably tell you have similar tastes to after listening to or reading their reviews over time.

      Delete
  4. I see that it is a joke, but don't get it. Sort of feels like reading Nietzsche.

    ReplyDelete
    Replies
    1. Because the BBC is a public broadcaster it technically isn't allowed to advertise any commercial product (though it cheats on this all the time), so if a BBC person mentions the name of a product live on air they are supposed to remind the audience that "other such-and-such products are available".

      Delete
  5. //Partly this is because I have never run anything I have bought as is, and can never really imagine how anybody would//

    "Nobody ever runs a module as-is" is demonstrably wrong, though. Some people do run modules as-is, and others run modules with only minimal alterations, and both groups strongly benefit from actual play reports.

    *I* have run games without alteration, and AP-based reviews have been useful for determining whether this would be possible. I have also run games with extensive alteration, and yet it is good to have AP-based reviews to look at beforehand, because I want to alter things because I see a way to improve on something good, not because the game is broken and I need to fix it.

    ReplyDelete
    Replies
    1. See above - I may have to write a post about what "as is" actually means.

      Delete
  6. I think the "at the table" criteria speaks more to how well the information has been organized by the author---is it done so that the DM can use it effectively without have to recreate extensive notes?

    Sure you will customize. But the criteria implies there is a focus on making the product a technical document (at least to some not-small degree) as oppose to a novel---which can also inspire but is not a "playable" product.

    ReplyDelete
    Replies
    1. Yeah, I suppose that does have value in reviews, but then again I'm also not sure I'd ever make a decision not to buy something that looked inspiring just because somebody said it was badly organized?

      Delete
  7. This idea is interesting, and I am pretty certain I agree. I think one of the most important aspects of being a good critic is having a consistent voice. It lets people know what your tastes are, your hang-ups, and lets them decide for themselves if your advice is worth taking to heart. But with an "actual-play" RPG review, or a review that emphasizes heavily the "feel" of the game at the table, the consistency of voice thing kinda falls apart. Everyone at the table brings their own palate of preferences to the experience, and since it's a collaborative game, the people who are actually there playing the game will necessarily be louder in their contributions than the authors. Is my critical integrity lessened if people find out that one of my players' characters was a Ferengi from Star Trek, and that he died when he provoked the medusa in our 5th Ed D&D game converted from an OSR module, and failed the Petrification saving throws that were based on rules I'd adapted from 4th Ed? I have no idea. But I'd still that OSR module a 10 out of 10.

    ReplyDelete
    Replies
    1. Yep. Consistency is vital for sure. Good film reviewers are good because you get a sense for whether their tastes are similar to yours over time.

      Delete
  8. I think you're making two different comments: that other factors impact play at the table and that no uses RAW/you use it for inspiration.

    On the first, I agree. If your review is "actual play" then you are reviewing your DM, your players, the mood everyone was in, and so on. A good DM and players fixe everything and a BAD DM and players have no solution in an adventure module. It's a session report, not a review.

    On the second I must FORCEFULLY disagree. It doesn't matter how YOU use the product but rather how its meant to be used. Once I turned my laptop on its side, partially opened, to use it as a stepstool. Would this be a valid rebuttal to a laptop review? Of course not. The purpose of the adventure module, I assert, is not inspiration but rather using it to run at the table. If you MUST take notes and use a highlighter in order to run it then its a bad adventure. If, however, you DO take notes, in order to change it, then that's your own business. More power to you. But it's not useful in any meaningful dialog about the adventure. (Well, ok, not true. "I had the orcs attack the caravan the day before" can be useful advice/how I did it.)

    Examine the inverse. If we allow for inspiration, as a criteria, then we essentially destroy all use of a review. It becomes a game of "Well _I_ got inspiration from it, sorry you did not." It could be argued that all review criteria is subjective, but I would assert that "I found a use encounter 3 in my home game after I switched out a bunch of stuff" is far far more subjective and far less useful than a focus on the core problems with 99% of RPG adventures: usability, interactivity, and evocativeness.

    ReplyDelete
    Replies
    1. Ah, that's very fair. You've clarified my thinking, thank you.

      I tend to look at adventure modules in a very right-brain dominant way. A good one, in my book, does all the left-brain's heavy lifting and allows me to be highly creative and relaxed at the table. But that's by no means a universal criteria for judging the usefulness of a thing, as you've stated. Merely my bias. I'll be mindful of this the next time I write a review.

      Delete
    2. Bryce, I know I am hardly going to convince you that single reviews are not useful! But I think you're wrong on a couple of points. The first is: "If you MUST take notes and use a highlighter in order to run it then its a bad adventure." Really? You want to come out and be so sure about that? I would probably need to take notes and use a highlighter if I wanted to run, say, Maze of the Blue Medusa. But I'm happy I have that book.

      The second is, inspiration as an invalid criteria which destroys all use for a review. Not true. I watch a lot of whisky reviews. You don't get anything more subjective or individual than smell and taste. That doesn't make reviewing worthless - it's an opportunity to discuss whisky in general and also compare whether the reviewer had the same experience as you. It's an opportunity for a kind of dialogue. That's what RPG reviews can be like too, if what they're about is inspiration.

      They're also about publicity and sorting the wheat from the chaff, both of which things you do very admirably on your blog.

      Delete
    3. I tend to agree with you, but not to the extreme to which I'm interpreting your remarks. These are not binary criteria, it's all a spectrum. I have a platonic ideal of a good adventure criteria and then muddy it up by comparing real-world things to it. So, in the sense that Blue Medusa took you, for example, notes and highlighter then yes, it's not a good adventure. [And by definition "adventure" means "running it at the table."]

      But, given the crap-tastic state of adventure design, the much more loosy-goosy "inspiration" criteria matters far FAR less than other criteria, those to which make the product immediately usable.

      Thus, I must stomp all over inspiration, as a platonic ideal for adventure design, and instead hold up other values. Which are almost certainly unobtainable and are therefore measured on a spectrum. Is this mix of elements good enough?

      Delete
    4. I dunno. I think inspiration counts for most of all. I sort of feel like the other things you talk about are mostly meaningless because I'll change them anyway or they'll be too restrictive.

      Delete
    5. And that's fine. For you. But to use the laptop analogy again, your remarks could be interpreted as "laptop reviews should focus on their use as a stepstool." No, they should not. Stool reviews should focus on their use as stepstool. Adventure reviews should focus on their primary purpose: to be run at the table.

      YOU might not care about that, but that IS the primary purpose of the product. I'd be more than happy for another category called "Adventure Inspiration", that can be completely ignored by folks looking for runnable adventures.

      Delete
    6. Bryce -- I'm not sure why your narrowly specific view of the "primary purpose" of adventure modules is anything other than an entirely personal preference.

      This isn't noisms proposing using a module for a purpose for which it was never intended and for which it is only used in vanishingly rare and bizarre circumstances (laptop as stool). Tons and tons of people buy and use adventure modules for inspiration. I bet more people use them for that purpose than for running with minimum prep out of the box.

      Instead it's more like you arbitrarily saying "the primary purpose of a laptop is a word processor. Sure you *can* use it for other things, but that IS the primary function, and reviews must focus on that."

      Delete