Obormot Discusses Goodhart's Law and WoW Raiding
<namespace> So I know I keep bringing up BBS: The Documentary.
<namespace> But there's just so much good shit here.
<namespace> Like on this one.
<namespace> https://www.youtube.com/watch?v=lhdYivbjiXU
<feepbot> BBS The.Documentary Part 3 - Make It Pay - YouTube (BBS The.Documentary Part 3 - Make It Pay)
<namespace> They talk about how having a sense of friendly competition, and a centralized publication that people read and a big convention/meeting place, spurred them to *really* try and put effort into improving themselves.
<namespace> Which squares with a lot of the stuff you were talking about with WoW guilds, and then sort of expands on it.
<Obormot\Sirius> Indeed
<Obormot\Sirius> And because that is a good and important point
<Obormot\Sirius> Let me tell you about another aspect of WoW's take on it
<Obormot\Sirius> And an important caveat / devil's-advocate thing
<Obormot\Sirius> So, the thing with WoW, especially WoW raiding, and most especially the raid boss encounters, is that it is very much a game of numbers and details
<Obormot\Sirius> Now, at first, in the very early days of WoW, people didn't necessarily appreciate this very well, nor did they have any good way to use that fact even if they did appreciate it
<Obormot\Sirius> And - this bit is a tangent, but an interesting one - a lot of superstitions arose about how game mechanics worked, which abilities had which effects, what caused bosses to do this or that, etc. - all the usual human responses to complex phenomena where discerning causation is hard
<Obormot\Sirius> And, more importantly and on-topic, there was no real good way to sift the good players from the bad; nor to improve one's own performance
<Obormot\Sirius> This hampered progression
<Obormot\Sirius> ("progression" is a WoW term of art for "getting a boss down, getting better at doing so, and advancing to the next challenge, rinse repeat")
<Obormot\Sirius> (hence "progression raiding" meant "working on defeating the currently-not-yet-beaten challenges")
<Obormot\Sirius> But one crucial feature of WoW is the combat log
<Obormot\Sirius> This is a little window that appears at the bottom of your screen, and into it the game outputs lines that report everything that happens to or around your character
<Obormot\Sirius> All damage done or taken, all hits taken or avoided, abilities used, etc. etc. - everything
<Obormot\Sirius> And this is in a specific format; and it can be parsed by the addon system
<Obormot\Sirius> Naturally, then, people soon began writing addons that did parse it - parse it, and organize it, and present various statistical and aggregative transformations of that data in an easy to view form - and live
<Obormot\Sirius> This gave rise to the category of addons known as "damage meters"
<Obormot\Sirius> (Of course they showed other things as well, but viewing damage output was the most popular and exciting use)
<Obormot\Sirius> And the better addons not only recorded this data, but also synchronized and verified it by communicating between instances of themselves running on the clients of all the people in the raid
<Obormot\Sirius> Which meant that NOW you could have a centralized display of just what exactly everyone in the raid was doing, and how, and how well.
<Obormot\Sirius> This was a great boon to raid leaders and raid guilds everywhere!
<Obormot\Sirius> You have a raid of 40 people, one of the DPSers ("damage per second", i.e. the people responsible for doing damage to the boss) is a noob, can't DPS for shit, or he's afk half the time, or he's just fucking around - who can tell?
<Obormot\Sirius> With damage meters - now you can tell.
<Obormot\Sirius> Sift the bad from the good, the conscientious from the moochers and slackers, and so on
<Obormot\Sirius> And more: someone's not performing well but seems to be trying, but failing? Well, now you look at his ability breakdown, you compare it to that of the top DPSers, you see what the difference is and you say - no, Bob, don't use ability X in this situation, use ability Y, it does more damage
<Obormot\Sirius> All of this is fantastic
<Obormot\Sirius> BUT this immediately and predictably began to be subverted by Goodhart's Law.
<Obormot\Sirius> If you are looking at the DPS meters but "maximize DPS" is not perfectly correlated with "kill the boss", then you got a problem
<Obormot\Sirius> Which may be obvious enough, but it is also instructive to consider the specific ways that those things can come uncoupled
<Obormot\Sirius> So let me try and enumerate them
<Obormot\Sirius> 1. The Thing is valuable but not the only valuable thing - there are other things that must be done, that are less glamorous, and may detract from doing the Thing, but which is a sine qua non of success
<Obormot\Sirius> (in WoW: the boss must be damaged, but also, adds must be kited - never mind what this means, know only that while a DPSer is doing THAT, he can't be DPSing)
<Obormot\Sirius> And yet more insidious elaborations on that possibility:
<Obormot\Sirius> What if, yes, this other thing must be done, but the maximally competent raid member must BOTH do that thing and ALSO the main thing? He won't DPS as well as he could, but he also can't just *not* DPS, because then you fail and die; you can't say "ok, JUST do the other thing and forget DPSing"
<Obormot\Sirius> i.e. what if the secondary task isn't just something you can put someone full-time on
<Obormot\Sirius> So e.g. in a software development context, suppose you're measuring commits, but also documentation must be written
<Obormot\Sirius> But you don't have a dedicated docs writer
<Obormot\Sirius> etc.
<Obormot\Sirius> Then other possibilities:
<Obormot\Sirius> 2. The Thing is valuable, but tunnel-visioning on The Thing means that you will forget to focus on certain other things, the result being that you are horribly doomed somehow - this is an individual failing, but given rise to by the incentives of the singular metric
<Obormot\Sirius> (The WoW example is: you have to DPS as hard as possible but you also have to move the fuck out the way when the boss does his "everyone in a 10 foot radius dies to horrible fire" ability)
<Obormot\Sirius> And yet more insidious versions of this one: yes, if this tunnel-vision dooms you, personally, in a predictable and unavoidable fashion, then it is easy enough to say "do this other thing or else you will predictably ALSO suffer on the singular metric"
<Obormot\Sirius> BUT the real problem comes in when neglecting such a secondary duty creates externalities; or when the destructive effect of the neglect can be pushed off on someone else
<Obormot\Sirius> (in WoW: "I won't run out of the fire and the healers can just heal me and I won't die and I'll do more DPS than those who don't run out"; in another context, perhaps "I will neglect to comment my code, or to test it, or to do other maintenance tasks; these may be done for me by others, and meanwhile I will maximize my singular metric")
<Obormot\Sirius> When in fact you always have the comparative advantage in doing the secondary thing that avoids the doom; if others have to pick up your slack there, it'll be way less efficient overall
<Obormot\Sirius> Then...
<Obormot\Sirius> 3. The Thing is valuable, but there are ways to *in fact* increase your level of the Thing, really do increase it, BUT at a non-obvious cost borne by others; yes, you are improving your effectiveness, but the price is that others, doing other things, now have to work harder, or waste effort on the consequences, etc.
<Obormot\Sirius> (Many examples of this in WoW, such as "start DPSing before you're supposed to, and risk the boss getting away from the tank and killing the raid" - in a general context, "taking risks, the consequences of which are dire, and the mitigation of which is a cost borne by others, not you")
<Obormot\Sirius> Then this one is particularly subtle and may be hard to spot:
<Obormot\Sirius> 4. The Thing is valuable, and doing it well brings judgment of competence, and therefore status; ALSO, there are *roles within the project's task allocation* that naturally give greater opportunities to maximize your performance of the Thing, and THEREFORE people seek out those roles preferentially - even when an optimal allocation of roles by relative skill or appropriateness to task, would lead them to be placed in roles
<Obormot\Sirius> that do not let them do the most of the Thing
<Obormot\Sirius> (in WoW: if the most skilled hunter is needed to kite the add, but there are no 'who kited the add best' meters, only damage meters, well, then maybe he says 'Bob over there can kite the add better' and as a result, because Bob actually is worse at that, the raid fails)
<Obormot\Sirius> (in other contexts... well, many examples, of course; glory-seeking in project participation, etc.)
<Obormot\Sirius> Of course there is also
<Obormot\Sirius> 5. The converse of #1 - if the Thing is valuable but you are bad at it, you might deliberately seek out roles in which there is an *excuse* for not performing it well (because the role's *primary* purpose is something else) - despite the fact that actually the ideal person in your role ALSO does the Thing (even if not as much as in a Thing-centered role)
<Obormot\Sirius> That's it for now, I think
