Bad stats -- "good enough"

I'm reading (or really listening to the audiobook during my commute) "Traffic: Why we drive the way we do" by Tom Vanderbilt.  It is a pretty good book and well researched and surprisingly interesting given the pretty dry nature of the material.

This morning, he was discussing what is a "safe road."  In particular, he was talking about small, curvy roads and whether they are made safer when the turns are labeled with reflective posts and recommended speed signs.

Proper signage is optional

Research has shown that the average speed on unlabeled roads is lower than on labeled roads.  The proposed reason is that the unlabeled road is observably dangerous, so drivers are more careful and drive more slowly.  The labeled road lulls drivers into a false sense of security and therefore they drive faster.  His conclusion is that the labeled roads are therefore more dangerous, as evidenced by the higher average speed.

Here's the thing though... average speed doesn't matter!

What matters is the occurrence of a driver entering a turn with a speed that exceeds the combination of his driving ability, the character of the turn (radius, traction, etc.) and the performance of his car.  If the "fly off the road threshold" (or put another way, "bad enough") isn't exceeded, the driver keeps right on going, no matter how close he might have gotten.  The conclusion drawn by the author is therefore baseless.

Further, if we imagine that the presence of the signage increases the average speed but decreases the standard deviation (or more specifically tightens up the distribution in one way or another), then we're probably less likely to hit the bad-enough threshold and have an accident, even though average speeds are higher.  This seems believable, since better information means that people don't have to guess how fast to go through the curve, and they should converge on something similar to the posted, recommended speed.

made with @Risk by Palisade

In the little simulation I did here, we have two distributions.  The blue represents the "no signs" scenario, where the driver has no idea what they're in for and tends to drive slower.  The red represents the good signs scenario, when everybody knows what's coming.  I set the "bad enough" threshold at 40 mph.  You can see that 0% of the good signs people crash, but 1.3% of the no signs people crash.

There is an analogous concept in marketing, called the "good enough" line.  The idea is that people have some threshold for purchase that depends on certain quality vectors.  In a car, for example, a driver might have a minimum requirement for horsepower, seating capacity, gas mileage, etc.  If your car doesn't meet that threshold, that person won't buy it.  If you exceed that threshold, they won't pay any more for the car and you don't monetize the value.  It's all or nothing.  This concept applies to lots of things.  While the car crashes when it exceeds the threshold, the customer buys something when their desire for the product exceeds the threshold.

So here's the problem.  Statistical process control (SPC) is all about removing variability.  Mass marketing and homogeneity of product offering is the way of the world.  What are the actual differences between different Android phones, for example?  As things move toward the red distribution, fewer and fewer outliers will exist in the tail, and fewer and fewer people will reach the purchase threshold for your product.

So the takeaway is that new products should desperately avoid the middle path.  If you're going to put out something new, put out something really new.  Expect most people to dislike it.  What you need is a few people who will like it, and you won't get there if you don't produce the wildly unpredictable product.

Growth and ungrowth - forcing the vote

Over the past 30 years the US economy has grown.

from http://data.worldbank.org/

At the same time, middle class incomes have stayed flat.

from: http://www.advisorperspectives.com/

I found a really great breakdown and analysis of earnings trends at this website.  It's interesting to note that the sources here are not political sources or think tanks.  This is the US census and a couple of investment advisers that have aggregated and charted the census data.

At the same time, there has been considerable growth in the income of middle class women, and some minorities.

http://www.bls.gov/opub/ted/2013/ted_20131104.htm

The majority of middle class men have not been realizing these gains.  Additionally, many recent college grads are not experiencing these gains.  Link to wikipedia on income

There is a significant body of research on happiness that points to the concept that an individual's sense of happiness is a function of how they feel they compare to their self-described set of peers.  When a person looks around at their friends they do a natural, precognitive comparison that establishes happiness on a sliding scale.  In the case of the economy, the rule is similar.  White men are still better off than most ethnographic groups in the US, but when they measure themselves based on growth, they feel left behind (they are left behind in growth).

Lastly, I'll leave this article on the slide of the American Middle class from the NYT.

Now I will make a logically unsubstantiated leap here and posit that a vast majority of Donald Trumps supports are accurately described by the above economic picture.  I believe that Trump's base is built on the frustrations of white men (and those who are close to them and care about them) who have legitimate economic grievances regarding their individual situation.

It is really a shame that Trump is their mouthpiece, however, because his root cause analysis is deeply flawed.  I don't think there are any credible economists out there who would blame illegal immigration for the trends I identified above.  Certainly there are no economists who would blame ISIS or gay rights or a woman's right to freedom from sexual assault or any of Trump's other defining positions.

The shame here is that he a) provides an unproductive distraction to the people who are getting screwed.  They're now getting screwed by him, too.  b) His angry rhetoric is producing a dangerous death spiral in the form of a reactionary populist echo chamber.  c) the result is that everybody who has been experiencing the gains in the economy assumes that those who are left behind are just racist crackpots.  The whole argument becomes illegitimate.

Ironically, the result is very similar to what you see with Black Lives Matter.  White men are not the subject of negative implicit bias and therefore are not systematically and consistently exposed to negative (and often lethal) encounters with the Police can't wrap their heads around the problem.  They look at the movement, find the worst in it, see some looters or rioters and decide the whole thing is bullshit.  Same story, different disenfranchised and marginalized set of experiences.

Metacognition, forecasts and estimating ability

tl;dr: Odds are pretty good that your opinions on everything are worse and more ill-informed than you think they are. Suspend judgment about the ability of others (positive or negative) because it is probably just an echo effect (they're brilliant if they agree with you and stupid if they don't). This is all especially true if you aren't an expert in the field under consideration. Don't apply this logic to others, apply it to yourself first.

Metacognition refers to the act of thinking about thinking. This is the same sort of concept as Freud's superego, or what phenomenologist philosophers call our consciousness. Kahneman might refer to it as "slow thinking".

In general, our brain has two methods for making a decision. By "decision" I mean just about any sort of conclusion about something. There is a fast method that runs ahead and draws its conclusions based on instinct, training, experience and mental shortcuts. The fast method is great for things that we are evolutionarily prepared for and for things where training is effective. Generally training is effective for things that require quick reflexes and that provide immediate, salient feedback. Fast thinking is gut feel and reflexes and is not something we are aware of consciously because it doesn't formulate in language -- it forms in action.

The slow method is what we think of as "thinking". It is logical, it handles new ideas and concepts. It is what we observe as "thought". It presents itself in our language and is shaped by the way our language works. Slow thinking spends a lot of its time confabulating a logical reason why the fast method came up with what it did. A lot of our thought is spent thinking about what we just did and justifying it to ourselves.

This where our ability to forecast comes in. Most of the time, when asked about the future, people immediately have a gut feel. That gut feel is hugely biased (too many forms of bias to go into now). As long as we don't get immediate and obvious feedback that the forecast was wrong, our brain has mechanisms in place to pat itself on the back for another great forecast. Even if the forecast is terrible, if the feedback is subtle or slow, our brain still thinks it did a great job and our metacognition formulates a story to back it up.

Unfortunately, most of the forecasting situations we face in the modern world don't provide immediate and obvious feedback.

This is the reason that we're all bad at estimating our ability. We are best at identifying problems when we have expertise with a subject. The better we are at something, the more capable we are of identifying when we screw it up. Conversely, the worse we are at something, the worse we are at identifying how bad we are. We blithely go along, thinking we're pretty awesome at the stuff we're worst at. Those who are the best at things are also the most aware of their shortcomings.

There are lots of examples of this in the world. I guarantee that Tiger Woods is far more critical of his golf game than a typical weekend duffer. Proof-reading the English language is another great example--if you don't know the rules of grammar very well, then you won't be aware of your errors. Driving is another example where people are notoriously bad at identifying their own skill (except race-car drivers, who are typically very slow drivers on city streets).

So what?

Well, this whole thing applies to business, sure. I think what's really interesting is how much it applies to politics. Look at how many people who know nothing about economics are judging the economic policy pronouncements of the Presidential candidates. Look at how many people who know nothing about police training are hypothesizing about their motives. Look at how many people who know nothing about life experience as a black American are denigrating their perspective. Science would tell us all those people are probably wrong and atribiliously wrong about it because they are even ignorant of their ignorance.

Mutual Admiration Society

Be careful what you hear, it may not have been what was said (or what was meant).

I just finished Malcolm Gladwell's wonderful podcast series "Revisionist History".  There are many interesting threads through the various episodes, but the big meta-theme that loomed large for me was the danger of confirmation bias.  I think it was the second episode  (the one about Saigon) that really hit on the issue.  It is worth the listen.

This problem is giant when you're building a business model.  It is very hard to keep somebody in the room who honestly thinks your idea is stupid, but that person is essential.  I'm not talking about some devil's advocate who will throw out half-baked objections before rolling over and giving you a false sense of accomplishment.  I mean somebody who really thinks the idea won't work and is willing to try hard to articulate why.

"Oh, you're so smart and your new business idea can't fail!"

What comes naturally is that you will find somebody else who is enthusiastic about your idea and will be supportive.  Often this is somebody you already know, and probably you're on friendly terms. After all, that's why you brought the idea to her in the first place.  She'll talk up how great your idea is and you'll recognize what a genius she is for liking your idea.  The two of you will add more friends until you've built up a real, proper mutual admiration society around self-love as represented in lavish praise for one another's brilliance.

"We're all so smart and funny and we're all going to be super-successful!!"

This is great when you're starting out!  Everybody except the most ruthlessly anti-social people need external approval of some kind.  The trick is knowing when it is time to stop bolstering your confidence and adding good news and start sorting out the potential problems, doing your pre-mortems with serious abandon and sorting out the real contingency plans.

At that point, it's essential to realize that sometimes the smartest people in the room are the ones who are willing to say "I don't get it" and your best friends are the ones who are willing to say "that's stupid."

Management Fail!

A developer in my org came to me with a technical question she was working on puzzling out around something we call "Decision Units" in portfolio.  Decision units itself is a topic worth its own post, and not really important here, but the gist of it boiled down to this:

Her: "We have a problem handling decision units that exist in multiple business units, where analysts in different groups aren't aware of all of the logic relationships that might exist for that project."

Me: "What's the problem?"

Her: "Well, the way we're handling things right now, if anybody makes a change to the logical relationships for a given project --if they add a dependency or some sort of optional relationship or anything-- it will wipe out all the stored values for that project.  Worse yet, the analyst who loaded them before won't know they were wiped out, and when they go back to load them again, there will be a bunch of new alternatives there that they never heard of!"

Me: "That is a problem!  What do you think we can do about it?"

She gave me a good explanation of the issue that indicated she understood both the technical character of the problem and the business character of the problem. 

She really had her head around the thing.  She then delivered a very technical, possible solution that involved setting some flags and having users determine several things in advance and some other business rules around the process flow.

Me: "I think we can do something simpler.  Here's what I think..."

And then I went on to sort of brainstorm out a couple of default states and business rules that eliminated most of the possible sources of trouble.  Lastly, I sketched out an approach that would handle the last situations elegantly and simply and without requiring the users to do anything they weren't already doing.

Her: "Wow! That was really great!  I'm glad I came to you.  Here I was, thinking of something super-complicated and all I needed was to talk to you and you had a simple answer right away!  Thanks!"

I felt pretty good about myself for a few minutes and then realized I had totally failed her as a manager.

With a little coaching I'm pretty sure that she could have come with the answer on her own.  Maybe a few questions, or at the very least I could have structured the conversation more collaboratively.  I sort of pontificated how I think it should work and we were done.  My developer was left maybe feeling good about my skills, but probably a little less confident in her own.  That's a big fail as a manager.

this is a clever metaphor!

I think this sort of thing is why it's very hard for managers to make those "Career Pipeline" transitions from independent contributor (IC) to manager.  We build up these technical skills and are rewarded for them, and often they become a source of pride and self-worth.  When problems come along that are in the sweet spot, we think immediately of a fix.  We miss the chance to do our new job, which is coach the person bringing the problem as they figure out a fix.

Intelligent Software Building: products, platforms, programs

On a regular basis I see folks who are trapped by the siren song of reusable code.

Now, don't get me wrong.  Reusing code is a wonderful thing, and it's a good goal, but it's important not to get carried away.  There are basically three levels of reuse:

1. Programs - this is when you reuse parts of your own code, on a personal sort of level.  You built these routines and snippets, documented them and like them.  They're something you've done before and know you want to do again, so you were kind and thoughtful to yourself and wrote it in a way that you would recognize.

2. Platforms - this is when you specifically plan for reuse when you're writing something.  Platforms refer to a broad area of capability that is important and that you'll want to build on.  Platforms require serious forethought.  They also require serious documentation and some support.  Typically this is something you will share with other members of your team.

3. Products - this refers to a software platform that you want to make broadly available.  When you create a software product you absolutely have to have clear documentation in and out of your code.  You have to plan for wide deployment and you have to have support people in place.  When changes are made, you have to have a thorough test apparatus in place to understand what the impact of your changes will be on others who have built their software with your product.

especially to yourself...

Be kind to the person who will inherit your code, because often that person is you. 

Many times I have cursed the me of weeks or months ago for leaving a mess for me.  When I'm kind to me, I always appreciate it.  Reuse of your programs is just good practice, and naturally happens when you're writing thoughtful and clean.

After you've experienced those benefits a few times, you'll start dabbling around with platform style reuse.  You'll think "hey!  that little block of code has been so useful to me, I bet others would like it!"  Before you know it, you have a whole set of interconnected things that you've put together and share.  You probably don't have any sort of obligation to support it, but it's fun and you're supporting your own work through the thing anyway, so it just makes sense. 

Ain't nothin' wrong with Platform style reuse.

Where you have to think long and hard is around the Product-style reuse.  This makes sense if your goal is to sell this code as a development tool.  This is something like D3, or Telerik, or the Unreal Engine, or Havok or anything like that.  More often than not, I see this when developers fall in love with their own, potentially reusable programs, assemble them into a platform and then decide the whole world needs it.  You will have tons of support work to do.  You will have to maintain forward and reverse compatibility.  

If this is some sort of exercise in self-love, you need to stop now and slowly back away.  Really, unless you're planning on making enough revenue from this (or if internal facing, saving enough effort) to justify several developers and a testing team, this is just a bad idea.

Sometimes a platform may naturally find demand in the market.  Maybe the platform you built is truly wonderful and well documented and bullet-proof and you really really want everybody talking about how awesome you are... Even then, don't do it unless you have the support staff.  The first time some package you're including gets depricated or MS rolls off some capability you're counting on or any other of a million possible things, you're going to have all those people who were singing your praises on your IM, demanding immediate fixes...

software III

Don't pee your pants for nobody

This one is pretty similar to the "you'll regret the hack" thing, and I see this a lot with green developers.  With deadlines looming and a rush to get things done, you may be tempted to degrade your work from "software product" to "proof of concept".  What this means is that you're no longer building a software platform you can add to later, instead you're building specific features to try and get somebody off your back.  It works out a lot like the old metaphor about peeing your pants for warmth.

you're warm for a while, anyway...

Essentially, you solve your short-term problem but wind up with a bigger, long-term problem.  This is bad when it's a team decision, and even worse when a developer does this without telling anybody.  I've personally been in the situation where one of my developers did this sort of thing and I wasn't close enough or careful enough to know that's what was happening.  Once we got past UAT (user acceptance testing) and I started rattling off the next steps and new features to add, he turned defensive and told me he needed a few months to rework everything because everything he had just done was purpose built to get past that first deadline.  He literally set us back 6 months and absolutely destroyed my trust in him.

Hire fast, check-in faster, fire fastest

Back in the games days I had an employment model that was built around the fact that I had no idea how to hire people.  In retrospect, I was probably ahead of my time, since research seems to indicate that interviewing is almost worthless, GPA is a poor predictor of work performance, and things like pedigree and credentials are over-rated.  This was before I had ever heard of a behavioral inteview, much less experiened or conducted one.  Since I worked in computer games, every job opening I had was flooded with candidates, and often these candidates were non-traditional.  They were often wiz-kids with no college or formal training.  Sometimes they were older folks with a passion for gaming.  Some were hobbyists who wanted some extra income (that was fine, I had plenty of contract-work jobs available, too).  The hard part wasn't finding people, the hard part was knowing which ones I could count on.

This was especially important for my projects with really short timelines and really tight budgets.  One really flaky developer could derail the entire project (and with it the budget and with the budget my paycheck for weeks or months).

The approach I came up with was "Hire fast, check in faster, fire fastest".

I googled "you're fired" and somehow this image came up.  It somehow seems better than anything else I could have put here.

Basically, I gave a job to anybody who wanted one.  Every new employee would get a single work product to deliver and two weeks to deliver it.  The work product was always something I actually needed and something that I knew could be done in two weeks.  Anybody who didn't get the work done on time was fired.  No questions asked, no conversation, no excuses.  What I had learned was that anybody who is willing to flub their very first impression will continue to disappoint thereafter.  Honestly, this got rid of around 80% of the people I hired.

I've applied this rule in a general sense in many ways since.  If I'm looking to hire a painter and he shows up late to discuss the quote, I take him out of the running.  I once had a buddy ask me to help him find a contractor to build a block wall around his house.  The guy showed up 30 minutes late and I told my friend not to hire him.  My friend did anyway and since I'm telling this story here, I'm sure you can imagine he never got his wall.

Hit the easy problems first

There's the rule when you're playing pool.  When one of your balls is sitting right on the pocket, you don't shoot that shot.  It will probably get knocked in on its own and you're really just wasting a shot.  That mindset is completely misplaced here.

a tempting but misleading metaphor!

Really this is probably just a good project management thing.  If you leave the little problems until they end, nothing good can come of it.  If they really are little, you'll just have a shit-ton of little problems to sort through and fatigue will build.  If any of them turn out not to be little problems (and some of them will end up a lot bigger than you originally thought) you suddenly have big problems that you weren't expecting, popping up right at the end of your project, when you don't have time to course-correct effectively.

Do those little, easy-looking things as early as you can.  Sprinkling in small wins helps to keep you from getting worn down on big projects.  Also, it gives you the chance to adjust your timelines or features if you wind up facing some big, unforeseen shitshow!

this thing again