If, when, and how often ... security flaws and fake news

When I was in middle school, my family moved from the sleepy rural town of Hurley, New York to the nearby city of Kingston.  While living in Hurley, my family had grown accustomed to the security standards of small communities where the nearest neighbors are quite a long distance from one another--that is to say no security standards.  My family almost never locked our doors, and my parents regularly left their keys in the car overnight.

Within a few weeks of moving to the comparatively populated Kingston, my father's car disappeared one morning.  He had left the car unlocked, with the keys in the ignition.  Some enterprising young man had noticed and taken advantage of the security hole.

A few weeks later we heard from the police at SUNY Albany.  Apparently, the guy had taken the car up to the campus and had been breaking into dorm rooms and storing stuff in the car (which he had also been living out of).  When the car came back, it was full of stolen stuff.  Amazingly, the SUNY police refused to deal with the stolen stuff because the car was originally stolen in Kingston and the Kingston PD refused to deal with the stolen stuff because the robberies happened on SUNY campus, so we were left with a whole bunch of stuff.  The guy who did all the stealing went to jail for a few months and my dad started taking his keys out of the ignition at night.

Amazingly, the day that the guy got out of jail, our car disappeared again!  Apparently, he had made himself a copy of the key.  That key was among his personal things when he was jailed and was among the personal things returned to him upon release.  He walked straight to our house, unlocked and started up the car and --creatively-- headed straight to SUNY Albany again and started robbing dorm rooms.  When the car came back the second time, Dad finally changed the locks.

I hope this is going somewhere...

When I was a teenager, I got involved with the early idea of online video games.  Back then, "online" meant dial-up, and specifically it meant direct dialing either a BBC or a friend's computer.  You would plug your phone line into the modem (external modem, of course) and then one of you would call the other one.  A few DOS commands later (actually a ton of inscrutably complex commands later) and you were DOOMing it up against one another.  In the '80s and early '90s, security considerations were a lot like Hurley... everything was unlocked and the keys were in the ignition.

Years later, at one of the first companies I started, I decided to host the email and web server for the company locally.  By "host locally" what I mean is that I got a static IP, stuck an old desktop in a closet and installed Apache.  Installing Apache was easy, but configuring security was complex and lengthy.  I thought back to the old BBC days and considered "what are the odds that somebody will discover this IP?"  The idea of learning about the security was interesting to me, though, and I put in a few days to educate myself, figure it all out, configure and install, etc. before activating the server.

What happened astonished me.  The first attack on the server came within a few seconds.  The next one just a few minutes after that.  By the end of the day, there had been dozens of intrusion attempts against this brand new server that was there to serve up a couple of lame web pages about a nobody video games company and to pass our completely uninteresting emails back and forth.  The pace of attacks didn't slow down, either.  It continued to increase as time went on.

At the time I started saying that security is serious business.  Security wasn't a question of IF you will get attacked or even WHEN, but HOW OFTEN will you get attacked?  This is probably more true now than ever before.

Over time, I have come to recognize that this equation is a function of network density.  The more dense a network of nodes are interconnected, the more interactions each node needs to expect.  These interactions can be anticipated, they can be serendipitous, or they can be hostile.  The more deeply and broadly we immerse ourselves as individuals into a more interconnected society, the more interactions we are exposed to.  The likelihood of "intrusion attempts" (hostile interactions) goes from "will it happen?" to "when will it happen?" to "how often will it happen?"

The next interesting question becomes "what will tomorrow's intrusion attempts look like?"  A decade ago everybody had at least a few Nigerian prince emails.  A few years ago, everybody was falling for phishing scams on Facebook.

This year was the year of fake news.  Possibly the most insidious of all hostile interactions, fake news plays on our human desire for an echo chamber.  Fake news spreads like a virus through the like-minded and hijacks our rational-thinking.  The Nigerian emails stole bank account numbers.  The phishing scams stole our identities.  Now fake news is stealing our ability to think.

The metaphor of the deer

Many moons ago (in college) I was driving on a small, country road in upstate New York ( somewhere around here, if I remember right ).  It was night and it was winter.  A dusting of snow was sprinkling down on the already frozen road, the sides of which were burmed with banks of tight-packed snow and ice with thick forest beyond.

This was college, and I was broke.  I was driving a Nissan Sentra Station Wagon, which was an absolute POS in case you're not familiar with it.

The POS in question

I was going around 60 and came over a hill to discover a deer hanging out in my lane, about half a mile away.  He was a big buck, facing to my left.  I seemed to have plenty of time to work out my options in my head, and it went like this:

a) Swerve around him to the left.  This would bring me into the empty oncoming lane, but I have always heard that a deer will jump forward if frightened, and that would mean he would jump right in front of my car.
b) Swerve around him to the right.  This would bring me right into the frozen snow bank.  Odds were really good that my junky little car would end up in the woods in the middle of nowhere.
c) Jump on my breaks.  The road was frozen and it seemed really likely that the bicycle tires on my car would lose traction and I would end up spinning out in the middle of the road.
d) Get off the gas, steer straight and hope for the best.

Now, it's worth mentioning that I was a truly terrible driver as a teenager.  These scenarios were not academic concepts to me, they were all experienced realities.  I had driven off the road several times (including a roll-over accident that crushed the roof of my car with a tree), I had lost control and entered into flat spins during bad weather, bad road conditions (and honestly sometimes under ideal conditions but bad choices).  I had wrecked 4 or 5 cars by this point IIRC.  I knew my way around crashing a car.

So, I did what they teach you to do in that circumstance.  Without any panicking, I took my foot off the gas and steered toward the right edge of the road.  I knew there wasn't room to get around the deer, but I was hoping he would jump out of the way.  In retrospect, I should have honked my horn, too.

About an hour later the local sheriff was there by the roadside, poking a buck with his foot and pronouncing "yup, he's dead alright.  Do you want to keep him?"

I spent the rest of the year driving a car with a completely smashed up front end, leaking Styrofoam from the bumper (did you know that cheap cars have Styrofoam bumpers??) and with a grill full of fur.

I tried a google image search for "furry bumper" and "fur in my bumper"... I don't recommend it.

There are lots of situations like this.  You can see them coming a mile away, but there isn't anything you can do about it.  In fact, anything you do is likely to make the situation worse.  When you look back at the outcome, it's easy to second-guess, but sometimes the right choice is to limit the downside--not to try to avoid it.

Bad stats -- "good enough"

I'm reading (or really listening to the audiobook during my commute) "Traffic: Why we drive the way we do" by Tom Vanderbilt.  It is a pretty good book and well researched and surprisingly interesting given the pretty dry nature of the material.

This morning, he was discussing what is a "safe road."  In particular, he was talking about small, curvy roads and whether they are made safer when the turns are labeled with reflective posts and recommended speed signs.

Proper signage is optional

Research has shown that the average speed on unlabeled roads is lower than on labeled roads.  The proposed reason is that the unlabeled road is observably dangerous, so drivers are more careful and drive more slowly.  The labeled road lulls drivers into a false sense of security and therefore they drive faster.  His conclusion is that the labeled roads are therefore more dangerous, as evidenced by the higher average speed.

Here's the thing though... average speed doesn't matter!

What matters is the occurrence of a driver entering a turn with a speed that exceeds the combination of his driving ability, the character of the turn (radius, traction, etc.) and the performance of his car.  If the "fly off the road threshold" (or put another way, "bad enough") isn't exceeded, the driver keeps right on going, no matter how close he might have gotten.  The conclusion drawn by the author is therefore baseless.

Further, if we imagine that the presence of the signage increases the average speed but decreases the standard deviation (or more specifically tightens up the distribution in one way or another), then we're probably less likely to hit the bad-enough threshold and have an accident, even though average speeds are higher.  This seems believable, since better information means that people don't have to guess how fast to go through the curve, and they should converge on something similar to the posted, recommended speed.

made with @Risk by Palisade

In the little simulation I did here, we have two distributions.  The blue represents the "no signs" scenario, where the driver has no idea what they're in for and tends to drive slower.  The red represents the good signs scenario, when everybody knows what's coming.  I set the "bad enough" threshold at 40 mph.  You can see that 0% of the good signs people crash, but 1.3% of the no signs people crash.

There is an analogous concept in marketing, called the "good enough" line.  The idea is that people have some threshold for purchase that depends on certain quality vectors.  In a car, for example, a driver might have a minimum requirement for horsepower, seating capacity, gas mileage, etc.  If your car doesn't meet that threshold, that person won't buy it.  If you exceed that threshold, they won't pay any more for the car and you don't monetize the value.  It's all or nothing.  This concept applies to lots of things.  While the car crashes when it exceeds the threshold, the customer buys something when their desire for the product exceeds the threshold.

So here's the problem.  Statistical process control (SPC) is all about removing variability.  Mass marketing and homogeneity of product offering is the way of the world.  What are the actual differences between different Android phones, for example?  As things move toward the red distribution, fewer and fewer outliers will exist in the tail, and fewer and fewer people will reach the purchase threshold for your product.

So the takeaway is that new products should desperately avoid the middle path.  If you're going to put out something new, put out something really new.  Expect most people to dislike it.  What you need is a few people who will like it, and you won't get there if you don't produce the wildly unpredictable product.

Growth and ungrowth - forcing the vote

Over the past 30 years the US economy has grown.

from http://data.worldbank.org/

At the same time, middle class incomes have stayed flat.

from: http://www.advisorperspectives.com/

I found a really great breakdown and analysis of earnings trends at this website.  It's interesting to note that the sources here are not political sources or think tanks.  This is the US census and a couple of investment advisers that have aggregated and charted the census data.

At the same time, there has been considerable growth in the income of middle class women, and some minorities.

http://www.bls.gov/opub/ted/2013/ted_20131104.htm

The majority of middle class men have not been realizing these gains.  Additionally, many recent college grads are not experiencing these gains.  Link to wikipedia on income

There is a significant body of research on happiness that points to the concept that an individual's sense of happiness is a function of how they feel they compare to their self-described set of peers.  When a person looks around at their friends they do a natural, precognitive comparison that establishes happiness on a sliding scale.  In the case of the economy, the rule is similar.  White men are still better off than most ethnographic groups in the US, but when they measure themselves based on growth, they feel left behind (they are left behind in growth).

Lastly, I'll leave this article on the slide of the American Middle class from the NYT.

Now I will make a logically unsubstantiated leap here and posit that a vast majority of Donald Trumps supports are accurately described by the above economic picture.  I believe that Trump's base is built on the frustrations of white men (and those who are close to them and care about them) who have legitimate economic grievances regarding their individual situation.

It is really a shame that Trump is their mouthpiece, however, because his root cause analysis is deeply flawed.  I don't think there are any credible economists out there who would blame illegal immigration for the trends I identified above.  Certainly there are no economists who would blame ISIS or gay rights or a woman's right to freedom from sexual assault or any of Trump's other defining positions.

The shame here is that he a) provides an unproductive distraction to the people who are getting screwed.  They're now getting screwed by him, too.  b) His angry rhetoric is producing a dangerous death spiral in the form of a reactionary populist echo chamber.  c) the result is that everybody who has been experiencing the gains in the economy assumes that those who are left behind are just racist crackpots.  The whole argument becomes illegitimate.

Ironically, the result is very similar to what you see with Black Lives Matter.  White men are not the subject of negative implicit bias and therefore are not systematically and consistently exposed to negative (and often lethal) encounters with the Police can't wrap their heads around the problem.  They look at the movement, find the worst in it, see some looters or rioters and decide the whole thing is bullshit.  Same story, different disenfranchised and marginalized set of experiences.

Metacognition, forecasts and estimating ability

tl;dr: Odds are pretty good that your opinions on everything are worse and more ill-informed than you think they are. Suspend judgment about the ability of others (positive or negative) because it is probably just an echo effect (they're brilliant if they agree with you and stupid if they don't). This is all especially true if you aren't an expert in the field under consideration. Don't apply this logic to others, apply it to yourself first.

Metacognition refers to the act of thinking about thinking. This is the same sort of concept as Freud's superego, or what phenomenologist philosophers call our consciousness. Kahneman might refer to it as "slow thinking".

In general, our brain has two methods for making a decision. By "decision" I mean just about any sort of conclusion about something. There is a fast method that runs ahead and draws its conclusions based on instinct, training, experience and mental shortcuts. The fast method is great for things that we are evolutionarily prepared for and for things where training is effective. Generally training is effective for things that require quick reflexes and that provide immediate, salient feedback. Fast thinking is gut feel and reflexes and is not something we are aware of consciously because it doesn't formulate in language -- it forms in action.

The slow method is what we think of as "thinking". It is logical, it handles new ideas and concepts. It is what we observe as "thought". It presents itself in our language and is shaped by the way our language works. Slow thinking spends a lot of its time confabulating a logical reason why the fast method came up with what it did. A lot of our thought is spent thinking about what we just did and justifying it to ourselves.

This where our ability to forecast comes in. Most of the time, when asked about the future, people immediately have a gut feel. That gut feel is hugely biased (too many forms of bias to go into now). As long as we don't get immediate and obvious feedback that the forecast was wrong, our brain has mechanisms in place to pat itself on the back for another great forecast. Even if the forecast is terrible, if the feedback is subtle or slow, our brain still thinks it did a great job and our metacognition formulates a story to back it up.

Unfortunately, most of the forecasting situations we face in the modern world don't provide immediate and obvious feedback.

This is the reason that we're all bad at estimating our ability. We are best at identifying problems when we have expertise with a subject. The better we are at something, the more capable we are of identifying when we screw it up. Conversely, the worse we are at something, the worse we are at identifying how bad we are. We blithely go along, thinking we're pretty awesome at the stuff we're worst at. Those who are the best at things are also the most aware of their shortcomings.

There are lots of examples of this in the world. I guarantee that Tiger Woods is far more critical of his golf game than a typical weekend duffer. Proof-reading the English language is another great example--if you don't know the rules of grammar very well, then you won't be aware of your errors. Driving is another example where people are notoriously bad at identifying their own skill (except race-car drivers, who are typically very slow drivers on city streets).

So what?

Well, this whole thing applies to business, sure. I think what's really interesting is how much it applies to politics. Look at how many people who know nothing about economics are judging the economic policy pronouncements of the Presidential candidates. Look at how many people who know nothing about police training are hypothesizing about their motives. Look at how many people who know nothing about life experience as a black American are denigrating their perspective. Science would tell us all those people are probably wrong and atribiliously wrong about it because they are even ignorant of their ignorance.

Mutual Admiration Society

Be careful what you hear, it may not have been what was said (or what was meant).

I just finished Malcolm Gladwell's wonderful podcast series "Revisionist History".  There are many interesting threads through the various episodes, but the big meta-theme that loomed large for me was the danger of confirmation bias.  I think it was the second episode  (the one about Saigon) that really hit on the issue.  It is worth the listen.

This problem is giant when you're building a business model.  It is very hard to keep somebody in the room who honestly thinks your idea is stupid, but that person is essential.  I'm not talking about some devil's advocate who will throw out half-baked objections before rolling over and giving you a false sense of accomplishment.  I mean somebody who really thinks the idea won't work and is willing to try hard to articulate why.

"Oh, you're so smart and your new business idea can't fail!"

What comes naturally is that you will find somebody else who is enthusiastic about your idea and will be supportive.  Often this is somebody you already know, and probably you're on friendly terms. After all, that's why you brought the idea to her in the first place.  She'll talk up how great your idea is and you'll recognize what a genius she is for liking your idea.  The two of you will add more friends until you've built up a real, proper mutual admiration society around self-love as represented in lavish praise for one another's brilliance.

"We're all so smart and funny and we're all going to be super-successful!!"

This is great when you're starting out!  Everybody except the most ruthlessly anti-social people need external approval of some kind.  The trick is knowing when it is time to stop bolstering your confidence and adding good news and start sorting out the potential problems, doing your pre-mortems with serious abandon and sorting out the real contingency plans.

At that point, it's essential to realize that sometimes the smartest people in the room are the ones who are willing to say "I don't get it" and your best friends are the ones who are willing to say "that's stupid."

Management Fail!

A developer in my org came to me with a technical question she was working on puzzling out around something we call "Decision Units" in portfolio.  Decision units itself is a topic worth its own post, and not really important here, but the gist of it boiled down to this:

Her: "We have a problem handling decision units that exist in multiple business units, where analysts in different groups aren't aware of all of the logic relationships that might exist for that project."

Me: "What's the problem?"

Her: "Well, the way we're handling things right now, if anybody makes a change to the logical relationships for a given project --if they add a dependency or some sort of optional relationship or anything-- it will wipe out all the stored values for that project.  Worse yet, the analyst who loaded them before won't know they were wiped out, and when they go back to load them again, there will be a bunch of new alternatives there that they never heard of!"

Me: "That is a problem!  What do you think we can do about it?"

She gave me a good explanation of the issue that indicated she understood both the technical character of the problem and the business character of the problem. 

She really had her head around the thing.  She then delivered a very technical, possible solution that involved setting some flags and having users determine several things in advance and some other business rules around the process flow.

Me: "I think we can do something simpler.  Here's what I think..."

And then I went on to sort of brainstorm out a couple of default states and business rules that eliminated most of the possible sources of trouble.  Lastly, I sketched out an approach that would handle the last situations elegantly and simply and without requiring the users to do anything they weren't already doing.

Her: "Wow! That was really great!  I'm glad I came to you.  Here I was, thinking of something super-complicated and all I needed was to talk to you and you had a simple answer right away!  Thanks!"

I felt pretty good about myself for a few minutes and then realized I had totally failed her as a manager.

With a little coaching I'm pretty sure that she could have come with the answer on her own.  Maybe a few questions, or at the very least I could have structured the conversation more collaboratively.  I sort of pontificated how I think it should work and we were done.  My developer was left maybe feeling good about my skills, but probably a little less confident in her own.  That's a big fail as a manager.

this is a clever metaphor!

I think this sort of thing is why it's very hard for managers to make those "Career Pipeline" transitions from independent contributor (IC) to manager.  We build up these technical skills and are rewarded for them, and often they become a source of pride and self-worth.  When problems come along that are in the sweet spot, we think immediately of a fix.  We miss the chance to do our new job, which is coach the person bringing the problem as they figure out a fix.

Intelligent Software Building: products, platforms, programs

On a regular basis I see folks who are trapped by the siren song of reusable code.

Now, don't get me wrong.  Reusing code is a wonderful thing, and it's a good goal, but it's important not to get carried away.  There are basically three levels of reuse:

1. Programs - this is when you reuse parts of your own code, on a personal sort of level.  You built these routines and snippets, documented them and like them.  They're something you've done before and know you want to do again, so you were kind and thoughtful to yourself and wrote it in a way that you would recognize.

2. Platforms - this is when you specifically plan for reuse when you're writing something.  Platforms refer to a broad area of capability that is important and that you'll want to build on.  Platforms require serious forethought.  They also require serious documentation and some support.  Typically this is something you will share with other members of your team.

3. Products - this refers to a software platform that you want to make broadly available.  When you create a software product you absolutely have to have clear documentation in and out of your code.  You have to plan for wide deployment and you have to have support people in place.  When changes are made, you have to have a thorough test apparatus in place to understand what the impact of your changes will be on others who have built their software with your product.

especially to yourself...

Be kind to the person who will inherit your code, because often that person is you. 

Many times I have cursed the me of weeks or months ago for leaving a mess for me.  When I'm kind to me, I always appreciate it.  Reuse of your programs is just good practice, and naturally happens when you're writing thoughtful and clean.

After you've experienced those benefits a few times, you'll start dabbling around with platform style reuse.  You'll think "hey!  that little block of code has been so useful to me, I bet others would like it!"  Before you know it, you have a whole set of interconnected things that you've put together and share.  You probably don't have any sort of obligation to support it, but it's fun and you're supporting your own work through the thing anyway, so it just makes sense. 

Ain't nothin' wrong with Platform style reuse.

Where you have to think long and hard is around the Product-style reuse.  This makes sense if your goal is to sell this code as a development tool.  This is something like D3, or Telerik, or the Unreal Engine, or Havok or anything like that.  More often than not, I see this when developers fall in love with their own, potentially reusable programs, assemble them into a platform and then decide the whole world needs it.  You will have tons of support work to do.  You will have to maintain forward and reverse compatibility.  

If this is some sort of exercise in self-love, you need to stop now and slowly back away.  Really, unless you're planning on making enough revenue from this (or if internal facing, saving enough effort) to justify several developers and a testing team, this is just a bad idea.

Sometimes a platform may naturally find demand in the market.  Maybe the platform you built is truly wonderful and well documented and bullet-proof and you really really want everybody talking about how awesome you are... Even then, don't do it unless you have the support staff.  The first time some package you're including gets depricated or MS rolls off some capability you're counting on or any other of a million possible things, you're going to have all those people who were singing your praises on your IM, demanding immediate fixes...

software III

Don't pee your pants for nobody

This one is pretty similar to the "you'll regret the hack" thing, and I see this a lot with green developers.  With deadlines looming and a rush to get things done, you may be tempted to degrade your work from "software product" to "proof of concept".  What this means is that you're no longer building a software platform you can add to later, instead you're building specific features to try and get somebody off your back.  It works out a lot like the old metaphor about peeing your pants for warmth.

you're warm for a while, anyway...

Essentially, you solve your short-term problem but wind up with a bigger, long-term problem.  This is bad when it's a team decision, and even worse when a developer does this without telling anybody.  I've personally been in the situation where one of my developers did this sort of thing and I wasn't close enough or careful enough to know that's what was happening.  Once we got past UAT (user acceptance testing) and I started rattling off the next steps and new features to add, he turned defensive and told me he needed a few months to rework everything because everything he had just done was purpose built to get past that first deadline.  He literally set us back 6 months and absolutely destroyed my trust in him.

Hire fast, check-in faster, fire fastest

Back in the games days I had an employment model that was built around the fact that I had no idea how to hire people.  In retrospect, I was probably ahead of my time, since research seems to indicate that interviewing is almost worthless, GPA is a poor predictor of work performance, and things like pedigree and credentials are over-rated.  This was before I had ever heard of a behavioral inteview, much less experiened or conducted one.  Since I worked in computer games, every job opening I had was flooded with candidates, and often these candidates were non-traditional.  They were often wiz-kids with no college or formal training.  Sometimes they were older folks with a passion for gaming.  Some were hobbyists who wanted some extra income (that was fine, I had plenty of contract-work jobs available, too).  The hard part wasn't finding people, the hard part was knowing which ones I could count on.

This was especially important for my projects with really short timelines and really tight budgets.  One really flaky developer could derail the entire project (and with it the budget and with the budget my paycheck for weeks or months).

The approach I came up with was "Hire fast, check in faster, fire fastest".

I googled "you're fired" and somehow this image came up.  It somehow seems better than anything else I could have put here.

Basically, I gave a job to anybody who wanted one.  Every new employee would get a single work product to deliver and two weeks to deliver it.  The work product was always something I actually needed and something that I knew could be done in two weeks.  Anybody who didn't get the work done on time was fired.  No questions asked, no conversation, no excuses.  What I had learned was that anybody who is willing to flub their very first impression will continue to disappoint thereafter.  Honestly, this got rid of around 80% of the people I hired.

I've applied this rule in a general sense in many ways since.  If I'm looking to hire a painter and he shows up late to discuss the quote, I take him out of the running.  I once had a buddy ask me to help him find a contractor to build a block wall around his house.  The guy showed up 30 minutes late and I told my friend not to hire him.  My friend did anyway and since I'm telling this story here, I'm sure you can imagine he never got his wall.

Hit the easy problems first

There's the rule when you're playing pool.  When one of your balls is sitting right on the pocket, you don't shoot that shot.  It will probably get knocked in on its own and you're really just wasting a shot.  That mindset is completely misplaced here.

a tempting but misleading metaphor!

Really this is probably just a good project management thing.  If you leave the little problems until they end, nothing good can come of it.  If they really are little, you'll just have a shit-ton of little problems to sort through and fatigue will build.  If any of them turn out not to be little problems (and some of them will end up a lot bigger than you originally thought) you suddenly have big problems that you weren't expecting, popping up right at the end of your project, when you don't have time to course-correct effectively.

Do those little, easy-looking things as early as you can.  Sprinkling in small wins helps to keep you from getting worn down on big projects.  Also, it gives you the chance to adjust your timelines or features if you wind up facing some big, unforeseen shitshow!

this thing again

software development II

You will always regret the hack

Every project gets hairy at some point.  Deadlines start to loom, customer requirements start to tighten, subcontractors and ingredient suppliers are late or flake out entirely.  The team gets stressed and the iron triangle reminds you of its immutable force:

the ol' iron triangle

Adding resources at this point never works.  Even if you have some ninja team of superhuman developers in your back pocket, they won't be able to parachute in at the last minute and do anything right away other than churn the project and introduce more problems.

Slipping the schedule is always tempting.  Reducing features is often what happens.  Sometimes we're smart about it and try to stage them together; reduce the features in the first release, with a schedule for releasing the remainder of the features in the future.

The worst thing that happens is when the decision is made to fight the triangle and keep the same schedule with the same features.  When that happens something else gives--Quality.

In software, when quality slips this usually means that the developers start to introduce hacks.

Magic numbers start to appear instead of declared variables.  Convoluted blocks of nested code get tucked into other functions instead of broken out into separate functions.  Blocks of code get commented out.  Blocks of code go undocumented.  Temporary variable names that make no sense get introduced.  Single purpose design takes over and expandable, scalable features fall by the way-side.  Unit tests and BAT tests go unwritten.

Your software product effectively gets downgraded to proof-of-concept.  When you need to build more later, you're instead spending time refactoring the junk you slapped together.  When users find errors it takes 5x as long to troubleshoot.  At the end of the day, you give up a lot more than you gained and always, always regret it.

Software Development

I've been doing professional software development of some kind or another since the mid-90's.  I wrote my first program on a Timex Sinclair when I was 10 years old, and taught myself Assembly to try to squeeze more into the 2K of RAM it came with.

pictured here with the 16K RAM upgrade!!  That changed everything!!

A lot has changed since then (thankfully).  Along the way I learned a lot of lessons--all of them the hard way.  Looking back at my first attempts at building a company, developing a brand, organizing a development methodology and cultivating people I realize I did things about as wrong as I could.

Here are some of those lessons.  Maybe you won't have to learn them the hard way, too...

Produce excellence, especially when you're not asked for it

My first commercial products were euphemistically called "Casual Games" or "Budget Extreme Sports Games" or more correctly "Shovelware".  Google "Extreme Paintbrawl" and read some of the reviews.  Enjoy the laughs and then come on back.

Extreme Paintbrawl and the games like it were precisely what our publisher hired us to do.  One important guy at a publisher told us "You could literally take a shit on the CD and I could sell it with the right licensing, just get them done as fast as possible."  So that's what we did.  We banged out junky games as fast as we could, because that's what we were hired to do.

This kept the lights on, and kept our customer satisfied, but it didn't lead anywhere.  Later on, when we tried to land bigger, higher-quality projects, nobody was interested.  We had a reputation for producing junk games at light speed.  Literally nobody would give us a shot at producing a quality game with more time and more budget.  In retrospect, they were probably right.  We had a shovelware culture and I'm not certain we could have turned the corner.

At the same time, a friend of mine with a very similar company and very similar starting point aligned his people around producing beautiful, compelling games.  He had the same customers, the same deadlines and the same budgets.  They were late all the time, the games were completely over-engineered relative to the spec, and they didn't make any money on those games, but they proved that his group could do really nice work.  They successfully segued into AAA games while I wound up throwing in the towel and leaving the industry.

Get a monkey on a football

On one of my early projects we had to develop a considerable amount of development infrastructure from scratch.  We were building a new rendering engine, new level design tools, new asset management tools and were working out of a basement.  Weeks went by with no visible progress to show the customer.  They started freaking out and threatened to cancel the contract.

I came up with a schedule of visible milestones to deliver to the customer.  For the first one I said "Look, I don't care if we render a monkey riding a flying football around the course, we have to show these guys something!!"  "A Monkey on a Football" became our catch-phrase for tangible customer deliverables that may not be completely required for the technical aspects of the development, but are completely required to make sure that your customers know you're moving forward.

It is a lesson I should have internalized from an earlier interaction with a guy named "Joris" in the Netherlands.  Joris had worked for us before and we hired him to write the AI for Santa in one of our early projects called "Nuclear Winter".  Joris kept telling us he was making progress and we kept believing him without demaning tangible proof.  When the time came for Joris to turn in his code, Joris disappeared.  Then he turned in corrupt zips, then disappeared again.  Eventually, Joris admitted that he hadn't done any work at all.  He had spent his entire advance on phonesex with some woman in Kansas that he was in love with and he needed more money from us and then he promised he would get right to work.  

So now, I always demand a monkey on a football from the folks who are working on my team, and I always give a monkey on a football to my stakeholders.

I have a few more, but my blogging time is up for this week... I'll post them another time.

options in the wild: part II

At the very beginning, I did a straightforward ROI type of analysis.  The decision seemed very clear from that perspective.  There was one line that took a little more work to get to, but was half as long and had twice as many windows operating.  It should have moved twice as fast.  Easy choice!

I hadn't considered the uncertainty in the model.

The people in line "d" were long a call option on all of the windows to the right.  If any of the windows from "e" through "h" opened up, they could easily exercise that option by walking over and moving to the new line.  I hadn't thought of that until it happened!

Once window "e" opened up, I sat there considering moving to line "e", thinking they might open up "f" or "g" or "h" and I didn't want to miss that opportunity, even though it meant moving to the back of the line.  I decided I was pot-committed to "b" when the other option in the mix made itself known...

The crew of any plane that happened by was long a call on window "a" and "b"!  Since I was in line "b", I was effectively the counterparty and short a call on those windows.  When the crew turned up and took over the windows, they exercised their call and my line came to a stop.

This story is a pretty good representation of the value of using a real options decision framework for a couple of reasons.  The most obvious reason is the importance of involving uncertainty in your decision.  If nothing had changed, my first choice was unquestionably the best.

The second reason is the importance of having a dynamic decision framework that can adapt to new information.  It's easy to make a straight-forward ROI type decision when you ignore the changing world.  One line was served by two windows and was half as long!  That's an idiot-proof decision!  A few minutes later, the decision was more complex.  I was at the end of a line that was roughly the same length as the others, but didn't have the advantage of more windows possibly opening up.  A few minutes after that, I was at a stand-still.  A robust, options-based decision framework that encapsulated the upcoming decisions would have helped.

From an executive decision making perspective, this sort of decision tool is almost never available.  What can be available, however, is some intelligence about what the inflection points are.  If I had any idea about the chances of another window opening up, or what the chances of a crew showing up, I could have produced some heuristics about comparative lengths of each line and made dynamic adjustments to my strategy when the real world changed in front of me.

Options in the wild

A few weeks ago I was travelling on an international business trip.  On the way back I came to the top of an escalator to face the immigration / passport check area.  It looked like this:

(only the lines were a lot longer)

I just jumped into the line that was right in front of me and then sort of observed the lines.  The mental schematic I put together was this:

So I moved through the crowd to line "b" because it was the shortest one I could join (since I don't hold a EU passport).  Also, I realized that there were to windows serving that lane... both window "a" and window "b".  No wonder the line was shorter!

A minute later, the situation looked like this:

As the efficient, German airport had recognized a long line and opened window "e" up, and a mad dash had taken place as people near the end of the line for window "d" scrambled to be at the front of the new line.  My old location at the end of line for window "d" would have put me in pole position for the newly opened window!

Then, a minute later, this happened:

When the crew for some flight showed up and took over priority on window "a" and window "b".

Being a ridiculous person I thought "This is fantastic!  What a great example of real world options analysis!!  I can't wait to write this up!"

What are the options?  Take a few minutes to think about it.  In a few days, I'll publish my thoughts on it, but I would really like to hear what others say, too...

Visualizing your business model

The elements of the business model

If you haven't read my last post about the elements of the business model, do so now.  There's not much sense in reiterating what a prospect is, or how it fits with a product, etc.

Now, check out this picture:

The visual business roadmap

There is a great deal of information in this drawing.

Along the bottom you have the projects associated with your technology (product development) roadmap.  In the drawing I have here, you have two layers--real business can have many more, of course.  The bottom row represents core technology programs.  The layer above that represent product development projects.  Each product development project relies on one or more core technology program.   This is very typical.

Along the left side are market development projects.  These projects are intended to change the appetites and buying behaviors of the markets in general--whether or not you will capture the business produced.  For example, when Apple introduced the ipod the TAM for portable MP3 players exploded.

Along the right side are all sorts of projects.  In a real portfolio this is where most of the decisions really are.  These projects can be things like customer enablement programs, directed marketing programs, and anything else that will impact your expectations around your prospect.  That is to say, anything you think will impact the market share and/or pricing expectations for that product with that market.

Often a project will have impacts at multiple elements of the drawing.  

For example, adding a cool, new feature that has never been seen before may encourage people who would never have bought a product like yours to enter the market (TAM expansion), will help you increase your share of that newly expanded market (MSS expansion) at an increased price (ASP increase) but with the downside of an increased product cost (Cost increase).  In this case, you would be impacting markets, products and prospects all at once!!

modeling your business - projects, products, prospects, markets

What are the fundamental elements of modeling a business?

I often help people figure out how to model and value their business concepts.  A really common concern is "My business is different!"  That rallying call often serves as the number one roadblock people have when trying to build a business analysis.

The fact is this: No matter how different your business is, it's different in one of the same few ways.

All businesses expect to have customers.  All business hope to deliver something of value to their customers.  All businesses do some up-front work in order to have that valuable thing and to make sure their customers know about that valuable thing and to actually deliver the thing to them.  Whether the thing is physical (like a car, a microchip, a refrigerator, a satellite or a pencil) or a service (like a hair cut, consulting, vacation experience, healthcare), or something intangible like software doesn't matter.  The fundamental economics are the same.

If your business doesn't look like this--well you don't really have a business.

So let's take that "my business is different" issue another step.  How is your business not different?

Products

You have (or will have) something that you want to sell.  If it is a tangible, physical product, it probably has some sort of manufacturing cost associated with it.  You should get an idea of how much each unit you sell will cost to produce and deliver to customers.  This is the variable cost of manufacture, or "unit cost" for short.  Unit Costs will sometimes scale as a function of learning curves or economies of scale, and sometimes they'll just stay flat.  Sometimes (like software) this number will be zero.

Fixed Costs

In addition to the variable cost of manufacturing, you'll have overhead.  Maybe you need to build a factory in order to produce your product.  Maybe you need to build a brick and mortar store as a venue for providing haircuts.  Whatever it is, you will have some fixed costs.   This is often called PPE for Property, Plant and Equipment.  PPE will almost always scale with your expected volume, but with the critical distinction that you have to make your investment in PPE before your volume starts to realize.

Projects

Your business will probably require some spending up front that doesn't go into PPE.  Sometimes this is called RND (Research 'n' Development), sometimes it's called NRE (Non-Recurring Engineering), sometimes it's just called "Spending".  Whatever you call it, this is what you have to invest in time and people-hours and other expense items in order to get to the point where you're ready to start bringing your product to your customers.

Markets

You have customers who are (presumably) going to find your product valuable and pay you for it.  Smart companies undergo an exercise called "Market Segmentation", where they try to group their customers together into meaningful groups called markets.  Members of these markets have similar buying behaviors.  They typically value the same aspects of the products and are willing to pay similar amounts.  For purposes of business forecasting, you should take an "outside in" look at your markets and a great deal of your analytical work will be done just trying to size these markets.  How many potential customers are there in each market?  How many of them will make a buying decision in any specific quarter (or year)?

Prospects

To build a predictive business model you need to make forecasts about how well the products you're creating will be received by the markets you have characterized.  This is a "Prospect" -- the specific economic expectations for an individual product with an individual market.  You will spend most of your business analysis time on forecasting prospects.  When will this product hit this market?  How much market share can we hope to achieve?  How quickly do we think we will achieve our peak market share?  How long before we see that peak share get eroded away--either cannibalized by our own subsequent products or defeated by competitive ones?  What sort of prices will we be able to command for the products?  How will the price and the market share forecasts interrelate?

These are the fundamental elements of analysis for any business.  

Before folks should start thinking about how their business is different and how their model won't fit into a typical analysis, they should address these questions and frame the business through these lenses.  After these basics are sorted out, you should start worrying about things like competitive timing, synergies and roadmap effects, market saturation and disruptive effects, complex pricing structures and so on.  Normally those sorts of details won't move the needle on the big analysis.

This isn't to say that the more detailed analyses aren't necessary.  If you're in the middle of negotiating tiered pricing discounts with an important customer you should really figure out what those will mean to your revenue and NPV expectations.   Those things can wait until later.

Next time I'll put up some materials one the mechanics of using these essential entities to create an NPV.

Politics and Market Segmentation: Why is Trump/Sanders a surprise?

The current primary season had me thinking about market segmentation this morning.  In particular, I was thinking about how the US population is traditionally understood:

If we assume that people are basically normally distributed along this one-dimensional scale, you end up with a distribution that would look sort of like this:

Where each of those little blue dots represents some voters (the data is made up, with a random, normal distribution).  There are some way to the right and there are some way to the left, but most of them are clustered in the middle, with increasing density as you move toward the middle.  The two major parties basically jockey over where the dividing line is and try to establish their candidates to capture as much territory as possible while playing chicken on the issues they think they can slide by on.

The fundamental problem is that this market segmentation is based primarily on how American politics is organized and not on how American individuals actually think.  For example, let's assign some typically assumed attributes to each party:

Do these clusters make sense?  If I believe strongly in Christian values does that mean I have to be opposed to affordable health care?  If I think that affordable education is important, does that mean that I must also be opposed to gun control?  This is a market segmentation strategy that is "Inside Out" -- it is based on how the governing parties are organized.

Smart market segmentation is "Outside In" and starts with the criteria people use to make decisions and then organizes around that.

Instead of this one-dimensional approach to segmenting voters, let's consider some two-dimensional approaches.  For example, let's consider some classics, such as personal freedom v. national security or government size v. economic stability.

Or, let's make this very simple and assume the same old Left/Right continuum and add a single new dimension of "Anti-establishment".  In this case, I might argue that the population looks more like this (this data is completely made up):

The distribution along the X-axis is the same here as in the previous drawing.  What's different is that I now have that Y-axis to represent that person's opinion regarding the political establishment.  With a wild majority of people effectively sick of the status quo candidates and more interested in finding an anti-establishment candidate than a traditional left/right candidate.  This is why Sanders and Trump are popular.  Both of them represent a break from the traditional choice that voters are given and are a chance to vote on a different (more meaningful) decision vector.

It's easy to chalk up Trump supports as racists and Sanders supporters as crackpots -- and maybe there's some truth in each, but the reason for their popularity is that the two choices that are available fail to satisfy the buying public.

Will your project survive in a big company?

This is based almost completely on work by this guy:
http://www.thomasthurston.com/

He found 8 questions that will effectively predict whether your new business line will succeed in a large organization.  I've paraphrased them and reworked them a little according to my experiences.  It has been a while since I've seen the originals...

1. Are the margins big enough to be interesting?  (Interesting >= current margins)
2. Are the markets you're serving big enough to be interesting? (Revenue contribution >= current revenue * 1%)
3. Does the project help sell products the biz is already selling?
4. Is the product in a market category we already participate?
5. Does the program help combat a top competitor?
6. Is the target market available through current primary channels?
7. Is there no perceived threat to an existing top customer?
8. Is there no perceived threat to an existing, power internal group?

If you can answer "YES" to everything, congrats!  You may survive the business funding cycle!

If you score one "YES" you're looking at a 50/50 chance of surviving each funding cycle.

If you score two "YES" answers... well, forget about it.

You might have a great business concept, a wonderful go-to-market strategy, and competitive products that serve a real market need.  If the corporate anti-bodies are going to reject you... you'll be rejected.

I have seen this happen several times.  There are reasons for a lot of this behavior.

For example, if this margins are too small, it will drag down the company's margins and starts to look like a growth trap.  From a strict-finance perspective, you're lowering the effective return for shareholders.

Most of what you see there are questions that are associated with diversifying the business.  Expanding into new markets, serving new customers (or pissing off old ones), attacking new competitors, etc. -- these are all outside the core and scary and difficult to do.

This is why big companies often fail to thrive.

The tyranny of an agenda

or, "How I learned to stop worrying and love the chaos"

or, "Don't cancel that meeting yet!"

If you work in a big company, chances are you have a lot of meetings.  I have lots of meetings.  I use Outlook religiously for meetings--even meetings with people outside my company.  Hell, I use my Outlook calendar to book time for things like dates with my wife, time with my kids, etc.  This is basically the only way I can ever remember to show up for anything.

I also have lots of work to get done on a regular basis.  I use Outlook to block out time to do that work.  If I think something will take me two hours of uninterrupted time to do it, I find a block of two hours and put a placeholder there.  Sometimes, when I'm in the middle of something I'll drop a block of work time on my calendar with "Todo" in the title, and in the body will be a bunch of other things I need to to block out time to do.  Blocking time to block time.  It's all very meta.

When that calendar starts to fill up, the first thing I usually try to do is skim it over, looking for optional meetings.  It usually goes like this?

1. Did I organize this meeting?  Am I running it?
2. Ok, I'm not running it, but will people be asking me for my opinion?
3. OK, I probably won't say anything, but will I learn something at this meeting?
4. double book right over it...

Sometimes though, I have a meeting that I did organize but doesn't have an agenda.  I might have booked the meeting a few weeks ago, expecting something to pop up.  Alternatively, the reason I booked it in the first place might have been taken care of.  These are really tempting candidates to cancel and get the time back.

Don't cancel them.

These sorts of no-agenda meetings with people who I apparently had sufficient cause to book a meeting at some point often end up the best conversations I have.  Ad-hoc meetings with no agenda are free to ramble a little.  They don't follow a script, so unanticipated topics organically bloom.  Some of my best projects have erupted from these unpredictable volcanoes.  I'm running out of metaphors to mix here.

Common Pitfalls - part 2

Last time I wrote about common problems in business plans that are associated with market sizing, forecasting and especially about estimating market share.

Other common issues are associated with pricing and margin expectations, defining the target market, project risks and customer contact.

Pricing forecasts are too aggressive

We see two sides of this coin.  For established products, we rarely test the limits of pricing for product line extensions, roadmap refreshes and similar product introductions that have the possibility of increasing our asking prices.  In these cases our models will always hold pricing flat, and thereby underestimate the revenue potential.

On the flip side, for new products in currently unserved or underserved market spaces we regularly overestimate the likely selling prices.  A common method to estimate these prices is to do a value chain analysis, some sort of should-cost and required margin analysis and then to determine how much profit pool is left in the portion of the value chain that the product will displace.  This makes a lot of sense and gives a good feel for how much the product could sell for.  Then, when it's finally ready, sales has a volume quota to meet, leadership gets worried about failing to land anchor customers and everybody sort of loses their spines.  The result is that we ask a laughably low price and the customer readily agrees!  This is a real "Flaw of Averages" sort of problem -- you will never get a higher price from your customers than the one you ask.

Margins are unrealistic for a new product

This one happens when we do a lot of market benchmarking to try and estimate prices based on our expected production costs and the product margins reported by competitors who are playing in the same (or comparable) markets.  Benchmarking is a great place to get started, but eager analysts and business managers often overlook the fact that incumbent competitors have had a head start to work out their cost structures, to improve their manufacturing flows and to feel out the best price points.  Their margins probably have improved over time--our first few years will not be so fat.  Similarly, fat margins happen in low-competition environments so by our very entry, everybody will have to tighten their belts and margins all around will probably come down.

Serviceable Market is too broadly defined

Often we will do market surveys that start with a "Total Available Market" this is broadly defined and we draw our numbers from commercially available sources like IDC or Gartner, etc.  For example, if we're working on a part intended to be included in new home construction as a sort of OEM Smart Home product, we will buy research around new housing starts for the next 10 years or so.  Ignoring the fact that these reports are almost never stochastic (they almost never include estimates of uncertainty or ranges for their estimates), they also almost always describe a very broad potential market.  For this reason, we try to define a portion of the total market as a "Servicable" market.  This is the subset of that market that could hypothetically meet their needs with the product we're intending to develop.  Often this segmentation is just as broad as the Total Available segmentation and we overlook the nuances in purchasing decisions between members of that sampling.

Sticking with our smart home example, if the key features of this product are all associated with climate control (like Nest or something) then you should anticipate that it won't be very attractive to new home builders in mild climates.  You can also safely assume that builders below a certain price point (like KB) will have a much lower sell-in rate than custom builders or luxury builders (like Toll Brothers).  It's common that a business manager won't consider all this and will just say "let's just assume we can address 80% of the market".

This approach is not only guaranteed to misjudge the size of the market that we could really address, it also skips past one of the most important parts of building up a business plan; actually talking to customers.

Business Plan never learned about actual customer preferences

I almost forgot this one is on the list, and in retrospect I'm not sure how I could forget it.  This is the case where an entire business plan is built up around a perceived use for a product that we already have--it just takes some modification or rebranding--with a new group of customers that we have little or no experience with.  It can be surprising how quick people can be to extrapolate their own experiences into conjecture about a broad segment of the buying public.  Compounding this problem is that (typically) between our business plan and that hypothetical buying public is a constellation of partners, manufacturers, retailers, channels, and co-travelers, any of whom can throw up a "I don't get it" roadblock.  We should have a really good answer to that objection before we even start to get serious with the rest of these forecasts.

Project risks are ignored

For most of the analyses I see, the actual Project spending isn't material to the overall NPV of the project.  By this I mean that the amount of the NPV is de minimis when compared to the margin dollars involved.  For this reason, it's common for teams to ignore project risk when building up their forecasts.  The business development managers involved will pulse the project managers for a general idea of how long it should take to develop and how many heads will be required and some real primitive multiplication-type analysis will take place and the RnD budget for the project is roughed out.

This approach really leaves a lot to be desired.  For one thing, "a failure to plan is a plan to fail" and RND project management really needs have at least the major serial stages of development roughed out, with the parallel activities in each stage identified and some sort of risk register built up.  Even this basic analysis will often show that the back-of-the-envelope timeline proposed above will really underestimate how long the project will take.  Sometimes the project timeline balloons to as much as 2x when taken in this sort of risk-adjusted, expected value perspective.

Whether or not the spending itself becomes material, there are usually other projects waiting for those people to finish up and move on to the next thing.  Even more importantly, a product that hits the market months (or even years) after the expected launch date is very likely to experience a very different competitive environment than the original forecasts of market share and market pricing called for.  Competitors may have produced improvements to their products, new competitors or new substitute products may have appeared, in some cases the very ecosystem purpose for the product may have rendered it obsolete.

Common problems in business models - Market Share

Common Pitfalls when estimating Market Share

There are a few things that seem to pop up on a regular basis when a business analysis / business plan is being developed.  More often than not, these are the product of biases and short-circuited heuristics on the part of the analysts and business owners involved.

When I'm doing an analysis review (and especially when I'm doing a long day of these reviews) I start with this list and immediately look to make sure these problems don't exist:

Competitive environment is poorly understood

Most of our business decisions address a competitive environment in which we are the incumbent.  This sometimes trips up business managers who are looking to extend our business into adjacencies or into transformational arenas.  If the plan doesn't comprehend the current competitive situation as well as how that competitive environment is evolving, it's time to go back and break out Porter's Five Forces again.

Market share ramps too quickly

I occasionally see some really enthusiastic projections.  I mean, really enthusiastic.  For example, a brand new product, that serves a market we're not familiar with, customers we don't currently do business with, through channels we're unfamiliar with, but we'll ramp up our volume faster than the last iPhone.

Peak market share is too aggressive

I see this one a lot.  The logic seems to be "if we can't capture first place in the market, why would we even enter?"  This might be true, but it doesn't mean that we will capture first place in the market.  If the peak market share for a product introduction is not consistent with the competitive environment, it needs to be adjusted.  Similarly, if the peak market share makes volume assumptions that we can't physically produce with our current manufacturing facilities, supply chain and channel structure, it needs to be adjusted, too.

(more to come)