The Forgotten Art of Estimation

My third grade class spent a lot of time studying “estimation”. Exercises were things like guessing how many jelly beans were in a jar, or how long it would take to fill up a gallon of water from a slow drip.

At the time, it seemed like a silly thing for eight and nine year olds to spend time on. Who needs to know how many jelly beans are in a jar, unless they literally want to become a bean counter?

Of course, I had no inkling that the Internet would emerge, and would be a dominant force in the world, and I’d spend most of my waking hours building products for it (give me a break, I was 9). So I didn’t know that estimation would be an essential skill — arguably the essential skill — for anyone deciding how and what to build online twenty-odd years later.

Why does estimation even matter? Well, if you can accurately estimate the effect of building Y or changing X, you wind up spending your time building things that your customers will use and that will help your business. It’s an especially valuable tool when you’re building something well-defined on top of your existing product. Twitter adding their own check-in feature? Should be pretty easy to make a good estimate of the effects. Google pushing out their glasses product? Much harder to estimate.

Many very smart people in Silicon Valley are surprisingly poor at estimation. LinkedIn is a rare visionary company, but in 2005 and 2006 we wasted a lot of time on features that had a very small probability of moving the needle for the business.

Let’s set the stage for those feature decisions. It’s 2006. LinkedIn has around 50 employees, and around 5 million registered users. We’re doing okay: our year-old job posting and premium subscription products are generating real revenues, and we’re getting close to profitability. But our new user growth is flat, most of our users have zero or one connection, and morale is only fair: half our engineering team left last summer for the hottest tech company around — Google.

The board and/or exec team decided to focus on LinkedIn’s usage more than on revenue or growth. That’s a debatable decision, but not necessarily a bad one; for this exercise, let’s just work under the assumption that building products to increase usage among existing users is the right thing for the team to do.

To estimate increases in user engagement, it’s valuable to understand and look at the individual levers which can lead to increases. For a social network like LinkedIn circa 2006, there were three major ways people would return to visit the site:

1) They’d directly type in the URL, or click on a bookmark in their browser.
2) They’d get pinged (emailed) after a friend or acquaintance performed an action: invite them to be a connection, respond to their comment, etc. They’d click through on that email.
3) They get pinged (emailed) independent of actions by other people, e.g., a marketing message or feature update. They’d click through on that email.

This isn’t exhaustive or current for LinkedIn or other sites: Google search is no doubt a huge driver of re-engagement for sites like Yelp; Like/Share/Tweet widgets across the web encourage lots of usage of Facebook/LinkedIn/Twitter; category 2 is much more built out than it was in 2006 with mobile notifications, Facebook and Twitter integration and more; mobile apps create huge amounts of category 1 usage today but didn’t in 2006.

Option #3 (non-social emails) wasn’t really on the table for LinkedIn in 2006: we’d had a couple of successful pure marketing emails, but no template for replicating that success (it likely would have been possible — sections 3 and 4 of my post on the value of large data sets explain how we did this at Circle of Moms — but would have been more of a pivoter move than a visionary one).

That left us two ways to increase engagement with a new feature. The feature could be so memorable that it would get someone to come back more frequently on their own, or it could generate more/more relevant pings by their friends, inviting them to connect or telling them about activity.

So now we have a baseline for estimation: anything we’d build to increase LinkedIn’s engagement would have to do one of those things.

Unfortunately, we built two major features in 2006 — Services and Answers — and neither one ever really had a chance to succeed at increasing engagement.

Services was a way for service providers — accountants, lawyers, auditors, and many more — to list their offerings. Providers would structure their LinkedIn profiles more carefully, so that they’d fit into a directory. Other users could then go and find, for instance, Sarbanes Oxley experts in the Bay Area.

Services, our team hoped, would provide one clear use case for the (many) people who asked “what do I use LinkedIn for?”: you use LinkedIn to find an accountant. So, that begs the question… how often do people need to find accountants? Considering that most Americans do their own taxes, and those who don’t probably keep their accountant for an average of at least a few years, it’s unlikely that over 20% of American households look for an accountant in a given year. So that means LinkedIn might have an average user look for an accountant once every five years.

Add in lawyers, Sarbanes Oxley experts, branding consultants and dozens of other specialist service providers; you get more frequency — but not much more. Best case, people find themselves searching for those sorts of specialists every few months. So even if LinkedIn got all of those searches — and I’ve heard about a site called Google where that search thing is popular — we would have brought our users back to the site maybe once every two months. Since there was nothing inherently social about the services product, friend pings wasn’t a viable means of increasing usage. In other words, we weren’t likely to get the deep engagement we were looking for.

The other product we built was Answers. Yahoo had recently built what was seen by Silicon Valley as a very successful product — Yahoo Answers had much better usage than Google Answers or any of its competitors — and LinkedIn saw opportunity in building out a professional version. Answers had a vision somewhat akin to Quora, but without functionality to follow (non-connection) users or follow topics.

Let’s again go back to the key engagement questions: would Answers provide a reason for people come back to LinkedIn regularly of their own accord, and/or because they were pinged by friends?

The vast majority of Internet users don’t contribute deep content, whether questions or otherwise. We didn’t have any reason to think that questions would be special; again a best case scenario would be that users might ask a question every few months.

That means two things. One, because most of our users had fewer than ten connections, it would be very rare that a typical one would get pinged saying “your friend has asked a question”. Second, if and when that user logged in, there would be the same issue: not many questions by friends.

In summary, we weren’t creating much directly relevant content and weren’t providing people with good reasons to come back to the site on their own. Not a winner.

[Aside: it's even debatable as to whether Quora's solved these problems, in spite of their better model (followed topics and followed users) that better facilitates discovery. I love the Quora product, and they have a tremendous amount of interesting content, but I'd guess that most of their users aren't regularly getting pinged about relevant stuff, nor are they logging in randomly to see what's going on.]

LinkedIn’s estimation process failed because the collective logic within the company was as follows:

1) It’s not clear to people how they use LinkedIn. How do we give them more obvious use cases?
2) Once we give them those use cases, they’ll use the product.

Unfortunately, the question “how and how many of those people will use the product for each use case?” didn’t really get asked, and we built a bunch of stuff that wound up getting scrapped. Good estimation generally happens when you ask good but simple questions, and follow them up with other good and relevant questions. What percentage of people will do X? Once they do X, what percentage will then do Y?

Of course, I didn’t manage to convince my peers that other features (most notably, those that increased the number of connections a typical user had) were more important than Answers or Services. I’m glad to report that LinkedIn before and after made many solid product decisions, and has of course attained great success in the process. Still, we had two major instances where the estimation process left something to be desired.

At Circle of Moms, we were okay but not great at estimation. That’s largely on my shoulders — it’s something that management needs to push people on by asking them to estimate the effects of product features and then justifying those assumptions. The process doesn’t need to be anything formal; the back of the envelope is literally fine. I’ve heard that Zynga is phenomenally accurate at guessing the effects of different features — in part because they’ve built so many games — and that serves as an enormous advantage. It’s also something that “business” types — most notably finance types — do well to size markets. In my next startup, I’ll make a conscious effort to bring the skill of estimation to my product team.

Mike Greenfield founded Bonafide, Circle of Moms, and Team Rankings, led LinkedIn's analytics team, and built much of PayPal's early fraud detection technology. Ping him at [first_name] at mikegreenfield.com.