No Job Left Behind / How to Not Shoot Yourself in the Foot

It was the day after we launched LinkedIn’s Jobs product in early 2005. Matt Cohler, the GM of our the new jobs product, was giddy. A day into the product’s life, companies were already posting hundreds of jobs on LinkedIn.

Upon seeing the numbers, Matt made a confident prediction: this would be my second consecutive Internet startup success. LinkedIn had over a million users, and our first path to real revenue — charging hiring managers to post job listings — looked extraordinarily promising.

Matt is frequently correct, and his prediction about LinkedIn wound up being a good one. But the path to success wouldn’t be quite as straight and simple as I think he anticipated.

For the first six weeks, it would be free to list jobs on LinkedIn. That led to lots more postings, but an imbalanced market: most listings attracted very few applicants and thus meant a poor experience for job posters. Still, we had reason to believe this dynamic would shift.

At the end of that period (March 2005), we started to charge $75 for job listings. Not surprisingly, this planned change led to a dramatic decrease in new listings. That drop meant our revenue wasn’t what Matt (who had just left to join a tiny startup called Thefacebook) and others had hoped for.

But that wasn’t the worst problem. Even with a small number of jobs in the system, most jobs were getting no more than a few applicants. That meant that hiring managers and recruiters were shelling out seventy-five bucks and getting relatively little in return; that didn’t bode well for the product’s future.

If employers had a great experience, they’d post more jobs and tell colleagues to do the same, and the market would even out. But with a poor experience, they’d try something else, and our job listing platform (and revenue stream) would suffer.

How could we remedy this problem and generate applicants for our job listings? I leapt at the challenge, which seemed like a great place to apply some interesting machine learning algorithms to an important business problem. We had millions of users; surely we could match at least a few hundred of them with each of the listings.

Within a few weeks, I built up a set of algorithms to find suggested jobs for LinkedIn’s users. Some of the predictions were quite good; others were clearly off-base. Matching users and jobs is a hard problem.

I shared those recommended jobs with my colleagues, and we decided to email our users with personalized job suggestions. Internally, we called the product No Job Left Behind (NJLB), oh-so-cleverly borrowing language from George W. Bush’s first-term education legislation. The first small NJLB batch was sent out with “Mike Greenfield” as the from name; thereafter they were sent out from “Nick Welihozkiy”, our account rep.

The NJLB results were okay but not great: lots of users clicked through on one or more recommendations, and a handful applied to the jobs. By emailing out suggestions to a targeted set of users, we were able to increase job applications by around ten percent.

However, we also got a handful of complaints about our occasionally off-target suggestions. This came to my attention when a Silicon Valley executive or two complained to my colleague Keith Rabois about their suggestions. As those who know him are well aware, Keith is less obliging toward mediocrity than most people. He of course conveyed his friends’ feedback: we were spamming some high-profile people with low-quality emails, in the process undermining the strong brand LinkedIn had built up.

At the time, I brushed this aside. To me, a user was a user, a job was a job, and a powerful friend of a VP wasn’t necessarily any more valuable to LinkedIn than anyone else. So I put in certain filters to make sure executives wouldn’t get job emails from us, and we continued as we had.

Sometimes, ignoring unsolicited advice from bigwigs is the right strategy. Many people building products are too eager to please a few powerful individuals whose opinions are no more relevant than those of an average user. If you’re building a product for mainstream moms, it’s probably safe to weight the opinion of a VC no more (and often less) than that of a typical mother. Most VCs aren’t experts — or even users — of products for moms, so optimizing for their experience is silly.

LinkedIn had a different dynamic, though. The hotshot execs serve as both the aspirational celebrities that “normal” professionals look up to, and the ultimate decision makers for many purchases which affect LinkedIn’s bottom line. And LinkedIn had a strong brand to protect: the site was (and is) seen as a place for high quality people and high quality (if sometimes boring) content.

If we sent an email to a large company CEO suggesting she apply for a job as a software engineer, we’d undermine that brand with the users most important to our product. In other words, Keith was absolutely correct.

Last week I got an email from Glassdoor that reminded me of that whole experience:

Glassdoor job recommendations

I’ve played a bunch of roles in my career, around both data (from growth hacking to scientist) and entrepreneurship (CTO, co-founder, investor, adviser, etc.), and I know from experience that people like me are a pain in the ass to match jobs for.

That said, Glassdoor’s suggestions don’t even have a unifying theme that suggests they’re making a true guess about what I am. Recommendations to be a retail sales rep, visual designer, and QA tester, all in the same list… seriously?

My assumption is that Glassdoor is sending suggestions purely based on my location, and not doing any advanced targeting. If that’s true, they should be framing the user’s expectations more clearly (DJ Patil, my successor at LinkedIn, explains this well in his e-book Data Jujitsu). Either way, the experience for me is poor. I’d guess that with these emails, Glassdoor generates a few additional applications but undermines their long term brand.

At LinkedIn, we realized that and ultimately turned off the No Job Left Behind emails. They carried substantial risk, and weren’t working well enough to justify that risk.

But progress has happened: the data team at LinkedIn revamped my old models, and seem to be producing a much better set of matches than I ever did. As a result they’ve resuscitated the “Jobs you may be interested in” emails, with a quality level that solidifies the LinkedIn brand.

Note that two of the recommendations are for positions at Glassdoor. Summing things up with a clever joke about that will be left as an exercise for the reader.

Mike Greenfield founded Bonafide, Circle of Moms, and Team Rankings, led LinkedIn's analytics team, and built much of PayPal's early fraud detection technology. Ping him at [first_name] at mikegreenfield.com.