There’s been much discussion of late on the accuracy and legitimacy of FiveThirtyEight. I’ve been a fan of FiveThirtyEight since just after it got started, but I find myself having some mixed feelings about the topic.

Here are three articles that are central to many of the arguments being made:

1) David Brooks, who works with FiveThirtyEight founder Nate Silver at the NY Times, writes of the futility of watching and using polls. His implications are twofold: that as an individual it’s a waste of time to track polls, and that it’s impossible to predict what will happen in a presidential election because it’s too complex (and “even experts with fancy computer models are terrible at predicting human behavior”).

2) Dylan Byers at Politico argues that Silver is a biased hack who throws around numbers that don’t make sense.

3) Ezra Klein at the Washington Post rebuts Byers, arguing that the betting markets clearly favor Obama, and that a bettor could use a better system to beat them… but only if a better system existed. Moreover, he makes the point that, unlike Silver, most pundits aren’t accountable, and that their main goal is traffic and/or attention, not accuracy. Hence he gives Silver a tentative endorsement.

I like elements of what Brooks, Klein, and Silver are saying and doing. On the other hand, I think Byers doesn’t understand probability and is completely full of crap.

My feelings can be summed up as follows:

1) As both Klein and Brooks point out, predicting elections is hard. As Klein notes, Silver doesn’t publish his formula (in his shoes, I wouldn’t either), so it’s tough to say definitively how good it is. There are a limited number of historical examples of presidential elections, so Silver must cleverly combine the results of different types of elections (governor, Senator) in a way that effectively multiplies the sample size by a lot. Does this work? Probably to some extent, but since there aren’t many past elections to test his methodology against, it’s tough to know for sure. And the results of this election will neither confirm nor refute his methodology (even though some will claim otherwise). As I’ve written in the past, big data trumps small data, and elections are small data.

2) The world is becoming more quantifiable and accountability-driven. At some point, that means that pundits like Byers are likely to get left behind.

3) Brooks’ comment about computer models doing a poor job at predicting human behavior suggests that he doesn’t really understand the difference between different types of models. There are certain situations where behavior can with near certainty be very well predicted: the same kind of kids who eat the marshmallow today will eat it tomorrow, and a predictive model can be quite accurate. On the other hand, a model that has implicit assumptions about the surrounding world — who is going to repay their mortgage — can be a very different story. It’s debatable where on this gradient an election falls.

4) I share some of Brooks’ concern about people (like me!) wasting time reading about polls. Silver brings a much needed rigor to the poll-watching, making it more intelligent. But ultimately, reading FiveThirtyEight is still mostly about “fast news” and addictive news consumption. I can read a few hundred words every day, and all I get out of it is a slightly better understanding of the horse race (there’s a great chance that Ohio is the deciding state!)

As both a data guy and someone with a vested interest in good journalism, that seems like a bit of a waste. Nate Silver’s a talented and resourceful data guy/journalist, and he’s almost exclusively focused on fast news-update me right now-horse race journalism.

Thanks to a few twists of fate, I sit on a panel that awards an investigative journalism prize (the Goldsmith Prize). Being on that panel, I read over a hundred “slow news” investigative stories from around the U.S. Every year, I see more and more data-centered reporting. That happens because additional public data sets become available and journalists (slowly) become more data-savvy. The end result is more good stories like this and this.

These big investigative reports — slow news — can be very valuable to our democracy. But historically, many such reports have been anecdotal and lacking in rigor. That’s changing: the Las Vegas Sun story on health care was wonderful in its depiction of individuals’ tales, but also painted a solid quantitative picture.

Each year, in spite of newsroom cuts, the public seems to get more and more stories combining slow news with data-driven rigor. That’s great for society, but it’s happening more slowly than it could be.

Hence my reaction to Silver is two-fold. On one hand, I applaud him for his statistical rigor towards a topic that’s too long been the domain of those who can speak loudest. I hope it serves as a model for others working in a variety of fields.

On the other hand, I question whether spending time on fast news like election coverage is the best use of his talent. I hope to see more and more folks like Nate Silver applying themselves to the deeper but slower news stories of our day.

In my recent post on what developers want, I wrote about which industries appeal to software developers.

The three areas at the top of developers’ lists are education, productivity, and human resources. The three areas least interesting to developers are entertainment, payments, and games.

After posting that, my friend Jeremy sent me an email suggesting there might be differences between companies that are desired by engineers and companies that are successful. (Belated credit: Jeremy was the person who complained to Keith about my job suggestions in this blog post.)

It was a question I found interesting. After four-plus years as a developer-founder (aka CTO), I’ve seen a bit of the other side via 500 Startups (where I help companies with data/growth), AngelList (great for understanding the venture ecosystem), and as an occasional individual angel investor. My gut instinct is straightforward: developers and investors don’t seem to have much in common.

Hence I thought Jeremy was likely right, but I wasn’t sure I’d be able to show it definitively. AngelList is a terrific data source, but it hasn’t been around for very long. As a result, the number of companies who have used it to both fundraise and hire is limited.

Fortunately, it’s comparatively easy to look at the fundraising success of companies within each industry to uncover trends. Breaking things down into industry categories, I could look at companies’ average level of fundraising success.

Because I didn’t want to overweight the importance of a few companies who raised a ton of money, I defined a set of fundraising steps a company could achieve. The lowest step consists only of raising non-zero capital; the highest step is raising a round of $5 million or more. I didn’t go higher than $5 million, because the percentage of companies raising that amount of money is tiny.

I only looked at investments that have taken place since 2010.

The X-axis denotes the mean number of fundraising steps taken by companies in the given industry (more steps means more money).

The top three categories favored by investors are Payments, Tech Infrastructure, and Education. Popular as it may be among investors, Payments is among the worst industries for recruiting engineers. Education and Tech Infrastructure, on the other hand, have it all: both do well at attracting developers.

The bottom three categories for investment are Productivity, Local Infrastructure, and Local Business. Productivity is among the most popular industries with developers, but does a lot less well with VCs. Local Business and Local Infrastructure are both in the middle of the pack with developers.

In other words, there’s not a huge amount of overlap between what investors want and what developers want. Here’s a graph mapping developer interest against VC interest:

There are three major outliers: Productivity (developers love, investors hate), Education (both love), and Payments (developers hate, investors love). There are no categories hated by both developers and entrepreneurs.

In my last post, I said that

Developers want to build something useful and societally valuable.

All things being equal, VCs may feel the same way. However, money usually chases returns, and that means that collectively, VCs are going to as well. Thus it stands to reason that VCs would likely behave in a more economically rational manner than the world of (depending on your perspective) naive and/or altruistic developers.

What about developers relative to other potential employees?

Let’s look at the same type of graph, with developer interest on the X axis and non-developer interest on the Y axis.

There’s an extremely high correlation (0.59) between a developer’s excitement about an industry and that of a non-developer. Productivity is the favorite industry of non-developers; it’s the second favorite industry of developers. Education is the favorite industry of developers; it’s well above average among non-developers. Games, Advertising, and Entertainment all fare poorly among both groups.

There’s only one major difference in this set: Payments. Developers generally dislike Payments companies, while Payments companies rate highly among non-developers.

So what to make of all this? Here are my top takeaways:

• Investors and employees tend to be driven by different factors.
• At least in startup-land, developers and non-developers are far more similar than they are different.
• Everyone loves Education (but it’s really hard to build an Education company).
• Non-devs and investors like Payments; developers don’t.
• Employees have soured on Games, Entertainment, and Advertising, even while investors have put a lot of money into those industries.
• Non-developers and VCs are more excited about data and analytics than developers are, but all three are positive about the field.
• Non-developers love Productivity companies and developers like them; VCs are much less excited about the industry.
• Way back when Facebook was a private company, they raised an insane sum of money. They (and a few others) have raised so much that trying to compute any sort of “average” for an industry winds up being an absurd exercise. Thank you, power law!
• And once again: investors and developers are really, really different.

A few months after we launched Circle of Moms, I hired my first superstar senior back-end developer. Brian had it all: his coding practices balanced speed and scalability, he’d been around the startup block as a founder and employee, and he had a quiet but kickass work ethic that would set the right tone for the team. Over the years, he’d re-architect our (messy) core technology, build many of the features most responsible for our growth, and patiently mentor younger employees. Hiring Brian may have been the best move I made at Circle of Moms.

I spent the next three years trying to find someone else like Brian. It never happened, for a variety of reasons: some people weren’t the right fit for us, some weren’t interested in building a product for moms, others wanted big-company salaries we couldn’t afford.

The experience underscored the importance of a strong team on a company’s success. On one hand, we were lucky to have someone like Brian; without him, I’m sure we would have been less successful. On the other hand — speaking as a Stanford Basketball fan spoiled by the dominance of both the Collins twins and the Lopez twins — I wish he’d come with a twin brother.

In the current developer-constrained world, hiring becomes a product marketing problem. There’s a limited supply of top people, and the startups that aren’t able to market their “product” (the career opportunities they offer) will have trouble hiring and growing.

To get big, today’s technology companies need to market themselves to three groups:

1) Customers/users
2) Investors
3) Developers

Many a book and blog post has been written on satisfying customers and raising capital. They’re both important and interesting topics. I’ve written about them in the past and will write more about them in the future.

Attracting Developers

Almost nothing has been written about attracting developers, and I wanted to figure out why it was so difficult for me at Circle of Moms.

Over the past few months, AngelList Talent has emerged as a strong marketplace for jobs, attracting both top companies and top developers (as well as non-developers).

(Disclosure: I’m an adviser to AngelList.)

Notably, the AngelList product doesn’t work like a traditional job site. Potential candidates don’t apply to work at a company, and companies don’t send messages to potential candidates. Instead, candidates see a summary of a company and its positions, and indicate — one-click, HotorNot style — whether they’re interested in the company. Companies do the same, viewing candidates one-by-one, and saying yes or no to each. Matches are then introduced.

This approach yields a large, clean data set with explicit information on which company profiles appeal to developers (and vice versa). Some people get off on fancy cars, fancy clothes, or fancy wines; for a data geek like me, it doesn’t get much better than large, clean, explicit indications of individual preference.

Top Companies

So who are the companies I should hate for hiring the people I wanted? At the time — 2010-2011 — the companies who “beat” us were Zynga, Google, BranchOut, and Federated Media (some of them probably aren’t having as much success now). Today, these are the companies that developers on AngelList like best:

1) Quora
2) Pocket
3) Path
4) Pulse
5) Instameet
6) ClassDojo
7) Ark
8) Rally
9) Locu
10) Clever

You’ve probably heard of most of these, but there still are some surprises. Pocket (disclosure: I’m an investor) has done very well as a business, but I still wouldn’t have expected them to be a notch above Path. I wasn’t familiar with Instameet, so seeing them in the top ten was unexpected. Meanwhile, some other more prominent companies — 42Floors, Kaggle, OUYA, Skillshare, wikiHow — find themselves just outside the top ten.

At the bottom of the list are five companies I’d never heard of (sorry, I’m not going to tell you who they are). Two have products aimed at improving people’s nightlife experience; one is a management tool for a very specialized audience; one is a mobile company focused on saving memories; one tries to improve experiences with physical devices. Like the top companies, most are in the Bay Area.

The difference between top companies and bottom ones is large. When someone expresses a preference between a top-five company and a bottom-five company (which has happened almost 500 times) they favor the more highly ranked company over 95% of the time.

(Quick note on methodology: I only included self-described developers in my analysis, I only looked companies with complete AngelList profiles, and I excluded votes from developers who either always or never express interest in startups. I also weighted all companies evenly, irrespective of the amount of interest they’ve received, and dampened the scores of outliers with only a small number of votes.)

Now for the tough part: what separates the top companies from the bottom ones?

Several factors don’t make a difference.

Having a technical co-founder, which one might think would make a company more attractive to developers, does not affect a company’s desirability. And somewhat surprisingly, the equity stake listed does not have any impact, though my analysis here does not account for company size.

Four factors matter a lot:

1) Industry. A company’s industry has a large impact on its ability to attract top developers. Nearly 40% of self-identified education startups show up in the top tier of companies most able to recruit, while under 10% of companies in games, entertainment, or advertising are in the top tier.

A quarter of games, payments, and entertainment companies find themselves at the back in competing for developers. By comparison, very few companies — under 10% — trying to improve personal finance, human resources/hiring, or personal productivity find themselves in that bucket.

(More blue and less red is good.)

Overall, personal finance, HR, personal productivity, and education stand out as the industries most attractive to developers, while entertainment, games, payments, and advertising stand out as the worst.

2) Investor Quality. Top investors certainly aren’t a guarantee of success, but they help a lot.

Roughly a third of companies that have the most highly rated investors (based on AngelList’s proprietary models) are among the very best at attracting talent. Interestingly, however, these companies aren’t immune to being among the worst at attracting talent. On the flip side, companies with the least prestigious investors are only slightly more likely to be among the bottom tier at attracting talent, but they are far less likely to find themselves among the top tier of companies like Quora, Pocket, and Path.

3) Founder Quality. Founder quality, using the same AngelList proprietary models as investor quality, can be a very good predictor of recruiting success. Companies with highly rated founders — like Path and Quora, both founded by early Facebook employees — are more likely to be highly attractive to developers than anyone else. However, such companies are relatively rare, as most companies are founded by first-time founders who don’t personally have strong track records.

As a result, with founders we see a large “middle”: there are lots of companies started by founders who, on paper, are hard to distinguish as especially good or especially bad. These companies, not surprisingly, are about average at hiring.

4) Salary. High salaries can make a substantial difference in a company’s hiring fortunes. Companies with the most attractive salary offerings — roughly corresponding to a high range of $125,000 — are more than five times as likely to be in the top tier at hiring as companies with the least attractive offerings. And companies offering high salaries are very unlikely (around 7%) to find themselves in the bottom tier.

On the opposite side, we see that it’s very difficult to be great at hiring — at least on AngelList — when salaries are on the low end. These days, good developers have lots of choices.

Other Factors

Salary, founder and investor quality, and industry all matter, but they certainly don’t explain away all of the differences in developer interest. Looking only at a statistical model that incorporates those four factors, one would expect that Quora would be the sixth most effective company at attracting developers. In fact, they’re a clear #1, and four of the other five companies the model likes most aren’t in the actual top ten (Clever is the exception).

So what’s missing from the model? My guess (which is completely non-quantitative) is that there are four things:

1) Company Reputation. This is an area where the top companies shine, even beyond the fact that most of them have successful founders and/or investors. Quora doesn’t really need to explain itself, since many top developers are using it all the time.

2) Well-Written Profile. One positive outlier less famous than Quora is LaunchRock, whose tagline is “We *get* users” and whose hiring page starts with “We like to party.” They then confidently describe their business and culture, leaving enough out to keep the reader intrigued. Developers show a lot of interest in working for them, placing them just outside the top ten.

Clarity, a big vision, and intrigue all help a lot. Empty buzzwords and corporate BS, on the other hand, aren’t terribly effective at attracting developers. Some of the worst-performing job pages have phrases that set off my BS filter and likely those of other developers as well: “virtual currency”, “revolutionizing”, “siloed”, “* as a service.”

3) Big and Important Problem for Society. I don’t think it’s a coincidence that personal finance, productivity, hiring, and education are at the top of the list, while games and entertainment are at the bottom. Developers want to build something useful and societally valuable. Industry correlates with “useful and societally valuable” but it’s not an perfect relationship. Many of the companies that are more successful attracting developers than the model would predict have a real chance to positively impact the world, while those that are less successful are often playing zero sum arbitrage games.

4) The Rest of the Hiring Process. We aren’t yet in a world where clicking on a button in AngelList commits a developer to a company: there are many additional stages. Initial intrigue is one thing, but as I learned at Circle of Moms, there’s a lot more to hiring than one “yes, I’m interested” click. Some companies may have a low “interested” rate, but a high rate of closing candidates; as a former (and likely future) hiring manager, I’d take that tradeoff.

Product and Marketing

The best companies have great products, well-marketed, at an attractive price for consumers.

The same applies to startups trying to appeal to developers.

The product — the existing team and its culture, the business’ goals and prospects, and the opportunity for the individual — is central. But the marketing — in this case, a clear articulation of the company’s vision — is also crucial, and the right price (salary) helps as well.

All that said, I’ll still offer a nice referral bonus to anyone who can clone a twin for Brian…