Skip to main content
ABC News
Three Rules to Make Sure Economic Data Aren’t Bunk

Mainstream economics coverage is full of numbers: unemployment, inflation, the trade deficit, etc. But stories only rarely stop to explain what those numbers measure, how they are collected or how reliable they are. Instead, readers are often left to accept numbers at face value or wade through conflicting indicators. If hiring accelerated but the unemployment rate rose, is the job market getting better or worse? If Democrats say inequality is rising and Republicans say it isn’t, whom to believe?

Economic indicators are often contradictory and always subject to uncertainty, but that isn’t a reason to dismiss them. Economics deals with subjects we encounter daily: jobs, spending, production, prices. It doesn’t take a Ph.D. or even a bachelor’s degree in economics to understand these concepts and how they interact with one another. It shouldn’t take a degree to interpret an economics report, either.

Armed with a few basic rules and some healthy skepticism, anyone can evaluate an economic claim. When we’re told that job growth is stalling, or that people are staying in their jobs for longer, or that entrepreneurship is dead, three simple rules should give us, if not a definitive conclusion, at least a sense of whether a claim is well-founded.

Rule No. 1: Question the data
When data run counter to conventional wisdom, it’s a good idea to question not just the wisdom, but also the data. That’s especially true in economics, where the data are volatile, subject to frequent revision, and often carry significant margins of error.

Take the monthly U.S. jobs report. The report is the most closely watched of any economic data release, with every deviation from economists’ expectations touted as a meaningful “miss” or “beat.” Yet the first estimate of employment growth is routinely revised by tens of thousands of jobs or more. In September 2011, for example, the Labor Department said the economy had added zero net jobs in the prior month, sparking fears the recovery had stalled. But a month later, the government revised the figure to a gain of 57,000 jobs — a bad month but not nearly so worrisome as the initial goose egg. After further revisions, the government now says hiring actually accelerated in August 2011, with employers adding 122,000 jobs.

Dismissing the original figure is easy in retrospect. The challenge is to tell whether the data are valid in real time. The markets certainly didn’t: The Dow Jones industrial average dropped more than 250 points the day the jobs report was released.

But even when the terrible report was first released, there were signs that it might be overblown. For one, jobs reports are notoriously subject to revision. The department makes no secret of that: The news release announcing the August number listed a margin of error of about 100,000 jobs and announced 58,000 jobs’ worth of revisions to the prior two months’ figures. Moreover, other data sources didn’t show the same slowdown. An estimate from the payroll processing firm ADP, for example, showed a gain of 91,000 private-sector jobs, which was much closer to the department’s final tally of 125,000. The Institute for Supply Management, which publishes another widely followed jobs index, also showed solid if unspectacular job gains in August. In fact, the same Labor Department report that included the zero-net-jobs figure also included another measure of employment, based on a separate survey, that showed significantly stronger gains for the month. None of that meant the disappointing number was necessarily false, but there was reason for skepticism.

Rule No. 2: Know what you’re measuring.
Numbers don’t have to be wrong to be misleading, however. That’s particularly true in economics, where even seemingly simple concepts can prove difficult to measure, and where different forces — business cycles, demographics, government policies, technological change — can be hard to separate from one another. To best evaluate an economic claim, then, look not just for signs that the numbers might be fishy, but also try to understand what’s really being measured and how.

For example, for all the talk of constant turnover and the end of the “lifetime job,” Labor Department data show that Americans are actually spending longer with their employers than at any point in the past 30 years. The median “tenure” of a worker — how long the typical worker has been with the same employer — rose by 14 percent between 1983 and 2006, to four years from 3.5. When the recession hit, the trend accelerated, with median tenure hitting 4.4 years in 2010 and 4.6 years in 2012. As counterintuitive narratives go, it would be hard to beat, “Job security continues to rise.”

Unlike the jobs report, the tenure data pass rule No. 1 without much trouble. The government releases the figures every two years, which means Labor Department statisticians aren’t trying to piece together incomplete or preliminary data. Nor are the numbers particularly volatile: The trend of rising tenure is fairly consistent over the years and not just a one-off blip.

But when you look closer, it becomes clear that this counterintuitive narrative is counterintuitive for a reason. The Labor Department’s data on tenure look only at people who are employed. That means that if a large number of recent hires lose their jobs at once — as tends to happen when a recession hits — median tenure will rise, even though people aren’t staying in their jobs for longer.

The prerecession trend of increased tenure turns out to be equally misleading in a different way. There are two major forces at work. The first is age: Older workers are more likely to have been in their jobs for longer, so the gradual aging of the U.S. population has pushed up workers’ average tenure. The second is the entrance of women into the workforce and, particularly, into career-track jobs. In 1983, the average woman had been with her employer a year less than the average man; 30 years later, their average tenures are nearly equal. If we set aside those factors and focus just on men in their prime working years, there was a decline in tenure in the years before the recession. This is one case where conventional wisdom holds up.

Rule No. 3: Look outside the data
The first two rules have to do with questioning the numbers — what they’re measuring, how they’re measuring it, and how reliable those measurements are. But when a claim passes both those tests, it’s worth looking beyond the data for confirmation. After all, economic data, no matter how clean or consistent, aren’t an end in themselves; they’re an effort to measure what’s happening in the real world. If a trend is real, we should see evidence of it, not just in spreadsheets but among the people and businesses that make up the economy.

To see how this works, consider this: According to data from the Census Bureau, the United States has become steadily less entrepreneurial over the past 30 years. In 1982, half of all U.S. companies had been founded in the previous five years. By 2011, that figure had fallen to 35 percent. But should we trust the data?

Start with the first two rules. Unlike the suspect August 2011 jobs report, multiple sources (data from the Census Bureau, Bureau of Labor Statistics, Bureau of Economic Analysis and Internal Revenue Service) tell the same basic story when it comes to entrepreneurship: that rates of new-business formation have been falling for decades. The data show a consistent pattern over years and even decades. And unlike with job tenure, a deeper dive into the data on entrepreneurship provides more support to the thesis, not less: The decline in startups cuts across virtually every major sector, from manufacturing to retail, and across states and regions too. Based on the data alone, the claim looks compelling.

Still, a decline in entrepreneurship seems counterintuitive in a world of 20-something Facebook billionaires, craft brewers and Etsy artisans. That’s not a reason to throw out the thesis, but it’s a reason to give it a closer look. Is there real-world evidence we might have missed?

That evidence isn’t hard to find. These days, the word “startup” tends to call to mind social media platforms or smartphone apps, but tech is a tiny part of the overall economy. And while Silicon Valley’s entrepreneurial culture has been booming, many larger sectors — at least in terms of number of companies — have been struggling. The number of new retail businesses — which account for roughly a quarter of all new companies in any given year — has fallen by 20 percent over the past 30 years, as small retailers have given way to big-box stores and, more recently, Amazon, eBay and other online retailers. Similarly, manufacturing, transportation, finance and even construction are increasingly dominated by established players.

Experts who study entrepreneurship — as opposed to politicians or Silicon Valley boosters — have identified the same trend. “Where have all the young firms gone?” asked the title of a 2012 paper sponsored by the Ewing Marion Kauffman Foundation; related research from the Census Bureau noted a “decline in business dynamism in the U.S.” and cited the falling startup rate as key evidence. Economists from the New America Foundation have noted a “slow-motion collapse of American entrepreneurship” and argued that the official statistics overstate the true startup rate, because they include independent contractors who are just employees by another name. The decline in entrepreneurship might be counterintuitive, but it’s close to the consensus view among experts.

No one rule serves as confirmation. Data are revised, experts are proven wrong, and real-world evidence can be cherry-picked, consciously or subconsciously. Evidence is often conflicting, and it isn’t always clear which argument to believe. These rules, then, aren’t meant as a mathematical formula. They’re a framework for how to think critically about evidence and a reminder to treat all economic pronouncements — including the ones made on this site — with appropriate skepticism.

Ben Casselman was a senior editor and the chief economics writer for FiveThirtyEight.