I’ve written here more than once before about the seemingly growing practice by law firms and others of using eye-catching statistics based on ‘quick and dirty’, dubious or even non-existent research to grab a few headlines for their organisation. All too many journalists – in both mainstream media and the specialist press – seem to lack a critical eye, and to simply regurgitate ostensibly ‘shocking’ statistics without question or analysis. And this week has been a bumper week, with no fewer than four questionable research ‘findings’ on different aspects of the labour market making headlines.

On Wednesday, the BBC, Guardian, Independent, Daily Mail and others reported that 58.8 per cent of UK university graduates are working in jobs that do not require a degree, according to research commissioned by the Chartered Institute of Personnel & Development (CIPD). This startling figure was the lead news story on BBC Radio 4’s flagship Today programme, though it appears no one at the BBC thought to query the spurious precision – are the good people at CIPD quite sure it’s not 58.7 per cent? – or to ask how many UK university graduates had actually been surveyed for the research, and when.

The 29-page CIPD report is wordy and no doubt worthy, and there’s no question as to its academic credentials. But you have to dig quite hard to discover that the 58.8 per cent figure is based on data from just one question in a 2010 European Social Survey, namely: “how many years of education someone would need to be hired for their current job”. To arrive at the 58.8 per cent figure, the CIPD researchers then assume “15–16 years of education as the minimum indicator for a graduate job”. That is, they were probably a graduate. But hey, let’s stick 58.8 per cent in the press release and see how it goes. Bingo!

Later the same day, the Daily Telegraph reported that, according to new research by Citizens Advice, 460,000 workers – one in 10 of the 4.6 million self-employed in the UK labour force – are cheated out of holiday pay, sick leave and pensions because “businesses have wrongly hired them as self-employed”. For its 22-page report, Citizens Advice surveyed a total of 491 of its clients, through two separate surveys (one online), and then determined from their answers that one in 10 “are on ‘bogus’ [self-employment] contracts, and should rightfully be appointed as company staff”. (Based on my 13 years at Citizens Advice, I’m surprised it’s only one in 10, but that’s another matter).

Then, on the basis that the “scale” of these 491 responses “provides a statistically-valid representation of the UK self-employment market (at a 95% confidence level, with a plus or minus 4% margin of error)”, despite the respondents to the online survey having self-selected, the Citizen Advice wonks extrapolate their ‘one in 10’ to the 4.6 million self-employed, do a few quick sums to show how much that is costing HM Treasury in lost National Insurance payments (£314m a year), and – Bob’s your uncle! – they have a headline in the Daily Telegraph and a piece on BBC TV’s supposedly cerebral Newsnight (at 20m15).

Except that, as any fule kno, and as – somewhat bizarrely – the very next sentence in the report acknowledges, Citizens Advice clients are not representative of the national labour force. On average, they are lower paid, lower skilled, work fewer hours, and – crucially – are far more likely to be working for an unscrupulous or ‘rogue’ employer. Furthermore, as already noted above, the online survey collected data from self-selecting respondents (the ‘methodology’ section of the report doesn’t even break down the number of respondents for each of the two surveys). Fortunately, not all journalists are as unquestioning as those at the Telegraph or Newsnight, so (for example) there is no mention of Citizens Advice’s ‘bogus’ 460,000 figure in the FT’s coverage of the report.

Come today, and the Independent reports that “zero-hours contracts make up one in four offers to [the] jobless”, according to research by recruitment website Glassdoor UK. However, a quick glance at Glassdoor’s own press release reveals that the ‘research’ says no such thing. According to that press release, 23 per cent of the 1,001 “unemployed people” surveyed online by market researchers Opinion Matters for Glassdoor in May reported having been “offered a zero-hours contract”. Which begs the obvious question: when, and how many times, were they offered such a contract?

The press release doesn’t say, and there is no link to any actual report that might explain the research methodology, so I contacted Glassdoor. They replied with a generic statement from Opinion Matters on how they ‘recruit’ such respondents: essentially, respondents are remunerated for being part of a panel that completes any number of surveys emailed to them by Opinion Matters. So we have no way of knowing whether, and if so how, the 1,001 respondents are representative of ‘unemployed people’, even if we can agree on a definition of ‘unemployed’ (e.g. unemployed and seeking work).

Glassdoor also sent me a spreadsheet containing the survey data. This shows that the question asked was simply: “Have you ever [sic] been offered a zero-hours contract”. So, the ‘finding’ that 77 per of the 1,001 respondents have never been offered a zero-hours contract – despite 299 (one in three) having been ‘unemployed’ for longer than three years, and 38 for more than a decade – tells us nothing at all about the proportion of job offers made to ‘the jobless’ in 2015 that are zero-hours contracts.

But it is the middle of the Silly Season. Which perhaps explains why leading law firm Slater & Gordon thought it a good idea to press release its new research with a claim that “almost six in 10 people have witnessed or suffered bullying in the workplace”. Again, there is no link to any actual ‘report’ that might explain the research methodology – despite the press release explicitly referring to “the report” – so I contacted Slater & Gordon to ask for a copy. They told me that the research report exits, but cannot be shared “externally at this stage as some of the details are commercially sensitive”, and referred me to the generic methodology of market researchers Censuswide, who conducted the survey of 2,000 of their 69,000 (remunerated) panel members for Slater & Gordon. And, subsequently, Censuswide told me:

We used an online quantitative methodology to achieve the overall 2,000 sample base of UK workers. In terms of this sample being robust enough to represent the UK workforce, based on the latest employment data from the ONS, there is a total of 31 million workers across the UK. With this in mind, a sample size of 2,000 is considered to be robust and representative (working to a low margin of error of 2.2% and a confidence level of 95%).

In short, Citizens Advice, Opinion Matters and Censuswide all appear to have worked on the highly questionable assumption that sample size alone is sufficient to ensure that their survey findings are representative of – so can be extrapolated to – the whole labour force. Which perhaps goes some way to explaining why the pollsters got it so wrong in May. For we know that, even if Citizens Advice surveyed all 30,000 of the self-employed people it advises through its “local offices” each year – Citizens Advice Bureaux seem to be a thing of the past, thanks to a recent brand makeover – the findings could still not be extrapolated to the national labour force. And how do we know that there isn’t a correlation between becoming a remunerated Censuswide panel member and having been bullied at work in the past, or between being ‘unemployed’ and becoming a remunerated Opinion Matters panel member?

Which is not to say that there are such correlations. We don’t know one way or the other. But if researchers don’t publish and explain the methodology used for their ‘research’, they must expect at least some people to question their headline-grabbing findings. And, perhaps more importantly, to question what they have added to the debate on labour market issues and appropriate policy responses.

[Postscript: Yesterday evening, Hetan Shah, executive director of the Royal Statistical Society, got in touch via Twitter to draw my attention to this fabulous “new free online stats course for journalists” that the Society has launched “to reduce just this kind of thing”. Managers at Citizens Advice might want to get their policy researchers to complete the course, too.]