A literature review on literature reviews of literature reviews

As previously established, content analysis has been a popular and influential research method for well over a century. This explains why it has also been the subject of several types of structured meta-analyses and literature reviews. In one of the first major studies of its kind, ^{Berelson (1952}) assessed a sample of articles published between 1935 and 1950. It included an overview of qualitative types of content analysis as well as a chapter on sampling, reliability, presentation, and modes of inference. He nonetheless explicitly focused on quantitative content analyses. By the 1970s, their increasing scholarly use within mass communication sciences had the attracted attention of peers (^{Perloff, 1976}). In their review of 486 articles published between 1971 and 1995 in the journal Journalism & Mass Media Quarterly, ^{Riffe and Freitag (1997}) identified “a twenty-five-year trend of increased content analyses”. 71% of all assessed articles focused on news or editorial content, with newspapers (47%) and television (24%) being the most popular mass media; 86% of the published studies analysed American media (pp. 517-518).

Later, ^{Lovejoy et al. (2014}) looked at 581 articles published in three flagship journals of major communication research associations between 1985 and 2010, and examined “whether a reliability test was conducted but also whether a reliability coefficient correcting for chance was used” (p. 208). They found that 24% of their body of articles did not include any information about reliability; a similar conclusion was reached previously by ^{Lombard et al. (2002}, p. 587), who stated that “mass communication researchers often fail to assess (or at least report) intercoder reliability”. In a follow-up study whose research scope was extended to 2014, Lovejoy et al. (2016) found that Krippendorff’s Alpha and Cohen’s Kappa were the most often used coefficients, with the use of Pi declining steadily throughout the last assessed decade.

The strong tradition of mass communications research in the United States led, practically by default, to a noted dominance of American-based studies. Only in recent decades have similar studies enjoyed increasing popularity and use, mostly among researchers in (Western) Europe, Australia, and New Zealand. In a rare European-centred meta-analysis of media content analyses, ^{Machill et al. (2006}) found that European Union-related topics accounted for “an extremely small proportion of reporting in the particular national media” in German, English and French news media (p. 78).

Based on the literature review, we have established several main gaps in scholarly knowledge on news content analyses. First, there is a shortage of studies which look at papers published in the 2010s, a decade in which technological innovation swept news production and platform dissemination possibilities, as well as computerised research methods dramatically. By limiting our time frame to articles published in the first two decades of the 21st century (2001-2010 and 2011-2020), we wish to fill this research gap. Second, we want to gauge changes in terms of the platforms chosen for news content assessment. Historically, newspapers have been the logical primary source of news content analyses. But, with their circulation figures and impact falling drastically in recent years, we seek to find out whether this has also led to more diversity in studied platforms. Third, we want to pinpoint continents and countries which are the topic of news content analyses of the 21st century. Thereby, we are interested in finding out whether the U.S. dominance remains or appears to be wavering or not, and what other countries or continents are increasingly becoming study objects.

Research questions and operationalisation

This study conducts a structured literature review of 2,909 paper abstracts, and aims to answer the following two research questions:

RQ1: From which platforms has news content been assessed most frequently in news content analyses published between 2001 and 2020?

RQ2: Which continents and countries are most represented in news content analyses published between 2001 and 2020?

With our research questions, we distinguish and operationalise two types of diversity which are relevant for our structured literature review, findings, and contributions to the state-of-the-art:

Platform diversity, or the content platform which was analysed by fellow peers (^{Chan et al., 2021}; ^{Diehl et al., 2019}), was defined by measuring which platforms were mentioned in studies. We differentiate between print newspapers, radio broadcasts, TV broadcasts and online news, as well as mixtures of two or more different platforms. Nearly all abstracts mentioned this explicitly. We coded numbers on content units and outlets for respectively 769 and 1,026 of the assessed abstracts, or 26.4% and 35.3% of the total n of 2,909 abstracts.

Location diversity, or the country or continent from which the assessed content emerged (^{Cicek et al., 2014}; ^{Xue et al., 2009}), was coded through logging mentions of country names in abstracts. In several cases, no explicit country name was stated but nevertheless easily deducible. For instance, a couple of abstracts referred to “the 2004 Presidential election”, which undoubtedly refers to the United States. This way, we were easily able to retrieve the country of origin of all but just 50 abstracts. Despite ‘only’ having analysed 2,859 abstracts, we nonetheless found 4,190 instances of studied countries. This is due to several studies comparing news content published across countries. These instances were all counted as separate entities in our location diversity analysis. To give just two examples here, ^{Tiffen et al. (2014}) carried out a comparative news content analysis in 11 different countries, ^{Schuck et al. (2016}) in 2021. Next, we grouped all countries into continents, as established by the United Nations’ Statistics division (United Nations - Statistics Division, n.d.), and used its most recent World population prospects report to contrast the dispersion of nations in our assessed abstracts with their respective numbers of inhabitants (United Nations - Population Division, 2019). This novel approach allows us to effortlessly denote countries and areas which are underor overrepresented in relevant scholarship.

A structured literature review using Scopus

We wish to answer the research questions by carrying out a mixed methods structured meta-analysis. Thereby, we follow ^{Rogers (1985}) and ^{Meier (2019}), who proposed a five-step approach for successfully and accurately executing this type of analysis. The processes behind steps one to three are outlined above and in the remainder of this Methodology section, with the latter two steps of the research design subsequently playing a leading role in the next parts of this paper.

For our analysis, we considered articles published in the vast Scopus database, as has been the case for several other recently appeared structured literature reviews from a host of different fields (e.g. ^{Bhimani et al., 2019}; ^{Secundo et al., 2020}) as it allows “for a broader search covering a wide range of disciplines with indexed content from more than 20,000 journals” (Bhimani et al., 2019, p. 253). We crafted a search string (see Figure 1) which allowed us to search for the terms “content analysis” as well as “news”, to better fit with the unique nature of articles we sought. An initial search yielded 7,036 results. After limiting the scope to articles published in the first twenty years of the 21st century, 6,041 document results remained. Next, we also excluded studies from fields outside social sciences and studies published in languages other than English. This resulted in 3,518 remaining publications, which were all considered for the second manual level of analysis¹.

Figure 1 Search string used in Scopus 

In the second phase of our process, we manually assessed and coded all 3,518 paper abstracts. We developed an extensive code book in a shared and protected Google Sheets document. Article abstracts were coded for an explicit mentioning of used research methods, the number of research units and outlets, the media outlet(s) used (e.g., print, radio, TV, online or a mixture of two or more types of platforms) and the country or countries the content analysis applied to (meaning that a study of American researchers studying Chinese news content would be classified as being applied to China). This way, the coding results were categorised into three main groups, which will be discussed separately in further parts of this paper and correspond to the three research questions: the presence or absence of methodological, platform and location diversity. This clustering approach was based on the papers’ abstracts, which is a commonly used research method in analysing scholarly content for longitudinal studies from an array of academic disciplines (e.g. ^{Gaur & Kumar, 2018}; ^{Rotgans, 2012}).

Finalising the data set

The coding process was undertaken in various steps. After a dry run in which all articles published between 2001 and 2005 (n = 242) were independently coded by both authors, the second author carried out the rest of the coding. The first author, who has prior experience with carrying out structured literature review, and two trained students also coded three randomised samples of articles independently. Differences in findings were settled during meetings, where consensuses were reached easily due to the straightforwardness of the coding variables. All variables were nonetheless tested on intercoder reliability using Cohen’s Kappa and can be interpreted as “substantial”, “almost perfect” or (nearly) “excellent” regarding agreement, with κ scores ranging between 0.68 and 0.93 (^{Gisev et al., 2013}; ^{Landis & Koch, 1977}).

At the end of this second phase, an additional 609 articles were excluded from further analysis. Following the definition of news content in the introduction section of this paper, we removed articles whose abstracts mentioned the study of such aspects as blogs (n = 22), editorials (n = 30), images (n = 42), advertisements (n = 49), social media posts (n = 67) or comments or letters (n = 74). Discounting the 609 excluded ones, the findings of our study are applicable to exactly 2,909 articles, unless stated otherwise (see Figure 2).

Figure 2 Flow diagram of article selection process 

In a next stage, we divided all articles into the two decades of our analysis time frame. One of the major trends over time identified throughout the entirety of the dataset was the increase of abstracts on news content as the years progressed. Especially significant was the increase at the start of the second decade of the dataset, prompting its division per decade to better identify and pinpoint relevant changes, as will be returned to in subsequent versions of the manuscript. This approach was previously applied in a similar structured literature review (^{Hendrickx et al, 2020}). Like the study at hand, it revealed that there were far more articles published in the second rather than in the first decade of the 21st century. In this analysis, 2,302 articles or 79% of the total n of 2,909 articles were published between 2011 and 2020, with just 607 or 21% having appeared between 2001 and 2010. We find a steady rise of published articles for nearly twenty consecutive years. This highlights the increased popularity of news content analysis as a research method which produces increasingly relevant contributions to scholarship, as we only assess peer-reviewed articles. Table 1 below shows the number of articles as found on Scopus, as excluded and as used for our analysis per year.

Table 1 Number of articles used for analysis, per year 

Platform diversity

When discussing platform diversity, we distinguished five distinct variables in our coding process: print newspapers, online news, radio news, TV news and a combination of two or more platforms. We were able to establish this for all 2,909 articles based on explicit mentions in the abstracts. We restricted our analysis to news content analyses carried out on platforms owned by recognised media brands and companies, including online-only news outlets. News via social media was excluded as we were unable to pinpoint from abstracts whether it was news produced for social media only (^{Hendrickx, 2021}) or dealt with regular online news shared via social media platforms (^{Haim et al., 2021}).

Table 2 Platform diversity in news content analyses of the 21st century (n = 2,909) 

Our findings, presented in Table 2 and applicable for 2001 to 2020, confirm the persistent dominance of print newspaper articles as assessed content units in news content analyses, in line with ^{Riffe and Freitag’s (1997}) results for 1971 to 1995. Television news broadcasts were surpassed by online news content only in the second decade of our time span. There is a notable lack of radio-based studies, and increased attention devoted to news originating from various platforms. Even when further narrowing our findings to recent studies from 2016 to 2020, the final five years of our analysis period, print articles still constitute 50.8% of content units. Within that more specific time frame, both online news articles (13.4%) and a combination of platforms (29.3%) gained additional scholarly popularity, at the continued expense of television news (6.4%) and with radio broadcasts effectively ignored (0.1%). A further significant trend visible in the last segment of our analysis is an inverted trend concerning analyses on combination of platforms and online platforms. Although causality is not seen as the cause of this inversion, it shows the significant rise in interest of online platforms as news sites, as expressed by the Reuters digital news report (^{Newman et al., 2021}).

Figure 3 Platform diversity in news content analyses (%) 

The selection of platforms for the research of content units by researchers appears to have become increasingly out of touch with the preferred platforms via which media users consume news, as well as the platforms to which media companies devote staff and funds. In recent years, the decline in print circulation figures and the dispersion of TV, radio and digital channels or platforms for news consumption have signalled a rise in online news consumption. This is supported by the 2021 Digital news report, which studied news consumption patterns among over 92,000 people in 46 countries worldwide and found that online platforms (including social media) were reported as the main sources of news in 43 of them (^{Newman et al., 2021}).

Of course, we do not mitigate the contribution of print newspaper-based content analyses. In many markets, they remain influential media outlets worth studying. However, scholarship has argued that the main reason why content analysis remains fixed on print media is its inherent logic being embedded in the affordances of analogue media, and written texts in particular (^{Karlsson & Sjøvaag, 2016}; ^{Krippendorff, 2004}). The former authors claim that content is still generally assumed to arrive in easily dissectible content bundles at the researcher’s convenience. Historically, this has typically been the case for print newspapers (^{Steensen & Westlund, 2020}), even though the use of scraping software and techniques is increasingly used as well for the study of online media (e.g. ^{Lewis et al., 2013}; ^{Vogler et al., 2020}). We acknowledge the important heritage of newspaper content analyses (^{Sumpter, 2001}) and their continued relevance to scholarship, but simultaneously welcome additional content analyses which address other platforms as well, in order to stay abreast with changes in platform popularity and actual (news) consumption patterns.

Location diversity

We found results applicable to 104 different countries worldwide. It is noteworthy that, country-wise, published news content analyses became much more diverse throughout time. Figure 3 below shows the evolution per year based on continents. We calculated this through our manual coding processes of the countries to which news content analysis results were applicable. The countries were grouped into (sub)continents following the classification of the Statistics Division of the United Nations as a more neutral means of dividing countries and laying bare tendencies for continents and to avoid discussions about disputed territories (United Nations - Statistics Division, n.d.). As shown below, the Americas (predominantly the United States) saw a huge drop in representation throughout the past two decades and has been surpassed by European countries. Whereas Oceania remained mostly stable, it is noteworthy that African and particularly Asian-based news content analyses steadily increased over time.

Figure 4 Representation of continents in news content analyses (%) 

Here too, we divided up our findings per decade to better pinpoint changes over time, and again as we found that structured analyses of the 2010s are lacking in scholarship. Between 2001 and 2010, we found studies assessing news content in just 50 countries, whereas this had increased to 102 in the 2011-2020 range. African, Asian, and European countries were assessed considerably more frequently in the latter decade of our analysis, which signals positive signs regarding location diversity and the advancement and professionalisation of social sciences in the respective continents. Notwithstanding this, and even though it is faltering, we still denote an overt Western bias, particularly geared towards the United States. In the first ten years of our study, the U.S. alone accounted for 53.6% of all instances of countries assessed in news content analyses. This decreased to 30.3% in the following decade; a sharp fall, but still significantly ahead of the second most ‘popular’ country, with a 6.3% ‘share’ between 2011 and 2020: the United Kingdom. In the overall ranking of countries assessed in the 21st century thus far, the top five is completed by the Netherlands (5.2%), Germany (4.4%) and Canada (3.9%). Table 3 below shows the (sub)continents and their respective countries which we retrieved at least once from our structured literature analysis.

Table 3 Numbers on location diversity per (sub)continent (n = 4,190) assessed in news content analyses of the 21st century 

The results in Table 3 facilitate hypotheses and speculations about news content analyses, the nations in which they were executed and the apparent yet alleged uneven balance for certain countries and regions. To offer more tangible evidence of this, we compared our ranking of 2001-2020 content analyses with an additional ranking, based on estimates of the world’s population within the same time frame, courtesy of the Population Division of the United Nations.

Table 4 Location diversity in our analysis and through estimates of the world’s population; right column: United Nations - Population Division (2019) 

The compared figures and calculated shares in Table 4 confirm the overand underrepresentation of continents and subregions. Unsurprisingly at this point, the Americas, Europe and Oceania belong to the former group, Africa and Asia to the latter. While there is obviously a disconnect between the representation of countries in terms of population and research on their respective news content, we do not wish to exacerbate these differences as they are largely understandable. Firstly, because we only considered English-language publications. Secondly, and most importantly, because of the presence (or absence) of a democratic political system, which allows both the press and academia to operate freely, as a mandatory prerequisite for both spheres. This is far from a given in multiple nations worldwide, notably in underrepresented regions in our sample, and helps to contextualise the sheer differences presented.

Finally, there is of course also the power of (scholarly) traditions to consider. As mentioned before in this paper, news content analyses originated in the United States (^{Sumpter, 2001}) and between 1971 and 1995, 86% of published studies in Journalism & Mass Media Quarterly analysed American media (^{Riffe & Freitag, 1997}). We find that this share has decreased to 30.3% between 2011 and 2020, following an analysis of 2,302 papers published in that decade. As opposed to lamenting the perceived lack of fair representation of countries and continents in news content analyses, we welcome the increased plurality in assessed nations. We fully expect this trend to continue to manifest in the coming years due to a wider internationalisation of both media markets and scholarly collaborations across both disciplines and countries. However, in the near future monitoring the freedom of movement for journalists and scholars will too remain a vital mechanism to ensure additional location diversity in content analyses and other relevant types of study.