Introduction

Technology has always been a transformative force in journalism (^{Pavlik, 2000}). Major innovations, like the production of mobile, immersive, and data stories brought about by new technologies, have restructured work processes and reshaped journalistic outputs (^{Lewis & Zamith, 2017}; Pavlik, 2000; ^{Tandoc, 2019}). While technological innovations may appear to be smoothly integrated in the news production process, tensions manifest due to various concerns regarding technology’s impact on ethics, job security, and information integrity (^{Flew et al., 2012}; Lewis & Westlund, 2015; ^{Van Dalen, 2012}; ^{Weaver & Willnat, 2016}). One innovation in journalism where such concerns are specifically salient is automated journalism.

The use of automated journalism, which some narrowly define as referring to computer-written news with little to no human input beyond initial programming (^{Carlson, 2015}; ^{Linden, 2017}), confronts the journalistic field with threats, opportunities, and ambiguities. For some, automated newswriting frees up human journalists from writing trivial, templatized articles and allows them to focus on writing news that requires more analysis and higher discernment. For others, introducing automated newswriting in the newsroom presents yet another challenge to human journalists’ ethical stance, editorial control, and job security, which are already all under threat. It presents ambiguities, as while organizations may benefit from a more efficient news production process, individual journalists fear job displacement or the need for retraining. Thus, automation in journalism has attracted a lot of scholarly attention, from those who sought the perspective of human journalists on how automation is unfolding in the field, to those who examined how audiences process algorithm-written news articles. To a large extent, many of these studies assume that human-written news articles differ from automated-written news articles. But do they?

This current study seeks to answer this question by comparing human-written and algorithm-written news articles published by financial media company Bloomberg, one of the early adopters of automated newswriting. Bloomberg publishes algorithmwritten news articles under the byline of “Bloomberg Automation.” Through the lens of ^{Bourdieu’s (1998}, ²⁰⁰⁵) field theory and a content analysis of 1,282 randomly selected news articles published by Bloomberg from 2016 to 2017, this study compares human-written and algorithm-written articles based on established benchmarks of traditional news, such as the presence of particular news values, the use of sources and typical formats, and the presence of interpretation, and discusses the possible impact of automated newswriting’s growth on the field of journalism.

Method

This study is based on manual content analysis of news articles published by Bloomberg across two years: 2016 and 2017. The unit of analysis is each individual news article. Content analysis allows the examination of the extent to which content elements recur, but this is limited to only manifest content (i.e., what is in the data), which potentially misses out on analyzing nuances and sensemaking that qualitative approaches engage in. Launched in the 1980s, Bloomberg has long been known for its information “terminal,” a computer software system designed to meet the needs of finance professionals such as bankers, analysts and traders who require real-time and newsworthy information related to the economy and financial markets. While it provides news and data services across other platforms like the internet, radio, print, and television, Bloomberg’s terminal offerings remain a core business (^{Stewart, 2019}), with more than 320,000 subscribers worldwide (CB Insights, 2016). Having established itself as a leading provider of global financial news, Bloomberg serves as a noteworthy case study in automated newswriting. The company has increasingly focused on its capacities in automated journalism, a domain wherein competitors such as Associated Press, The Washington Post and Reuters are also developing expertise.

Using constructed week sampling (^{Riffe et al., 1993}), our analysis involves 1,280 articles: 650 articles are human-written articles, and 630 articles are algorithm-written articles. We sourced the articles from our university’s subscription to Bloomberg’s computer terminal. This paid subscription provides access to Bloomberg’s news reports, among other types of content. In this study, we focused on news articles in the terminal. We collected a representative sample of articles using constructed week sampling by constructing two full weeks for each year. To construct one full week for 2016, we randomly selected one Monday, one Tuesday, and so forth (e.g., using random selection, we randomly picked one Monday and then sampled all articles published on that day). We repeated this process to construct a second week for 2016, ensuring that no two dates are repeated (Riffe et al., 1993). The same process was done to sample articles from 2017.

Based on the literature, we developed a coding manual and trained three student coders in using the manual to analyze the sampled articles. Following the training, an intercoder testing exercise was conducted, where the coders independently analyzed 20 articles, evenly split between human and automated and excluded from the final sample. While intercoder reliability among the three coders was achieved in most categories, issues were found for some variables, including coding for the dominant source (Krippendorf’s α = .62) and the news value of proximity (Krippendorf’s α = .32). Proximity is defined as the geographical or psychological closeness of an issue or event to the newsroom (^{Martin, 1988}); however, Bloomberg operates several bureaus which is difficult to ascertain based on content alone. Thus, we decided to exclude this variable in our analysis. We conducted another round of coder training and subsequently conducted another round of intercoder testing, which yielded acceptable intercoder reliability results (Krippendorf’s α = .70 and above), allowing us to proceed with the actual coding of the articles based on the following categories:

Author type. First, we coded the articles if they were written by a human, using automation, or a combination of both human and automation. Bloomberg labels automated news articles with a byline that says “Bloomberg Data News” as well as a sentence at the end of the article stating that: “This story was produced by the Bloomberg Automated News Generator.” Since this was a straightforward categorization, and no article had combined authorship, the coders achieved perfect reliability (Krippendorf’s α = 1.0).

News values. The articles were coded for the presence or absence of five news values common in traditional journalism (^{Caple and Bednarek, 2015}; ^{Harcup and O’Neill, 2001}; ^{Rogers, 2004}). Negativity refers to when the article focuses on the negative aspects of an issue or event (Krippendorf’s α = 0.74). Timeliness refers to whether the news article is about a recent issue or event, and we coded each article based on its date of publication, so that an article is coded as having the news value of timeliness if the event or issue mentioned in the article is temporally close to the publication date (Krippendorf’s α = 1.0). Impact refers to whether the article is about an issue or event that has significant effect or consequences on a large number of people (Krippendorf’s α = 0.71). Novelty refers to whether the issue or event depicted in the article has new or unexpected aspects (Krippendorf’s α = 1.0). Finally, superlativeness refers to whether the issue or event itself has a large scope or scale, such as involving a high number of participants, etc. (Krippendorf’s α = 0.74). Eliteness or prominence was excluded from the analysis as the coders yielded consistently low reliability in both rounds of testing.

Topic. We adopted Bloomberg’s own categorization of topics as specified in the terminal. Thus, the articles were coded if they were about business news, general news, legal affairs, or sports. The coders recorded a reliability level slightly lower than the study’s threshold (Krippendorf’s α = 0.69)-this limitation must be considered when interpreting the results.

Dominant source. The articles were coded for the dominant human source used. We only focused on the main source, based on who is quoted in the lead or cited in most paragraphs. We also decided to narrow down to just the human source, although an article can also cite document sources. The dominant source can be from government, politics, or law enforcement; business; civic society; culture, arts, sports, or entertainment; academe; or ordinary people. We also coded for the absence of any human source. Following a second training session and intercoder reliability testing, reliability improved from the first round (Krippendorf’s α = .90).

Story format. This refers to the story structure, which can be in the form of a listicle, chronology, reverse chronology, narrative, or the commonly used inverted pyramid (^{Tandoc, 2018}). The coders achieved perfect reliability. We also coded for story length based on number of words, where the coders achieved acceptable reliability (Krippendorf’s α = .78).

Providing interpretation. Based on previous studies that argued about the limitations of automated news and the advantages of human authors, we also coded the articles on whether they include opinion (e.g., personal interpretation of an issue or event), context (e.g., background information or description of the bigger picture), and analysis (e.g., explanations or critical perspectives on the event). An article got a score of 1 for each of these elements; the scores across the three elements were added, so that an article can get a maximum score of 3 if all these elements were present. The coders achieved acceptable reliability (Krippendorf’s α = .74).

Results

Using manual content analysis, we compared Bloomberg articles written by human authors with those written by its automated news generator based on news values, topic, source, format, and interpretation, which are usually considered markers of traditional news, long been assumed to be produced by humans.

First, we focus on news values: negativity, timeliness, impact, novelty, and superlativeness. The analysis found that all articles analyzed contained the news values of timeliness and novelty, except for one algorithm-written article. This may be due to the nature of Bloomberg’s subscription-based news service that focuses on sending out news quickly to subscribers. Thus, we proceed to the three other news values: negativity, impact, and superlativeness. The analysis found significant associations between type of author (human vs. machine) and the presence of the three news values (Table 1).

Table 1 Comparing human-written and algorithm-written articles 

Note. a p < .001.

In terms of negativity (χ2[1, N = 1280] = 63.18, p < .001), human-written articles (49.1%) had a higher likelihood of containing negativity than algorithm-written articles (27.5%). Next, in terms of impact (χ2[1, N = 1280] = 59.93, p < .001), human-written articles (20.3%) had a higher likelihood of containing the news value of impact than algorithm-written articles (5.7%). Finally, in terms of superlativeness t (χ2[1, N

= 1280] = 67.25, p < .001), human-written articles (21.2%) had a higher likelihood of containing the news value of superlativeness than algorithm-written articles (5.6%).

Second, the analysis found significant association between type of author (human vs. automation) and broad topic, based on Bloomberg’s own categorization, (χ2[1, N= 1280] = 408.37, p < .001; see Table 1). Human-written articles were almost evenly split into business news (48.3%) and general news (52.7%) while algorithm-written news was mostly about business news (97.9%).

Third, the analysis found significant association between type of authorship and the use of human source (χ2[1, N = 1280] = 1108.61, p < .001; see Table 1). While we have coded for specific types of sources, we had to dummy code the variable into with human vs no human source to be able to run meaningful chi-square test of association, as algorithm-written articles were mostly devoid of human sources. While human-written articles were predominantly relying on human sources (93.5%) such as businesspeople, almost all algorithm-written articles did not mention a human source (99.4%). Fourth, in terms of story format, most human-written articles used the inverted pyramid format (96.2%) while all the algorithm-written articles except one (99.8%) used inverted pyramid (see Table 1). In terms of story length, there was a significant difference between human-written and algorithm-written articles, t (1278) = 23.61, p < .001. Human-written articles (M = 476, SD = 296) were significantly longer than algorithm-written articles (M = 189, SD = 72).

Finally, when it comes to providing interpretation-inclusion of opinion, context, and analysis-we found a significant, albeit small, difference between human-written and algorithm-written articles, t (1258) = 11.11, p < .001. Human-written articles were slightly more likely to exhibit various types of interpretation (M = 1.54, SD = .82) than algorithm-written articles (M = 1.13, SD = .44). The smaller variation in scores among algorithm-written articles also shows that algorithm-written articles tend to be more uniform, while human-written articles exhibited more variability in the extent to which forms of interpretation are included (see Table 1 for a detailed comparison).

Discussion

This current study compared human-written with algorithm-written news articles published by Bloomberg and archived in its terminal, which supplies a range of content, including news articles, to subscribers. Guided by the frameworks of field theory and journalistic boundaries, we compared the news articles based on traditional markers of news, which have been examined for a long time with the assumption that news is produced and constructed by human journalists. By using these traditional markers, we found that algorithm-written news shares some similarities with human-written news, such as focusing on timely or recent events and using the inverted pyramid format. Beyond these, we also found differences. First, human-written news articles tend to display more negativity and impact than algorithm-written news articles. Human-written news articles are also more likely to include interpretation, which many scholars studying automation in news have argued based on interviews with human journalists. Algorithm-written articles also tend to be shorter in length and contain no human sources.

Automation in the news has been initially welcomed for its promise of efficiency and speed. Indeed, we see this in the news value of timeliness, that algorithm-written news can assemble details faster, using automated processes. Since they are automatically generated using algorithms and templates that were initially programmed by humans, they also usually contain news elements that are easily templatized, such as story format. These findings show that machines can be programmed to mimic human output, at least to some extent. Across other aspects, however, we still find divergence between human-written and algorithm-written outputs.

First, we found that human-written news articles are more likely to provide interpretation. But a closer scrutiny of the data also reveals that algorithm-written articles also provide background information, a form of context (see Table 1). Background information can also be templatized, at least in business news, such as providing annual financial trends to contextualize daily reports. But human-written articles contain more analysis and opinion. Indeed, unlike machines, human journalists can inject opinion, analysis, and context into their work (^{Wu et al., 2018}). Second, and closely related to the earlier point, human-written articles tend to be longer, which can be explained by the injection of analysis and opinion. Differences in length may also be due to the use of human sources, common in human-written articles but absent in algorithm-written articles. Third, we found that Bloomberg uses automation almost exclusively for business news, while human-written articles represent a wider range of topics. This may be representing what previous studies have argued, that automation in journalism can free up human journalists to pursue more important, and perhaps more diverse, stories.

These findings also seem to show automation, at least in Bloomberg, is kept in its place by human managers-the humans continue to control the types of articles the machines are tasked to write, overseeing their work, and perhaps ensuring that they do not take on tasks that significantly threaten the position of human journalists in the newsroom hierarchy. Machines are delegated articles that are number-oriented and easily templatized, i.e., business news rather than general news, and that have less impact and are of a smaller scope or scale. These algorithm-written articles tend also to be devoid of human perspectives and have less interpretation. Human managers in the newsroom, it seems, are still controlling the rate at which machines are allowed to transform the news industry. That said, if automated news writing continues to grow in scale and improve in its ability to mimic human writing, how will it affect variety and diversity in news coverage? For example, our findings show that while human-written articles have a lot of variance when it comes to length and injection of interpretation; algorithm-written articles tend to be more uniform, displaying less variety. News organizations keen to utilize such automated technologies to reduce their dependence on manpower and increase their output may actively make decisions to machine-write a greater number of articles. Concerns then may arise, as news produced by machines tend to contain less critical thought, inquiry, and investigation (^{Abu-Fadil, 2016}). Output quantity may come at the expense of quality. Of course, the findings presented here must be understood in the context of several limitations. First, while manual content analysis allows empirical comparison of textual elements, it cannot capture the contexts and processes behind the production of manifest content. For example, our results cannot ascertain what kinds of editorial policies are in place that guide automated news writing (e.g., what kinds of stories are “assigned” to algorithms). Still, the findings presented here can complement interview-based studies that explored what journalists think about and do with automation. Second, we focused on a specific news organization, one that has the resources to experiment with and fine-tune the embedding of automation in its work processes, which limits the applicability of the insights gleaned from our analysis. Future studies can build on our findings to examine the content produced by automated processes implemented in other types of news organizations as well as articles. For example, other news organizations use automated newswriting for earthquake articles and sports news articles. Finally, our sample of news articles was from 2016 to 2017, and the use of automated newswriting may have changed since then, both within and outside Bloomberg, especially during the COVID-19 pandemic, where a lot of news coverage focused on pandemic-related statistics. Still, we hope the findings we presented here will be useful for future studies as we continue to examine and understand how automation is (or is not) transforming the journalistic field.