We systematically reviewed literature on near-term ecological forecasting to both determine how best practices have been implemented over time and compare forecastability across scales and variables. First, we used Web of Science searches and abstract review to identify papers that report near-term ecological forecasts (described in Literature search below). Two reviewers then independently read and analyzed each selected paper using a standardized matrix of criteria (Matrix analysis) and we recorded forecast skill when reported (Forecast skill).
Literature search (Dataset 1)
We began by querying Web of Science Core Collection [v.5.34] for “forecast*” in the title, abstract, or keywords of papers published in 301 ecological journals, then manually screened abstracts of all resulting papers. We conducted the Web of Science search on 18 May 2020 and limited the search to articles and proceedings papers (hereafter, ‘papers’) published in English. This yielded 2711 results.
We screened the abstracts of all 2711 papers and selected those that met three criteria:
(1) Papers had to include at least one forecast, which we defined as a prediction of future conditions from the perspective of the model; forecasts could be developed retroactively (i.e., “hindcasts”) but could only use driver data that was available before the forecast date (e.g., forecasted or time-lagged driver variables).
(2) The forecast had to be near-term, which we defined as predicting ≤ 10 years into the future.
(3)The forecast had to be ecological, which we defined as predicting a biogeochemical, population, or community response variable. This definition therefore excludes physical (e.g., streamflow or water temperature) and meteorological forecasts. Forecasts of human disease were only included if there was an animal vector.
If the abstract indicated that the paper met all three criteria, it was moved to a second round of screening. Here, a second reviewer read the full paper to ensure that at least one forecast in the paper met all three criteria. Through this screening process, we identified 142 near-term ecological forecasting papers out of the 2711 Web of Science results.
Because ecological forecasts may be published in journals that are not categorized as “ecological” by Web of Science, we then searched all papers that were cited by the near-term ecological forecast papers we identified, as well as all papers that cited these studies. We selected those that were published in English and included “forecast*” in the title, abstract, or keywords, then screened the abstracts to ensure they met our three criteria. Finally, we read the papers themselves for confirmation.
Matrix analysis (Dataset 2)
We analyzed each of the 252 papers selected in our systematic search using a standardized matrix of questions (SI Table 1). This matrix was co-developed over several months of iteration and discussion by all authors within an Ecological Forecasting graduate seminar at Virginia Tech (January–May 2020). The final matrix used for this study included 65 questions about the model, evaluation, cyberinfrastructure, archiving, and decision support (SI Table 1).
Throughout the graduate seminar, we read and analyzed 10 papers as a group, ensuring that all reviewers understood how to interpret and answer questions in a consistent manner. Reviewers also screened several papers individually and checked their responses with another reviewer prior to the start of this analysis, helping to ensure consistency between reviewers. For the analysis described in this paper, all 252 papers were read and analyzed independently by two reviewers, and reviewers then compared any differing answers to reach consensus on a final set of responses for each paper.
During the intensive matrix analysis, 74 papers were determined to not meet our criteria of being near-term ecological forecasts, despite having passed the initial rounds of screening—these papers typically used one or more data sources that would not have been possible to know before the forecast date, and were difficult to identify without reading the entire paper in detail. These papers were excluded from the analysis, leaving 178 papers in the final dataset.
Forecast skill (Dataset 3)
We gathered all Pearson's r and R2 data reported in papers in the dataset. Pearson’s r values were squared to yield R2
