The Index Investor
March 2019

Key Takeaway

Many asset classes remain significantly overvalued.

Market stress indicators have declined somewhat since last month, however the balance of three month returns for assets expected to perform best under different regimes indicates a continuing level of investor uncertainty.

We have again increased the probability that 12 months from now the macro system will still be in the High Uncertainty Regime (60%), and decreased the probability of the Persistent Deflation Regime to 40%.

This reflects new evidence on the delayed impact of strengthening deflationary forces (e.g. the slow rate at which business is implementing rapidly improving labor substituting automation and artificial intelligence technologies), continued increases in US housing costs, the prospect of a US/China trade deal, and the decision of the European Central Bank to join the Federal Reserve in delaying planned rate increases in the face of slowing growth.

Asset Class Valuation and Momentum Indicators (@28Feb19)

Note: The language we use to describe our estimated likelihood of asset class over or undervaluation is based on US Intelligence Community Directive 203 on Analytic Standards, which includes the following table:

Market Stress Indicators (@28Feb19)

Market Stress Indicators: Forecast Discussion

We view financial markets as a complex adaptive system. The size of changes generated by such a system follows a power law rather than a normal (Gaussian) distribution. The critical point is that large changes are much more common in complex adaptive systems than most people’s intuition leads them to believe.

While predicting the behavior of complex adaptive systems remains far more art than a science, various researchers have found that large changes in such systems are often preceded by subtle warning signs, as stress accumulates within them. While this research is not definitive, we believe that five warning signs are worth monitoring as potential indicators of growing stress within financial markets that could suddenly give rise to large changes in asset class valuations.

Our first indicator is the month-to-month autocorrelation of broad asset class returns (i.e., the relationship of this month’s returns to last month’s). A system under increasing stress loses resiliency, causing it to take longer to recover from perturbations; hence, autocorrelation increases as it approaches a critical transition (see, “Early Warning Signals for Critical Transitions” by Scheffer, et al).

The one-month autocorrelation of returns for the broad asset classes we monitor fell sharply to .28, versus (.86) last month. This indicates that financial markets are becoming less ordered and are potentially further away from a critical transition point (which would most likely be accompanied by sudden and substantial changes in asset class values) than they were last month.

The second market stress indicator we monitor is the Economic Policy Uncertainty Index published by the Federal Reserve Bank of St. Louis (via its FRED economic database), which is based on research by Baker, Bloom, and Davis (see their paper, “Measuring Economic Policy Uncertainty”). The index is based on automated text analysis of leading newspapers and magazine publications, to identify the frequency with which words and phrases are used that indicate uncertainty.

In humans’ evolutionary past, when uncertainty increased the probability of survival was enhanced by staying close to a group. All of us still have that instinct. Research has found that as uncertainty increases, we have an unconscious bias towards higher conformity of our own views with those of a larger group (i.e., reduction in cognitive diversity). Behaviorally, heightened uncertainty induces more “social copying” of others, likely due to both conformity bias and the rational belief that others may be acting on the basis of superior information. This increase in conformity and copying makes a social system more ordered as uncertainty increases, and also reduces its responsiveness to perturbations (i.e., increases autocorrelation) because of delays in the social copying process.

The key point is that increasing uncertainty induces more, not less order in social systems, and in so doing primes them for sudden non-linear change.

Based on our continued research into the insights this index can provide, in 2019 we are focusing on the number of days, in the previous 30 days, that this index was in the top quartile of all values since the index series begins at the start of 1985. We then compare this statistic to the full set of rolling 30-day periods, and calculate its percentile at the end of the most recent month. At the end of February, our rolling 30 day count of top quartile values was in the 40thth percentile – a substantial fall from last month, which indicates the macro system is less primed for sudden, non-linear change.

Our third market stress indicator is the spread between the yield on AAA rated bonds and the 10-year US Treasury. This is a proxy for the level of investor concern about financial system funding liquidity. At the end of February 2019, this spread stood at 1.13%, (the 44th percentile since the series began in 1983), down slightly from last month’s 1.16% spread. This indicates essentially no change in market stress.

Our fourth market stress indicator is the yield spread between speculative BB rated bonds and the ten-year US Treasury. Throughout history, excessive credit growth has been a root cause of many financial crises. An indicator of such growth is falling credit spreads, particularly in the case of riskier borrowers. In contrast, rising BB spreads indicate growing investor concern about the consequences of such growth, and the financial distress lower rated companies could experience in an economic downturn. At the end of February 2019, this spread was 2.39% (21st percentile since the series began in 1996), down from the 3.60% spread at the end of December. This is a very low level for this late in what is already an exceptionally long period without a serious economic downturn. As such, it likely indicates the further buildup of hidden stresses in credit markets.

Our fifth market stress indicator is what we term the “political risk premium” that is implicit in the price of gold. Our starting point for estimating this premium is the three different roles that gold plays. First, gold is a store of value in a world of fiat currencies. When the rate of money supply growth exceeds the growth of nominal GDP, gold’s price should increase to maintain its purchasing power. Between 2007 and 2017, the US money supply (M2) grew by about 86%, while nominal US GDP grew by 35%. The stock of gold grew by 18%, based on mine production over this period. We therefore infer that 33% of the increase in the price of gold represented the maximum potential gold price change that could be attributed to a desire to hedge inflation risk (86% less 35% less 18%).

Second, gold is a unit of account. We take this to mean that the annual change in GDP expressed in terms of physical gold (i.e., nominal GDP divided by the price of gold) should equal the change in real GDP calculated using the GDP price deflator to account for actual inflation over the period. A key challenge is the point at which to start this calculation.

We chose the price of gold in 1995/1996. In that period, the change in real global GDP measured using the IMF’s price deflator just about equaled the change in GDP measured in terms of physical gold. We interpret that coincidence as indicating that at that point in time, concerns about future inflation and political risk were minimal, and the change in the price of gold was mostly driven by its role as a unit of account. We calculated a subsequent series of gold prices that would produce the same change in “gold GDP” as the actual real GDP as calculated by the IMF. Between 2007 and 2017, “gold as a unit of account” warranted a 21% increase in its price.

Gold’s third role is as a hedge against inflation and what we term “political disaster” risk. We subtract the 21% estimated compensation for actual inflation from the 33% “gross” inflation risk hedge to derive an apparent 12% increase in the gold price that reflected the true risk premium to hedge against possible future inflation. However, between 2007 and 2017 the price of gold actually increased by 81%. This implies that 48% of this (81% less 21% less 12%) represented a premium for some other type of uncertainty at the end of 2017. The interesting question is the nature of the uncertainty for which gold is believed by some investors to be a superior hedge than traditional ports in a storm like short-term US government securities, or similar securities issued by other developed countries.

The logical inference is that the uncertainty in question must reflect a situation in which short term US Treasuries would be a less effective hedge than gold. This could be a world of widespread hyperinflation, capital controls, and/or radical changes in nations’ governments (of course, this would also imply a preference for investing in gold coins rather than bullion, as while the latter may be a store of value, it is far less convenient as a means of paying for transactions).

To put this in further perspective, this gold price “disaster risk” premium sharply increased from 2008 to 2012, then declined before sharply increasing again after 2016. Arguably, a significant part of the former increase reflects concerns about the potential inflationary consequences of dramatic quantitative easing by central banks. But this is not likely to be the case after 2016.

Over the last month, the price of gold rose by 0.22%, Since the end of 2017, it has increased by 2.23%.Therefore, on a rough approximation, the political uncertainty premium at the end of February 2019 stood at about 50% (48%+2.23%) compared to a low of about 39% at the end of August 2018. This is another indicator of rising market stress.

To conclude, three of our indicators – asset class return autocorrelation, top quartile values for the Economic Uncertainty Index, and the AAA spread over Treasuries– point to falling levels of underlying market stress this month. In contrast, our metrics for the BB spread and gold price indicate rising stress in financial markets.

Macro Regime Forecast and Implications for Asset Class Values

Summary: Why Did We Change Our Regime Probabilities?

Perhaps the wisest insight I’ve come across in 40 years of forecasting is this quote by the late economist Rudi Dornbusich: “Crises take a much longer time coming than you think, then happen much faster than you would have thought.”

With that quote in mind, I’ve increased the probability that 12 months from now we will still be in the High Uncertainty Regime from 50% to 60%, and decreased the probability that we’ll be in the Persistent Deflation Regime from 40% to 30%.

An interesting mix of new high value evidence drove this forecast change.

On the one hand, there was more evidence of the buildup of deflationary pressures in the economy, including more announcements of impressive technology announcements in areas of potentially labor substituting artificial intelligence technologies, a comprehensive new report from the OECD that provided further details on the extent of corporate debt problems that could emerge in an economic downturn, and indicators that aggregate demand is slowing at an accelerating pace (even without a full-blown US-China Trade War).

But on the other hand, there were indicators that the deflationary impact of the forces building in the system might be delayed. These included (1) new reports on the relatively slow pace at which corporations are implementing rapidly improving new automation and artificial intelligence technologies; (2) new US CPI data which showed a still strong 3.2% trailing 12 month increase in the cost of owner occupied and rental housing (“shelter” has a 33% weight in the CPI); (3) indicators that both Xi Jinping and Donald Trump may opt for some type of face saving trade deal instead of a worsening trade war; (4) a new report from McKinsey which found negative productivity growth in the US Medical Services sector (which has a 7% weight in the CPI); and (5) the announcement by the European Central Bank that it would join the US Fed in delaying planned interest rate increases in light of a surprising slowdown in aggregate demand growth (which we interpret as the logical outcome of past uncertainty shocks).

All of these factors argue not for a return to the Normal Regime or a transition to the High Inflation Regime, but rather for a continuation of the High Uncertainty Regime for longer than we had previously expected.

Additional evidence also provided further support for the High Uncertainty Regime. In the US, the political split in the US Democratic Party became harder to ignore with the announcement of the Progressive “Green New Deal”. The uncertainty created by radical proposal like this is further enhanced by the lack of proposals by centrists from either party that have a credible chance of successfully addressing the challenges facing the US middle class. Absent such proposals, movement toward the political extremes, and the uncertainty this creates, seems likely to continue.

In the UK, the split in Labor broke into the open with the departure of MPs to form the new Independent Group. And as EU negotiators refused to budget, a “no deal” Brexit looked increasingly likely. And the Trump saga continued, with damaging congressional testimony by Michael Cohen, his former lawyer, that essentially guarantees that investigations will continue throughout the runup to the 2020 election (unless, of course, Trump departs from office before then).

More removed from the political and economic drivers of uncertainty, but important nonetheless was new evidence about the increasing conflict between western governments and companies and “state actors” in cyberspace, including the evolution of offensive cyberstrategies (see this month’s Evidence File note on “cyberblitzkrieg”).

Finally, another indicator of increasing uncertainty was new evidence of Vladimir Putin’s declining domestic popularity and the worsening performance of the Russian economy, at a time when the European Union is increasingly in disarray. Such circumstances seem tailor made to tempt Putin to seek another challenge to the West.


Forecast Methodology

The focus of our monthly macro forecast is twofold. First, the probability of a change in financial market regime that causes changes of 20% or more in asset class valuations over the next year. Second, contingent on such a change taking place, the probability of a subsequent transition to other regimes.

Our analysis focuses on four possible macro regimes: (1) Normal Times, where equity asset classes perform well; (2) a High Uncertainty regime that is usually short and transitory, where asset classes like short-term government bonds perform best and equities suffer significant declines; (3) High Inflation (which we deem 5% or more, year-on-year), where commercial property, real return bonds and other traditional hedges are favored; and (4) Persistent Deflation (a year-on-year decline in the US CPI), which up to now has only been seen in Japan, and in which the relative performance of different asset classes remains uncertain, but will likely favor high quality bonds and the consumer staples equity sector.

Our forecasting methodology is derived from our experience on the Good Judgment Project, as described in the book, “Superforecasting” by Gardner and Tetlock, as well as a range of other sources, from the intelligence community to systems dynamics and complex adaptive systems to statistics and political economy.

We start with base rate/reference case data about the historical probability of large changes in equity and bond valuations. We then analyze the current situation from both a quantitative and qualitative perspective. In the latter, we focus on the key endogenous drivers of macro regime change, including technological, economic, national security, social, and political trends and uncertainties. We also focus on three potential sources of exogenous shocks that could also produce a macro regime change, caused by environmental, disease, and cyber related events.

While most of our attention typically focuses on various flows (e.g., economic growth, change in the price level, sales, earnings, job creation, etc.), endogenously caused regime changes result when those flows push key stocks beyond a critical threshold or tipping point, often setting off non-linear reactions across multiple areas. As noted by Hyman Minsky and others, a classic example is the steady accumulation of outstanding debt until it reaches the point where it can no longer be serviced and triggers a crisis.

Base Rate Data

Since the end of World War Two, there have been fifteen months where a downturn in the US equity market began that eventually reduced asset class value by 20% of more. That is a hazard rate of about 1.75% per month. Put differently, in any given month there is a 98.25% probability that a 20%+ downturn won’t occur, or, in a given year, an 81% probability.

However, as the time without a 20%+ downturn extends, the compound probability that one will not occur shrinks. At the end of August 2018, it is more than nine years since the last equity market decline of 20% or more. The probability of that happening is only 15%.

To estimate the base rate for a 20% fall in bond prices (which historically has been caused by a sharp increase in inflation, as we saw in the late 1970s and early 1980s), we analyzed monthly historical AAA bond yields since 1919. For consistency, we used them to calculate the price of a ten-year zero coupon bond. We then calculated the probability of a price decline of 20% or more over three different holding periods: 12, 18, and 24 months. In any month, the annualized probability of a decline of 20% or more over the subsequent 12 months is 12%; over 18 months, 20%, and over 24 months, 25%.

The Current State of Quantitative Regime Predictors

Our quantitative methodology focuses on the level and change in three-month returns, over the most recent and previous three-month periods, for those asset classes which should perform best under different regimes.

As you can see in the following table, based on three month returns to the end of February 2019, this analysis indicates that the balance of expectations across all four regimes was in rough balance, which we consider to be another indication of high underlying uncertainty.

That said, comparing the change between the two three month periods, positive momentum is strongest for both the Normal and the High Inflation Regimes.

Like Professors Andrew Lo, Doyne Farmer and others, we regard financial markets as a complex adaptive system (CAS), that exist as part of a larger macro system comprised of other CAS between which there are multiple feedback loops. These other systems include those that produce technology innovations, and economic, environmental, national security (including cyber), social, demographic, and political outcomes.

We also find that these systems tend to operate and generate effects in a rough chronological sequence, albeit with many feedback loops between them. The following chart highlights that the changes we observe in different areas at any point in time are actually part of a much more complex evolutionary process.

While most media coverage of these systems focused on flows (e.g., the size of the government deficit), rapid non-linear change in complex adaptive systems is often caused by a key stock (e.g., the amount of outstanding government debt) exceeding a critical threshold.

The next table highlights the key macro system stocks that we monitor.

In the next section, we will discuss information received over the past month that is related to these stocks, and which we believe is significant to our assessment of the probabilities that a critical threshold will be reached and a regime change will occur. We will conclude with our estimate, at the end of this month, of how close the macro system is to these critical thresholds, and the implications for financial market regime change probabilities.

In our methodology, we classify new information as significant and highly valuable if either it (1) is an “indicator”, which reduces or increases our uncertainty about the value of a parameter in our mental model for making sense of the dynamic macro system, or (2) it is a “surprise” which increases our uncertainty about, and causes us to revaluate the structure of our mental model.

How Close is the Macro System to One or More Critical Thresholds?

As we have noted, the macro drivers of financial market regime changes typically follow a rough chronological sequence, from technology to economic, security, social, and political causes and effects. Yet there are many feedbacks loops between them, creating complex root causes for many of the critical thresholds we have identified.

Understanding the time dynamics in this complex system is critical to avoiding substantial downside investment risk.

We use the UK Met Office Warning Model to communicate our assessment of these time dynamics. We estimate the time remaining before a critical macro system threshold is reached that could trigger a regime change, which is usually accompanied by substantial changes in asset class valuations.

The model uses three increasingly serious levels of warning, from “Be Aware” (condition yellow), to “Be Prepared” (condition orange), to “Take Action” (condition red).

For our purposes, we denote as “Be Aware” (yellow) critical thresholds that we assess to be three or more years away. We estimate that “Be Prepared” (orange) thresholds could be reached within 1 to 3 years. “Take Action” thresholds are very likely to be reached within one year.

Given their nature, we also note that in our three “wildcard” areas (Environment and Energy related; Disease and Human Caused Bioevents; and Cyber and Electromagnetic Events), our forecasts have higher levels of uncertainty.

The following charts summarize our current estimate of the time remaining before different critical thresholds will be reached.

Our current forecast question is this: what is the probability we will either remain in the High Uncertainty Regime or transition to another regime over the next twelve months?

We currently estimate there is a 60% chance of remaining in the High Uncertainty Regime over the next 12 months (an increase of 10% from last month), which will see the commencement of what promises to be a tumultuous US presidential campaign, and what looks increasingly likely to be a so-called “Hard Brexit” with the UK leaving the EU without a transitional agreement.

We also conclude that over the next 12 months, the probability of returning to the Normal Regime is slight, at 5%, as is the probability of entering the High Inflation regime over the next 12 months, at only 5%.

We estimate that the probability of entering the Persistent Deflation Regime over the next 12 months has declined 10% to 30%, for reasons noted earlier in this month’s issue.

While we expect the High Uncertainty Regime will produce declines of 20% or more in equity asset classes, it is unlikely that any other asset class will experience a gain of 20% or more. We estimate there is a roughly even chance that gold could be the exception, with that increase heavily tied to continued global confidence in the US government and economy. While the apparent inflation and political uncertainty premia in the gold price today are high relative to the last 25 years, they are still below the peak reached in 2012, and it is possible that herding in the face of increasing uncertainty could produce 20% price gainsAt the highest level, we believe the global macro system can be in one of four states, based on its degree of order versus disorder, and degree of social cooperation versus conflict. We believe that the system is currently in its most uncertain state, characterized by high degrees of underlying disorder and social conflict.

Pre-Mortem Analysis

One of the most important forecasting disciplines is to ask yourself why your forecast could be wrong. Dr. Gary Klein’s research has shown that a very powerful and insightful way to do this is via a “pre-mortem analysis.” This method asks you to assume that it is a point in the future, and your forecast has been proven wrong (or your strategy or company has failed). You are then asked to look backward from this imagined point in the future, to explain why you failed, what you missed, and what you could have done differently to avoid your fate.

The pre-mortem method takes advantage of the fact that humans reason much more concretely and in more detail when explaining the past than they do when trying to forecast the future.

So let us assume that it is one year from now, and our current forecast has turned out to be wrong.

How did this happen? What developments did we fail to anticipate?

• At the end of February 2019, the leaders of the world’s three major powers – Xi Jinping, Donald Trump, and Vladimir Putin are all facing weakening economies and declining political popularity. History teaches us that this can lead to increased “foreign adventurism” to distract the public from worsening domestic conditions, as a nation rallies around its leader in a period of heightened external conflict. Should such a conflict develop between China and the United States, or between Russia and one or more European countries, it would generate a sharp increase in uncertainty that would likely cause a sharp economic slowdown and, given high debt levels, speed the arrival of the Persistent Deflation Regime.

• With the UK “crashing out” of the EU at the end of March – i.e., “Hard Brexit” – looking more likely by the day, it could also be possible that we have underestimated the economic disruption that will result. This could also cause a sharp increase in uncertainty, not the least because it will further undermine public confidence – perhaps on both sides of the Atlantic – in current political leaders’ ability to successfully navigate an increasingly complex and disordered environment. This could set the stage for later electoral wins by populist politicians on either end of the political spectrum, such as Jeremy Corbyn or Marine LePen. This scenario too would increase the probability of entering the Persistent Deflation Regime.

Note: Combining this Forecast with Others and Extremizing the Result Should Increase Predictive Accuracy

Research has found that three steps can improve forecast accuracy. The first is seeking forecasts based on different forecasting methodologies, or prepared by forecasters with significantly different backgrounds (as a proxy for different mental models and information). The second is combining those forecasts (using a simple average if few are included, or the median if many are). The final step, which significantly improved the performance of the Good Judgment Project team in the IARPA forecasting tournament, is to “extremize” the average (mean) or median forecast by moving it closer to 0% or 100%.

Forecasts for binary events (e.g., the probability an event will or will not happen within a given time frame) are most useful to decision makers when they are closer to 0% or 100% than the uninformative “coin toss” 50%. As described by Baron et al in “Two Reasons to Make Aggregated Probability Forecasts More Extreme”, forecasters will often shrink their probability estimates towards 50% to take into account their subjective belief about the extent of potentially useful information that they are missing.

When you average multiple forecasters’ estimates, you are including more information, which should increase forecast confidence and push the mean estimate closer to 0% or 100%. However, this doesn’t happen when you use simple averaging. For this reason, forecast accuracy is increased when you employ a structured “extremizing” technique to move the mean estimate closer to 0% or 100%.

You can download an extremizing model from our website to use when combining the forecasts you use in your decision process. The extremizing factors in our model are those that the Good Judgment Project found maximized the accuracy of combined forecasts. Note that the extremizing factor is lower when average forecaster expertise is higher. This is based on the assumption that a group of expert forecasters will incorporate more of the full amount of potentially useful information than will novice forecasters.

Over the years, I’ve learned a lot about forecasting the behavior of complex socio-technical systems (e.g., organizations, industries, financial markets, nation states, and global systems) from a wide range of very smart people, including Pierre Wack, Philip Tetlock, Doyne Farmer, Marvin Cohen, Gary Klein, and many others. Clearly I am a fox, not a hedgehog!

To help our subscribers become better forecasters, this month’s feature article focuses on three issues, the importance of which is often underappreciated. These are (1) How to define forecasting questions, (2) How to establish a prior or baseline forecast, and (3) How to weigh new qualitative evidence and update your prior.

Defining Your Forecasting Questions

Many of the questions about the future that we encounter are broadly defined – for example, “How is the US-China relationship going to evolve?” or, “Will I outlive my retirement savings?”

Answering these questions requires that we break them down into sub-questions that are more amenable to the application of forecasting methods.

Developing these sub-questions requires us to have or develop some type of causal model that includes the key elements in a situation and how they interact with each other to produce an answer to the broad question we are trying to predict.

Effective sub-questions have three other characteristics: (1) they are phrased in a way that allows the use of probabilities to answer them; (2) they have a specific time horizon; and (3) they have clear criteria for establishing the accuracy of a forecast at or before the end of that time horizon.

For example, to forecast the relative returns on different asset classes, we focus on four sub-questions

(1) What is the probability that, 12 months from now, we will be in the High Uncertainty Regime (defined as a decline in the FTSE All-World equity index, expressed in US Dollars, of at least 20% over the previous 12 months)?

(2) What is the probability that, 12 months from now, we will be in the High Inflation Regime (defined as an increase of 5% or more in the US Consumer Price Index over the previous 12 months)?

(3) What is the probability that, 12 months from now, we will be in the Persistent Deflation Regime (defined as a decline in the US Consumer Price Index over the previous 12 months)?

(4) What is the probability that, 12 months from now, we will be in the Normal Regime (defined as not being in any of the other regimes)?

In our causal model, these macro regimes are the primary driver of asset class returns. In turn, these macro regimes result from the interplay of a series of higher-level drivers that operate in a rough chronological sequence (with multiple feedback loops), including changes in technological, economic, national security, social, and political conditions.

It is important to note that most socio-technical systems are both complex and adaptive. Forecasting the future outcomes produced by such systems is extremely difficult, not only because they typically have many cause-effect relationships, but also because many of these relationships are time-delayed and/or non-linear. Moreover, the adaptive actions of agents (both human and algorithmic) within such systems often cause the structure of the system itself to evolve over time.

In modeling terms, the sources of uncertainty include randomness, the parameter values for different variables and relationships, and the structure of the model itself.

As Robert Hoffman and Gary Klein have shown (in their series of papers on “Explaining Explanation”) the challenge posed by the complexity of socio-technical systems also extends to our attempts to explain their past behavior, which is often a key source of the mental models we use to forecast their future outcomes.

To be sure, advancing technology is slowly making this task easier (e.g., agent based modeling and simulation and a variety of artificial intelligence methods). But using models to forecast the future behavior of complex socio-technical systems still has a very long way to go (e.g., “Priority Challenges for Social and Behavioral Research and Modeling”, by Davis et al from RAND).

At this point, when trying to anticipate the behavior of complex adaptive socio-technical systems the best we can hope for are forecasts that are “coarse-grained”, but accurate, particularly as the time horizon extends. Our goal is to produce accurate forecasts of broad macro regimes, rather than very specific events that could contribute to the development of a given regime.

Establishing Your Prior/Baseline Forecast

Having defined your forecasting questions, the next challenge is developing your prior (to use the Bayesian term) or initial/baseline forecast. In many cases, this turns out to be more difficult than it first seems.

In the easiest case, a baseline forecast can be constructed using easily obtained data on the frequency of comparable historical events – e.g., the percentage of times a US equity market index fell by 20% or more over rolling 12 month periods since a given starting date.

When direct evidence like this is unavailable, you must use other techniques. One approach I have found very useful starts with the search for analogies to the forecasting problem at hand and determining if there is better historical data available for them.

If there is not, I ask myself what would be the likely shape of the distribution of outcomes if they were available. This is a very important step, because our baseline mental model of what a distribution looks like – the normal (Gaussian, Bell Curve) distribution we learned in statistics class – often does not accurately describe the distribution of results produced by complex social systems (as Nassim Taleb reminded us in Fooled by Randomness and The Black Swan).

Instead, outcomes produced by systems in which social learning or influence occur are best described by a power law function or Pareto-type distribution, with a large number of small outcomes and a few very large ones (e.g., see, Boisot and McKelvey, “Extreme Events, Power Laws, and Adaptation” and Andriani and McKelvey, “From Gaussian to Paretian Thinking”).

However, the histories of complex social systems are also filled with plausible counterfactuals, because the outcomes we observe are the result of interacting situational forces, human decisions, and randomness (or, if you will, luck). For this reason the use of analogical reasoning to establish a prior must be balanced with an analysis of the forces at work in the case at hand, and the forecast probabilities they imply. As a final step, the probability produced by analogy must be combined with the probability derived from analysis of the current situation. As a general rule, the more similar the case is to the analogy used, the greater the weight that should be put on the latter.

A third way to establish a prior is to ask yourself what the conventional wisdom is on the question you are trying to forecast. This approach can be particularly useful in situations of high uncertainty, where, as John Maynard Keynes noted, commonly accepted “conventions” are used when no better information is available. Evidence that is subsequently gathered can then be used to test the accuracy of the conventional forecast, as, for example, described by Rappaport and Mauboussin in their 2001 book, Expectations Investing.

Weighing New Evidence and Updating Your Prior Probabilities

Having established your baseline/prior forecast probability estimate, the next challenge is determining how you should adjust it after receiving new information.

In my experience on the Good Judgment Project, I found that this process of evidence collection, evaluation, and weighting was critical to the accuracy of my forecasts.

In a world of information overload, the first challenge is deciding which information to target for collection. I found that doing a pre-mortem on my baseline forecast was extremely useful in this regard. A pre-mortem uses a technique called “prospective hindsight” to highlight weaknesses in our forecasts. Research has shown that when we attempt to explain the past, we get to a much more specific level of detail than we do when trying to forecast the future. Hence, the pre-mortem technique asks you to imagine that at some time in the future, your forecast has been shown to be wrong. It then asks you to write down why this occurred – the important information or dynamics you missed, and what you could have done differently to avoid being wrong. Having used it both individually and in groups, I can assure you it is a very powerful technique (as more systematic research has also found).

As a general rule of thumb, the greater the number of assumptions you use in your argument, and the more uncertain they (and perhaps the forecast logic itself) are, the lower should be the resulting forecast probability.

Doing a pre-mortem on our forecast highlights the assumptions and/or logic in your forecast about which you are most uncertain, and which would therefore most benefit from collecting additional information.

There are three questions that must be asked about each new piece of information you obtain, before it is combined with others and weighed to determine by how much you should adjust your most recent forecast probability.

The first question is whether the information is relevant to the forecasting question at hand. If your collection effort is targeted, it should be.

The second question is whether the information is credible. In a world of fake news and echo chambers, this has become a very non-trivial issue. While intentionally deceptive information has always been a concern for professional intelligence analysts, it is now a concern for everyone.

Assuming new information is relevant and credible, the third step is to weigh its value and impact on your prior forecast probabilities.

Broadly, there are three systematic approaches to weighing evidence (which itself is an ambiguous phrase, with no agreed upon meaning).

In the 17th century, Sir Francis Bacon posited that the weight of evidence for or against a hypothesis depends on both how much relevant and credible evidence you have, and on how complete your evidence is with respect to those matters that you believe are critical for evaluating a hypothesis.

Bacon recognized that we can be “out on an evidential limb” if we draw conclusions about the probability a hypothesis is true based on our existing evidence without also taking into account the number relevant questions that are still not answered by the evidence in our possession. We typically fill in these gaps with assumptions, about which we have varying degrees of uncertainty. In this context, the value of a new piece of information depends on the degree to which it either reduces the number of assumptions you use, or increases your confidence that they are accurate.

In the 18th century, Reverend Thomas Bayes invented a quantitative method for using new information to update a prior estimated probability (degree of belief) in the truth of a hypothesis.

”Bayes Theorem” says that given new evidence (E), the updated (posterior) probability that, in light of this new evidence, a hypothesis is true p(H|E) is a function of the conditional probability of observing the evidence given the hypothesis p(E|H), times the prior probability that the hypothesis is true p(H), divided by the probability of observing the new evidence p(E).

The “Likelihood Ratio” is a critical Bayesian concept. It is equal to the probability of observing a piece of evidence if a hypothesis is true, (E|H), divided by the probability of observing the evidence if the hypothesis is false, p(E|not-H). The greater the Likelihood Ratio for a piece of new evidence, the greater is its information value, and thus the larger should be the difference between your prior and posterior probabilities. When it comes to evaluating new evidence, the Likelihood Ratio is a very valuable heuristic that is easy to intuitively apply.

In the 20th century, Arthur Dempster and Glenn Shafer developed a new theory of evidence weighing.

Assume a set of competing hypotheses. For each of these hypotheses, a new piece of evidence is assigned to one of three categories: (1) It supports the hypothesis; (2) It disconfirms the hypothesis (i.e., it supports “Not-H”); or (3) it neither supports nor disconfirms the hypothesis. To relate this back to Bayes, in cases (1) and (2), the evidence has a high Likelihood Ratio; in the case (3) it does not.

The accumulated and categorized evidence can then be used to calculate a lower bound on the belief that each hypothesis is true (based on the number of pieces of supporting evidence and their quality), as well as an upper bound (equal to one less the probability that the hypothesis is false, again, based on the evidence that disconfirms the hypothesis, and its quality). This upper bound is also known at the plausibility of each hypothesis.

The difference between the upper (plausibility) and lower (belief) probabilities for each hypothesis is the degree of uncertainty associated with it. Hypotheses can then be ranked based on their degrees of uncertainty. Similar to the Likelihood Ratio, in the Dempster-Shafer context the value of a new piece of information is proportional to the change it produces in the uncertainty associated with one or more hypotheses.

While there are quantitative methods for applying all of these theories, they can also be applied qualitatively, to quickly produce an initial conclusion about which of a given set of hypotheses is most likely to be true, and by how much you should adjust a forecast probability.

For example, in our analytical process, we use the same probability categories as the US Intelligence Community:

Almost Certain: 95% or more
Very Likely: 80% - 95%
Likely: 55% - 80%
Even Chance: 45% - 55%
Unlikely: 20% - 45%
Very Unlikely: 5% - 20%
Almost No Chance: 5% or less

These categories provide a starting point for using evidence to formulate and later update estimated forecast probabilities. The probability adjustments should be proportional to the relevance, credibility, and information value of new evidence that is received.

However, one of the key lessons from the Good Judgment Project was that the best forecasters make probability distinctions that are finer than these seven broad categories (see, “Small Steps to Prediction Accuracy” by Atanasov et al).

Another lesson was that individual forecasters’ probability estimates were often too close to the 50% “toss-up” category that is the least useful to policymakers. In other words, individuals typically did not give enough weight to the information they used to make their forecast. The GJP team compensated for this (and increased forecast accuracy) by combining individual forecasts and then making the resulting probability more extreme. Subscribers can download this “extremizing” equation from our website.

The final – and critical – step in using new evidence to update your prior probability (to a so-called “posterior”) is to repeat the pre-mortem process.

This has two important benefits in addition to focusing your subsequent information collection activities. First, it forces you to make explicit the logic, evidence, and assumptions that underlie your forecast. Second, by forcing you to recognize the greatest uncertainties in your forecast, it reduces your overconfidence in its accuracy.

Caveat #1: The Dangers of Social Information

Broadly speaking, there are two kinds of social information you can receive. One will help improve your forecast accuracy, and the other very likely will not, particularly in highly uncertain situations.

The first is information from other people preparing forecasts on the same issue you are, which details their logic, evidence, assumptions, and conclusions, and/or challenges your own. The Good Judgment Project showed that this team-based approach can improves forecast accuracy.

The second type of social information simply tells you which is the most popular forecast. This is likely to worsen the accuracy of your forecast, especially in highly uncertain situations where accurate forecasts are most valuable. The reason for this lies deep in our evolutionary past, when increased uncertainty raised our anxiety about being cast out from our group, and hence made us much more likely to conform to its prevailing opinion. This herding process can lead a large number of people to accept a forecast that was originally made on the basis of very little information, or using weak logic and assumptions.

Caveat #2: The Critical Role of Surprise

Surprise is fleeting feeling that has underappreciated importance for updating forecast probabilities. As Daniel Kahneman explained in his book, “Thinking Fast and Slow”, the experience of feeling surprised is transitory because human beings naturally try to minimize energy and time intensive cognitive effort.

Surprise is triggered by our perception of information that either is at an extreme end of the range of outcomes that our mental models of the world predict (such as very rapid or large change), or by an observation that is inconsistent with the models themselves (e.g. awakening to a green sky). To minimize cognitive effort and reduce uncertainty, we subconsciously try to adjust our mental models to enable surprising information to cohere with our existing beliefs. Most of the time, this process occurs automatically; it is only when coherence can’t quickly be reestablished that we become conscious of feeling surprised.

When we are trying to accurately forecast a future outcome, and particularly when the distribution of that outcome follows a power law, it is easy to see how our natural suppression of surprise can get us into trouble.

For this reason, when startled by surprise, you should immediately try to write down what triggered it, before the feeling disappears. This enables you to later more carefully consider the implications of that trigger.