Analyst Optimism Explains the Excess Volatility Puzzle

An important financial paradox is the excess volatility puzzle.

First identified by Robert Shiller in 1981, the excess volatility puzzle says that asset prices fluctuate much more than information about their fundamental value would suggest. The traditional finance explanation is that the volatility is caused by rational time-varying risk premiums.

Nonrational Expectations

Pedro Bordalo, Nicola Gennaioli, Rafael La Porta, Matthew O’Brien, and Andrei Shleifer, authors of the August 2023 NBER study “Long Term Expectations and Aggregate Fluctuations,” took a different approach, one that allowed for expectations to be nonrational.

“Key to our strategy is the use of data on stock analysts’ consensus expectations of the earnings growth of S&P 500 firms,” they wrote. “One measure turns out to be critical: the analysts’ forecast of a firm’s long-term earnings growth, which captures expectations of fundamentals over a three- to five-year horizon. Our main variable is the consensus LTG forecast, aggregated across firms in the S&P 500 Index.”

Their data sample covered the period from 1980 to 2022. The following is a summary of their key findings:

• Volatility in the stock market and real economic activity are linked by expectations of long-term profits.
• Analysts’ optimism about the long-term growth of S&P 500 firms is associated with a near-term boom in major US financial markets, real investment, and other business cycle indicators.
• When long-term earnings growth is high relative to historical standards, analyst forecasts of short- and long-term profits are systematically disappointed in the future, inconsistent with rationality.
• High long-term growth also correlates with higher survey expectations of stock returns, in contrast with standard theories, in which investors expect low returns in good times. High long-term growth, thus, proxies for excess optimism; it points to investors being too bullish about future profits and stock returns.
• The same optimism predicts disappointing earnings growth and a contraction in financial markets and real activity one to two years later.
• The explanatory power of long-term growth reaches beyond the stock market: Higher growth predicts near-term increases and long-term declines in short- and long-term interest rates, and the reverse pattern for credit spreads.
• Long-term growth connects to real activity. One standard deviation increase in growth fuels an investment boom; growth in the investment/capital ratio is 3% higher than conventional levels in the following year, corresponding to a 0.4 standard deviation increase. The investment boom sharply reverts two years later, and that reversal is fully explained by the predictable disappointment of the initially high long-term earnings growth.
• In the short term, higher long-term earnings growth acts like a positive shock: It predicts growth in consumption, employment, and wages. However, it also predicts a longer-term reversal in these variables.

High Expectations and Volatility

Their findings led the authors to conclude: “In the long run, a current increase in long-term growth is associated with reversals whose magnitude is comparable to that of the initial boom. These dynamics mimic those of real investment and financial markets, confirming that expectations of long-term growth can reconcile financial and real volatility.”

They explained: “The present value of short- and long-term expected earnings for S&P 500 firms, computed using a constant required return, fully explains observed stock market fluctuations.”

They added: “A directly measured and clearly interpretable variable, changes in the long-term profit expectations of individual firms, predicts aggregate boom-bust comovement among macro variables as well as with financial variables.”

They further explained: “High long-term growth captures periods of excess aggregate optimism systematically predicts positive forecast errors and, thus, future disappointment of earnings growth expectations. Disappointment persists at least four quarters out, suggesting that long-term growth is a source of persistent excessive optimism, which eventually reverts. In contrast, expectations about short-term growth do not predict forecast errors. This finding strengthens the interpretation of excess stock price volatility as being due to the excess volatility of long-term beliefs. It also suggests that excess volatility of beliefs may drive volatility in real investment, because high long-term growth captures persistent optimism about the full-term structure of expectations, proxying for times in which the perceived returns to investment are high.”

Investor Takeaways

Using analyst expectations of long-term earnings growth for individual US-listed firms, Bordalo et al. demonstrated that the connection between financial markets and the macroeconomy is attributable to the influence of nonrational expectations on both. Long-term expectations exhibit excess volatility, which in turn correlates with movements of stock prices and returns, interest rates, and credit spreads, as well as with the cyclical behavior of investment and other real quantities.

Thus, they showed: “Belief overreaction arises as an important ingredient that appears both qualitatively and quantitatively important to understand volatility, particularly predictable long-term reversals.”

This is of particular importance given that US equity valuations are currently at historically high levels: As of May 22, 2025, the CAPE 10 stood at 35.8. Such high levels of optimism have tended to lead to lower future returns. Forewarned is forearmed.