A primer on U.S. stock price indices

A primer on U.S. stock price indices – includes related article on the relationship between stock indices and the market

Peter Fortune

The measurement of the “average” price of common stocks is a matter of widespread interest. Investors want to know how “the market” is doing, and to be able to compare their returns with a meaningful benchmark. Money managers often have their compensation tied to performance, typically measured by comparing their results to a benchmark portfolio, so they and their clients are interested in the benchmark portfolio’s returns. And policymakers want to judge the potential for sudden adjustments in stock prices when differences from “fundamental value” emerge.

The most widely quoted stock price index, the Dow Jones Industrial Average, has been supplemented by other popular indices that are constructed in a different way and pose fewer problems as a measure of stock prices. At present, a number of stock price indices are reported by the few companies that we will consider in this paper. Each of these indices is intended to be a benchmark portfolio for a different segment of the universe of common stocks. This paper discusses some of the issues in constructing and interpreting stock price indices. It focuses on the most widely used indices: the Dow Jones Industrial Average, the Standard & Poor’s 500, the Russell 2000, the NASDAQ Composite, and the Wilshire 5000.

The first section of this study addresses issues of construction and interpretation of stock price indices. The second section compares the movements of the five indices in the last two decades and investigates the relationship between the returns on the reported indices and the return on “the market.”

Our results suggest that the Dow Jones Industrial Average (Dow 30) has inherent problems in its construction. Even so, while it overstates the performance of its segment of the stock market during the 1990s, adding to the popular impression that the blue chips have been the star performers, it does a fairly good job of reflecting the performance of large companies. We also find that the five indices that we examine appear to fall into two groups. The Dow 30, S&P 500, and Wilshire 5000 have high correlations among themselves, and all have a beta coefficient less than 1.0 relative to the market, indicating that these indices represent relatively “conservative” investments with lower volatility. The NASDAQ Composite and the Russell 2000 have higher correlation with each other than with the other three indices and have beta coefficients greater than 1.0, indicating that they represent relatively aggressive investments with above-average volatility. Thus, the first and second groups of indices reflect different segments of the market, but indices within each group convey roughly the same information and are interchangeable.

I. The Construction of Stock Price Indices

This section discusses some important aspects of the construction of stock price indices and their implications for measuring the rate of return on stocks. We use the two most popular indices to illustrate our discussion: the Dow Jones Industrial Average (Dow 30) and the Standard & Poor’s 500 Composite Index (S&P 500). In a later section we discuss other widely used measures of stock prices.

Total Return versus Capital Appreciation

In the absence of income tax considerations, investors would be indifferent between receiving a dollar of cash dividends and a dollar of capital appreciation. They would base their choice among common stocks on their assessment of the portfolio’s total return, the simple sum of the cash dividend yield and the rate of capital appreciation. While this might apply to tax-exempt investment accounts like IRAs and 401(k)s, most investors will care whether their returns come from dividends or appreciation because of the U.S. income tax code. For most of its history, the tax code has applied a lower tax rate to capital gains than to dividend income. At present, investors pay the ordinary income tax rate on both dividends and gains until they are in a tax bracket greater than 20 percent. Investors in a higher tax bracket will prefer to receive their returns in capital gains, taxed at a maximum of 20 percent, rather than dividends, taxed as ordinary income.

Although investors are concerned about total return, each of the indices we examine is an index of prices, allowing us to determine the rate of capital appreciation but not the total return. The reason is probably that over short periods, cash dividends are a steady and predictable component of returns, while the rate of capital appreciation is highly variable. However, over long periods, cash dividends are a significant component of the value of stocks. This is clear in Figure 1, which shows the reported S&P 500 stock price index and the S&P 500 stock wealth index, each scaled so that January 1980 = 100. The stock price index shows changes arising from capital appreciation. The stock wealth index is calculated as the accumulated value of reinvested dividends and of capital appreciation, assuming that all cash dividends are reinvested in the stocks in that index. By the end of September 1998, a $100 investment in the S&P 500 portfolio made in January 1980 grew to $960 from capital appreciation alone. However, if reinvested dividends had been considered, the accumulated value would have been $1,470. Thus, while it is true that our attention is drawn to the rapidly moving hare of stock prices, the slow but steady tortoise of reinvested dividends is a very important component of our wealth.

The neglect of cash dividends would be unimportant in a comparison of indices if the dividend yield were the same for all indices. However, this is not the case. For example, according to the Frank Russell Company (1998), the median dividend yield for the Russell 2000 was zero, while the median dividend yield for the S&P 500 was 1.36 percent. Small company stocks tend to pay low or no dividends, while the stocks of large and mature companies tend to carry higher dividend yields. Thus, a comparison of stock price indices for small and large companies (for example, the Russell 2000 and the Dow 30) gives a more positive view of the relative performance of small companies than would a comparison of stock wealth indices. The focus of indices on prices rather than accumulated values can distort our view of relative performance!

Despite this problem, the remainder of this section will address stock indices as they are constructed, without consideration of dividends. Cash dividends will be drawn in only when they are pertinent to the specific issue being discussed.

Desirable Characteristics of Stock Price Indices

Stock price indices differ according to the number and characteristics of the stocks included in the index, as well the weights given to each stock. While a stock price index measures the level of stock prices, its practical application is to compare values at different points in time, that is, to measure the rate of appreciation (excluding cash dividends) on common stocks. This discussion of five desirable characteristics of stock price indices will focus on implications for measuring the rate of return on stocks.

First, any index is designed to tell investors in the same segment of the market how well their stock portfolio should be performing. That is, an index is a benchmark against which a “representative investor” can compare performance. If an index is to enjoy widespread use, both the stocks selected and the weights employed should be close to those of some significant portion of investors, and the rate of return should correspond closely to the rate of return experienced by the representative investor. Some indices are so representative that they have become the basis for “index funds,” mutual funds that structure their portfolios to mimic the performance of a specific index.

Second, an index is intended to measure the price changes arising from forces affecting “the market.” However, stock prices are also affected by events unique to the firm, such as supply disruptions, introduction of competitive products, or labor unrest. These events give rise to firm-specific risks which investors can avoid by proper diversification. A well-diversified index will reflect less idiosyncratic risk than does an index with few firms. Thus, other things equal, the larger the number of firms represented in the index, the better the index will represent its segment of the market.

Third, a stock price index should normally be shielded from the effects of internal decisions that affect the firm’s share price without altering investors’ risk and return. Stock splits, for example, reduce the price of the stock for reasons unrelated to market forces, but they do not alter the subsequent returns to investors. An index that automatically corrects for stock splits is, other things equal, desirable. On the other hand, a firm’s decision to merge or to acquire other firms will change the character of the firm’s business and call for active intervention by the index manager to determine if the firm should be dropped from an index and replaced by other firms.

Fourth, as stock prices change, each stock’s share of a representative investor’s portfolio will change: The weights for stock with appreciation greater than average will rise, the weights for below-average performers will fall. An index that automatically corrects for the repercussions of price changes upon the weights themselves, without requiring buying or selling of shares to restore weights, will, other things equal, be more useful. For example, an equally weighted index, like the Dow Jones Industrial Average, will become increasingly less useful as a benchmark, or increasingly more expensive to mimic, because an investor must sell his best-performing stocks and reinvest in the worst performers in order to maintain equal weights.

A final consideration is the trading frequency of stocks included in an index. A stock index that includes too many infrequently traded stocks will fail to accurately reflect the market, especially over short intervals when prices are changing rapidly. Even indexes that include actively traded stocks, such as the S&P 500, can reflect “stale” prices if markets are closed by circuit breakers, if trading in individual stocks is halted, or if volume is so high that quotes are delayed.

The Dow Jones Industrial Average

The Dow Jones indices, the oldest and most frequently cited indices, are constructed for the Transportation, Utilities, and Industrial sectors, as well as for a 65-stock Composite. In addition, Dow Jones & Co. produces 25 indices for the U.S. Real Estate Investment Trust industry and international stock indices for 13 global regions.

Initiated as a 12-stock index in 1896, then increased to 30 stocks in 1928, the companies in the Dow Jones Industrial Average (Dow 30) account for one-quarter of the market value of all stocks traded on the New York Stock Exchange. The Dow 30 is, therefore, an index of large, mature, “blue chip” companies with actively traded stocks.

The Dow Jones indices are all constructed as a simple sum of the prices of the included stocks, divided by an “index divisor.” Thus, the Dow 30 stock price index is an equally weighted index calculated as

[DJ30.sub.t] = [summation of] [P.sub.it]/[A.sub.t] where i = 1 to 30 (1)

where [A.sub.t] is the index divisor and [P.sub.it] is the ith stock’s price at time t. This can also be expressed as

[DJ30.sub.t] = [a.sub.t] [summation of] [P.sub.it]/30 where i = 1 to 30 where [a.sub.t] = 30/[A.sub.t] (1[prime])

so the Dow 30 is proportional to the average stock price. The multiplier, [a.sub.t], measures the response of the index to a $1 change in the average price of the 30 stocks. For example, on Friday, March 13, 1998, the Dow 30 closed at 8602.52 and the divisor was 0.25089315. The simple average of the Dow 30 stock prices was $71.94, so a $1 increase in any one stock’s price induced a 3.98 unit increase in the index, and a $1 increase in the average price of the 30 stocks raised the index by the multiplier, 119.57.

As noted above, the index divisor (or multiplier) is used to adjust the index for corporate actions that would affect the index level but would not reflect changes in market conditions. Dow Jones alters its indices to reflect two events: the splitting of shares, and the substitution of a new firm for an old firm in the index. Because a split reduces the stock price in inverse proportion to the increase in the number of shares, it does not affect the market value of the firm’s equity or the wealth of its shareholders. However, to prevent the split from affecting the index level and giving a misleading signal of a declining market, the index divisor is reduced so that its value on the trading day before the split, using the new prices, is the same as the actual index on that date.(1) Later we shall see that while this approach adjusts the level of the Dow 30 index for splits, it might harm the quality of the index as a measure of stock returns over time.

While stock splits are the most frequent reason for divisor adjustments, infrequently substitutions are made of new firms for old firms. These substitutions might arise from mergers or acquisitions that either terminate a Dow 30 company’s identity or change it dramatically. Occasionally, Dow Jones will decide to remove a firm’s stock from the index and replace it with another firm’s stock because the new firm is more representative of a sector. In either case, the new firm’s share price invariably differs from the old firm’s price and the divisor is adjusted to compensate. Since 1980 Dow Jones has made substitutions in the Dow 30 stocks on only three dates, though multiple substitutions were made on each date.(2) While substitutions are infrequent, when they occur they have affected an average of four stocks, or 13 percent of the stocks in the index.

The Dow Jones 30 index is a measure of the level of stock prices for an equally weighted portfolio of blue chips. However, as noted above, few investors are interested in the level of stock prices alone. Rather, they are interested in comparing levels at different points in time, or in computing the rate of return on stocks. If we use equation (1) to calculate the formula for the rate of return over a short period (say, a day or a week) the resulting rate of appreciation is (as an approximation):

[R.sub.t] = [summation of] [w.sub.i,t-1] ([Delta][P.sub.i,t]/[summation of] [P.sub.i,t-1] where i = 1 to 30) where i = 1 to 30 – [Delta][A.sub.t]/[A.sub.t-1], [w.sub.i,t-1] = [P.sub.i,t-1]/[summation of] [P.sub.i,t-1] where i = 1 to 30 (2)

where [Delta][P.sub.i,t]/[summation of] [P.sub.i,t-1] where i = 1 to 30 is the rate of change in the ith stock’s price, [w.sub.i,t-1] is the weight on the ith stock’s return, and [R.sub.t] is the rate of return on the index. This reveals that the rate of return on stocks, measured by the Dow 30, is a weighted average of the proportional change in stock prices less the proportional change in the index divisor. The weights, [w.sub.i,t-1], are calculated using the previous trading day’s prices of the individual stocks; hence, although the Dow 30 is an equally weighted index of prices, it is a price-weighted average of returns, with high-priced stocks getting proportionally larger weight. Following popular usage, we will refer to an equally weighted index as “price-weighted” whether it is an index of price levels or of appreciation rates.

The Standard & Poor’s Indices

Standard & Poor’s produces 10 U.S. stock price indices: the S&P 500, the S&P 100, four broad sector indices (Utilities, Transportation, Financials, and Industrials), a mid-cap index for middle-sized companies and a small-cap index for small companies, a REIT index, and a 1500-stock SuperComposite index. The S&P 500 and S&P 100 are frequently cited measures of “the market.”

The Standard & Poor’s indices are constructed by computing the market value of each firm’s shares (price times shares outstanding), adding these together, then dividing by a number (the index divisor) that makes the index level at the initial date equal to some arbitrary value. For example, the S&P 500 is set so that the average 1941-43 value is 10. Hence an S&P 500 index of 1000 means that the current market value of S&P 500 stocks is 100 times the value of the average 1941-43 value.

The S&P 500, consisting of 500 mid-cap and larger companies, is calculated as

[SP500.sub.t] = [summation of] [P.sub.it][N.sub.it]/[B.sub.t] where i = 1 to 500 (3)

where [N.sub.it] is the number of shares outstanding for the ith firm and [B.sub.t] is the S&P 500 index divisor. After some manipulations, the S&P 500 index can be rewritten as

[Mathematical Expression Omitted] (3[prime])

showing that it is proportional to a share-weighted average of common stock prices. The multiplier, [b.sub.t], varies directly with the aggregate shares outstanding for all firms and inversely with the index divisor, [B.sub.t], but [b.sub.t] should be approximately 1.0 because the divisor, [B.sub.t], is calculated to adjust for changes in the number of shares.

While stock splits are the most frequent corporate action requiring index divisor adjustment for the Dow Jones 30, splits require no divisor adjustment for a share-weighted index like the S&P 500. Only the market value of a firm’s stock enters the index, and the market value is unaffected by a stock split. In short, a share-weighted index automatically corrects for stock splits. The factors that lead to divisor adjustments in the Standard & Poor’s indices are those events, other than stock splits, that affect the number of shares. Standard & Poor’s reports that they adjust the divisor when stock issuance or stock repurchase changes the number of shares by 5 percent or more, or when there is a rights issue.(3)

Converting a share-weighted index like the S&P 500 to its rate-of-appreciation form leads to the approximation:

[R.sub.t] = [summation of] [w.sub.i,t-1] ([Delta][P.sub.i,t]/[summation of] [P.sub.i,t-1] where i = 1 to 500) where i = 1 to 500

+ [[summation of] [w.sub.i,t-1] ([Delta][N.sub.i,t]/[summation of] [N.sub.i,t-1] where i = 1 to 500) where i = 1 to 500, – [Delta][B.sub.t]/[B.sub.t-1], (4)

where the weights, [w.sub.i,t-1] = [P.sub.i,t-1][N.sub.it-1]/[summation of] [P.sub.i,t-1][N.sub.it-1] where i = 1 to 500, are the ith company’s share of aggregate market value on the previous day. The second term, in brackets, should be zero if the change in the divisor accurately captures the effect of changes in outstanding shares. Thus, the rate of appreciation on a share-weighted average like the S&P 500 is a value-weighted average of proportional changes in stock prices. Following popular usage, we will refer to a share-weighted index as “value-weighted,” whether it is an index of price levels or of appreciation rates.

Price-Weighted versus Value-Weighted Average Returns

A value-weighted average return has all of the desirable features listed above. Its weights approximate the weights of the “representative investor” in that collection of stocks because, by definition, the average investor holds stocks in proportion to their share in the aggregate market value of all stocks in the index. If the market value of XYZ’s stock is 5 percent of the market value of all stocks, it must be true that the average investor in those stocks is holding 5 percent of her stock portfolio in XYZ: XYZ should receive a 5 percent weight in calculations of the return on the portfolio represented by the index.

A value-weighted average return automatically adjusts for the effect of stock splits, the most frequent form of corporate policy decision affecting stock prices. A (say) 3-for-1 split reduces the price of a share to 1/3 of its previous level, leaving the market value of the firm’s shares, and the firm’s weight in the average return, unchanged. Also, no transactions are required to maintain market value weights when individual stock prices change: If XYZ is initially 5 percent of the market value of a group of stocks and its stock price increases so that its market share is, say, 7 percent, the average investor will have achieved the new value-weight of 7 percent through price changes alone. Finally, an index like the S&P 500 is dominated by actively traded companies, so stale prices are not a particularly important problem.

A price-weighted average return, like the return on the Dow 30, has few of the desirable attributes listed above. A stock’s weight in the average return is its relative price, [P.sub.i,t-1]/[Sigma] [P.sub.i,t-1], so the representative investor for a price-weighted index is assumed to invest in direct proportion to the stock’s price: If one stock is priced at three times the other, the Dow 30’s representative investor should invest three times as much in the first stock as in the second. Doing so would mean putting most of the eggs into a high-priced basket, a strategy that has no logical foundation.

Second, in spite of adjustments to the level of a price-weighted index via the divisor, events like stock splits will affect the weights assigned to each stock: Returns on a stock that splits will be given a lower weight, while non-splitting stocks get a higher weight. This can lead to a systematic bias in a price-weighted average return if the future returns are correlated with a stock’s price. If, as many investors believe, stocks that split typically enjoy unusually high post-split returns, a price-weighted index, by giving lower weight to the splitting stocks, will discount the best performers and bias the return downward. If, on the other hand, splitting stocks have lower future returns, a price-weighted average return will give greater weight to the best performers.

The Dow 30 does have two desirable characteristics. The stocks of very large companies are actively traded, so the Dow 30 is rarely stale unless trading is halted in one of the stocks. Furthermore, price weights do automatically adjust for changes in the value of the Dow 30 stocks, so a mutual fund mimicking the Dow 30 will not be forced into transactions simply to maintain the weighting scheme.

The Importance of Price versus Value Weights

One way to assess the importance of price weights versus value weights is to compare price-weighted and value-weighted indices constructed from the same common stocks. We have done this for both the Dow 30 and the Wilshire 5000. To construct a value-weighted Dow 30 we obtained from Dow Jones the daily aggregate market value of the 30 stocks in the Dow 30, going back to January 1988. This allows us to construct a share-weighted price index, called [DJ30.sup.*] and defined as [Mathematical Expression Omitted]; following our language convention, we call this a value-weighted index. The divisor of this index, [C.sub.t], is set so the value of the index was 100 on January 4, 1988. The divisor was then changed on the three dates in the period when new firms displaced old firms in the Dow 30 (see footnote 2). It was not changed when stock splits occurred because a value-weighted average return is not affected by splits.

The price-weighted and value-weighted Wilshire 5000 indices were a bit easier. The actual Wilshire 5000 is the value-weighted index, while the price-weighted index is reported at the Wilshire Associates website (www.wilshire.com), available as far back as 1971.

Figure 2 shows the month-end levels of the ratio of price-weighted to value-weighted indices for both the Dow 30 and the Wilshire 5000. These were constructed so that both ratios were 1 in January 1988, the starting point for the Dow Jones data. Two important points emerge from Figure 2. First, the price-weighted index generally increases at a faster rate than the value-weighted index. The differential increase is very small for the Dow 30 but is considerably greater for the Wilshire 5000. Second, the ratio of the price-weighted and value-weighted indices varies considerably over time, again especially true for the Wilshire 5000. Indeed, the Wilshire 5000 ratio grew from 1 in January 1988 to almost 1.7 in early 1996, then fell back to about 1.3 in September 1998. The Dow 30 ratio rose slightly, from 1 to 1.1, and showed much less variability.

Two major factors underlie the movements shown in Figure 2. The first is that small companies, which have low value weights in any index of returns, tend to grow more rapidly than larger companies. Thus, they start with a small weight in the index and their weights rise over time. As a result, they will be given greater weight in a price-weighted index, but this effect will tend to diminish as the company grows. This probably explains why the equally weighted Wilshire 5000 index increases much more rapidly, and displays more variability, than the equally weighted Dow 30. The Wilshire 5000 includes many small companies, whose performance would be exaggerated in a price-weighted index, while the Dow 30 consists of large companies for which equal weights and share weights are not as different.

The second factor is that, as noted above, the correlation between a firm’s absolute stock price and its subsequent rates of appreciation is important in a price-weighted index. One of the common sources of declines in a company’s stock price is a stock split, which is usually intended to get the stock’s price back down to a normal target level after a sustained period of price increase. If this correlation is positive, stocks that split get lower weight in a price-weighted index of returns and that index is biased downward relative to a value-weighted index. A seminal study of stock splits (Fama et al. 1969) confirmed that stocks with sharply rising prices tended to be the ones that split, but it found that after a split the returns on a stock remain at the normal rate of return for the stock’s risk level. Thus, no correlation exists between a stock’s price and its future appreciation rate, and this source of bias should be small.

Price-weighted indices appear to show greater returns than do value-weighted indices, primarily because the price-weighted index gives higher weight to small companies that tend to have higher rates of appreciation. Since the mid 1990s, however, small company stocks have not performed as well as large company stocks, contributing to the decline in the value of the price-weighted Wilshire 5000 relative to its value-weighted counterpart.

Other Popular Price Indices(4)

In previous sections we discussed the Dow 30 and the S&P 500. Here we will cover the chief characteristics of other popular stock price indices: the NASDAQ Composite, the Russell 2000, and the Wilshire 5000. Table 1 shows descriptive statistics for each index. The Dow 30 and the S&P 500 have the largest firms among the five indices. The mean market value is $65 billion for the Dow and $15 billion for the S&P 500, far higher than any other index. These two indices are heavily weighted by industrial stocks.

The National Association of Securities Dealers publishes 13 indices: the NASDAQ Composite, including almost 6,000 of the nearly 6,400 stocks traded on NASDAQ; the NASDAQ National Market Composite, with those stocks listed in the National Market Tier of the NASDAQ; the NASDAQ National Market Industrial Index, with stocks in the industrial sector index that are also in the National Market tier; the NASDAQ-100, with the 100 largest nonfinancial stocks; and the NASDAQ Financial-100, with the 100 largest financial stocks traded on NASDAQ. All NASDAQ stocks are also allocated to one of eight sector indices (Banks, Biotechnology, Computers, Industrials, Insurance, Other Finance, Telecommunications, and Transportation). All NASDAQ indices are value-weighted.

The NASDAQ Composite has long been used to measure prices in the “over-the-counter” (OTC) market, the market for stocks not traded on registered exchanges. The NASDAQ Composite’s early history was as a measure of prices of “small-cap” stocks, but this image has been weakened by the rise of some very large firms, like Microsoft and Intel, that are traded on the NASDAQ. The Composite has recently become a popular measure of the high-tech segment of the market.

In 1984, the Frank Russell Company created the Russell 1000, the Russell 2000, and the Russell 3000. These value-weighted indices include only common stocks of corporations domiciled in the United States or one of its territories. The Russell 1000 is the largest third (by market capitalization) of the Russell universe, and the Russell 2000 contains the smallest two-thirds of the universe. The Russell 2000 is the most widely used of the three indices, having become a popular measure of the “small-cap” segment of common stocks. In 1998 the median market value of the Russell 2000 companies was $500 million, while the average was $592 million and the smallest was $221 million. About two-thirds of the Russell 2000 are industrial stocks and almost one-quarter are financial companies. Russell also produces international stock price indices for Japan, Australia, and Canada. The Russell indices are all value-weighted.

Wilshire Associates prepares the Wilshire 5000, [TABULAR DATA FOR TABLE 1 OMITTED] along with large-cap, mid-cap, and small-cap indices, each having style variations.(5) Wilshire also has three REIT indices. The Wilshire 5000, created in 1974, was initially reported only monthly, but daily calculations began in December 1979. According to Wilshire Associates, in March of 1998 the index included 7,412 companies, the largest with a market value of $282 billion and the smallest with a market value less than $1 million. The mean size of $1.6 billion greatly exceeded the median size of $136 million, indicating a very skewed distribution with a few large firms. The Wilshire 5000 covers the widest range of firm sizes of any index we examine. Almost 65 percent of the companies were traded on NASDAQ and 27 percent were traded on the NYSE. The business sector breakdown is similar to the Russell 2000, with slightly less weight in the industrial and financial sectors. The Wilshire indices are all value-weighted.

II. Comparison of Stock Price Indices

Any index is useful only if it adds to information available on stock prices. If each index followed precisely the same path, we could use any index to represent all indices. However, indices do display divergent movements, suggesting that each carries some independent information and that different indices might represent different segments of the U.S. stock market.

In this section we compare the five stock price indices discussed above to determine which appear to provide independent information about segments of the market. The simplest way to do this is a visual comparison of movements over time. Figure 3 shows the level of each price index at the end of each month, through September 1998, assuming a value of 100 on January 31, 1980. The use of a ratio scale means that the slope of the line represents the rate of growth in the index. Throughout the entire period, the slope of a line connecting the endpoint to the starting point is higher for the Dow 30 index than for any other index. This means that its average rate of return over the two decades was higher than for any other index.

Throughout the early 1980s, the Russell 2000 and the NASDAQ Composite indices moved closely together, as would be expected since both were used as measures of the price of small-capitalization stocks. The other three indices also appeared to move together, suggesting that the Dow 30, the S&P 500, and the Wilshire 5000 measured prices of essentially the same stocks – those of large-capitalization companies.

After September 1987, this dichotomy appeared to break down. In early 1987 the five indices had roughly the same value, and in the years 1987 to 1989 they all marched in a near lockstep. During the 1990-91 recession, over which stock prices generally declined, large-cap stocks appeared to perform better than small-caps, probably a reflection of the lower betas attached to large company stocks. In 1990 the performance ranking, from highest to lowest, was that the Dow 30, S&P 500, and Wilshire 5000 performed best, while the NASDAQ and Russell 2000 performed least well.

The 1990s have been a different story. The strongest market indices have been the NASDAQ and Dow 30, with the S&P 500 and Wilshire 5000 in an intermediate position and the Russell 2000 taking last place. Thus, the 1990s have been dominated by the very large-cap blue chip stocks and by technology stocks. The long-observed small firm effect has been noticeably absent.

Table 2 shows the correlations between monthly returns on each index for the entire period and for two subperiods: 1980-1989 and 1990-1998. For both the entire period and the two subperiods, the NASDAQ’s correlation with the Russell 2000 is particularly high, while the S&P 500 is most highly correlated with the Dow 30 and the Wilshire 5000. This suggests that the NASDAQ and Russell 2000 indices carry similar information, and the Dow 30, S&P 500, and Wilshire 5000 also carry similar information. In short, it is not unreasonable to use either the NASDAQ or the Russell 2000 as a measure of returns on one market segment, while the Dow 30, S&P 500, and Wilshire 5000 represent the returns on another segment. But what distinguishes these segments?

The Relationship between Stock Index Returns and Returns on “the Market”

If information on the market portfolio were available, we could avoid the multiplicity of indices and implement many of the propositions of financial theory with impunity. We could also better understand why our five indices seem to represent two different market segments.

In an effort to develop a measure of the market portfolio, and to better understand the link between each index and “the market,” we use the Capital Asset Pricing Model of an efficient market. We treat each index as providing some information about the market, then employ a method of estimating relationships between observable and unobservable variables to infer the relationship of each index to “the market.” This method allows us to derive a measure of the return on the market portfolio as well as to examine the links between observed stock price indices and “the market.”(6)

Table 2

Correlations among Returns on U.S. Stock Price Indices

Monthly, January 1980 to September 1998

NASDAQ Russell S&P Dow Wilshire

Composite 2000 500 30 5000


1980-1998 1.00 .94 .86 .82 .89

1980-1989 1.00 .98 .89 .86 .91

1990-1998 1.00 .90 .82 .76 .88

Russell 2000

1980-1998 1.00 .86 .82 .89

1980-1989 1.00 .89 .86 .90

1990-1998 1.00 .79 .76 .88

S&P 500

1980-1998 1.00 .96 .98

1980-1989 1.00 .97 .97

1990-1998 1.00 .95 .99

Dow 30

1980-1998 1.00 .92

1980-1989 1.00 .94

1990-1998 1.00 .90

Wilshire 5000

1980-1998 1.00

1980-1989 1.00

1990-1998 1.00

Note: Data are returns, exclusive of cash dividends, from

end-of-month to end-of-month. This is computed as the average

logarithmic daily return in a given month times 365.

According to the CAPM, the excess return on each security or portfolio is a linear function of the excess return on the market, that is,

([R.sub.i,t] – [r.sub.t]) = [[Alpha].sub.i] + [[Beta].sub.i]([R.sub.M,t] – [r.sub.t]) + [[Epsilon].sub.it] i = 1, . . . , k (5)

where [R.sub.i,t] is the return in period t on the ith index, [r.sub.t] is the risk-free rate of return in the tth period, [R.sub.M,t] is the return in period t on “the market,” and [[Epsilon].sub.it] is the unsystematic risk of the index. This is the risk that can, in principle, be completely diversified away by combining the index portfolio with all other securities in the market portfolio. Each return is defined as the total return, including all cash dividend payments.

An index portfolio’s beta coefficient is the slope coefficient for equation (5). Note that the average beta for all securities in the market portfolio is 1.0, where “the market portfolio” is interpreted as the efficiently diversified portfolio of all possible securities, having no unsystematic risk. A beta coefficient can be interpreted in several ways. It measures the systematic risk of the return on the portfolio represented by that index, that is, the variability of the index’s return attributable to movements in the market rather than to non-market factors specific to the firm (which are the source of unsystematic risk). It is the unavoidable risk associated with holding that index. By this we mean that any risks not associated with movements in the market (unsystematic risks) can be diversified away by an appropriate mixture of the index with other securities. There is a reward, or risk premium, for bearing the systematic risk because it is unavoidable. An investor who holds a portfolio with unsystematic risk will earn no extra return because that risk is avoidable. The CAPM tells us that the risk premium on a stock or on a portfolio will be [Lambda][Beta], where [Lambda] is the market price of risk (always positive) and [Beta] is the beta coefficient for the index.

An index’s beta can also be interpreted as the amount by which including the index portfolio in “the market” portfolio will increase the risk of the market portfolio. For example, if an index’s beta is 0.75, the addition of the last bit of an index’s portfolio to the market portfolio increases the market portfolio’s risk by 0.75 units. A positive beta means that the securities in the index add to “the market’s” risk, and a negative beta means that the index’s securities reduce the market’s overall risk.

Stocks can be classified according to their beta values. A stock (or an index) with a negative beta is a “defensive” investment because buying some of it will actually reduce the investor’s risk. These rare investments will have a negative risk premium (earning less than the risk-free interest rate) because they actually reduce overall risk. Stocks (or indices) with a beta that is positive but less than 1.0 can be classified as “conservative” investments because they increase the market’s risk by less than the average stock. Stocks (or indices) with a beta that is greater than 1.0 can be classified as “aggressive” investments because they increase the market’s risk by more than the average stock.

The alpha coefficient for each index measures the risk-adjusted rate of return on the index, that is, the return on the security above or below the level attributable to the security’s systematic risk. In an efficient market, the alpha coefficient would be zero because each portfolio should earn a return that differs from the riskless rate only because of risk.

The conventional application of this model uses statistical methods to derive estimates of the alpha and beta coefficients for individual securities or for portfolios of securities. This is done by linear regression of the return on an individual security or portfolio upon the return on a stock price index, typically the S&P 500, which is used as a proxy for “the market.” Our goal is to estimate the alpha and beta coefficients of returns on stock price indices relative to the unknown and unobservable “market” return. We do this by assuming that the market return is a random walk, that is, the best estimate of today’s return is the return yesterday. This very common assumption about returns on securities or on portfolios in an efficient market means that

[Mathematical Expression Omitted] (6)

describes the path of the market return. It is a random walk with constant variance [[Sigma].sup.2].

The details of our method are summarized in the Box. We cast the model into a state-space form, treating the market return as an unobserved state variable. Using a Kalman filter, the path of the market return associated with any set of parameter values is generated. The method of maximum likelihood is used to pick the parameter values that fit the data best. The final solution of the state-space model generates the path of the market return that gives the best fit in tying the five indices to the market return through the five alpha and five beta coefficients. The results are estimates of the five alpha coefficients and of the five beta coefficients, along with their t-statistics, as well as an estimate of the path of the market return (or state variable) implied by the data on stock price indices. From this path we can easily calculate the variance of the shock to the market return, [[Sigma].sup.2], and we can reconstruct the market portfolio index by applying the path of the market return to some initial value.

The parameter estimates are reported in Table 3. Each of the five alpha coefficients is small, and none are significantly different from zero. This is consistent with the efficient market theory’s prediction that no index portfolio earns an unusual return on a risk-adjusted basis, and that each index portfolio receives returns determined solely by the riskiness of that portfolio.

The five beta coefficients are all positive, meaning that an investor in each index can expect a return above the riskless rate because each index is positively correlated with the market return and, therefore, adds to the market portfolio’s risk. The betas are all close to 1.0, indicating that the systematic risk of each index is close to the risk of the market portfolio. However, the beta coefficients fall into two groups. The betas for the NASDAQ Composite and the Russell 2000 are 15 to 20 percent greater than 1.0, while the betas for the Dow 30, the S&P 500, and the Wilshire 5000 are all 5 to 10 percent below 1.0. The differences between the index betas and the market beta of 1.0 are all statistically significant.

Table 3

Relationship between Returns on Reported Indices and “The Market”

February 1980 to September 1998

Stock Price Index

NASDAQ Russell S&P Dow Wilshire

Parameter Composite 2000 500 30 5000

[Alpha] -.0126 -.0115 .0056 .0063 .0107

(-1.14) (-1.13) (.35) (.40) (1.05)

[Beta] 1.2074 1.1649 .9112 .8958 .9438

(74.60) (63.77) (47.10) (52.87) (53.60)

Note: The alpha and beta coefficients are estimates of the mean

risk-adjusted return and of the systematic risk for each index

relative to the return on “the market.” They are analogous to the

intercept and slope coefficients in a regression of each index

return on the market return. The market index is not observable, but

inferences about it are derived from estimation of a state-space

model (see the Box). The sample data are monthly log changes

expressed as percentages at annual rates. T-statistics are in


Thus, the NASDAQ Composite and Russell 2000 indices measure returns on stocks that, as a group, represent aggressive investments. The other three indices contain stocks that are, as a group, conservative investments with below-market risk.

Our results also confirm the correlation tests in Table 2. The Dow 30, S&P 500, and Wilshire 5000 behave in a similar fashion and appear to carry roughly the same information about stock market performance. The NASDAQ and Russell 2000 also behave in a similar fashion, and appear to carry similar information. The latter group is a measure of more volatile segments of “the market” than is the first group. Thus, the market segments represented by the two groups appear to be the conservative and aggressive investments.

Characteristics of the Market Return and the Market Index

Our primary interest is in estimating the beta coefficients for each of the five indices to see how these indices relate to the general market. However, our method also traces out the path of the “market return” that is implied by our five measured indices. This path can be used to construct a path for the “market index” by using the formula [M.sub.t] = [M.sub.t-1](1 + [R.sub.M,t]), M denoting the market index; we assume [M.sub.0] = 100.

The Dow 30, the S&P 500, and the Wilshire 5000 follow close to the market index, and the NASDAQ Composite and the Russell 2000 are the best and worst performing indices over the entire period. Figure 4 shows the market index implied by our state-space model, along with the NASDAQ Composite and the Russell 2000. During the recent correction, from the end of July through the end of September, the market index has declined by 10.4 percent, more than the S&P 500’s 9.3 percent decline but considerably less than the Russell 2000’s 13.4 percent contraction.

Our state-space model was estimated with the assumption that the volatility of the market return, [Sigma], is constant. In Figure 5 we show the absolute value of the change in the market return. This is the value of [Sigma] plus a random shock reflecting the noise in the market return; we call it the “sample volatility” of the market return. A six-month centered moving average is also shown in Figure 5.

Figure 5 shows significant spikes in the sample volatility, but the average value displays no trend, suggesting that our assumption of a constant volatility is not far off the mark. However, some short-term movements in sample volatility are worth noting. The spike in 1987 is the October Crash. During the first half of the 1990s, the sample volatility is a bit less than in the 1980s; this was a period of widespread remarks about the historically low volatility of returns on common stocks. Since 1996 there has been a rebirth of volatility, and the 1998 spike in sample volatility is on the same scale as the October 1987 spike. We now hear about the historically high volatility of stock returns.

III. Summary

This study addresses issues of construction and interpretation of U.S. stock price indices. An equally weighted index, like the Dow Jones Industrial Average, is compared with share-weighted averages like the S&P 500, the Wilshire 5000, the NASDAQ Composite, and the Russell 2000. We show that an equally weighted index of price levels is equivalent to a price-weighted average of rates of price appreciation, while a share-weighted average of price levels is equivalent to a value-weighted average of rates of appreciation. We argue that the Dow Jones Industrial Average is, in principle, a poor measure of stock price behavior because it is a price-weighted index of returns, giving more weight to high-priced stocks than to low-priced stocks in computations of the average rate of return on stocks.

Whether this is of practical importance depends on the correlation between stock prices and subsequent rates of return. If, as many investors believe, stocks that split will have higher than normal future returns, the Dow Jones Industrial Average should be biased downward, creating an impression of slower price increases than actually occur if the index is value-weighted. We show that the Dow Industrials have actually overstated the returns to the Dow 30 list of common stocks since the early 1990s, reflecting a slight positive correlation between stock prices and subsequent rates of return. Even so, the Dow Jones Industrial Average has been a decent measure of stock returns in the last two decades, in that it has performed over time in a fashion close to that of broader value-weighted indices like the S&P 500 and the Wilshire 5000. The reason is that there has been little correlation over that longer period between the prices of individual stocks and their future rates of return.

The study also compares the movements in the five popular indices in the last two decades. We examine the correlation between returns on each of the stock price indices. This shows that returns on the Dow 30, S&P 500, and Wilshire 5000 indices move closely together. These returns are less closely correlated with returns on the NASDAQ Composite and on the Russell 2000, which are highly correlated. This suggests that the Dow 30, S&P 500, and Wilshire 5000 are similar and capture the movements in a different segment of the market than do the NASDAQ Composite and Russell 2000.

In order to understand what these two market segments represent, we use an econometric method for estimating the movements of unobservable variables to estimate the path for the “market return” over the sample period. This method is built on the Capital Asset Pricing model, and it generates estimates of the alpha and beta coefficient linking each index to the market return. We find that no alpha coefficients are significantly different from zero, as the efficient market theory predicts. We also find that the beta coefficients linking returns on the Dow 30, S&P 500, and Wilshire 5000 to the market return are each less than one, while the beta coefficients for the NASDAQ Composite and the Russell 2000 are both greater than one. This confirms the earlier result that our five indices fall into two groups, and it tells us that the differences in the two groups are primarily in the risk levels of the securities in the indices. The Dow 30, S&P 500, and Wilshire 5000 stocks tend to be more conservative, with systematic risk below the market level, while the NASDAQ Composite and Russell 2000 stocks tend to be more aggressive, with above-market systematic risk.

Estimating the Relationship between Stock Indices and “The Market”: A State-Space Model

In the text we derive the following model relating excess returns on stock indices to the excess return on the market.

[Mathematical Expression Omitted] (1)

where, assuming k stock price indices, [y.sub.t] is the k x 1 vector of excess returns, [Alpha] and [Beta] are k x 1 vectors of parameters to be estimated, and [Sigma] is the volatility of the market return, a scalar, also to be estimated. Note that while the market return is unobservable, the stochastic properties of its movements over time are known: By assumption, the mean is zero, and the market return is a random walk with changes having a zero mean and a constant variance, [[Sigma].sup.2].

This model can be put into state-space form as follows. First, form a matrix Z by horizontally appending a k x k identity matrix to the k x 1 vector of beta coefficients, that is, Z = [Beta] [similar to] I. Then form a (k + 1) x 1 column vector, called [[Theta].sub.t], of “state variables” by vertically appending the k x 1 alpha coefficient vector to the market return at time t, [R.sub.Mt], thus creating the vector [[Theta].sub.t] = [R.sub.Mt] [where] [Alpha]. Finally, form a vector [v.sub.t] by vertically appending the random innovation in [R.sub.Mt] to the innovation in the alpha vector. Because the alphas are constants and have zero innovations, this gives [v.sub.t] = [Eta][where]0.

To be specific, if there are three indices (K = 3), the results would be

[Mathematical Expression Omitted]

The state-space form of (1) is

[y.sub.t] = -[r.sub.t][Beta] + Z[[Theta].sub.t] + [Epsilon] E([[Epsilon].sub.t]) = 0 E([[Epsilon].sub.t][[Epsilon][prime].sub.t]) = [[Sigma].sup.2]H

[[Theta].sub.t] = [[Theta].sub.t-1] + [v.sub.t] E([v.sub.t]) = 0 E([v.sub.t][v[prime].sub.t]) = [[Sigma].sup.2]V

[v.sub.11] = 1 [v.sub.ij] = 0 for all i [not equal to] j

The first system, consisting of k equations, is called the “measurement model” and the second system, consisting of k + 1 equations, is the “transition model.” The matrices H and V are the covariance matrices for the measurement equations and the transition equations; each is scaled by the market return variance. We assume that the equation error for each index has a variance equal to the market return variance, so H is an Identity matrix. We also assume that V has all zeros except that the (1,1) element is one, that is, the only randomness in the state vector is attributable to the market return.

Estimation of the [Alpha] and [Beta] vectors and of the entire path of the market return, [R.sub.Mt], can be done by applying a Kalman Filter to the state-space model and using the method of Maximum Likelihood. Once the likelihood-maximizing path of the market return is found, the variance [Mathematical Expression Omitted] can be calculated as [Mathematical Expression Omitted]. Also, armed with a value of the market return for each month, the market index level can be calculated for each period as [M.sub.t] = [M.sub.t-1] (1 + [R.sub.M,t]); this requires a starting value, which we choose to be [M.sub.0] = 100. Further description is beyond the scope of this paper, and the interested reader can refer to any upper-level econometrics textbook, such as Hamilton (1994).

1 For example, on November 19, 1997, Travelers Group paid a 3-for-2 stock split. On November 20, the ex-distribution date, the price of Travelers Group shares dropped. Dow Jones reduced the index divisor from 0.2545 to 0.2513, the amount required to make the index level on the previous trading day (November 19), using post-split prices, the same as it was on that date using pre-split prices.

2 On October 30, 1985, American Express, Chevron, Inco, and McDonald’s replaced American Tobacco, General Foods, International Nickel, and Johns-Manville. On May 6, 1991, Caterpillar, J.P. Morgan, and Walt Disney replaced Navistar, Primerica, and USX. On March 17, 1997 Travelers Group, Hewlett-Packard, Johnson & Johnson, and Wal-Mart Stores replaced Westinghouse Electric, Texaco, Bethlehem Steel, and Woolworth. On October 8, 1998 Citigroup replaced Travelers Group and Citibank as a result of the merger of those two companies; this did not change the divisor.

3 In a rights offering, existing shareholders are given the right to buy newly issued shares. These rights have value, and when a stock goes ex-rights the rights attached to it expire and the price drops to reflect the loss of the value of the rights.

4 We do not discuss the NYSE Composite index. This share-weighted index, which includes only stocks listed on the New York Stock Exchange, was started in 1966. It is not widely used, though it is often reported.

5 Wilshire Associates separates each of the size-related indices (large cap, mid cap, and small cap) into a growth style index and a value style index. The “growth style” contains high book-to-market-value stocks, and the “value style” contains low book-to-market-value stocks.

6 As Richard Roll (1977) reminds us, the absence of a direct measure of “the market” impedes our ability to apply financial theory. Roll argues that the CAPM is inherently untestable because it applies to the relationship between individual securities and “the market,” but we have no measure of “the market” portfolio. The use of any price index – or of a multiplicity of indices – to construct a proxy for returns on “the market” is necessarily misleading because the underlying theory (CAPM) cannot be validated.


Fama, Eugene F., Lawrence Fisher, Michael C. Jensen, and Richard Roll. 1969. “The Adjustment of Stock Prices to New Information.” International Economic Review, vol. 10, no. 1 (February), pp. 1-21.

Frank Russell Company. 1998. Russell US Equity Indexes. Second Quarter. Tacoma, WA.

Hamilton, James D. 1994. Time Series Analysis. Princeton, NJ: Princeton University Press.

Roll, Richard. 1977. “A Critique of the Asset Pricing Theory’s Tests.” Journal of Financial Economics, vol. 4, pp. 129-76.

Peter Fortune is a Senior Economist and Advisor to the Director of Research at the Boston Fed. He is grateful to Richard Kopcke and Lynn Browne for constructive advice.

COPYRIGHT 1998 Federal Reserve Bank of Boston

COPYRIGHT 2004 Gale Group