The raw measures analyzed thus far were "bottom-line" -- based on risks and returns calculated using either fund total returns or excess returns. While excess returns are calculated by taking the difference between two returns, the return that is subtracted from a fund's return is that of Treasury bills, not the return on a benchmark considered to be similar to the fund itself. While it is true that the relative measures computed by normalizing raw measures and/or ranking results for funds within an asset class or category can be considered "benchmarked", at a theoretical level this is subject to criticism.
Take, for example, the category rating. It incorporates a risk measure that is appropriate if the fund in question constitutes the entirety of the investor's portfolio. By normalizing this and the measure of return, then ranking all the funds in a category, one obtains an answer to the following question: If I had chosen one fund and had limited my choice to funds in this category, which would have been the best? In principle, at least, this is very different from the more common question: If I had put some of my money in a fund in this category, which would have been the best? As we will argue, to answer the latter question, a different type of measure is called for -- one involving the difference between a fund's return and that of an appropriate benchmark.
We have defined a fund's selection return in month t as the difference between its return and that of a benchmark:
SelRetit = Rit - Rbp,t
where:
SelRetit = the selection return for fund i in month t
Rit = the return on fund i in month t
Rbp,t = the return on a benchmark portfolio in month t
As we will show, Morningstar's alpha measures are, in effect, measures of mean selection returns.
To compute its measure of alpha, Morningstar performs a regression analysis to fit the following equation:
ERit = ai + msBetai * ERindex,t + eit
where:
ERit = the excess return on fund i in month t
ERindex,t = the excess return on the index in month t
ai = the regression intercept
msBetai = the regression slope coefficient
eit = fund i's residual return in month i
The alpha value for fund i is then computed as an annualized value of the regression interecept:
1 + msAlphai = ( 1 + ai ) 12
To show that ai is equal to a mean monthly selection return, we begin by substituting the components of the two excess returns in the regression equation:
Rit -Bt = ai + msBetai * ( Rindex,t - Bt ) + eit
where:
Bt = the return on a Treasury bill in month t
Rearranging and simplifying gives:
Rit - [ ( 1 - msBetai ) * Bt + msBetai * Rindex,t ] = ai + eit
The term in the brackets is the return on a portfolio with (1-msBetai) invested in Treasury bills and msBetai invested in the index. This can be considered a benchmark portfolio, thus:
Rbp,t = b1 * Bt + b2 * Rindex,t
where:
b1 = 1 - msBetai
b2 = msBetai
Note that the proportions invested ( b1 and b2 ) sum to one, although one of the coefficients may be negative (representing a short position) and the other greater than one (representing investment of the initial amount plus the proceeds from the short sale).
We can now interpret results from the regression equation as selection returns:
SelRetit = Rit - Rbp,t = ai + eit
Over a period of T months, the mean monthly selection return will be:
MMnSelReti = ( 1 / T ) * sumt=1..T { SelRetit }
Standard linear regression methods select parameters (here, ai and msBetai ) to minimize the variance of the residuals ( here, the variance of the eit terms). As a byproduct, the average value of the residuals ( eit's ) will be zero. Hence the mean difference between the return on the fund and that of the benchmark will equal ai. But this is the monthly mean selection return. Thus:
MMnSelReti = ai
Finally, note that the minimand in a standard regression analysis is the same as that in a standard style analysis. The only difference concerns the upper and lower bounds, which are not present in a regression analysis. This leads to the interpretation of Morningstar's procedure for calculating a fund's alpha value as equivalent to (1) determining the mean selection return obtained from a style analysis in which the assets are the index used for the asset class (for diversified equity funds, the S&P500) and Treasury bills, with upper and lower bounds set at sufficiently extreme values (e.g. plus 10 and minus 10 respectively) to be non-binding, then (2) annualizing the resulting value.
Our measure of mean selection return uses a simple annualization:
MnSelReti = 12 * MMSelReti
MnSelReti = Annualized mean selection return
This differs slightly from Morningstar's procedure, which assumes compounding. However, the magnitudes of monthly mean selection returns tend to be quite small -- hence any differences due solely to the choice of an annualization method will be slight.
We wish to analyze three measures of mean selection returns:
msAlphai : Morningstar's Alpha
msBFAlphai Morningstar's Best-Fit Alpha
MnSelReti : Ten-asset Style Analysis Mean Selection Return
Except for slight differences in annualization methods, all can be considered measures of mean selection return using benchmarks obtained from in-sample style analyses. The key differences concern the assets allowed in the analyses and the upper and lower bounds employed. For our group of diversified equity funds the alternatives are:
Morningstar's Alpha:
Assets:
Treasury Bills
Standard and Poor's 500 indexShort sales allowed
Morningstar's Best-Fit Alpha:
Assets: Treasury Bills and one of:
Lehman Brothers Long-term Treasury Index
First Boston High-Yield Bond Index
Standard and Poor's 500 Index
Standard and Poor's MidCap 400 Index
Russell 2000 Index
Wilshire 4500 Index
Morgan Stanley Capital International All Country Index
MSCI Europe, Australia and South East Asia Index
Morgan Stanley Capital International Europe Index
Morgan Stanley Capital International Pacific Index
Morgan Stanley Capital International Pacific ex Japan Index
Morgan Stanley Capital International World ex U.S. Index
JSE Gold Index
Wilshire Real Estate Investment Trust Index
Short sales allowed
Ten-asset Style Analysis Mean Selection Return
Assets:
Vanguard Money Market Reserves -- Prime Portfolio Vanguard Bond Index Fund -- Short-term Portfolio Vanguard Bond Index Fund -- Intermediate-term Portfolio Vanguard Bond Index Fund -- Long-term Portfolio Vanguard Index -- Value Vanguard Index -- Growth Vanguard Index Extended Market Vanguard International Equity Index -- European Vanguard International Equity Index -- Pacific Vanguard International Equity Index -- Emerging Market No short sales allowed
As described earlier, the Best-Fit Alpha is obtained by performing a series of two-asset style analyses, each involving Treasury Bills and one of the listed asset classes. From the results, the one with the best fit (lowest variance of eit values) is then selected.
Having described the similarities among these three measures, we are now ready to examine their characteristics.
The figure below shows the distribution of Morningstar Alpha values for the 1,268 funds in our three-year sample for which the value was between -10% and +10% per year.
Morningstar Alphas, 1,268 funds, 1994-1996
The results are dramatic. For the full set of 1,286 funds the mean alpha value was -2.14% per year. Moreover, 77.0% of the funds had negative alpha values. While one expects the average active fund to underperform a suitably chosen passive benchmark, the magnitudes of these results suggest that the S&P500 was not an appropriate benchmark for many of these funds.
The distribution of fund betas relative to the S&P500 is also of interest. It is shown below:
Morningstar Betas, 1,286 funds, 1994-1996
The average beta was 0.916. None was negative, but 335 of the funds had beta values greater than 1.0. In each such case, the benchmark portfolio required borrowing at the Treasury bill rate with investment of the proceeds as well as the initial amount in the S&P500 index. To the extent that actual borrowing would be more expensive, the net return on the benchmark would be smaller and the resultant alpha larger. For these funds, at least, the Morningstar alpha results may be biased downward relative to values that could reasonably be obtained in practice.
Finally, there is the question of the extent to which diversified fund excess returns can be explained by their comovement with the S&P500. This is measured by the Morningstar R-squared. The distribution of the values for our three-year sample is shown below:
Morningstar R-squared values, 1,286 funds, 1994-1996
The average R-squared value was 68.0%. Thus movements in the S&P500 explained only 67.9% of the variation in monthly returns for the average diversified equity fund in this time period.
While the S&P500 may not be the best benchmark for some funds, it may be better than a number of alternatives if only one index is to be used along with Treasury bills for the purpose. When selecting a best-fit benchmark for an equity fund, Morningstar evaluates 14 different indices. However, only four represent domestic equity securities, and these differ only in capitalization. Surprisingly, given its emphasis on value, growth and blend categories, Morningstar does not employ any value or growth indices for this analysis.
Only six of the 14 indices were selected in the analysis. The table below shows the number of funds for which each of these was the best-fit index, the average R-squared value for such funds based on the S&P500 results, the average R-squared value based on the best-fit index, and the improvement in R-squared associated with the use of the best-fit index instead of the S&P500. The bottom line provides summary values for all 1,286 funds in our sample.
Best-Fit Index Number Avg R2 Avg Best-Fit R2 Improvement Standard and Poor's 500 Index 584 87.6 87.6 0 Wilshire 4500 Index 241 45.5 77.3 31.8 Standard and Poor's MidCap 400 Index 263 68.0 78.5 10.5 Russell 2000 Index 191 35.7 77.6 41.9 Morgan Stanley Capital International All Country Index 5 69.8 72.8 3.0 Lehman Brothers Long-term Treasury Index 2 23.0 35.5 12.5 All 1,286 67.9 82.2 14.4
As can be seen, the S&P500 Index was the best for only 45% of the funds (584 out of 1,286). For the vast majority of the remaining funds a smaller-capitalization U.S. stock index was the best of the allowed set. Improvements in fit were substantial, with the overall average R-squared increasing from 67.9% to 82.2%. The difference can be seen in the distribution of the Best-Fit R-squared values, which is shown below.
Morningstar R-squared values, 1,286 funds, 1994-1996
It should be remembered that all these results are in-sample and that no adjustments are made for lost degrees of freedom, hence it is not surprising that the R-squared values are greater for many of the funds. However, the magnitudes of the improvements are large enough that it seems highly improbable that they would be found to be insignificant in a more relevant out-of-sample test.
We turn now to the best-fit alpha values. The figure below shows the distribution for the 1,268 funds in our three-year sample for which the value was between -10% and +10% per year.
Morningstar Best-Fit Alphas, 1,269 funds, 1994-1996
These results differ considerably from those based on comparisons with only the S&P500 Index. Here, the average alpha value is -0.58%, roughly consistent with the underperformance expected given fund expenses. Moreover, only 59.6% of the funds had negative best-fit alphas.
This illustrates the dangers associated with use of inappropriate indices. The typical diversified equity fund has an average capitalization well below that of the S&P500 Index. By comparing the performance of every fund with that of this index one mixes results due to differences in style (here, capitalization) with those due to differences in selection. During the 1994-1996 period small stocks tended to underperform large stocks, so the average alpha against the S&P500 was very negative. In other periods the bias could be reversed. In either case, comparisons with more appropriate benchmarks could better measure the extent to which a manager had outperformed or underperformed a passive strategy invested in the same sector of the market. The Best-Fit alpha is clearly a step in the right direction.
Finally, we show the distributions of Best-Fit Betas.
Morningstar Best-Fit Betas, 1,286 funds, 1994-1996
The average best-fit beta was 0.970. None was negative, but 425 of the funds had beta values greater than 1.0 and thus involved comparisons with benchmark portfolios requiring borrowing at the Treasury bill rate, with the resultant possibility that the computed value was biased downward.
We turn now to the results obtained using style analysis with the ten Vanguard funds as asset classes. Note that the assets used in this case are investable funds and that the returns are net of all recurring costs. This makes the comparisons more relevant (and also more favorable for the mutual funds) than those involving indices that may or may not be investable and returns that do not take into account any of the costs that could be incurred when investing passively in the associated sectors.
The table below shows the average exposures to the assets for the 1,286 funds, based on returns from 1994 through 1996.
Asset Average Exposure (%) Vanguard Money Market Reserves -- Prime Portfolio 4.1 Vanguard Bond Index Fund -- Short-term Portfolio 1.6 Vanguard Bond Index Fund -- Intermediate-term Portfolio 1.4 Vanguard Bond Index Fund -- Long-term Portfolio 1.0 Vanguard Index -- Value 20.4 Vanguard Index -- Growth 20.0 Vanguard Index Extended Market 46.1 Vanguard International Equity Index -- European 1.7 Vanguard International Equity Index -- Pacific 1.5 Vanguard International Equity Index -- Emerging Market 2.1
Summarizing by major type of asset:
Cash 4.1 Bonds 4.0 U.S. Stocks 86.5 Non-U.S. Stocks 5.4
It is notable that the average "diversified U.S. equity fund" has an economic exposure to portions of the U.S. stock market equal only to that of 86.5 cents per dollar invested. From an economic standpoint, allocation to funds in this sector is likely to bring exposure to fixed income instruments and to stock markets outside the United States.
The style results are consistent in a major respect to those obtained in the Best-Fit analysis. The average fund has more exposure to the stocks in the bottom 30% of the market (represented by the Wilshire 4500 index) than to those in the top half (represented by the S&P500 index). Due to the lack of index funds specializing in small-capitalization value or growth stocks, we are unable to determine whether or not the average small-capitalization fund has a growth or value tilt. However, the results suggest that at least in the large-capitalization sector, exposures are more or less evenly divided between value and growth emphases
We turn now to the counterpart of the Morningstar alpha measures -- the mean selection returns for the funds. The figure below shows the distribution for the 1,267 funds for which the results were between -10% and +10% per year.
.
Ten-Asset Mean Selection Returns, 1,267 Funds, 1994-1996
In this case the results are more evenly distributed around zero. In fact, the average mean selection return for the full set of 1,286 funds is only -0.09% per year and only 53.6% of the funds have negative values. Such differences are due in no small measure to our use of actual index fund returns net of costs as surrogates for asset classes.
Not surprisingly, the counterparts of R-squared values are high using this procedure.
Ten-Asset Style Analysis R-squared values, 1,286 funds, 1994-1996
The average value was 83.2% -- slightly higher than that obtained in the best-fit analysis.Of course, these are in-sample values with no attempts made to correct for lost degrees of freedom. Since the sample properties of statistics obtained using inequalities or selection from among a set of separate regression results are not easily specified, questions concerning "true" fit are best left for out-of-sample studies in which benchmarks are chosen before-the-fact.
Go to Table of Contents for this paper
