While some of Morningstar's measures are similar to those used in academic papers and industry analyses, there are differences. For purposes of comparison, we introduce three key alternative performance measures:
excess return Sharpe ratio
mean selection return
selection Sharpe ratio
For emphasis, the sections describing these measures are highlighted in green:
Most academic studies compute historic Sharpe Ratios by dividing the mean of the differences between two return series by the standard deviation of the differences. In the case of monthly excess returns:
MERSRi = AMERi / SDMERi
where:
AMERi = the arithmetic monthly average excess return on fund i during period T
SDMERi = the standard deviation of monthly excess returns for fund i during period T
MERSRi = the monthly excess return Sharpe ratio for period T
For purposes of comparisons, such measures are often scaled to give annualized equivalents, assuming no serial correlation and ignoring compounding. Under such conditions, the mean increases with the number of periods per year (here, 12), while the standard deviation increases with the square root of the number of periods per year. As a result, the annualized Sharpe ratio equals the original value times the square root of the number of periods per year. In this case:
ERSRi = sqrt(12) * AMERi / SDMERi
where:
ERSRi = the annualized monthly excess return Sharpe ratio for period T
Later, we will compare this measure with the slightly different one computed by Morningstar.
Two of Morningstar's performance measures can be characterized as summarizing information about differences between the returns on a fund and those of a benchmark portfolio. Such differences can be termed selection returns. Let:
SelRetit = Rit - Rbp,t
where:
SelRetit = the selection return for fund i in month t
Rit = the return on fund i in month t
Rbp,t = the return on a benchmark portfolio in month t
Different approaches can be used to select an appropriate benchmark portfolio for a fund. Such a portfolio might include only a single fund or index. Alternatively, it might include a fund plus a cash position.Or it might include several funds or indices. Morningstar's alpha and best-fit alpha measures fall in the second category (one index plus a cash position). Below, we describe a method that can utilize benchmark portfolios of as few as one or as many as several funds. Later, we compare the characteristics of measures using alternative approaches for the selection of an "appropriate" benchmark for each fund.
Once a benchmark is selected for a fund, selection returns can be computed for each of the T months in a period of interest (for example, the last 36 months). The average, or mean, can then be used to summarize the extent to which the fund "beat its benchmark" (if the mean is positive) or was "beaten by its benchmark" (if the mean is negative):
MMnSelReti = ( 1 / T ) * sumt=1..T { SelRetit }
where:
MMSelReti = the mean monthly selection return for fund i over period T
This value can be annualized with or without taking compounding into account. The corresponding results are given by the formulas:
MnSelReti = 12 * MMSelReti
( 1 + CMnSelReti) = ( 1 + MMSelReti ) 12
where:
MnSelReti = Annualized mean selection return
CMnSelReti = Annualized compounded mean selection return
While Morningstar uses the compounded version (CMnSelReti) we will use the other version (MnSelReti) for the sake of simplicity, and refer to it simply as the mean selection return.
The mean selection return summarizes the average performance of a fund relative to its benchmark but does not provide any indication of the consistency with which that performance was achieved. To measure the variation of the selection returns around the mean, we compute the monthly selection standard deviation:
VMSelReti = ( 1 / T ) * sumt=1..T { ( SelRetit - MMnSelReti ) 2 }
SDMSelReti = sqrt { VMSelReti }
where:
VMSelReti = the variance of monthly selection returns for fund i during period T
SDMSelReti = the standard deviation of monthly selection returns for fund i during period T
To take into account both the added (or subtracted) value provided by a fund vis-a-vis its benchmark and the added risk, we divide the monthly mean selection return by the standard deviation of monthly selection returns. This gives a monthly selection Sharpe ratio:
MSelSRi = MMnSelReti / SDMSelReti
where:
MSelSRi = Monthly selection Sharpe ratio for fund i during period T
To annualize this, we multiply by the square root of 12. The result will be termed simply the selection Sharpe ratio:
SelSRi = sqrt { 12} * MSelSRi
where:
SelSRi = the annualized selection Sharpe ratio
In addition to the methods used by Morningstar to select benchmark portfolios, we introduce results obtained using the method often termed returns-based style analysis, described in detail in Sharpe, Asset Allocation: Management Style and Performance Measurement. In this case we wish to find a portfolio of up to m index funds. The return on any such portfolio in time t can be written as:
Rbp,t = b1*R1t + b2*R2t + ... + bm*Rmt
where:
b1, b2, ..., bm = the proportions invested in index funds 1,2,.. m, respectively
R1t, R2t, ... Rmt = the returns on index funds 1,2,...m in period t
Rbp,t = the return on the benchmark portfolio in period t
In order for the results to correspond to a portfolio, the sum of the b's must equal 1.0. In addition we impose the constraint that each of the b's must lie between 0.0 and 1.0, inclusive. Of course, a great many portfolios meet this requirements. Of all possible such portfolios, returns-based style analysis selects the one that will give the smallest selection standard deviation when used as the benchmark for the fund in question. This must be done using mathematical programming procedures, due to the presence of inequality constraints regarding the values of the b's. In effect, the analysis finds the combination of indices that "moved most like" the fund during the period analyzed.
To provide results comparable to those produced by Morningstar, we compare the performance of each fund with that of the benchmark selected using returns-based style analysis on data covering the same period. Thus we use 36 months of data on the ten index funds and a selected mutual fund to (1) determine a benchmark portfolio with the most similar style, (2) compare the fund's returns with those of this benchmark portfolio for each of the 36 months to determine the monthly differences (selection returns), then (3) summarize the fund's relative performance by computing the associated mean selection return and selection Sharpe ratio.
Note that this procedure is "in sample" -- the fund is compared with a strategy that could not have been identified before-the-fact, since the characteristics of the strategy are determined using data covering the period over which the performance is evaluated. This is also the case for the Morningstar measures. A more relevant study would use only "out-of-sample" measures of performance in which comparisons were made with investment alternatives that could be identified in advance and implemented, if desired. However, the task at hand is to compare alternative methods with those used by Morningstar on roughly equal ground. The far more important question of predicting future performance is left for other studies.
To make the benchmark as relevant as possible, our analysis uses the following ten index funds offered by the Vanguard group:
Vanguard Money Market Reserves -- Prime Portfolio Vanguard Bond Index Fund -- Short-term Portfolio Vanguard Bond Index Fund -- Intermediate-term Portfolio Vanguard Bond Index Fund -- Long-term Portfolio Vanguard Index -- Value Vanguard Index -- Growth Vanguard Index Extended Market Vanguard International Equity Index -- European Vanguard International Equity Index -- Pacific Vanguard International Equity Index -- Emerging Market
Since the emerging market fund was introduced in early 1994, we use returns for International Financial Corporation's Investable Index (minus 2 basis points per month) for the first four months of 1994 as estimates of the returns that might have been achieved during that period. Otherwise all returns are net returns obtained by investors in the funds (omitting, however, the one-time transactions fees charged by a few of the funds during the period).
Further information about these funds can be found on the Vanguard home page. For the actual returns of the funds, see the monthly returns page.
The extent to which a benchmark portfolio's returns explain a fund's returns can be measured by one minus the ratio of (1) the variance of the selection returns to (2) the variance of the fund's returns. In the case of a benchmark portfolio determined using returns-based style analysis::
1 - SAR2i = var ( SelReti ) / var ( Ri )
where:
SAR2i = style analysis R-squared for fund i
This is comparable to the Morningstar R-squared measure, except that in the latter the variance of the fund's excess returns is used in the denominator, rather than the variance of the fund's total returns.
All variances, standard deviations and R-squared values are computed from the data at hand without any adjustment for lost degrees of freedom. This follows Morningstar's practice. The statistics should thus be interepreted as summary measures of historic data, rather than as unbiased estimates of future values.
Go to Table of Contents for this paper