With Year 2 of the Charlie Weis tenure officially in the books, the search is on for metrics by which to judge his performance. What numerical data can we use to check our biases and provide context? Over the long term, every Notre Dame coach will be judged by whether or not he delivered a national championship. However, while five different coaches have won national championships at Notre Dame, none of these championships was won in the coach's first two years. Thus it's too early to apply this standard to Weis.
One of the obvious measures by which to judge a coach in the first few years of his tenure is his performance relative to his predecessor. In his first two years, Charlie Weis guided Notre Dame to 19 wins. In the two preceding years, Tyrone Willingham guided Notre Dame to 11 wins. While Irish fans certainly appreciate those eight extra wins (and may appreciate the eight fewer soul-crushing losses even more), it's not clear how much this tells us about Weis at Notre Dame.
To provide context, we decided to look at the change in win totals for all the Division I coaches hired following the 2004 season. Twenty-two new head coaches were hired that offseason. Because there was such a large amount of turnover (changes at almost one-fifth of the D-I schools), we should have a decent-sized sample to examine. A Dennis Dodd article from early 2005 provided a comprehensive list of the new hires at that time, and that list is reproduced below. (Dodd's ranking of the hires is also included, for entertainment purposes only.)
|Dodd's Rank||Coach||School||Dodd's Rank||Coach||School|
|1||Urban Meyer||Florida||12||Frank Solich||Ohio|
|2||Steve Spurrier||South Carolina||13||Dave Wannstedt||Pitt|
|3||Ed Orgeron||Mississippi||14||Walt Harris||Stanford|
|4||Mike Gundy||Oklahoma State||15||Dick Tomey||San Jose State|
|5||Les Miles||LSU||16||Hal Mumme||New Mexico State|
|6||Kyle Whittingham||Utah||17||Terry Hoeppner||Indiana|
|7||Tyrone Willingham||Washington||18||Bronco Mendenhall||BYU|
|8||Charlie Weis||Notre Dame||19||Shane Montgomery||Miami (OH)|
|9||Ron Zook||Illinois||20||Skip Holtz||ECU|
|10||Greg Robinson||Syracuse||21||Brent Guy||Utah State|
|11||Mike Sanford||UNLV||22||Bill Cubit||Western Michigan|
Now let's compare the win totals at these programs in the two years preceding the new coach's arrival and in the two years following his arrival. The following list ranks the coaches by this change in win total:
|Coach||School||Wins (03-04)||Wins (05-06)||Delta||Dodd's Rank |
|Bill Cubit||Western Michigan||6||15||+9||22|
|Charlie Weis||Notre Dame||11||19||+8||8|
|Dick Tomey||San Jose State||5||12||+7||15|
|Steve Spurrier||South Carolina||11||15||+4||2|
|Brent Guy||Utah State||6||4||-2||21|
|Hal Mumme||New Mexico State||8||4||-4||16|
|Mike Gundy||Oklahoma State||16||11||-5||4|
|Shane Montgomery||Miami (OH)||21||9||-12||19|
What do these numbers tell us, and how much significance should we attach to these deltas? Let's dispense with the low-hanging fruit. The first conclusion we can draw is that Dodd's prognostication abilities are lacking, and in this respect I think the numbers are fairly reliable. Dodd ranked just three of the nine coaches that improved on the prior two years' win total in the top half of new hires. Even worse, three of the six worst totals were produced by coaches Dodd ranked among the six best hires. While (for reasons that will be discussed below) one cannot look solely at the delta in evaluating coaching performance, the numbers are not flattering to Dodd.
Moving on to the more substantive analysis, the numbers suggest that one cannot look at the delta in isolation. The programs that had the highest win totals in 2003-2004 generally garnered fewer wins in 2005-2006 following the coaching change. Conversely, the teams with the fewest win totals in 2003-2004 generally improved their win total in 2005-2006 following the coaching change. I believe there are two phenomena at work here. First, the three programs in this group that had 20+ wins in '03-04 were left searching for new coaches in the winter of 2004 because their coach had left for greener pastures. Thus the new coaches at LSU, Utah, and Miami (OH) were all following coaches who had exceeded the historical performance standards at their program. One would expect matching such standards would be difficult.
Second, I believe it is harder to improve on a mediocre win total than an abysmal win total for three reasons: (1) when the win total is higher in the previous years, there are fewer opportunities to garner additional wins; (2) as the previous win total increases, the quality of teams against whom additional wins can be garnered will tend to increase; and (3) increases on truly bad performances may simply represent regression to the mean. This suggests that an individual coach's delta will be "inflated" relative to his peers where his absolute total of wins in 2005-2006 is middle-of-the-pack, and an individual coach's delta will be "undervalued" relative to his peers when his absolute total of wins in 2005-2006 is high.
A few examples illustrate this point. Les Miles, Tyrone Willingham, and Ron Zook all failed to improve on the preceding two years' win total. However, even if Miles had won every game the past two years, he would only have improved on Saban's record by four wins. Saban's record in his final two years at LSU was 22-4 and Miles matched this record in his first years. Yet if Miles had notched these four wins, he would have won back-to-back national titles, and would be regarded more highly than any other coach on this list. Conversely, Willingham matched Keith Gilbertson's 7-16 record and Ron Zook matched Ron Turner's 4-19 record. Willingham and Zook inherited bad situations, but there are many losses to bad teams in Willingham and Zook's tenures at Washington and Illinois. Would it have been too much to ask Willingham to beat, say, Air Force? The truth is that matching Nick Saban's performance is simply a greater achievement than matching Gilbertson or Turner's, even if the result is the same delta.
Urban Meyer's numbers tell a similar story. Four coaches improved their school's record by more wins than Meyer, but it is Meyer that has reached the mountaintop. That Florida's players were able to win 15 games in 2003-2004 despite Zook's mismanagement speaks volumes about just how much talent Meyer inherited. Yet to generate a +7 improvement in wins, Meyer had to defeat teams like Ohio State and LSU, teams that finished in the top five. Again we see that the difficulty of achieving marginal improvement increases with the previous success of a given program.
(I recognize that many attribute much of Miles' and Meyer's recent success to Bo Pelini, Jimbo Fisher, and Charlie Strong. While I can agree with this to an extent, a head coach deserves credit for hiring strong coordinators and must share the blame for hiring weak coordinators.)
In conclusion, while I think the change in a team's win total tells us something about coaching quality, such numbers must be viewed in light of the starting point. The raw delta is not as informative as it might initially appear. In this respect, I do see a positive sign for the Notre Dame faithful. As down as Irish fans are following the Sugar Bowl loss and the last-minute defection of a handful of recruits, the long-term trend for the program remains positive. Only three coaches in this sample were able to deliver improvement on a double-digit win total: former national champion coach Steve Spurrier (+4 on 11 wins), current national champion coach Urban Meyer (+7 on 15 wins), and our own Charlie Weis (+8 on 11 wins). Time will tell whether Weis can build the defense necessary to meet the real measuring stick of great coaches - a national championship.