Office of Information Technology

E-mail Outage & Performance Issues - August 2006

For the outage, I have attached a report for that incident. This issue was about a hardware problem that is very similar to the June outage. They may have the same root cause, but that has not been determined. I can tell you that after this recent problem and the subsequent investigation by the vendor, we have since swapped hardware without incident.

The recent performance problems are more problematic. These started after some rather large migrations were completed. It appears that these migrations may have pushed the system beyond the limits based on the current configuration. However, we have known of some configuration flaws for several weeks and have been working on plans to correct these problems. The growth of the system was faster than we had anticipated and the results have been bad performance.

The issues are related to the volume of mail we have in the Exchange system. There are several problems with the volume. They include:

• Hundreds of mailboxes are over the proposed 250MB quota size
• Over 50 mailboxes or over 1GB with the top at over 4GB
• Many mailbox folders exceed Microsoft's recommended items per folder (3500 per folder)
• The mail store allocation is too large to manage effectively
• The disk layout needed more separation to reduce competition for disk I/O
• The operating system needed several other tuning parameters implemented

We have several solutions planned - some have been planned for months. They include:

• Implement the quota - This will be done for students immediately and be phased in for Faculty and Staff.
• Implement an Extended Storage Option - We have Enterprise Vault and plan to implement it as an option for those people who need more than 250MB in their mailboxes. This will take a few weeks. The cost of the service may be passed to those who choose the service. We would like to have this in place before quotas are fully implemented.
• Implement a separate storage option for attachments - SharePoint Services for Windows offers an attachment storage option, but this requires additional time and effort. This is a subset of SharePoint Portal.
• Reformat the mail store databases - This effort is underway. The August 3 outage was caused by adding the additional storage needed for this effort.
• Tune the system - Microsoft has reviewed the system and plans to return for another review. We are implementing the changes they suggest as well as changes we found and were recommended by UT Martin who has run Exchange for a couple years.
• Reduce items per folder - This is an awareness and training issue. OIT is discussing how to share more client-side tuning recommendations.

Mike McNeil