Friday, August 31, 2007

Speed issues at 10AM, 12PM, 2PM, 4PM EST

We had a customer report a slow down at the above mentioned times during our hosting system. Those slowdowns last under 15 minutes and happen repeatedly. Those times are selected for scheduled 120 minute incremental backups of the hosting environment.

We have a sophisticated system in place that not only allows us to backup frequently, it also allows us to restore in case of a massive failure quickly. In fact, a standard restore would take 24-48 hours, we can get it done in way under 120 minutes. Hopefully we never have to try it, but its good to know the system is protected.

Performance problems with this type of backup strategy are rare, but can happen when large amounts of data are being changed during the day - as we have seen. Given the new information we will adjust the backup routine to increase performance.

Next week we will address the issue in two stages to see which solution works best (unfortunately we need to do a little trial/error). First stage will be to up the frequency of the backups to once every 15 minutes, and see if doing more "smaller" backups works better than less "larger". If the system experiences lag we will shift in the opposite direction and do less backups alltogether, and schedule them when usge is not at its peak.

We appreciate your cooperation.

No comments: