July 19-20 Weekend Upgrades

Parts of the hosting system will be unavailable as we're expanding the virtualization environment.   We'll be bringing down the entire server farm to add additional RAM as well as upgrade the virtualization software.  The entire upgrade process will last less than 1 hour but we need to complete full system backups which will take significantly more time. Keep an eye on this blog as we'll post our activities as they happen.  Most of the downtime is expected Sunday morning.

 

-Karl J

Hosting SQL Server failure 7/10/08

The primary SQL Server HOSTSERV1 suffered a failure last night preventing users from launching TrialWorks.  The issue was corrected approximately 9:05 AM but some databases did not immediately come online - this behavior is by design.  When SQL Server recovers from a failure all databases are placed in a "Recovery" mode and checked for consistency before being brought online. This process may take several minutes to complete for each database.   At the time of this e-mail approximately 5% of the databases were still in process of coming online.

The failure was caused by a Microsoft update which failed to properly install at 3:00 AM EST.  We apologize for any inconvenience this may have caused.

 

 

Administrative Note (cause of issue):

 

Event Type: Error
Event Source: MsiInstaller
Event Category: None
Event ID: 1023
Date:  7/10/2008
Time:  3:03:35 AM
User:  NT AUTHORITY\SYSTEM
Computer: HOSTSERV1
Description:
Product: Microsoft SQL Server 2005 - Update 'GDR 3068 for SQL Server Database Services 2005 ENU (KB948109)' could not be installed. Error code 1603. Additional information is available in the log file C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9_Hotfix_KB948109_sqlrun_sql.msp.log.

 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 7b 31 33 30 41 33 42 45   {130A3BE
0008: 31 2d 38 35 43 43 2d 34   1-85CC-4
0010: 31 33 35 2d 38 45 41 37   135-8EA7
0018: 2d 35 41 37 32 34 45 45   -5A724EE
0020: 36 43 45 32 43 7d 20 7b   6CE2C} {
0028: 31 44 37 38 35 45 42 33   1D785EB3
0030: 2d 45 36 44 44 2d 34 46   -E6DD-4F
0038: 41 41 2d 39 34 39 43 2d   AA-949C-
0040: 44 32 45 35 44 33 37 36   D2E5D376
0048: 43 37 39 32 7d 20 31 36   C792} 16
0050: 30 33                     03     

 

/Author: Karl J

 

 

LogOn and LogOFF speed will be severly affected by changes this weekend

March 15-17th we will be enabling Roaming Profiles and Document Redirection to facilitate the access by all our users to multiple servers. The first few times users logon/off with the policies in place the system will have to move large amounts of data from the current stores to the new locations. The result will be a delay in logging on and off. Please be patient and do not interrupt the process.

What is a user profile?
Your user profile stores your individual settings under Windows. It is used to re-create your preferred system environment when you log on. This includes everything from your wallpaper and desktop, to your My Documents folder, to your email settings and network connections.

Why use profiles?
User profiles allow multiple users to share a Windows system and still maintain their own preferences. You can set your background to chartreuse without affecting anyone else who uses the machine. Some of the things you can set that are stored in your profile include:

• Wallpaper Screen saver Display properties - colors, fonts and sizes Shortcuts on the desktop Whether to display or not to display web content A personalized Taskbar Desktop Toolbars Settings for some applications

What is a roaming profile?

A roaming profile is stored on a central server which can be accessed from all domain computers. This allows you to have the same environment settings on every machine to which you log on.

Expanded Storage!

During the month of March we will also be expanding our storage infrastructure. Along with new servers we've also added network attached storage to improve transfer speeds and increase capacity. However, we will also implement policies and limits. We'll be contacting those customers whose data stores are excessively large to help reduce consumption and improve manageability.

Faster Outlook and TrialWorks for Hosting Customers!

Through the weekend of March 7th -9th users will once again experience some interruptions in data accessibility as we expand infrastructure. TrialWorks (specifically SQL Server) will be most affected during that time. The end result, however, will offer hosting customers significant improvements in the speed of Microsoft Outlook (due to the performance gains for Microsoft Exchange Server) as well as TrialWorks (since SQL Server will be upgraded).

After the upgrades we will institute harsher limits on mailbox size to maintain an improved customer experience. All users are urged to start cleaning up their mailboxes now.

Load Balancer is Online

One of the most recent advances has been the addition of the LoadBalancer. We suspect it was also one of the most anticpated ones as it will rapidly improve performance on our networks. It is a gateway server that will monitor performance of all servers in the "farm" and route users to the one that is least busy (fastests).

There are some changes to our hosting users.

1. The RDP Connection Address Has Changed. All Users will need to change their RDP Connection icon to point to the LoadBalancer, rather than the existing server connections.
   
2. TechSupport must confirmn Roaming Profile. In some cases roaming profiles have been disabled for hosting users. Each account must be confirmed and re-activated with Roaming Profiles before they will be allowed to connect through the Load Balancer.
   
   Users with lots of data in their My Documents will likely experience delays in logging on/off because of the synchronizations. Please work to reduce the amount of data in MY DOCUMENTS and on DESKTOPS. Our network team will be performing audits to assess the data usage.
   

There are certainly other factors at play when users are hopping through a network of servers. We will work to ensure that all machines are as similar as possible with software and updates; as our server farm grows so does a chance for slight mis-matches in software versions. We'll work quickly to resolve them.

It's been a while...

I have not kep this blog current but in the last few months lots of exciting things have happened; mainly we got lots of new hardware. We're in the process of deploying it this month (started back about two weeks). It's a tough process as we try to do most of the work off-hours and weekends. In addition, the process is very complex, so we have to do it in pieces - but we know you are all patiently waiting.

Our Communications Provider will Cause an OUtage on 2/21/2008

Dear Customer,

This email notice is to inform you of scheduled maintenance occurring in the Miami, FL area affecting your XO Data/Access service. 

On 2/21/2008, between the hours of 11:00 PM EST [2/21] to 6:00 AM EST [2/22], you will experience a 45 minute interruption of service. XO Communications Customer Care and Network Teams will be monitoring this activity to ensure necessary downtime is kept to a minimum while the maintenance is completed. 

At your convenience, feel free to reply to this message or call us at the toll free number listed on your XO bill, as our Customer Care teams will be available to handle any questions, comments, or concerns. 

Please reference Maintenance Request 48394 when contacting Customer Care. 

Thank You,

XO Communications