Monday, April 17, 2006
Learnings at Work - Applying Windows Security Patch!
On Friday, April 14th, we recieved an email from our company's Network Adminstrator team stating that they would be applying a Windows Security patch to all the desktops and the servers. We all went home, after making sure that all our desktops are logged off.
We came back on Monday to work. Everything was seemed to be working fine. By 10am, we started getting calls from our application users stating that they were experiencing performance issues and runtime errors with some of our web applications. We did our part of the checking to make sure that no one had made any changes to the production environment. By lunch time, some of the team leads had decided to do a "Reboot" of all the application servers and DB servers to get things under control. We all went to lunch, after restarting our servers. We thought the problem was over.
We came back from lunch to see that the problem was still there. We had databases on multiple servers and our applications were using the data from all these servers. We were having major performance issues because of just one server. We called the Network guys who performed the patch update and asked them to remove all the patches from all the servers to see, if that was the root cause. But, it wasn't !!
Then we started applying the patches again. When we reached the server, which was causing us more headache, we found that the server was missing one of the important pre-requisite patch needed for the current patch to be applied. When we manually started applying the patch, we realized this fact and brought the server in sync with all the remaining servers. Bingo!!! All the applications started working fine. This particular server was missing some MDAC patches and hence we were facing so many issues.
Learning: Don't blindly apply security patches to all the servers. Check what are the pre-requisites and make sure that the serves have them all before letting the new patches to be installed. Let the team also follow the process and publish the procedure to be followed for patch updates. Take extra precautions while applying patches to application and database servers.
I would like to hear, how such situations are being managed to make sure that all the machines in an organization are up-to-date with the same patch? Any response would be a new learning for me !!
We came back on Monday to work. Everything was seemed to be working fine. By 10am, we started getting calls from our application users stating that they were experiencing performance issues and runtime errors with some of our web applications. We did our part of the checking to make sure that no one had made any changes to the production environment. By lunch time, some of the team leads had decided to do a "Reboot" of all the application servers and DB servers to get things under control. We all went to lunch, after restarting our servers. We thought the problem was over.
We came back from lunch to see that the problem was still there. We had databases on multiple servers and our applications were using the data from all these servers. We were having major performance issues because of just one server. We called the Network guys who performed the patch update and asked them to remove all the patches from all the servers to see, if that was the root cause. But, it wasn't !!
Then we started applying the patches again. When we reached the server, which was causing us more headache, we found that the server was missing one of the important pre-requisite patch needed for the current patch to be applied. When we manually started applying the patch, we realized this fact and brought the server in sync with all the remaining servers. Bingo!!! All the applications started working fine. This particular server was missing some MDAC patches and hence we were facing so many issues.
Learning: Don't blindly apply security patches to all the servers. Check what are the pre-requisites and make sure that the serves have them all before letting the new patches to be installed. Let the team also follow the process and publish the procedure to be followed for patch updates. Take extra precautions while applying patches to application and database servers.
I would like to hear, how such situations are being managed to make sure that all the machines in an organization are up-to-date with the same patch? Any response would be a new learning for me !!
# posted by Vinay Barigidad @ 10:01 PM 
