Best Practice for Patching Individual Packages in PCM
| Document Number | 1-1F6K86 |
| Product | Platform OCS/PCM |
| Affected Version | 1.2* 2.0* 2.1* |
| OS | Linux |
| Category | Installation/Upgrade |
| Date Created | Jan 21 2009, 10:15 PM |
| Last Update | Nov 10 2010, 11:14 AM |
| Keywords | Update, patch, rhn, repopatch, kernel, |
Topic
Best Practice for Patching Individual Packages in PCM
Issue Description
I need to update the kernel package on my PCM cluster with the latest and greatest avaialbe on RHN.
Solution Detail
Often, it is necessary to apply patches to your PCM cluster. Some patches fix security issues, while other patches offer bug fixes and feature improvements. Patches can be applied in two ways.
1. One way is to use kusu-repopatch tool to download and install all available patches for your Operating System. This has the advantage of having your cluster nodes always at the "latest and greatest" version level. There are a few down sides with this approach, namely that updating large number of packages is more risky and can break your production environment, and also if you are using Platform OFED for InfiniBand support, kusu-repopatch cannot be used due to dependency issues. This last point is only valid for PCM versions < 2.1. In 2.1, this issue is addressed, and kusu-repopatch can safely be used even with Platform OFED kit installed.
2. The other way is to download the individual packages manually using yum command, and then add them to your PCM repository. The advantage is that updates can be done very quickly and relatively safely. The disadvantage is that not all of your packages are at the "latest and greatest" version level.
This article describes the second approach, That is, how to patch individual packages in your PCM cluster. As a working example, the article shows how to patch the kernel package.
If you are interested in using kusu-repopatch to patch all the packages, please refer to the article titled "Best Practice for Patching PCM Clusters with kusu-repopatch."
0. Before you begin
Before applying any patches to your production environment, you should take some precautions in case something goes wrong. In your PCM cluster you can take advantage of repository snapshots and node group templates to apply your updates in a safe way.
NOTE: In older version of PCM (< 2.0) the above commands did not have the "kusu-" prefix. For example, repoman vs. kusu-repoman. To determine which version of PCM cluster you are using,you can look at the output of cat /etc/kusu-release.
1. Use kusu-repoman command to get the name of your production repository. For example,
pcm201installer:~ # kusu-repoman -l
Repo name: sles-10.3-x86_64
Repository: /depot/repos/1000
Installers: 192.168.1.1;172.27.1.43
Ostype: sles-10-x86_64
Kits: base-2.0-1-x86_64, sles-10.3-x86_64,
nagios-2.12-7-x86_64, ofed-2.0-1-x86_64,
PCM_GUI-2.0-1-x86_64, platform-hpc-2.0-2-x86_64,
platform-isf-ac-1.0-
platform-mpi-7.1-1-x86_64, platform-rtm-2.0.1-1-x86_64
Repo name: sles-10.3-x86_64
Repository: /depot/repos/1000
Installers: 192.168.1.1;172.27.1.43
Ostype: sles-10-x86_64
Kits: base-2.0-1-x86_64, sles-10.3-x86_64,
nagios-2.12-7-x86_64, ofed-2.0-1-x86_64,
PCM_GUI-2.0-1-x86_64, platform-hpc-2.0-2-x86_64,
platform-isf-ac-1.0-
platform-mpi-7.1-1-x86_64, platform-rtm-2.0.1-1-x86_64
The name of the repository is sles-10.3-x86_64, and it has an ID of 1000. The repository is located under /depot/repos/1000/ directory, and it was created during PCM installation.
2. Create a snapshot of your production repository with kusu-repoman command. This will create a new repository with ID 1001 under /depot/repos/1001/ directory.
# kusu-repoman -r -s
3. Create a new node group which is a copy of your production node group with kusu-ngedit command. You can use kusu-nghosts -l command to determine the name of your production node group, first. Then, use kusu-ngedit to create a copy called compute-patchtest, like this:
# kusu-ngedit -c compute-rhel-5.4-x86-64 -n compute-patchtest
4. Associate the compute-patchtest node group with snapshot repository using the kusu-ngedit command.
5. Move a single node to compute-patchtest node group using kusu-nghosts command.
1. Register with Red Hat Network (RHN)
Fist step is to to register your cluster to Red Hat Network. If you are using SLES, then please use equivalent procedure to register for online updates. If you are using CentOS or SCL, then you do not need to register your cluster, but you might need to enable the online repository in /etc/yum.repos.d/ directory.
Do following to register your cluster with RHN:
1. run rhn_register on your Installer node
2. Update the /opt/kusu/etc/updates.conf file and enter valid username/password to login to RHN as well as Server ID (a.k.a RHN System ID).
NOTE: Server ID is not needed if you are running PCM version < 2.0.
2. Download and Install the Patches
Yum will, by default, download and install the updated packages. However, since PCM will be responsible for installing the updated packages on cluster nodes, we just need yum to download the package, and not install them. For this, you need to download the downloadonly plugin for yum. Follow these steps to download and install the kernel patches.
1. Download the downloadonly plugin from RHN:
# yum -y install yum-downloadonly
2. Create a snapshot of your production repository. Best practice is to always test the updates prior to applying them to production repository. Therefore, you should first create a snapshot of your production repository and .....
3. Download latest kernel packages from RHN. There are three packages which you must update: kernel, kernel-devel and kernel-headers. The packages must be downloaded to /depot/contrib/1001/ directory; this is where PCM will look for custom RPMs associated with snapshot repository (id = 1001).
# yum update -y kernel kernel-devel kernel-headers --downloadonly --downloaddir=/depot/contrib/
4. Update the snapshot repository:
# kusu-repoman -ur "name of snapshot repository"
This will create appropriate symbolic links in /depot/repos/1001/Server/ directory, and it may take up to 10 minutes to complete. The symbolic links to the latest kernel packages are shown below:
[root@pcm201installer ~]# ll /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 59 Oct 21 12:13 /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 65 Oct 21 12:13 /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 67 Oct 21 12:13 /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 59 Oct 21 12:13 /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 65 Oct 21 12:13 /depot/repos/1000/Server/
lrwxrwxrwx 1 root root 67 Oct 21 12:13 /depot/repos/1000/Server/
5. Install the patches on nodes in the compute-testpatch node group. Use the kusu-cfmsync command to install the patches "on-the-fly" - there is no need to re-provision the nodes to install the patches in PCM. However, if you have patched the kernel, you need to reboot the nodes using kusu-boothost command to boot them with the new kernel.
# kusu-cfmsync -n compute-testpatch -u
# kusu-boothost -r "hostname of patched node"
# kusu-boothost -r "hostname of patched node"
3. Test the Patches and Install in Production Environment
The only thing that remains is to test the patches work ok for nodes in compute-testpatch node group. Do as much testing as required to convince yourself that your production applications will work with OK after the patches were applied. For example, you may wish to do the following tests:
- Test that nodes in compute-testpatch node group can be re-provisioned correctly
- Test that patched packages are installed after re-provisioning. For kernel patches, run uname -r command to make sure kernel version is the latest one.
- If you have patched the Installer node as well, then you can run command like kusu-genconfig nodes and make sure you get a list of all node hostnames in you cluster. This will confirm that PCM tools can communicate to the cluster database
Once you are satisfied that the patches are safe to roll out to your production environment, you can follow the steps outlined in section 2 "Download and Install the Patches" to patch your production repository. The only difference is that now you should specify your production node group and installer node group name when you apply the updates via kusu-cfmsync -u
No comments:
Post a Comment