Ruth Lee

Engineering Manager by day, Linux Acolyte by night

Yum Transaction Ordering

19 May 2017 » linux

Today I got an interesting question from a team member about yum’s behaviour around the yuminstall_only= field in yum.conf

We had a server with a small /boot partition that was mainly full. It had the standard 5 retained kernel versions kept there, and we had reduced this to 2 by changing this in yum.conf. My colleagues’ idea was that we could run the yum update command and not worry about the low space on /boot since he assumed that yum would drop the 3 extra old kernels before starting to unpack the new one.

That didn’t make much sense to me logically and I wanted to give him a clear answer for why it wouldn’t do that which wasn’t just “because I think doing it that way would be dumb”. And thus, I spent some of Friday morning investigating it with one of the guys.

The ordering of yum transactions is interesting. Technically, after some digging into the source code, this is done by rpm being called out to by yum, so that’s what’s taking care of the order in which packages are manipulated. To be honest making sense of that would have been a lot of work so we can actually work out yum’s behaviour more simply.

Step 1: What do we think is going to happen?

From a purely logical perspective, if I’m writing the code for yum, i’m going to want to remove the old packages after i’ve installed the new ones. That’s a pretty standard safety check, don’t scrap your rollback plan before you have confirmed you’ve successfully completed the new install. If we removed these before then, in a circumstance where our transaction failed to complete for some reason we could end up with less than the number of retained kernels specified in that config file which seems counter intuitive.

Step 2: Prove what I theoretically think happens does actually happen (because Lord knows Linux doesn’t always do things logically!). Couple of ways to do this, so lets look at all of them.

Method A: Look at the dependencies declared during an install (semi-convincing proof) One of the major things we know yum and rpm are doing is managing dependencies. Dependencies can be specified for two purposes mainly: (a) declared by a package as a dependency, for example mod_ssl requires httpd. (b) used by yum to deal with ordering of the installs of groups of packages, yum will do some cursory ordering of groups of packages to be installed in an intelligent manner, and it does this by createing a tree of dependencies and using them to ensure packages aren’t modified in the wrong order. Knowing that, let’s look at a transaction log from a transaction I quit out of where I try to install another kernel: 2.6.32-696

[root@athena ~]# yum install kernel-2.6.32-696.el6.x86_64 --disableexcludes=all --showduplicates
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-696.el6 will be installed
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-642.15.1.el6 will be erased
---> Package kernel.x86_64 0:2.6.32-696.1.1.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================
 Package                 Arch                    Version                              Repository                 Size
======================================================================================================================
Installing:
 kernel                  x86_64                  2.6.32-696.el6                       base                       32 M
Removing:
 kernel                  x86_64                  2.6.32-642.15.1.el6                  @updates                  131 M
 kernel                  x86_64                  2.6.32-696.1.1.el6                   @updates                  131 M

Transaction Summary
======================================================================================================================
Install       1 Package(s)
Remove        2 Package(s)

Total download size: 32 M
Is this ok [y/N]: ^CExiting on user Command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx-2017-05-19-10-42kpqPll.yumtx
[root@athena ~]# cat /tmp/yum_save_tx-2017-05-19-10-42kpqPll.yumtx
321:948eb17674fd22d90b6de338e86184c6cdb98499
0
5
base:6706:1490724196
epel:12304:1495064580
extras:64:1489012224
pgdg96:307:1494498242
updates:282:1494864180
3
mbr: kernel,x86_64,0,2.6.32,696.1.1.el6 20
  repo: installed
  ts_state: e
  output_state: 40
  isDep: False
  reason: user
  reinstall: False
  depends_on: kernel,x86_64,0,2.6.32,696.el6@a
mbr: kernel,x86_64,0,2.6.32,696.el6 70
  repo: base
  ts_state: i
  output_state: 30
  isDep: False
  reason: user
  reinstall: False
mbr: kernel,x86_64,0,2.6.32,642.15.1.el6 20
  repo: installed
  ts_state: e
  output_state: 40
  isDep: False
  reason: user
  reinstall: False
  depends_on: kernel,x86_64,0,2.6.32,696.el6@a

So, from the above we can basically see the following. We’re installing “2.6.32-696.el6”, but we can see that the old kernels that will be removed (due to the retention limit set in yum conf) have the kernel that we are installing listed as a dependency. This isn’t a hard dependency as declared by the package, but rather a dependency that’s added by yum for this transaction (i.e. look don’t touch the older kernels unless the kernel we’re installing here is present)

Now that doesn’t 100% answer our question because it’s not clear whether we mean ‘is present in the transaction thus allowing successful dependency resolution’ or whether we mean ‘is literally installed at this point in time’. So, question now becomes, what order do transactions take?

Method B: Use debug mode and trust the yum output (More convinced but still not iron clad) We can look to the output of a yum transaction with debug enabled to see some more information around this. We know that yum provides a log of its actions, because if we are doing an update and we ctrl c out of it while it’s installing a specific package we can look to that output to see what was successfully installed and what wasn’t, as well as what we quit out of. We can surmise then that it’s a chronologically accurate log of the actions the program is taking.

Switching debug mode on we can see this:

Running Transaction
  Installing : kernel-2.6.32-696.el6.x86_64                                                                       1/3
  Cleanup    : kernel.x86_64                                                                                      2/3
warning:    erase unlink of /lib/modules/2.6.32-642.15.1.el6.x86_64/modules.order failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-642.15.1.el6.x86_64/modules.networking failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-642.15.1.el6.x86_64/modules.modesetting failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-642.15.1.el6.x86_64/modules.drm failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-642.15.1.el6.x86_64/modules.block failed: No such file or directory
  Cleanup    : kernel.x86_64                                                                                      3/3
warning:    erase unlink of /lib/modules/2.6.32-696.1.1.el6.x86_64/modules.order failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-696.1.1.el6.x86_64/modules.networking failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-696.1.1.el6.x86_64/modules.modesetting failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-696.1.1.el6.x86_64/modules.drm failed: No such file or directory
warning:    erase unlink of /lib/modules/2.6.32-696.1.1.el6.x86_64/modules.block failed: No such file or directory
/var/cache/yum/x86_64/6/base/packages/kernel-2.6.32-696.el6.x86_64.rpm removed
  Verifying  : kernel-2.6.32-696.el6.x86_64                                                                       1/3
  Verifying  : kernel-2.6.32-696.1.1.el6.x86_64                                                                   2/3
  Verifying  : kernel-2.6.32-642.15.1.el6.x86_64                                                                  3/3
VerifyTransaction time: 2.277
Transaction time: 67.796

Removed:
  kernel.x86_64 0:2.6.32-642.15.1.el6                        kernel.x86_64 0:2.6.32-696.1.1.el6

Installed:
  kernel.x86_64 0:2.6.32-696.el6

Looking at this it you can see then basically that it’s first installing, and then cleaning up (removing the older kernels to bring us in line with the limit set in yum.conf)

Method C: Test it! (Can’t get more definitive than this!)

We could also prove this by testing it on a box, first let’s arbitrarily reduce the space on the /boot partition. I did this with fallocate, fallocate -l xxxM yum-test-file where the value was something that would bring us to only having around 20MB of space free on the fs. I then set the number of kernels to be retained down to 2, so theoretically, if it erased prior to the kernel installation process there should have been space free to unpack the kernel. As we can see this fails out:

[root@localhost boot]# yum install kernel-3.10.0-514.16.1.el7 --disableexcludes=all
Loaded plugins: fastestmirror
base                                                                                           | 3.6 kB  00:00:00
epel/x86_64/metalink                                                                           |  22 kB  00:00:00
epel                                                                                           | 4.3 kB  00:00:00
extras                                                                                         | 3.4 kB  00:00:00
updates                                                                                        | 3.4 kB  00:00:00
Loading mirror speeds from cached hostfile
 * base: centos.serverspace.co.uk
 * epel: www.mirrorservice.org
 * extras: centos.serverspace.co.uk
 * updates: mirror.sov.uk.goscomb.net
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-514.16.1.el7 will be installed
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-327.el7 will be erased
---> Package kernel.x86_64 0:3.10.0-327.36.3.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================
 Package                Arch                   Version                                Repository                 Size
======================================================================================================================
Installing:
 kernel                 x86_64                 3.10.0-514.16.1.el7                    updates                    37 M
Removing:
 kernel                 x86_64                 3.10.0-327.el7                         @anaconda                 136 M
 kernel                 x86_64                 3.10.0-327.36.3.el7                    @updates                  136 M

Transaction Summary
======================================================================================================================
Install  1 Package
Remove   2 Packages

Total size: 37 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  installing package kernel-3.10.0-514.16.1.el7.x86_64 needs 15MB on the /boot filesystem

Error Summary
-------------
Disk Requirements:
  At least 15MB more space needed on the /boot filesystem.

So, there’s our answer. Basically the documentation for this sucks but we can pretty much trust best practice most of the time, and there’s a couple of different ways we can double check our hunch is right.

To be honest, I was hoping to hero level this and read the python code for yum and be like ‘well clearly in line blah blah blah we can see it does this’ but it turns out yum is complicated and my python is poor at best! Luckily I think doing this just by observing yum’s behaviour and running a few tests still gives us a definitive answer. Now, as for whether it behaves in the same way for regular non-kernel packages, that’s another question. (To which I suspected the answer is yes, but processes transactions in little chunks since the source code indicates its using something called transaction groups, so that it’s only ever doubling up the necessary disk space for a subset of packages in the total transaction).

In the end after forcing my much better at python friend to read through a lot of the yum and rpm source he found this in the rpm source code:

/** \ingroup rpmts
* Determine package order in a transaction set according to dependencies.
*
* Order packages, returning error if circular dependencies cannot be
* eliminated by removing Requires's from the loop(s). Only dependencies from
* added or removed packages are used to determine ordering using a
* topological sort (Knuth vol. 1, p. 262). Use rpmtsCheck() to verify
* that all dependencies can be resolved.
*
* The final order ends up as installed packages followed by removed packages,
* with packages removed for upgrades immediately following the new package
* to be installed.
*
* @param ts            transaction set
* @return              no. of (added) packages that could not be ordered
*/
int rpmtsOrder(rpmts ts);

So I think that’s case shut on this one. We install and then we remove, and the grouping of packages provided by yum’s sorting of dependencies provides us the ability to do this without doubling the disk space we’re consuming during the time it takes for the various stages of the transaction to complete.

Not too bad for a Friday.