Tales from the Gryphon

Recently, I have made fairly major changes to kernel-package, and there were some reports that I had managed to mess up cross compilation. And, not having a cross-compilation tool chain handy, I had to depend on the kindness of strangers to address that issue. And, given that I am much less personable than Ms Vivien Leigh, this is not something I particularly look forward to repeating.

At the onset, building a cross compiling tool chain seems a daunting task. This is not an activity one does frequently, and so one may be pardoned for being non-plussed by this. However, I have done this before, the most recent effort being creating one to compile rockbox binaries, so I had some idea where to start. Of course, since it is usually years between attempts to create cross-compiling tool chains, I generally forget how it is all done, and have to go hunting for details. Thank god for google.

Well, I am not the only one in the same pickle, apparently, for there are gobs of articles and HOWTOs out there, including some pretty comprehensive (and intimidating) general tool sets to designed to create cross compilers in the most generic fashion possible. Using them was not really an option, since I would forget how to drive them in a few months, and have a miniature version of the current problem again. Also, you know, I don’t feel comfortable using scripts that are too complex for me to understand – I mean, without understanding, how can there be trust?

Also, this time around, I could not decide whether to cross compile for arm-elf, as I did the last time, or for the newfangled armel target. A need for quickly changing the target for the cross compiler build mechanism would be nice. Manually building the tool chain makes a wrong decision here expensive, and I hate that. I am also getting fed up with having to root around on the internet every time I wanted to build a cross compiler. I came across a script by Uwe Hermann, which started me down the path of creating a script, with a help option, to store the instructions, without trying to be too general and thus getting overly complex. However, Uwe’s script hard coded too many things like version numbers and upstream source locations, and I know I would rapidly find updating the script irritating. Using Debian source packages would fix both of these issues.

I also wanted to use Debian sources as far as I could, to ensure that my cross compiler was as compatible as I could make it, though I did want to use newlib (I don’t know why, except that I can, and the docs sound cool). And of course the script should have a help option and do proper command line parsing, so that editing the script would be unnecessary.

Anyway, all this effort culminated in the following script: build cross toolchain, surprisingly compact. So I am now all set to try and cross compile a kernel the next time a kernel-package bug comes around. I thought that I would share this with the lazy web, while I was at it.

Enjoy.

The next thing, of course, is to get my script to create a qemu base image every week so I can move from user mode Linux to the much more nifty kvm, which is what all the cool kids use. And then I can even create an arm virtual machine to test my kernels with, something that user mode linux can’t easily do.

With tonight’s upload of kernel-package, the recent flurry of activity on this package (8 uploads in 6 days) is drawing to a close. I think most of the functionality I started to put into place is now in place, and all reported regressions and bugs in the new 12.XX version have been fixed. The only known deficiency is in the support of Xen dom0 images, and for that I am waiting for kernel version 2.6.30, where Linus has reportedly incorporated Xen patches. In the meanwhile, kernel-package seems to be working well, and I am turning my attention to other things.

But, before I go, here is another example kernel postinst hook script (which, BTW, looks way better with syntax highlighting CSS on my blog than it does in a rss feed or an aggregator site).

    1  #! /bin/sh
    2  
    3  set -e
    4  
    5  if [ -n "$INITRD" ] && [ "$INITRD" = 'No' ]; then
    6      exit 0
    7  fi
    8  version="$1"
    9  vmlinuz_location="$2"
   10  
   11  
   12  if [ -n "$DEB_MAINT_PARAMS" ]; then
   13      eval set -- "$DEB_MAINT_PARAMS"
   14      if [ -z "$1" ] || [ "$1" != "configure" ]; then
   15          exit 0;
   16      fi
   17  fi
   18  
   19  # passing the kernel version is required
   20  [ -z "$version" ] && exit 1
   21  
   22  if [  -n "$vmlinuz_location" ]; then
   23      # Where is the image located? We'll place the initrd there.
   24      boot=$(dirname "$vmlinuz_location")
   25      bootarg="-b $boot"
   26  fi
   27  
   28  # Update the initramfs
   29  update-initramfs -c -t -k "$version" $bootarg
   30  
   31  exit 0

With the new kernel-package hitting Sid today, and the fact that it no longer does symlink handling by default, I thought it was time that we had an example script that shows how to do that. This is a fairly full featured script, feel free to cull down to use just what you want.

I’ll post a couple of ther scripts, if there is interest in this. BTW, this script does far more than the old kernel-package postisnt script ever did.

Have fun.

    1  #!/bin/sh -
    2  #                               -*- Mode: Sh -*- 
    3  # 
    4  # This is an example of a script that can be run as a postinst hook,
    5  # and manages the symbolic links in a manner similar to the kernel
    6  # image default behaviour, except that the latest two versions (as
    7  # determined by ls -lct) are kept. You can modify this script 
    8  # 
    9  # Copyright 2003, 2004, 2005, 2006, 2007, 2008, 2009 Manoj Srivastava
   10  # Copyright 2009 Darren Salt
   11  
   12  set -e
   13  
   14  # The dir where symlinks are managed
   15  SYMLINKDIR=/
   16  
   17  if [ $# -ne 2 ]; then
   18      echo Usage: $0 version location
   19      exit 2
   20  fi
   21  
   22  version="$1"
   23  vmlinuz_location="$2"
   24  vmlinuz_dir="$(dirname "$2")"
   25  
   26  cd $SYMLINKDIR || exit 1
   27  
   28  if [ -n "$DEB_MAINT_PARAMS" ]; then
   29      eval set -- "$DEB_MAINT_PARAMS"
   30  fi
   31  
   32  if [ -z "$1" ] || [ "$1" != "configure" ]; then
   33      exit 0;
   34  fi
   35  
   36  rm -f vmlinuz vmlinuz.old vmlinuz-rd vmlinuz-rd.old initrd.img initrd.img.old 
   37  
   38  # Create a temporary file safely
   39  if [ -x /bin/tempfile ]; then
   40      outfile=$(tempfile -p outp -m 0600);
   41  else
   42      set -e
   43      mkdir /tmp/kernel-image-$version-$$
   44      outfile=/tmp/kernel-image-$version-$$/output
   45  fi
   46  
   47  (cd "$vmlinuz_dir" && ls -ct vmlinuz-*) > $outfile
   48  
   49  STD="$(head -n 1 $outfile |             sed 's/vmlinuz-//')" 
   50  OLD="$(head -n 2 $outfile | tail -n 1 | sed 's/vmlinuz-//')" 
   51  
   52  if [ "X$STD" = "X" ]; then
   53      exit 0;
   54  fi
   55  
   56  # If you want version-specific links, here's how to start
   57  STD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true
   58  OLD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true
   59  
   60  STD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true
   61  OLD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true
   62  
   63  echo Booting $STD, old is $OLD
   64  
   65  if [ -f "$vmlinuz_dir/"initrd.img-$STD ] ; then 
   66     ln -s "$vmlinuz_dir/"initrd.img-$STD initrd.img
   67     ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz-rd
   68  else
   69     ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz
   70  fi
   71  
   72  if [ "X$OLD" != "X" ]; then
   73      if [ -f "$vmlinuz_dir/"initrd.img-$OLD ] ; then
   74      ln -s "$vmlinuz_dir/"initrd.img-$OLD initrd.img.old
   75      ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz-rd.old
   76      else
   77      ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz.old
   78      fi
   79  fi
   80  
   81  # if [ "X$STD24" != "X" ]; then
   82  #     if [ -f "$vmlinuz_dir/"initrd.img-$STD24 ] ; then 
   83  #     ln -s "$vmlinuz_dir/"initrd.img-$STD24 initrd24.img
   84  #     ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24-rd
   85  #     else
   86  #     ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24
   87  #     fi
   88  # fi
   89  # if [ "X$OLD24" != "X" ]; then
   90  #     if [ -f "$vmlinuz_dir/"initrd.img-$OLD24 ] ; then
   91  #     ln -s "$vmlinuz_dir/"initrd.img-$OLD24 initrd24.img.old
   92  #     ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24-rd.old
   93  #     else
   94  #     ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24.old
   95  #     fi
   96  # fi
   97  
   98  # Run boot loaders here.
   99  #lilo
  100  
  101  rm -f $outfile 
  102  if [ -d /tmp/kernel-image-$version-$$ ]; then
  103      rmdir /tmp/kernel-image-$version-$$
  104  fi
  105  
  106  exit 0

A new version of kernel-package in Incoming at the time of writing adds support for creating a package which contains the Linux kernel debug image. This means the debugging information for the modules in the kernel image package, and the uncompressed vmlinux image. This builds on suggestions and code from Troy Heber, Theodore Y. Ts’o, and Dann Frazier.

As support for kexec/kdump support becomes more real, it will be very useful to be able to build kernels that have debugging information available, but not necessarily to install the debugging information on every single client system.

The .ko object files are stripped of the debugging information before they are placed in the standard linux-image installation package. However, before that, a copy of the vmlinux and unstripped .ko files are saved in a “debuginfo” package, and everything except for the debugging information is removed from them using objcopy --keep-only-debug. This means that if someone ends up with a crash dump, they can send it to a support engineer and only the support engineer needs to install the debuginfo package and use the “crash” utility to debug the crash dump. It’s also useful for developers, since the debuginfo information can be stored somewhere outside of /lib for storing its debug information, for example. This is useful for keeping the size of the root partition small, for those who keep / separate from /usr.

The locations used are compatible also with SystemTap, which provides free software infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.

Also, I had to clear out some FUD about kernel-package from the ircbot dpkg on the OFTC Debian IRC channel, since someone had implied that kernel-package was some how obsolete. As can be seen, it is being actively developed, and features are being added apace.

A few hours ago, a new version of kernel-package was uploaded to experimental. This is a major change,and I would appreciate it if folks took it out for a spin, kicked the tires, and provide feedback about where this version is lacking.

This is only part of the way along in this development cycle. I would like to add a debug-info separation, either in a different directory than / in the image packages, or a separate package by itself. I would also like to create an overlay directory for /usr/share/kernel-package/, so people can inject code or override the defaults for kernel-package easily. I am also willing to make any changes to standardize the handling of hook scripts for kernel packages in Debian.

I have been thinking some more about how to improve ucf. One of the things that struck me was that based on my earlier analysis there are only five actions that ucf can take, and the decision about the actions depends on the state it finds the configuration file in on the target machine, and there are only eight of those. Now, thinking back to my days as a VLSI designer back in the halcyon days of electrical engineering, This is a pretty simple state machine. It is not as neat as it could be (where just three variables would be needed to keep track of things, but still, it bears investigation. This would be a way for converting the current procedural ucf into a functional programming model.

Hop over here for a look at how that went — it was fun, and afforded me an opportunity to demonstrate how well org handles snippets.

There has been some discussion on the Debian development mailing list about adding hooks into ucf, to allow people to do things like committing files into different SCM branches. So, I thought I would help people out by letting them tell me where hooks would be useful, and so decided to do an activity diagram for ucf. Gawd, what a mess. I mean, I wrote this thing, and it boggles even my mind. See the figure for how horrendous code can get when it grows organically.

So, I decided to re-factor/redesign ucf, see if I could create a less complex activity diagram. On analysis, it turns out that ucf has just five actions it may perform, and which action it takes depends on which of eight possible states the configuration file is in.

I have long been interested in moving the debian-policy package away from CVS, but had never quite managed to gather enough motivation to do the switch. Debian policy has long had an Alioth project, but I finally managed to file a support ticket, and nag Wichert into creating the arch project for policy.

The first step was to convert the CVS version into an arch repository, and this is where cscvs comes in. cscvs is nice, but I am told that Canonical has a better, private version, and helpful folks there offered to do a managed conversion to arch for me using these cutting edge tools. I declined, being persnickety enough to want to convert Debian technical policy using tools in Debian itself. And, apart from two change-sets (numbers 117 and 125), cscvs managed to do the conversion to arch (the bazaar flavour) nicely on its own (well, after a few false starts as I climbed the learning curve). Came to 283 change-sets. Have a look at my Debian policy branch. It can be registered at http://arch.debian.org/arch/private/srivasta/archive-etch/.

The next step was to create a baz archive on arch.debian.org, and use Clint’s ACL recipe to allow people in the dbnpolicy group to have write access. And then, since I wanted this branch to also have the full set of distinct patch logs, I cycled through all 286 patches in my local branch, replayed and committed them into the remote branch one by one. You may browse the public, or release, branch as well. This version can be registered at http://arch.debian.org/arch/dbnpolicy/etch/.

I have been thinking about informally going through NM (perhaps with an AM who wants some practice) for a while now (and have been procrastinating about it like many real-life DD candidates :-). I have planned on using Devotee (DEbian VOTe EnginE) as the package I use to get through NM. So, pursuant to packaging Devotee, I am trying to add a test suite to the package. In order to run a decent test, I need a set of ballots — signed ballots — at least some of which are signed by keys in the Debian key-ring. I have put together a dummy vote on my own box, with the topic being voted upon being What is your favorite color of the rainbow?. Here is the ballot. Please note that any emailed ballot is likely to end up in a GPL’d package, open for all the world to see, so please consider that before voting. Oh, and the interesting stats are here.

So, why am I embarking on this strange course of going through NM even though I am already a Debian developer? Well, I’ve been in the project since well before there was a new maintainer process. I just felt like contributing, and, one day, just emailed the devel list, and I was in. No checks, no key signatures required (not even a key was needed, unless my memory is failing me). I have a theoretical knowledge of what NM entails, but no real gut feeling for what the various stages feel like, nor how hard they are to get through.

I have always wanted to contribute to the NM effort, but I have never felt comfortable with trying to change — or even participate as an account manager — with so little understanding of what an applicant is going through.

One of the concerns with me trying to go through NM is that I would be taking valuable resources away from the list of real candidates who are stuck in the queue, and mostly for my own selfish reasons. However, I have been assured by current AM’s that adding another candidate to their list would not significantly alter their workload, so I have decided to proceed.

So please, do help develop a test suite for Devotee by voting for your favorite color. Pretty soon I shall be looking for a sponsor for Devotee, and an advocate.