Posted in the wee hours of Monday night, May 5th, 2009
License: GPL
Posted in the wee hours of Monday night, May 5th, 2009
License: GPL
So, recently our email overlords graciously provided means for us minions to help them in their toils and help clean up the spammish clutter in the mailing lists by helping report the spam. And the provided us with a dead simple means of reporting such spam to them. Now, us folks who knoweth that there is but one editor, the true editor, and its, err, proponent is RMS, use Gnus to follow the emacs mailing lists, either directly, or through gmane. There are plenty of examples out there showing how to automate reporting spam to gmane, so I won’t bore y’all with the details. Here I only show how one serves our list overlords, and smite the spam at the same time.
Some background, from the Gnus info page. I’ll try to keep it brief. There is far more functionality present if you read the documentation, but you can see that for yourself.
The Spam package provides Gnus with a centralized mechanism for detecting and filtering spam. It filters new mail, and processes messages according to whether they are spam or ham. There are two “contact points” between the Spam package and the rest of Gnus: checking new mail for spam, and leaving a group.
Checking new mail for spam is done in one of two ways: while splitting incoming mail, or when you enter a group. Identifying spam messages is only half of the Spam package’s job. The second half comes into play whenever you exit a group buffer. At this point, the Spam package does several things: it can add the contents of the ham or spam message to the dictionary of the filtering software, and it can report mail to various places using different protocols.
All this is very plugin and modular. The advantage is, that you can use various plugin front ends to identify spam and ham, or mark messages as you go through a group, and when you exit the group, spam is reported, ham and spam messages are copied to special destinations for future training of your filter. Since you inspect the marks put into the group buffer as you read the messages, there is a human involved in the processing, but as much as possible can be automated away. Do read the info page on the Spam package in Gnus, it is edifying.
Anyway, here is a snippet from my
etc/emacs/news/gnusrc.el file, which can help automate
the tedium of reporting spam. This is perhaps more like how Gnus
does things than having to press a special key for every spam, and
which does nothing to help train your filter.
1 (add-to-list 2 'gnus-parameters 3 '("^nnml:\\(debian-.*\\)$" 4 (to-address . "\\1@lists.debian.org") 5 (to-list . "\\1@lists.debian.org") 6 (admin-address . "\\1-request@lists.debian.org") 7 (spam-autodetect . t) 8 (spam-autodetect-methods spam-use-gmane-xref spam-use-hashcash spam-use-BBDB) 9 (spam-process '(spam spam-use-resend)) 10 (spam-report-resend-to . "report-listspam@lists.debian.org") 11 (subscribed . t) 12 (total-expire . t) 13 ))
Recently, I have made fairly major changes to kernel-package, and there were some reports that I had managed to mess up cross compilation. And, not having a cross-compilation tool chain handy, I had to depend on the kindness of strangers to address that issue. And, given that I am much less personable than Ms Vivien Leigh, this is not something I particularly look forward to repeating.
At the onset, building a cross compiling tool chain seems a daunting task. This is not an activity one does frequently, and so one may be pardoned for being non-plussed by this. However, I have done this before, the most recent effort being creating one to compile rockbox binaries, so I had some idea where to start. Of course, since it is usually years between attempts to create cross-compiling tool chains, I generally forget how it is all done, and have to go hunting for details. Thank god for google.
Well, I am not the only one in the same pickle, apparently, for there are gobs of articles and HOWTOs out there, including some pretty comprehensive (and intimidating) general tool sets to designed to create cross compilers in the most generic fashion possible. Using them was not really an option, since I would forget how to drive them in a few months, and have a miniature version of the current problem again. Also, you know, I don’t feel comfortable using scripts that are too complex for me to understand – I mean, without understanding, how can there be trust?
Also, this time around, I could not decide whether to cross
compile for arm-elf, as I did the last time, or for
the newfangled armel target. A need for quickly
changing the target for the cross compiler build mechanism would be
nice. Manually building the tool chain makes a wrong decision here
expensive, and I hate that. I am also getting
fed up with having to root around on the internet every time I
wanted to build a cross compiler. I came across a script by Uwe
Hermann, which started me down the path of creating a script,
with a help option, to store the instructions, without trying to be
too general and thus getting overly complex. However, Uwe’s
script hard coded too many things like version numbers and upstream
source locations, and I know I would rapidly find updating the
script irritating. Using Debian source packages would fix both of
these issues.
I also wanted to use Debian sources as far as I could, to ensure that my cross compiler was as compatible as I could make it, though I did want to use newlib (I don’t know why, except that I can, and the docs sound cool). And of course the script should have a help option and do proper command line parsing, so that editing the script would be unnecessary.
Anyway, all this effort culminated in the following script: build cross toolchain, surprisingly compact. So I am now all set to try and cross compile a kernel the next time a kernel-package bug comes around. I thought that I would share this with the lazy web, while I was at it.
Enjoy.
The next thing, of course, is to get my script to create a qemu base image every week so I can move from user mode Linux to the much more nifty kvm, which is what all the cool kids use. And then I can even create an arm virtual machine to test my kernels with, something that user mode linux can’t easily do.
Posted Wednesday night, April 22nd, 2009
License: GPL
This is a continuation from before. I am digressing a little in this post. One of the things I want to get out of this exercise is to learn more about Ontologies and Ontology editors, and on the principle that you can never learn something unless you build something with it (aka bone knowledge), so this is gathering my thoughts to get started on creating an Ontology for package building. Perhaps this has been done before, and better, but I’ll probably learn more trying to create my own.
Also, I am playing around with code, an odd melange of my
package building porcelain, and gitpkg, and other
ideas bruited on IRC, and I don’t want to blog about
something that would be embarrassing in the long run if some of the
concepts I have milling around turn out to not meet the challenge
of first contact with reality.
I want to create a ontology related to packaging software. It should be general enough to cater to the needs any packaging effort in a distribution agnostic and version control agnostic manner. It should enable us to talk about packaging schemes and mechanisms, compare different methods, and perhaps to work towards a common interchange mechanism good enough for people to share the efforts spent in packaging software.
The ontology should be able to describe common practices in packaging, concepts of upstream sources, versioning, commits, package versions, and other meta-data related to packages.
I am doing this ontology primarily for myself, but I hope this might be useful for other folks involved in packaging software.
So, here follow a set of concepts related to packaging software, people who like pretty pictures can click on the thumbnail on the right:
Posted Saturday night, April 18th, 2009
License: GPL
This is a continuation from before.
Before I go plunging into writing code for a generic
vcs-pkg implementation, I wanted to take a close look
at my current, working, non-generic implementation: making sure
that the generic implementation can support at least this one
concrete work-flow will keep me grounded.
One of the features of my home grown porcelain for building
package has been that I use a fixed layout for all the packages I
maintain. There is a top level directory for all working trees.
Each package gets a sub-directory under this working area. And in
each package sub-directory, are the upstream versions, the checked
out VCS working directory, and anything else package related. With
this layout, knowing the package name is enough to locate the
working directory. This enable me to, for example, hack away at a
package in Emacs, and when done, go to any open terminal window,
and say stage_release kernel-package or
tag_releases ucf without needing to know what the
current directory is (usually, the packages working directory is
several levels deep —
/usr/local/git/debian/make-dfsg/make-dfsg-3.91, for
instance.
However, this is less palatable for a generic tool – imposing a
directory structure layout is pretty heavy. And I guess I can
always create a function called cdwd, or something, to
take away the tedium of typing out long cd
commands.
Anyway, looking at my code, there is the information that the scripts seem to need in order to do their work.
rpm based sources, look for the
spec filedebian/rulesspec or
debian/rules in the current directory, and parse
either the spec file or
debian/changelog.tla tree-rootbzr infogit rev-parse --show cduphg rootdebian directory, and
changelog and rules files existThen, look for the spec file or
debian/rules in the base directory
spec or changelog files.pristine-tar is in use, given two trees
(branches, commits. etc), namely:
The tree can be generated
tar archive.So, if I do away with the whole working area layout convention, this can be reduced to just requiring the user to:
dpkg-buildpackage imposes this too).pristine-tar or have the upstream
tar archive in the parent directory of the working
directoryHmm. One user specified directory, where the results are dumped.
I can live with that. However, gitpkg has a different
concept: it works purely on the git objects, you feed it upto three
tree objects, the first being the tree with sources to build, and
the second and third trees being looked at only if the upstream tar
archive can not be located, and passes the trees to pristine tar to
re-construct the upstram tar. The package name and version are
constructed after
the source-tar archive is extracted to the staging area. I like the
minimality of this.
This is continued here.
Posted Thursday afternoon, April 16th, 2009
License: GPL
I have been involved in vcs-pkg.org since around
the time it started, a couple of years ago. The discussion has been
interesting, and I learned a lot about the benefits and
disadvantages of serializing patches (and collecting integration
deltas in the feature branches and the specific ordering of
the feature branches) and maintaining integration branches (where
the integration deltas are collected purely in the integration
branch, but might tend to get lost in the history, and a fresh
integration branch having to re-invent the integration deltas
afresh).
However, one of the things we have been lax about is getting
down to brass tacks and getting around to being able to create
generic packaging tools (though for the folks on the serializing
patches side of the debate we have the excellent quilt
and the topgit packages).
I have recently mostly automated my git based work-flow, and
have built fancy porcelain around my git repository setup. During
IRC discussion, the gitpkg script came up. This seems
almost usable, apart from not having any built-in
pristine-tar support, and also not supporting
git submodules, which make is less useful an
alternative than my current porcelain.
But it seems to me that we are pretty close to being able to create a distribution, layout, and patch handler agnostic script that builds distribution packages directly from version control, as long as we take care not to bind people into distributions or tool specific straitjackets. To these ends, I wanted to see what are the tasks that we want a package building script to perform. Here is what I came up with.
The first and third steps above are pretty straight forward, and fairly uncontroversial.
The upstream sources may be handled by one of these three alternatives:
The command to run may be supplied by the user in a
configuration file or option, and may default based on the native
distribution, to dpkg-buildpackage or
rpm. There are a number of already mature mechanisms
to take a source directory and upstream tar archive and produce
packages from that point, and the wheel need not be
re-invented.
So the hardest part of the task is to present, in the staging area, for further processing, a directory tree of the source package, ready for the distribution specific build commands. This part of the solution is likely to be VCS specific.
This post is getting long, so I’ll defer presenting my evolving
implementation of a generic vcs-pkg tool,
git flavour, to the next blog post.
This is continued here.
Posted Wednesday night, April 15th, 2009
License: GPL
There are a lot of little git scripts and tools being written by a lot of people. Including a lot of tools written by people I have a lot of respect for. And yet, they are mostly useless for me. Take git-pkg. Can’t use it. Does not work with git submodules. Then there is our nice, new, shiny, incredibly bodacious “3.0 (git)” source format. Again, useless: does not cater to submodules.
I like submodules. They are nice. They allow for projects to take upstream sources, add Debian packaging instructions, and put them into git. They allow you to stitch together disparate projects, with different authors, and different release schedules and goals, into a coherent, integrated, software project.
Yes, I use git submodules for my Debian packaging. I think it is
conceptually and practically the correct solution. Why submodules?
Well, one of the first things I discovered was that most of the
packaging for my packages was very similar – but not identical.
Unfortunately, the previous incarnation of my packages with a
monolithic rules file in each ./debian/ directory, it
was easy for the rules files in packages to get out of sync – and
there was no easy way to merge changes in the common portions an
any sane automated fashion. The ./debian/ directories
for all my packages package that they are instrumental in
packaging. So, since I make the ./debian/ directories
branches of the same project, it is far easier to package a new
package, or to roll out a new feature when policy changes – the
same commit can be applied across all the branches, and thus all my
source packages, easily. With a separate debian-dir
project, I can separate the management of the packaging rules from
the package code itself.
Also, I have abstracted out the really common bits across all my
packages into a ./debian.common directory, which is
yet another project, and included in as a submodule in all the
packages – so there is a central place to change the common bits,
without having to duplicate my efforts 30-odd times.
Now people are complaining since they have no idea how to clone
my package repositories, since apparently no one actually pays
attention to a file called .gitmodules, and even when
they do, they, and the tools they use, have no clue what to do with
it. I am tired of sending emails with one off-cluebats, and I am
building my own porcelain around something I hope to present as a
generic vcs-pkg implementation soon. The firs step is
a wrapper around git-clone, that understands git
submodules.
So, here is the
browsable code (there is a link in there to the downloadable
sources too). Complete with a built in man page. Takes the same
arguments as git-clone, but with fewer options. Have
fun.
With tonight’s upload of kernel-package, the recent
flurry of activity on this package (8 uploads in 6 days) is drawing
to a close. I think most of the functionality I started to put into
place is now in place, and all reported regressions and bugs in the
new 12.XX version have been fixed. The only known deficiency is in
the support of Xen dom0 images, and for that I am waiting for
kernel version 2.6.30, where Linus has reportedly
incorporated Xen patches. In the meanwhile,
kernel-package seems to be working well, and I am
turning my attention to other things.
But, before I go, here is another example kernel postinst hook script (which, BTW, looks way better with syntax highlighting CSS on my blog than it does in a rss feed or an aggregator site).
1 #! /bin/sh 2 3 set -e 4 5 if [ -n "$INITRD" ] && [ "$INITRD" = 'No' ]; then 6 exit 0 7 fi 8 version="$1" 9 vmlinuz_location="$2" 10 11 12 if [ -n "$DEB_MAINT_PARAMS" ]; then 13 eval set -- "$DEB_MAINT_PARAMS" 14 if [ -z "$1" ] || [ "$1" != "configure" ]; then 15 exit 0; 16 fi 17 fi 18 19 # passing the kernel version is required 20 [ -z "$version" ] && exit 1 21 22 if [ -n "$vmlinuz_location" ]; then 23 # Where is the image located? We'll place the initrd there. 24 boot=$(dirname "$vmlinuz_location") 25 bootarg="-b $boot" 26 fi 27 28 # Update the initramfs 29 update-initramfs -c -t -k "$version" $bootarg 30 31 exit 0
With the new kernel-package hitting Sid today, and the fact that it no longer does symlink handling by default, I thought it was time that we had an example script that shows how to do that. This is a fairly full featured script, feel free to cull down to use just what you want.
I’ll post a couple of ther scripts, if there is interest in
this. BTW, this script does far more than the old
kernel-package postisnt script ever
did.
Have fun.
1 #!/bin/sh - 2 # -*- Mode: Sh -*- 3 # 4 # This is an example of a script that can be run as a postinst hook, 5 # and manages the symbolic links in a manner similar to the kernel 6 # image default behaviour, except that the latest two versions (as 7 # determined by ls -lct) are kept. You can modify this script 8 # 9 # Copyright 2003, 2004, 2005, 2006, 2007, 2008, 2009 Manoj Srivastava 10 # Copyright 2009 Darren Salt 11 12 set -e 13 14 # The dir where symlinks are managed 15 SYMLINKDIR=/ 16 17 if [ $# -ne 2 ]; then 18 echo Usage: $0 version location 19 exit 2 20 fi 21 22 version="$1" 23 vmlinuz_location="$2" 24 vmlinuz_dir="$(dirname "$2")" 25 26 cd $SYMLINKDIR || exit 1 27 28 if [ -n "$DEB_MAINT_PARAMS" ]; then 29 eval set -- "$DEB_MAINT_PARAMS" 30 fi 31 32 if [ -z "$1" ] || [ "$1" != "configure" ]; then 33 exit 0; 34 fi 35 36 rm -f vmlinuz vmlinuz.old vmlinuz-rd vmlinuz-rd.old initrd.img initrd.img.old 37 38 # Create a temporary file safely 39 if [ -x /bin/tempfile ]; then 40 outfile=$(tempfile -p outp -m 0600); 41 else 42 set -e 43 mkdir /tmp/kernel-image-$version-$$ 44 outfile=/tmp/kernel-image-$version-$$/output 45 fi 46 47 (cd "$vmlinuz_dir" && ls -ct vmlinuz-*) > $outfile 48 49 STD="$(head -n 1 $outfile | sed 's/vmlinuz-//')" 50 OLD="$(head -n 2 $outfile | tail -n 1 | sed 's/vmlinuz-//')" 51 52 if [ "X$STD" = "X" ]; then 53 exit 0; 54 fi 55 56 # If you want version-specific links, here's how to start 57 STD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true 58 OLD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true 59 60 STD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true 61 OLD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true 62 63 echo Booting $STD, old is $OLD 64 65 if [ -f "$vmlinuz_dir/"initrd.img-$STD ] ; then 66 ln -s "$vmlinuz_dir/"initrd.img-$STD initrd.img 67 ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz-rd 68 else 69 ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz 70 fi 71 72 if [ "X$OLD" != "X" ]; then 73 if [ -f "$vmlinuz_dir/"initrd.img-$OLD ] ; then 74 ln -s "$vmlinuz_dir/"initrd.img-$OLD initrd.img.old 75 ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz-rd.old 76 else 77 ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz.old 78 fi 79 fi 80 81 # if [ "X$STD24" != "X" ]; then 82 # if [ -f "$vmlinuz_dir/"initrd.img-$STD24 ] ; then 83 # ln -s "$vmlinuz_dir/"initrd.img-$STD24 initrd24.img 84 # ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24-rd 85 # else 86 # ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24 87 # fi 88 # fi 89 # if [ "X$OLD24" != "X" ]; then 90 # if [ -f "$vmlinuz_dir/"initrd.img-$OLD24 ] ; then 91 # ln -s "$vmlinuz_dir/"initrd.img-$OLD24 initrd24.img.old 92 # ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24-rd.old 93 # else 94 # ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24.old 95 # fi 96 # fi 97 98 # Run boot loaders here. 99 #lilo 100 101 rm -f $outfile 102 if [ -d /tmp/kernel-image-$version-$$ ]; then 103 rmdir /tmp/kernel-image-$version-$$ 104 fi 105 106 exit 0
Posted late Thursday afternoon, April 9th, 2009
License: GPL
A new version of kernel-package in Incoming at the
time of writing adds support for creating a package which contains
the Linux kernel debug image. This means the debugging information
for the modules in the kernel image package, and the uncompressed
vmlinux image. This builds on suggestions and code from Troy Heber,
Theodore Y. Ts’o, and Dann Frazier.
As support for kexec/kdump support becomes more real, it will be very useful to be able to build kernels that have debugging information available, but not necessarily to install the debugging information on every single client system.
The .ko object files are stripped of the debugging
information before they are placed in the standard
linux-image installation package. However, before that, a
copy of the vmlinux and unstripped .ko files are saved
in a “debuginfo” package, and everything except for the
debugging information is removed from them using objcopy
--keep-only-debug. This means that if someone ends up with a
crash dump, they can send it to a support engineer and only the
support engineer needs to install the debuginfo package and use the
“crash” utility to debug the crash dump. It’s also useful
for developers, since the debuginfo information can be stored
somewhere outside of /lib for storing its debug
information, for example. This is useful for keeping the size of
the root partition small, for those who keep /
separate from /usr.
The locations used are compatible also with SystemTap, which provides free software infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.
Also, I had to clear out some FUD about kernel-package from the
ircbot dpkg on the OFTC Debian IRC channel, since
someone had implied that kernel-package was some how
obsolete. As can be seen, it is being actively developed, and
features are being added apace.
A few hours ago, a new version of kernel-package was uploaded to
experimental. This is a major change,and I would
appreciate it if folks took it out for a spin, kicked the tires,
and provide feedback about where this version is lacking.
This is only part of the way along in this development cycle. I
would like to add a debug-info separation, either in a different
directory than / in the image packages, or a separate package by
itself. I would also like to create an overlay directory for
/usr/share/kernel-package/, so people can inject code
or override the defaults for kernel-package easily. I am also
willing to make any changes to standardize the handling of hook
scripts for kernel packages in Debian.
./debian/ is ephemeralmake-kpkg removes and re-creates
./debian on every invocation. This started as an
exercise to protect ourselves from the upstream builddep
functionality, that randomly cleaned out ./debian
whether or not it had created it, effectively making it impossible
to run dpkg-buildpackage easily (which is ok, if all
you care about is the image package)
This does make the kernel-package far more nimble;
we now offer less surprise to users who did not expect stampts that
the kernel-packagge used to not do duplicate work. Now, if you edit
a couple of files in the kernel source, and run
make-kpkg, the kernel will build as expected. There
are no more “version mismatch” errors, and the kernel version can
be modified using localconfig as one desires. With
this, kernel-package can routinely be used to build kernels out of
the git tree.
The con is that we no longer cater to official kernels, or to
anyone who expected content in ./debian to persist. At some point,
there are plans to implement an overlay directory that will shadow
/usr/share/kernel-package/ruleset, but that is not yet
implemented. In any case, the kernel team in Debian regards
kernel-package to be broken, and have been bad
mouthing it and deprecating it for a few years now, so this will
not be a loss for them.
The patch the kernel facility was adding complexity, and failing
to provide the flexibility required for a generic patching
facility. It used to be useful at one point, but in the modern
parlance, witht he widespread use of distribute version control
systems, and various facilities to manage source and patch them,
the built in version was clunky. This means the
--added-patches option of make-kpkg is
gone, the work-around is to prepare the kernel sources before calling
make-kpkg.
For the longest tine (well, ever since Herbert Xu too over
building kernel images from me), kernel-package has
carried specal case code for official images. This has caused some
problems, recently, since the need to preserve
./debian has caused no end of problems when the
version changed out from under ./debian, or when
people wanted to edit a file and expected kernel-package to do a
minimal recompile.
However, sometime in the Etch release cycle, the kernel team
deprecated kernel-package as the means of building
official kernels. They have recently started saying they think
kernel-package is broken, and have their own
recommendation for how to build kernel packages. Therefore, a full
release cycle later, we can get rid of the special case rules used
for official packages. Also, this allows us to drop
./debian at the drop of a hat, and recreate it with an
version that reflects the current state of the kernel sources.
/usr/srcInstead, ship an example shell script that replicates the old behaviour. This script can then be deployed on the target machines, and could be a part of a locally created kernel configuration package, if one needs to deploy the same behavior across a cluster of machines.
This is a shift from previous behaviour. Any symbolic link
manipulation must now be done with hook scripts in
/etc/kernel/*.d directories.
Firstly, modern boot loaders scan the boot directory for kernel images, and the user no longer has to code in the path to the symbolic links that the kernel image package used to manipulate.
Secondly, hardcoding the behaviour into the postinst made for a very rigid policy; and user wanted more flexibility than that. There is an example shipped with the package that shows a more flexible scheme that kept two symbolic links for version 2.4 kernels, and two symbolic links for 2.6 kernels; it can be easily modified to keep two links for 2.9 kernels and two links for 2.8 kernels, or one of each, or whatever the user wants.
Please note that this was already the case for grub, one of the more popular boot loaders.
Now that we have a mechanism for running arbitrary scripts when the image packages are manipulated, we can stop embedding the boot loader actions in the package itself. This means that lilo, elilo, etc will no longer be run directly by the post install script, and all the code related to detecting the boot loader, managing the configuration, and adding bits about bootloader documentation is all removed from the postinst. This allows the image package to be more flexible, since the end user is no longer restricted to the actions encoded in the image package. This is a fairly large change.
It also opens the door for the user to easily use non-standard bootloaders, if they so desire.
Instead, there are example scripts provided that will perform the task. These scripts will work for official kernel images as well.
The initramfs scripts provided work with the
make-kpkg images as well as the official images, and
are thus better than the script shipped with
initramfs-tools themselves, as they offer a super set
of functionality.
This also demonstrates how the posts install script communicates with the initramfs creation scripts so that no initramfs is generated in case you do not want it.