Tales from the Gryphon/ archives/

Tales from the Gryphon

Archives for 2009

Manoj's hackergotchi
Add a new post titled:
Tuesday 05 May
2009
Link: Debian list spam reporting the Gnus way

Posted early Tuesday morning, May 5th, 2009

Debian list spam reporting the Gnus way

#+TITLE: Debian spam reporting the Gnus way #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ So, recently our email overlords graciously provided means for us minions to help them in their toils and help clean up the spammish clutter in the mailing lists by helping report the spam. And the provided us with a dead simple means of reporting such spam to them. Now, us folks who knoweth that there is but one editor, the true editor, and its, err, proponent is RMS, use Gnus to follow the emacs mailing lists, either directly, or through [[http://www.gmane.org][gmane]]. There are plenty of examples out there showing how to automate reporting spam to /gmane/, so I won't bore y'all with the details. Here I only show how one serves our list overlords, and smite the spam at the same time. Some background, from the Gnus info page. I'll try to keep it brief. There is far more functionality present if you read the documentation, but you can see that for yourself. #+BEGIN_QUOTE The Spam package provides Gnus with a centralized mechanism for detecting and filtering spam. It filters new mail, and processes messages according to whether they are spam or ham. There are two "contact points" between the Spam package and the rest of Gnus: checking new mail for spam, and leaving a group. Checking new mail for spam is done in one of two ways: while splitting incoming mail, or when you enter a group. Identifying spam messages is only half of the Spam package's job. The second half comes into play whenever you exit a group buffer. At this point, the Spam package does several things: it can add the contents of the ham or spam message to the dictionary of the filtering software, and it can report mail to various places using different protocols. #+END_QUOTE All this is very plugin and modular. The advantage is, that you can use various plugin front ends to identify spam and ham, or mark messages as you go through a group, and when you exit the group, spam is reported, ham and spam messages are copied to special destinations for future training of your filter. Since you inspect the marks put into the group buffer as you read the messages, there is a human involved in the processing, but as much as possible can be automated away. /Do/ read the info page on the Spam package in Gnus, it is edifying. Anyway, here is a snippet from my ~etc/emacs/news/gnusrc.el~ file, which can help automate the tedium of reporting spam. This is perhaps more like how Gnus does things than having to press a special key for every spam, and which does nothing to help train your filter. #+BEGIN_HTML [[!syntax language="Common Lisp" linenumbers=1 bars=1 text=""" (add-to-list 'gnus-parameters '("^nnml:\\(debian-.*\\)$" (to-address . "\\1@lists.debian.org") (to-list . "\\1@lists.debian.org") (admin-address . "\\1-request@lists.debian.org") (spam-autodetect . t) (spam-autodetect-methods spam-use-gmane-xref spam-use-hashcash spam-use-BBDB) (spam-process '(spam spam-use-resend)) (spam-report-resend-to . "report-listspam@lists.debian.org") (subscribed . t) (total-expire . t) )) """]] #+END_HTML

Manoj

Monday 04 May
2009
Link: Reflections on streetcars

Posted terribly early Monday morning, May 4th, 2009

Reflections on streetcars

#+TITLE: Reflections on streetcars #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ Recently, I have made fairly major changes to #+BEGIN_HTML kernel-package, #+END_HTML and there were some reports that I had managed to mess up cross compilation. And, not having a cross-compilation tool chain handy, I had to depend on the kindness of strangers to address that issue. And, given that I am much less personable than Ms Vivien Leigh, this is not something I particularly look forward to repeating. At the onset, building a cross compiling tool chain seems a daunting task. This is not an activity one does frequently, and so one may be pardoned for being non-plussed by this. However, I have done this before, the most recent effort being creating one to compile [[http://www.rockbox.org][rockbox]] binaries, so I had some idea where to start. Of course, since it is usually years between attempts to create cross-compiling tool chains, I generally forget how it is all done, and have to go hunting for details. Thank god for google. Well, I am not the only one in the same pickle, apparently, for there are gobs of articles and HOWTOs out there, including some pretty comprehensive (and intimidating) general tool sets to designed to create cross compilers in the most generic fashion possible. Using them was not really an option, since I would forget how to drive them in a few months, and have a miniature version of the current problem again. Also, you know, I don't feel comfortable using scripts that are too complex for me to understand -- I mean, without understanding, how can there be trust? Also, this time around, I could not decide whether to cross compile for ~arm-elf~, as I did the last time, or for the newfangled ~armel~ target. A need for quickly changing the target for the cross compiler build mechanism would be nice. Manually building the tool chain makes a wrong decision here expensive, and I _hate_ that. I am also getting fed up with having to root around on the internet every time I wanted to build a cross compiler. I came across a script by *Uwe Hermann*, which started me down the path of creating a script, with a help option, to store the instructions, without trying to be /too/ general and thus getting overly complex. However, Uwe's script hard coded too many things like version numbers and upstream source locations, and I know I would rapidly find updating the script irritating. Using Debian source packages would fix both of these issues. I also wanted to use Debian sources as far as I could, to ensure that my cross compiler was as compatible as I could make it, though I did want to use [[http://sources.redhat.com/ml/newlib/][newlib]] (I don't know why, except that I can, and the docs sound cool). And of course the script should have a help option and do proper command line parsing, so that editing the script would be unnecessary. Anyway, all this effort culminated in the following script: #+BEGIN_HTML build cross toolchain, #+END_HTML surprisingly compact. So I am now all set to try and cross compile a kernel the next time a /kernel-package/ bug comes around. I thought that I would share this with the lazy web, while I was at it. Enjoy. The next thing, of course, is to get my script to create a #+BEGIN_HTML qemu #+END_HTML base image every week so I can move from user mode Linux to the much more nifty #+BEGIN_HTML kvm, #+END_HTML which is what all the cool kids use. And then I can even create an /arm/ virtual machine to test my kernels with, something that user mode linux can't easily do.

Manoj

Thursday 23 April
2009

Ontologies: Towards a generic, distribution agnostic tool for building packages from a VCS

#+TITLE: Ontologies: Towards a generic, distribution agnostic tool for building packages from a VCS #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ This is a continuation from #+BEGIN_HTML before. #+END_HTML I am digressing a little in this post. One of the things I want to get out of this exercise is to learn more about Ontologies and Ontology editors, and on the principle that you can never learn something unless you build something with it (aka bone knowledge), so this is gathering my thoughts to get started on creating an Ontology for package building. Perhaps this has been done before, and better, but I'll probably learn more trying to create my own. Also, I am playing around with code, an odd melange of my package building porcelain, and ~gitpkg~, and other ideas bruited on =IRC=, and I don't want to blog about something that would be embarrassing in the long run if some of the concepts I have milling around turn out to not meet the challenge of first contact with reality. I want to create a ontology related to packaging software. It should be general enough to cater to the needs any packaging effort in a distribution agnostic and version control agnostic manner. It should enable us to talk about packaging schemes and mechanisms, compare different methods, and perhaps to work towards a common interchange mechanism good enough for people to share the efforts spent in packaging software. The ontology should be able to describe common practices in packaging, concepts of upstream sources, versioning, commits, package versions, and other meta-data related to packages. #+BEGIN_HTML vcs-pkg concept diagram #+END_HTML I am doing this ontology primarily for myself, but I hope this might be useful for other folks involved in packaging software. So, here follow a set of concepts related to packaging software, people who like pretty pictures can click on the thumbnail on the right: - *software* is a general term used to describe a collection of computer programs, procedures and documentation that perform some tasks on a computer system. - *software* is what we are trying to _package_ - *software* has /names/ - *software* may exist as + /source code/ + /executable code/ + /packaged code/ - *source code* is any collection of statements or declarations written in some human-readable computer /programming language/. - *source code* is usually held in one or more text files (/blobs/). - A large collection of *source code* /files/ may be organized into a directory tree, in which case it may also be known as a source /tree/. - The *source code* may be _converted_ into an /executable format/ by a compiler, or executed on the fly from the human readable form with the aid of an interpreter. - *executable format* is the form /software/ must be in in order to be run. Running means to cause a computer "to perform indicated tasks according to encoded instructions." - software *source code* has one or more /lines of development/. Some Common specific /lines of development/ for the software to be packaged are: + *upstream* /line of development/ + *feature branch* is a /line of development/ related to a new feature under development. Often the goal is to _merge_ the feature branches into the /upstream/ /line of development/ + usually, all *feature branches* are _merged_ into the /integration branch/, and the /package/ is created from the /integration branch/. + *integration branch* is the /line of development/ of software that is to be packaged - some software *lines of development* have /releases/ - *releases* have /release dates/ - some *releases* have /release versions/ - *source code* may be _stored_ in a version control /repository/, and maintain history. - *Trees* are a collection of /blobs/ and other /trees/ (directories and sub-directories). A tree object describes the state of a directory hierarchy at a particular given time. - *Blobs* are simply chunks of binary data - they are the contents of files. - a *tree* can be _converted_ into an /archive/ and back - In git, *directories* are _represented_ by /tree/ object. They refer to blobs that have the contents of files (file name, access mode, etc is all stored in the tree), and to other trees for sub-directories. - *Commits* (or "changesets") mark points in the history of a /line of development/, and references to /parent commits/. - A *commit* refers to a tree that represents the state of the files at the time of the commit. - *HEAD* is the most recent commit in a /line of development/ or /branch/. - A *working directory* is a directory that corresponds, but might not be identical, to a /commit/ in the version control /repository/ - *Commits* from the version control system can be _checked out_ into the /working directory/ - *uncommitted changes* are changes in the working directory that make it different from the corresponding /commit/. Some call the working directory to be in a "dirty" state. - *uncommited changes* be _checked in_ into the version control system, creating a new /commit/ - The *working directory* may contain a /ignore file/ - *ignore file* contains the names of files in the /working directory/ that should be "ignored" by the version control system. - In git, a *commit* may also contains references to /parent commits/. + If there is more than one /parent commit/, then the /commit/ is a /merge/ + If there are no /parent commits/, it is an /initial commit/ - references, or heads, or *branches*, are movable references to a /commit/. On a fresh /commit/, the head or /branch/ reference is moved to the new /commit/. - *lines of development* are usually _stored_ as a /branch/ in the version control /repository/. - A new *branch* may be created by _branching_ from an existing /branch/ - a *patch* is a file that contains difference listings between two /trees/. - A *patch* file can be used to transform (_patch_) one /tree/ into another (/tree/). - A *quilt series* is a method of representing an /integration branch/ as a collection of a series of /patches/. These patches can be applied in sequence to the /upstream/ branch to produce the /integration branch/. - A *tag* is a named reference to a specific /commit/, and is not normally moved to point to a different /commit/. - A *package* is an /archive/ format of /software/ created to be installed by a package management system or a self-sufficient installer, derived by _transforming_ a /tree/ associated with an /integration branch/. - *packages* have /package names/ - *package names* are related to /upstream/ /software names/ - *packages* have /package versions/ - *package versions* may have + an /upstream version/ component + a distribution or packaging specific component - *package versions* are related to upstream /software versions/ - *helper packages* provide libraries and other support facilities to help _compile_ an /integration branch/ ultimately yielding a /package/

Manoj

Sunday 19 April
2009

Looking at porcelain: Towards a generic, distribution agnostic tool for building packages from a VCS

#+TITLE: Looking at porcelain: Towards a generic, distribution agnostic tool for building packages from a VCS #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ This is a continuation from #+BEGIN_HTML before. #+END_HTML Before I go plunging into writing code for a generic =vcs-pkg= implementation, I wanted to take a close look at my current, working, non-generic implementation: making sure that the generic implementation can support at least this one concrete work-flow will keep me grounded. One of the features of my home grown porcelain for building package has been that I use a fixed layout for all the packages I maintain. There is a top level directory for all working trees. Each package gets a sub-directory under this working area. And in each package sub-directory, are the upstream versions, the checked out VCS working directory, and anything else package related. With this layout, knowing the package name is enough to locate the working directory. This enable me to, for example, hack away at a package in Emacs, and when done, go to any open terminal window, and say =stage_release kernel-package= or =tag_releases ucf= without needing to know what the current directory is (usually, the packages working directory is several levels deep -- =/usr/local/git/debian/make-dfsg/make-dfsg-3.91=, for instance. However, this is less palatable for a generic tool -- imposing a directory structure layout is pretty heavy. And I guess I can always create a function called ~cdwd~, or something, to take away the tedium of typing out long ~cd~ commands. Anyway, looking at my code, there is the information that the scripts seem to need in order to do their work. - *Staging area*. This is where software to be built is exported (and this area is visible from my build virtual machine). + User specified (configuration) - *Working Area*. This is the location where all my packaging work happens. Each package I work on has a sub-directory in here, and the working directories for each package live in the package sub-directory. /Note: Should not be needed/. + User specified. - *Working directory*. This is the checked out tree from the VCS, and this is the place where we get the source tree from which the package can be built. + Since we know the location of the working are, if the package name is known, we can just look in the package's sub-directory in the working area. * For =rpm= based sources, look for the ~spec~ file * For Debian sources, locate ~debian/rules~ + If package name is not known, look for ~spec~ or ~debian/rules~ in the current directory, and parse either the ~spec~ file or ~debian/changelog~. + If in a VCS directory, look for the base of the tree - ~tla tree-root~ - ~bzr info~ - ~git rev-parse --show cdup~ - ~hg root~ - You have to climb the tree for subversion + If you are in a ~debian~ directory, and ~changelog~ and ~rules~ files exist Then, look for the ~spec~ file or ~debian/rules~ in the base directory - *package name* + User specified, on the command line + If in the working directory of the package, can be parsed from the ~spec~ or ~changelog~ files. - *upstream tar archive* + Usually located in the parent directory of the working directory (the package specific sub-directory of the working area) + If ~pristine-tar~ is in use, given two trees (branches, commits. etc), namely: * a tree for upstream (/default: the branch ~upstream~/) * a tree for the delta (/default: the branch ~pristine-tar~/) The tree can be generated * Given an upstream tree (/default: the branch ~upstream~/), a tar archive can be generated, but is likely to be not bit-for-bit identical to the original ~tar~ archive. So, if I do away with the whole working area layout convention, this can be reduced to just requiring the user to: - Specify *Staging area* - Call the script in the working directory (=dpkg-buildpackage= imposes this too). - Either use ~pristine-tar~ or have the upstream ~tar~ archive in the parent directory of the working directory Hmm. One user specified directory, where the results are dumped. I can live with that. However, ~gitpkg~ has a different concept: it works purely on the git objects, you feed it upto three tree objects, the first being the tree with sources to build, and the second and third trees being looked at only if the upstream tar archive can not be located, and passes the trees to pristine tar to re-construct the upstram tar. The package name and version are constructed _after_ the source-tar archive is extracted to the staging area. I like the minimality of this. This is continued #+BEGIN_HTML here. #+END_HTML

Manoj

Thursday 16 April
2009

Towards a generic, distribution agnostic tool for building packages from a VCS

#+TITLE: Towards a generic, distribution agnostic tool for building packages from a VCS #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ I have been involved in =vcs-pkg.org= since around the time it started, a couple of years ago. The discussion has been interesting, and I learned a lot about the benefits and disadvantages of serializing patches (and collecting integration deltas in the feature branches *and* the specific ordering of the feature branches) and maintaining integration branches (where the integration deltas are collected purely in the integration branch, but might tend to get lost in the history, and a fresh integration branch having to re-invent the integration deltas afresh). However, one of the things we have been lax about is getting down to brass tacks and getting around to being able to create generic packaging tools (though for the folks on the serializing patches side of the debate we have the excellent =quilt= and the =topgit= packages). I have recently mostly automated my git based work-flow, and have built fancy porcelain around my git repository setup. During IRC discussion, the =gitpkg= script came up. This seems almost usable, apart from not having any built-in =pristine-tar= support, and also not supporting =git submodules=, which make is less useful an alternative than my current porcelain. But it seems to me that we are pretty close to being able to create a distribution, layout, and patch handler agnostic script that builds distribution packages directly from version control, as long as we take care not to bind people into distributions or tool specific straitjackets. To these ends, I wanted to see what are the tasks that we want a package building script to perform. Here is what I came up with. 1. Provide a copy of one or more upstream source tar-balls in the staging area where the package will be built. This staging area may or may not be the working directory checked out from the underlying VCS; my experience has been that most tools of the ilk have a temporary staging directory of some kind. 2. Provide a directory tree of the sources from which the package is to be built in the staging area 3. Run one or more commands or shell scripts in the staging area to create the package. These series of commands might be very complex, creating and running virtual machines, chroot jails, satisfying build dependencies, using copy-on-write mechanisms, running unit tests and lintian/puiparts checks on the results. But the building a package script may just punt on these scripts to a user specified hook. The first and third steps above are pretty straight forward, and fairly uncontroversial. The upstream sources may be handled by one of these three alternatives: 1. compressed tar archives of the upstream sources are available, and may be copied. 2. There is a pristine-tar VCS branch, which in conjunction with the upstream branch, may be used to reproduce the upstream tr archive 3. Export and create an archive from the upstream branch, which may not have the same checksum as the original branch The command to run may be supplied by the user in a configuration file or option, and may default based on the native distribution, to =dpkg-buildpackage= or =rpm=. There are a number of already mature mechanisms to take a source directory and upstream tar archive and produce packages from that point, and the wheel need not be re-invented. So the hardest part of the task is to present, in the staging area, for further processing, a directory tree of the source package, ready for the distribution specific build commands. This part of the solution is likely to be VCS specific. This post is getting long, so I'll defer presenting my evolving implementation of a generic =vcs-pkg= tool, ~git~ flavour, to the next blog post. This is continued #+BEGIN_HTML here. #+END_HTML

Manoj

Thursday 16 April
2009
Link: The glaring hole in most git tools, or the submodule Cinderella story

Posted in the wee hours of Wednesday night, April 16th, 2009

The glaring hole in most git tools, or the submodule Cinderella story

#+TITLE: The glaring hole in most git tools, or the submodule Cinderella story #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ There are a lot of little git scripts and tools being written by a lot of people. Including a lot of tools written by people I have a lot of respect for. And yet, they are mostly useless for me. Take git-pkg. Can't use it. Does not work with git submodules. Then there is our nice, new, shiny, incredibly bodacious "3.0 (git)" source format. Again, useless: does not cater to submodules. I like submodules. They are nice. They allow for projects to take upstream sources, add Debian packaging instructions, and put them into git. They allow you to stitch together disparate projects, with different authors, and different release schedules and goals, into a coherent, integrated, software project. Yes, I use git submodules for my Debian packaging. I think it is conceptually and practically the correct solution. Why submodules? Well, one of the first things I discovered was that most of the packaging for my packages was very similar -- but not identical. Unfortunately, the previous incarnation of my packages with a monolithic rules file in each ~./debian/~ directory, it was easy for the rules files in packages to get out of sync -- and there was no easy way to merge changes in the common portions an any sane automated fashion. The ~./debian/~ directories for all my packages package that they are instrumental in packaging. So, since I make the ~./debian/~ directories branches of the same project, it is far easier to package a new package, or to roll out a new feature when policy changes -- the same commit can be applied across all the branches, and thus all my source packages, easily. With a separate =debian-dir= project, I can separate the management of the packaging rules from the package code itself. Also, I have abstracted out the really common bits across all my packages into a ~./debian.common~ directory, which is yet another project, and included in as a submodule in all the packages -- so there is a central place to change the common bits, without having to duplicate my efforts 30-odd times. Now people are complaining since they have no idea how to clone my package repositories, since apparently no one actually pays attention to a file called ~.gitmodules~, and even when they do, they, and the tools they use, have no clue what to do with it. I am tired of sending emails with one off-cluebats, and I am building my own porcelain around something I hope to present as a generic =vcs-pkg= implementation soon. The firs step is a wrapper around =git-clone=, that understands git submodules. So, #+BEGIN_HTML here #+END_HTML is the browsable code (there is a link in there to the downloadable sources too). Complete with a built in man page. Takes the same arguments as =git-clone=, but with fewer options. Have fun.

Manoj

Tuesday 14 April
2009
Link: Yet another kernel hook script

Posted terribly early Tuesday morning, April 14th, 2009

Yet another kernel hook script

#+TITLE: Yet another kernel hook script #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ With tonight's upload of ~kernel-package~, the recent flurry of activity on this package (8 uploads in 6 days) is drawing to a close. I think most of the functionality I started to put into place is now in place, and all reported regressions and bugs in the new 12.XX version have been fixed. The only known deficiency is in the support of Xen dom0 images, and for that I am waiting for kernel version =2.6.30=, where Linus has reportedly incorporated Xen patches. In the meanwhile, ~kernel-package~ seems to be working well, and I am turning my attention to other things. But, before I go, here is another example kernel postinst hook script (which, BTW, looks way better with syntax highlighting CSS on my blog than it does in a rss feed or an aggregator site). #+BEGIN_HTML [[!syntax language=Bash linenumbers=1 bars=1 text=""" #! /bin/sh set -e if [ -n "$INITRD" ] && [ "$INITRD" = 'No' ]; then exit 0 fi version="$1" vmlinuz_location="$2" if [ -n "$DEB_MAINT_PARAMS" ]; then eval set -- "$DEB_MAINT_PARAMS" if [ -z "$1" ] || [ "$1" != "configure" ]; then exit 0; fi fi # passing the kernel version is required [ -z "$version" ] && exit 1 if [ -n "$vmlinuz_location" ]; then # Where is the image located? We'll place the initrd there. boot=$(dirname "$vmlinuz_location") bootarg="-b $boot" fi # Update the initramfs update-initramfs -c -t -k "$version" $bootarg exit 0 """]] #+END_HTML

Manoj

Sunday 12 April
2009
Link: Sample kernel symlink postinst hook script

Posted in the wee hours of Saturday night, April 12th, 2009

Sample kernel symlink postinst hook script

#+TITLE: Sample kernel symlink postinst hook script #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ With the new kernel-package hitting Sid today, and the fact that it no longer does symlink handling by default, I thought it was time that we had an example script that shows how to do that. This is a fairly full featured script, feel free to cull down to use just what you want. I'll post a couple of ther scripts, if there is interest in this. BTW, this script does far more than the old ~kernel-package~ =postisnt= script ever did. Have fun. #+BEGIN_HTML [[!syntax language=Bash linenumbers=1 bars=1 text=""" #!/bin/sh - # -*- Mode: Sh -*- # # This is an example of a script that can be run as a postinst hook, # and manages the symbolic links in a manner similar to the kernel # image default behaviour, except that the latest two versions (as # determined by ls -lct) are kept. You can modify this script # # Copyright 2003, 2004, 2005, 2006, 2007, 2008, 2009 Manoj Srivastava # Copyright 2009 Darren Salt set -e # The dir where symlinks are managed SYMLINKDIR=/ if [ $# -ne 2 ]; then echo Usage: $0 version location exit 2 fi version="$1" vmlinuz_location="$2" vmlinuz_dir="$(dirname "$2")" cd $SYMLINKDIR || exit 1 if [ -n "$DEB_MAINT_PARAMS" ]; then eval set -- "$DEB_MAINT_PARAMS" fi if [ -z "$1" ] || [ "$1" != "configure" ]; then exit 0; fi rm -f vmlinuz vmlinuz.old vmlinuz-rd vmlinuz-rd.old initrd.img initrd.img.old # Create a temporary file safely if [ -x /bin/tempfile ]; then outfile=$(tempfile -p outp -m 0600); else set -e mkdir /tmp/kernel-image-$version-$$ outfile=/tmp/kernel-image-$version-$$/output fi (cd "$vmlinuz_dir" && ls -ct vmlinuz-*) > $outfile STD="$(head -n 1 $outfile | sed 's/vmlinuz-//')" OLD="$(head -n 2 $outfile | tail -n 1 | sed 's/vmlinuz-//')" if [ "X$STD" = "X" ]; then exit 0; fi # If you want version-specific links, here's how to start STD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true OLD24="$(grep vmlinuz-2.4 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true STD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | sed 's/vmlinuz-//')" || true OLD25="$(grep vmlinuz-2.5 $outfile | head -n 1 | tail -n 1 | sed 's/vmlinuz-//')" || true echo Booting $STD, old is $OLD if [ -f "$vmlinuz_dir/"initrd.img-$STD ] ; then ln -s "$vmlinuz_dir/"initrd.img-$STD initrd.img ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz-rd else ln -s "$vmlinuz_dir/"vmlinuz-$STD vmlinuz fi if [ "X$OLD" != "X" ]; then if [ -f "$vmlinuz_dir/"initrd.img-$OLD ] ; then ln -s "$vmlinuz_dir/"initrd.img-$OLD initrd.img.old ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz-rd.old else ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz.old fi fi # if [ "X$STD24" != "X" ]; then # if [ -f "$vmlinuz_dir/"initrd.img-$STD24 ] ; then # ln -s "$vmlinuz_dir/"initrd.img-$STD24 initrd24.img # ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24-rd # else # ln -s "$vmlinuz_dir/"vmlinuz-$STD24 vmlinuz24 # fi # fi # if [ "X$OLD24" != "X" ]; then # if [ -f "$vmlinuz_dir/"initrd.img-$OLD24 ] ; then # ln -s "$vmlinuz_dir/"initrd.img-$OLD24 initrd24.img.old # ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24-rd.old # else # ln -s "$vmlinuz_dir/"vmlinuz-$OLD vmlinuz24.old # fi # fi # Run boot loaders here. #lilo rm -f $outfile if [ -d /tmp/kernel-image-$version-$$ ]; then rmdir /tmp/kernel-image-$version-$$ fi exit 0 """]] #+END_HTML

Manoj

Thursday 09 April
2009
Link: kernel-package: Add support for kernel image debug package

Posted Thursday night, April 9th, 2009

kernel-package: Add support for kernel image debug package

#+TITLE: kernel-package: Add support for kernel image debug package #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ A new version of ~kernel-package~ in Incoming at the time of writing adds support for creating a package which contains the Linux kernel debug image. This means the debugging information for the modules in the kernel image package, and the uncompressed vmlinux image. This builds on suggestions and code from Troy Heber, Theodore Y. Ts'o, and Dann Frazier. As support for kexec/kdump support becomes more real, it will be very useful to be able to build kernels that have debugging information available, but not necessarily to install the debugging information on every single client system. The =.ko= object files are stripped of the debugging information before they are placed in the standard /linux-image/ installation package. However, before that, a copy of the vmlinux and unstripped =.ko= files are saved in a "/debuginfo/" package, and everything except for the debugging information is removed from them using ~objcopy --keep-only-debug~. This means that if someone ends up with a crash dump, they can send it to a support engineer and only the support engineer needs to install the debuginfo package and use the "/crash/" utility to debug the crash dump. It's also useful for developers, since the debuginfo information can be stored somewhere outside of ~/lib~ for storing its debug information, for example. This is useful for keeping the size of the root partition small, for those who keep ~/~ separate from ~/usr~. The locations used are compatible also with *SystemTap*, which provides free software infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. *SystemTap* eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data. Also, I had to clear out some FUD about kernel-package from the /ircbot/ *dpkg* on the OFTC Debian IRC channel, since someone had implied that ~kernel-package~ was some how obsolete. As can be seen, it is being actively developed, and features are being added apace.

Manoj

Tuesday 07 April
2009
Link: Not your father's kernel-package

Posted at teatime on Tuesday, April 7th, 2009

Not your father's kernel-package

#+TITLE: Not your father's kernel-package #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ A few hours ago, a new version of kernel-package was uploaded to ~experimental~. This is a major change,and I would appreciate it if folks took it out for a spin, kicked the tires, and provide feedback about where this version is lacking. This is only part of the way along in this development cycle. I would like to add a debug-info separation, either in a different directory than / in the image packages, or a separate package by itself. I would also like to create an overlay directory for =/usr/share/kernel-package/=, so people can inject code or override the defaults for kernel-package easily. I am also willing to make any changes to standardize the handling of hook scripts for kernel packages in Debian. ** =./debian/= is ephemeral =make-kpkg= removes and re-creates =./debian= on every invocation. This started as an exercise to protect ourselves from the upstream builddep functionality, that randomly cleaned out ~./debian~ whether or not it had created it, effectively making it impossible to run ~dpkg-buildpackage~ easily (which is ok, if all you care about is the image package) This does make the ~kernel-package~ far more nimble; we now offer less surprise to users who did not expect stampts that the kernel-packagge used to not do duplicate work. Now, if you edit a couple of files in the kernel source, and run ~make-kpkg~, the kernel will build as expected. There are no more "version mismatch" errors, and the kernel version can be modified using ~localconfig~ as one desires. With this, kernel-package can routinely be used to build kernels out of the git tree. The con is that we no longer cater to official kernels, or to anyone who expected content in ./debian to persist. At some point, there are plans to implement an overlay directory that will shadow =/usr/share/kernel-package/ruleset=, but that is not yet implemented. In any case, the kernel team in Debian regards ~kernel-package~ to be broken, and have been bad mouthing it and deprecating it for a few years now, so this will not be a loss for them. ** Get rid of the facility to patch kernel sources The patch the kernel facility was adding complexity, and failing to provide the flexibility required for a generic patching facility. It used to be useful at one point, but in the modern parlance, witht he widespread use of distribute version control systems, and various facilities to manage source and patch them, the built in version was clunky. This means the ~--added-patches~ option of =make-kpkg= is gone, the work-around is to prepare the kernel sources _before_ calling =make-kpkg=. ** Remove special case code for official kernels For the longest tine (well, ever since Herbert Xu too over building kernel images from me), ~kernel-package~ has carried specal case code for official images. This has caused some problems, recently, since the need to preserve ~./debian~ has caused no end of problems when the version changed out from under ~./debian~, or when people wanted to edit a file and expected kernel-package to do a minimal recompile. However, sometime in the Etch release cycle, the kernel team deprecated ~kernel-package~ as the means of building official kernels. They have recently started saying they think ~kernel-package~ is broken, and have their own recommendation for how to build kernel packages. Therefore, a full release cycle later, we can get rid of the special case rules used for official packages. Also, this allows us to drop ~./debian~ at the drop of a hat, and recreate it with an version that reflects the current state of the kernel sources. ** Header package no longer create symbolic links in =/usr/src= Instead, ship an example shell script that replicates the old behaviour. This script can then be deployed on the target machines, and could be a part of a locally created kernel configuration package, if one needs to deploy the same behavior across a cluster of machines. ** The postinst no longer manipulates symlinks This is a shift from previous behaviour. Any symbolic link manipulation must now be done with hook scripts in ~/etc/kernel/*.d~ directories. Firstly, modern boot loaders scan the boot directory for kernel images, and the user no longer has to code in the path to the symbolic links that the kernel image package used to manipulate. Secondly, hardcoding the behaviour into the postinst made for a very rigid policy; and user wanted more flexibility than that. There is an example shipped with the package that shows a more flexible scheme that kept two symbolic links for version 2.4 kernels, and two symbolic links for 2.6 kernels; it can be easily modified to keep two links for 2.9 kernels and two links for 2.8 kernels, or one of each, or whatever the user wants. ** Image postinst no longer runs a boot loader Please note that this was already the case for grub, one of the more popular boot loaders. Now that we have a mechanism for running arbitrary scripts when the image packages are manipulated, we can stop embedding the boot loader actions in the package itself. This means that lilo, elilo, etc will no longer be run directly by the post install script, and all the code related to detecting the boot loader, managing the configuration, and adding bits about bootloader documentation is all removed from the postinst. This allows the image package to be more flexible, since the end user is no longer restricted to the actions encoded in the image package. This is a fairly large change. It also opens the door for the user to easily use non-standard bootloaders, if they so desire. ** The image postinst no longer creates an initramfs Instead, there are example scripts provided that will perform the task. These scripts will work for official kernel images as well. The initramfs scripts provided work with the =make-kpkg= images as well as the official images, and are thus better than the script shipped with =initramfs-tools= themselves, as they offer a super set of functionality. This also demonstrates how the posts install script communicates with the initramfs creation scripts so that no initramfs is generated in case you do not want it.

Manoj

Tuesday 31 March
2009
Link: Fighting FUD: Working with openssl

Posted late Tuesday evening, March 31st, 2009

Fighting FUD: Working with openssl

#+TITLE: Fighting FUD: Working with openssl #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ Unfortunately, there is so much FUD associated with doing your own certificates, either based on how complex the operation is (which led to my previous supervisor insisting I use something like ~tinyca~), and now to my employer succumbing to the FUD and shelling out several hundreds, perhaps several thousands, of dollars a year for something we could well have handled in house. Public key infrastructure, in the form of the =X509= standard, is the underpinning of most of the secured communications over the 'net these days. The big winner in the transport protocols, ~TLS~, and its predecessor, ~SSL~, support =X509= certificates. There are several ways of getting your own services their own ~X509~ certs; one of which I am exploring below. One may, or course, opt to get a commercially signed certificate, and various companies are eager to do just that for you. They also charge about $400 per annum per certificate for the privilege of doing so. While there is some marginal benefit of doing so (some web browsers come with the commercial public certificate built in, allowing for an out of band distribution of the public cert), the benefit accrued is in the order or pennies, in my opinion, not hundreds of dollars, unless you are providing banking or retail services, where the end users might be justified in being paranoid. Why is there so much FUD your own certificates? Especially about how hard it is do to your own? As you can see below, it only takes three commands you have to master in order to set up your own private certifying authority, and sign your own certificates. The only marginal issue is that the user needs to verify your certificate out of band (if, really, they want to bother). Most people just accept the certificate, in my experience. The sole benefit that commercial entities provide is that they verify the identity of the person asking for the certificate, with varying degrees of diligence. For a Class 1 cert the CA usually just verifies that the email address of the requester was confirmed. For $400/year. For a Class two cert, they look up the company in a credit bureau records. A class 3 certs does an ID check with a notary public present, or a government issued ID. So, a class 3 cert is somewhat less diligent than becoming a Debian developer. Or getting your key signed at a Debian conference. As to the security aspects, or wondering whether to trust the information present on a designated web site, I have no idea how it helps verify any of those things in any way. So the web site is run by a person with a government provided ID, and who has a few hundred dollars to burn. So what? Me, I just sign my own certificates. And I think most small business web sites and mail servers are perfectly well served by using their own certificates. And there are just three simple commands that enable them to do this, in the Linux world. So what are these three commands? #+BEGIN_HTML Gory practical details and recipes hidden here #+END_HTML In conclusion, creating your own certifying authority is trivial, and certainly not worth several hundreds of dollars every year, and the functionality provided is identical.

Manoj

Sunday 01 March
2009
Link: Rethinking ucf redux

Posted late Sunday evening, March 1st, 2009

Rethinking ucf redux

#+TITLE: Rethinking ucf redux #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ I have been thinking some more about how to improve ucf. One of the things that struck me was that based on my earlier #+BEGIN_HTML analysis #+END_HTML there are only five actions that ucf can take, and the decision about the actions depends on the state it finds the configuration file in on the target machine, and there are only eight of those. Now, thinking back to my days as a VLSI designer back in the halcyon days of electrical engineering, This is a pretty simple state machine. It is not as neat as it could be (where just three variables would be needed to keep track of things, but still, it bears investigation. This would be a way for converting the current procedural /ucf/ into a functional programming model. Hop over #+BEGIN_HTML here #+END_HTML for a look at how that went --- it was fun, and afforded me an opportunity to demonstrate how well /org/ handles #+BEGIN_HTML LaTeX.png #+END_HTML snippets.

Manoj

Wednesday 25 February
2009
Link: A day in the life of a Debian hacker

Posted terribly early Wednesday morning, February 25th, 2009

A day in the life of a Debian hacker

#+TITLE: A day in the life of a Debian hacker #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta@debian.org #+DATE: #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ I have been meaning to write this up for a long time now, since I #+BEGIN_HTML Packaging activity diagram #+END_HTML vaguely made a promise to do so last Debconf. I have also been wondering about the inefficiencies in my work-flow, but I kept postponing my analysis since there were still large gaps in my packaging automation since I moved off Arch as my SCM of choice. However, recently I have taken a sabbatical from Debian, so I've had time to complete bits and pieces of my package building framework, enough so that I could no longer justify putting off the analysis. I tried writing it up, but the result confused even me; so I instead recorded every shell command during a recent series of packaging tasks, and converted that into a nice, detailed, activity diagram that you see over here. This is as efficient a work-flow as I have been able to come up with. #+BEGIN_HTML details here #+END_HTML Along with a git commit hook script, that parses the commit log and adds pending tags to bugs closed in the commit, the figure above represents my complete work-flow -- down to the details of every /cd/ command I executed. I think there are too many steps still. Feedback and commentary would be appreciated, as well as any suggestions to improve efficiency.

Manoj

Tuesday 24 February
2009
Link: Rethinking ucf

Posted Tuesday night, February 24th, 2009

Rethinking ucf

#+TITLE: Rethinking UCF #+AUTHOR: Manoj Srivastava #+EMAIL: srivasta\@debian.org #+DATE: <2009-02-24 Tue> #+LANGUAGE: en #+OPTIONS: H:0 num:nil toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:nil tags:not-in-toc #+INFOJS_OPT: view:showall toc:nil ltoc:nil mouse:underline buttons:nil path:http://orgmode.org/org-info.js #+LINK_UP: http://www.golden-gryphon.com/blog/manoj/ #+LINK_HOME: http://www.golden-gryphon.com/ There has been some discussion on the Debian development mailing list #+BEGIN_HTML Tortured activity diagram #+END_HTML about adding hooks into ucf, to allow people to do things like committing files into different SCM branches. So, I thought I would help people out by letting them tell me where hooks would be useful, and so decided to do an activity diagram for ucf. Gawd, what a mess. I mean, I wrote this thing, and it boggles even my mind. See the figure for how horrendous code can get when it grows organically. So, I decided to re-factor/redesign ucf, see if I could create a less complex activity diagram. On analysis, it turns out that ucf has just five actions it may perform, and which action it takes depends on which of eight possible states the configuration file is in. #+BEGIN_HTML Gory details follow #+END_HTML

Manoj


Webmaster <webmaster@golden-gryphon.com>
Last commit: terribly early Sunday morning, June 8th, 2014
Last edited terribly early Sunday morning, June 8th, 2014