A serious security issue in paludis was brought to my attention recently, and I feel I should make you all aware. Apparently someone, with root access to a machine, can gain root access by installing or editing paludis config files.

For those interested, this is how it happened (times are GMT+1):

22:34 <@ferdy> bonsaikitten: can you give me any details regarding that
 security bug in paludis?
22:35 <+bonsaikitten> ferdy: it's so obvious you should have found it already
22:37 <@ferdy> bonsaikitten: I should, but I probably haven't
22:37 <+bonsaikitten> ferdy: well, as I am a moron I'm unable to coherently explain :)
22:37 <@ferdy> bonsaikitten: I mean, depends on whether we are talking about
a real security issue or about something we should document to avoid people
shooting themselves in the foot
22:39 <@ferdy> bonsaikitten: is that all you are going to tell me?
22:39 <+bonsaikitten> ferdy: come on, it's obvious. You're supposed to be smart ...
22:39 * bonsaikitten is not in a mood to explain
22:40 <@ferdy> bonsaikitten: you aren't really talking about the paludisbuild issue, are you?
22:41 <+bonsaikitten> mmh no, that's a different one
22:41 <@ferdy> k
22:41 <@ferdy> bonsaikitten: what are we talking about?
22:42 <@ferdy> bonsaikitten: you don't need to explain it... just say, in general 
terms, what the issue is
22:50 <@ferdy> bonsaikitten: so? care to give any useful hint?
22:50 <+bonsaikitten> ferdy: doesn't happen in portage compatibility mode
22:51 <+bonsaikitten> but I blame the vodka, hard to explain when *burp* *giggle*
22:52 <@ferdy> bonsaikitten: what's the impact?
22:53 <+bonsaikitten> ferdy: depends on how annoying the other person is
22:54 <+bonsaikitten> ferdy: worst case random file modification
22:58 <@ferdy> bonsaikitten: and we already agreed that we aren't talking about
the paludisbuild issue, right?
22:59 <@ferdy> bonsaikitten: if we aren't, I'll need more hints....
23:05 <@ferdy> bonsaikitten: can I get an attack vector?
23:05 <@ferdy> that shouldn't need lots of explaining... I can figure out that
part myself
23:19 <@ferdy> bonsaikitten: have you got that attack vector for me?
23:24 <+bonsaikitten> ferdy: look at configuration files, maybe you notice that
there's some exquisit code execution possible there
23:29 <@ferdy> bonsaikitten: you mean those config files that only root can
edit? I must be missing something here
23:29 <+bonsaikitten> ferdy: you are :)
23:29 <+bonsaikitten> not much, and it's basically the same flaw bashrc is
for portage
23:29 <+bonsaikitten> only that bashrc is config_protect'ed ...
23:30 <@ferdy> bonsaikitten: but for a package to clover those files, it must be
in a repo root added, right?
23:31 <+bonsaikitten> someone in the package mangler group, but yes
23:35 <@ferdy> bonsaikitten: but if you can change those files in the first place,
why clover them by adding a malicious repo with a malicious package that changes
those files?
23:35 <+bonsaikitten> ferdy: because it's very subtle
23:36 <@ferdy> moreover, if you can already do that, why not just make the
package install whatever backdoor you want?
23:37 <@ferdy> I mean, it is subtle, but why would anyone go the 'convoluted'
route? he is already able to edit those files (since he had to add that repo)
23:38 <+bonsaikitten> 'cause only paludis is affected and you will find it very
hard to trace
23:38 <+bonsaikitten> that makes it so tempting ...
23:40 <+bonsaikitten> just don't be surprised if it suddenly unmerges itself :)
23:41 <@ferdy> yeah... well...
23:41 <@ferdy> bonsaikitten: mind if I disclose this vulnerability in
 planet.gentoo.org?
23:42 <+bonsaikitten> go ahead
23:42 <@ferdy> ta
23:42 <+bonsaikitten> 't is even on the features page of the package mangler :)

This is a good lesson to learn today:

If you can edit files owned by root in a machine, you can get root access to that machine.

So the bottom line is: There is no vulnerability, if you can mangle paludis config files, you are already root so you don't need to edit a file to run any command you want. Another lesson one can learn by reading that log is how to be really cooperative.

Ah, and before someone with a need to use cheap psychology asks, the intention of this blag post is to stop the FUD.

- ferdy

Leave a comment

...you get up, fire your mail client just to find:

https://bugs.gentoo.org/show_bug.cgi?id=64840
https://bugs.gentoo.org/show_bug.cgi?id=168573
https://bugs.gentoo.org/show_bug.cgi?id=161368

Please take the time to read those 'Excerpts', they are really enlightening. Politics is so much fun.

Some facts:

• All of them work in Paludis
• Some of them have been vocal against how #gentoo and forums.gentoo.org are managed and its politics.
• Using the image I posted yesterday as an avatar is one of the reasons one of them got the boot.

I wonder who is going to be next. I also wonder who 'started' the process. It is difficult to know, because the bug is restricted ( https://bugs.gentoo.org/show_bug.cgi?id=216219 ).

Way to go Gentoo. No love for you today.

- ferdy

Leave a comment

I hear this image got someone a Developer Relations complain. Way to go, Gentoo. Way to go.

You are clearly doing it wrong.

(For those that aren't aware of this rather old issue: have fun).

- ferdy

Leave a comment

I'm proxy-posting this to Gentoo Planet from yoswink:

---

During the last week, the people of Isla Cristina (a beautiful town in the Atlantic coast with probably the best beaches of all Spain) organized an event called '1st Technological days of Isla Cristina'. They invited me to participate and give a talk about Gentoo.

On Friday I was there explaining what makes our distribution different, what the 'All about choice' is and the technology behind it. I shared the day with Spanish developers of other open source projects such as OpenBSD, NetBSD and KDE.

The day organization was perfect, any more to say, it couldn't be better. I want to thanks the people (teachers and students) from IES Mirabent, who fought to make this idea a reality in a small town of 22k people (is not so common to find this kind of event in Spain with official devs from big OSS projects even in the big cities or universities), and to the youth department of Isla Cristina for all they have done for me.

If someone want to take an example about how an event should treat a lecturer, please contact with them.

P.D: thanks to Julio Merio from NetBSD for lending me the computer (I forgot the damn Mini-DVI cable) and sorry to Antonio Larrosa from KDE, I ate some minutes of his time :(

---

Leave a comment

Apparently, people are bashing Joshua Jackson (a.k.a. tsunam) for posting his opinion (which happens to be shared among lots of us). So just in case someone hasn't read what he said yet, I'm going to link his posts here. Please, do read them:

http://tsunam.org/2008/01/12/in-response/
http://tsunam.org/2008/01/14/clarifications/
http://tsunam.org/2008/01/14/tool-box/

I simply can't trust Daniel Robbins after what he tried to do the last time he tried to come back. Has everybody forgotten that? I hope not.

No love.

- ferdy

Leave a comment

After a fair time of using the hook described in the previous post I realized that it noticeably slows down the uninstalls of packages with lots of files like sys-kernel/gentoo-sources and hence I decided to rewrite it as a Python hook to see how it performs:

UNINSTALL_PROTECT=["/lib64/modules/"]

def hook_run_unmerger_unlink_file_override(env, hook_env):
    for path in UNINSTALL_PROTECT:
        if hook_env["UNLINK_TARGET"].startswith(path):
            return "skip"
        else:
            return ""

.hook hook ~ 8m
python hook ~ 1m 15s
no hook ~ 1m

Leave a comment

I was inspired by a post on the Command Line Warriors blog to encrypt my /home directory. Unfortunately, the directions given in that post don’t quite work. Here is the process I followed to set everything up.

We’re setting up a basic LUKS volume encrypted with AES with a 256-bit key. This means we make a special “filesystem” on the disk partition which encrypts our real filesystem on disk, and makes it available unencrypted via the device-mapper interface (/dev/mapper/).

First, you need to have a partition available for your /home directory. In my case, I decided to nuke my Windows install, but most people will probably need to use parted to resize some existing partitions. If you’re resizing your root (”/”) partition, you’ll need to run it from a LiveCD. For the partition, I chose the “Linux” type, but I’m not sure that really matters.

Once you’ve allocated the partition, you’ll need to create the LUKS partition. You do this with the cryptsetup command. But, before you can use this, you’ll need to make sure you’ve compiled these settings into your kernel: CONFIG_DM_CRYPT, CONFIG_CRYPTO_CBC, CONFIG_CRYPTO_SHA256, and CONFIG_CRYPTO_AES.

After configuring your kernel and you’ve rebooting, if need be, you’ll need to install sys-fs/cryptsetup. Don’t install sys-fs/cryptsetup-luks… it’s old. The newest (>1.0) versions of cryptsetup are based on the -luks version, and are what you should be using.

Now, use cryptsetup to format the luks partition:

  cryptsetup luksFormat -c aes-cbc-essiv:sha256 /dev/hda4

Substitute /dev/hda4 with whatever partition you created earlier. It will ask you to enter a password. Use a strong one, and don’t just write it on a piece of paper by your computer, or you’ve defeated the whole point of this.

Now, we need to open the partition so we can create our encrypted data partition. We do this with the following:

  cryptsetup luksOpen /dev/hda4 crypt-home

This will make /dev/mapper/crypt-home, which is the device you use to mount your /home. Currently that partition is unformatted, so use your mkfs of choice to format it.

  mke2fs -j /dev/mapper/crypt-home

Next, make a temporary place to mount this so you can copy over your data, and mount it.

  mkdir /mnt/crypt-home
  mount /dev/mapper/crypt-home /mnt/crypt-home

And then, copy all your data from your current /home to the new one:

  rsync -tarv /home/* /mnt/crypt-home

Now would also be a good time to back up your important data to some other location.

Before we continue, we should configure the system to mount our new /home at boot. This requires editing /etc/fstab:

  # /etc/fstab
  # ... your other stuff ...
  /dev/mapper/crypt-home /home ext3 noatime 0 2

and /etc/conf.d/dmcrypt:

  # /etc/conf.d/dmcrypt
  # This file has all sorts of comments in it already
  # just uncomment the following:

  ## /home with passphrase
  target=crypt-home
  source='/dev/hda4'

Now all that is left is to remove the unencrypted copy of /home. First, you should go through and rm -rf anything that doesn’t contain sensitive information, like open source project code, your mp3s, etc.

The last step is to use the shred command to securely delete all the remaining files. Shred works by overwriting a file many times with different patterns to make recovering them extremely difficult. Use the following commands to securely delete all the files in home, and then remove all the empty directories:

  find -H /home/*/ -type f -exec shred -u -v {} \;
  rm -rf /home/*/

Now, reboot and if everything went well, you should be prompted for your password, and then everything should just work as it did before.

I just set up a commits list for Paludis’ subversion repository. You can subscribe to it at http://lists.pioto.org/mailman/listinfo/paludis-commits

Once we've converted native types we face the task of converting arbitrary classes and types. Another thing we'll need is extract the C++ type of a Java object (jobject in JNI).

For that task, we will follow the convention of calling the native pointers in Java classes _ptr. With that, we can define the following templates:

template <typename T_>
inline T_ * get_native_ptr(JNIEnv * env, const char * const class_name, jobject obj)
{
    jclass cls(env->FindClass(class_name));
    jfieldID ptr_field(env->GetFieldID(cls, "_ptr", "J"));
    jlong ptr(env->GetLongField(obj, ptr_field));
    env->DeleteLocalRef(cls);
    return from_java_ptr<T_>(ptr);
}

template <typename T_>
inline tr1::shared_ptr<T_> get_native_sptr(JNIEnv * env, const char * const class_name, jobject obj)
{
    jclass cls(env->FindClass(class_name));
    jfieldID ptr_field(env->GetFieldID(cls, "_ptr", "J"));
    jlong ptr(env->GetLongField(obj, ptr_field));
    env->DeleteLocalRef(cls);
    return from_java_ptr_sptr<T_>(ptr);
}

This should really be doing more error checking, but it is good enough for ilustrating how nice are things when using proper tools (both languages and libraries).

Converting arbitrary types also uses some template magic:

template <typename T_>
struct NativeToJavaTypeMapper
{
    jobject operator() (JNIEnv *, const T_ &);
};

Now we need specializations for each type we want to convert, for instance, converting a paludis' FSEntry into a java.io.File looks like the following:

template<>
struct NativeToJavaTypeMapper<FSEntry>
{
    jobject operator() (JNIEnv * env, const FSEntry & f)
    {
        jclass cls(env->FindClass("java/io/File"));
        jmethodID constructor(env->GetMethodID(cls, "<init>", "(Ljava/lang/String;)V"));
        jobject ret(env->NewObject(cls, constructor, to_java_string(env, stringify(f))));
        env->DeleteLocalRef(cls);
        return ret;
    }
};

Neat and clean.

For containers, a java.util.LinkedList would be used for paludis' Sequence; and java.util.TreeSet for paludis' Set:

template <typename T_, typename It_>
jobject range_to_list(JNIEnv * env, It_ begin, It_ end)
{
    jclass list_class(env->FindClass("java/util/LinkedList"));
    jmethodID constructor(env->GetMethodID(list_class, "<init>", "()V"));
    jobject our_list(env->NewObject(list_class, constructor));
    jmethodID add_method(env->GetMethodID(list_class, "add", "(Ljava/lang/Object;)Z"));

    for (It_ i(begin) ; i != end ; ++i)
        env->CallBooleanMethod(our_list, add_method, NativeToJavaTypeMapper<T_>()(env, *i));

    env->DeleteLocalRef(list_class);

    return our_list;
}

template <typename T_, typename It_>
jobject range_to_set(JNIEnv * env, It_ begin, It_ end)
{
    jclass set_class(env->FindClass("java/util/TreeSet"));
    jmethodID constructor(env->GetMethodID(set_class, "<init>", "()V"));
    jobject our_set(env->NewObject(set_class, constructor));
    jmethodID add_method(env->GetMethodID(set_class, "add", "(Ljava/lang/Object;)Z"));

    for (It_ i(begin) ; i != end ; ++i)
        env->CallBooleanMethod(our_set, add_method, NativeToJavaTypeMapper<T_>()(env, *i));

    env->DeleteLocalRef(set_class);

    return our_set;
}

And now defining NativeToJavaTypeMapper specializations for containers is quite easy:

template<>
template <typename T_>
struct NativeToJavaTypeMapper<tr1::shared_ptr<const Sequence<T_> > >
{
    jobject operator() (JNIEnv * env, const tr1::shared_ptr<const Sequence<T_> > & s)
    {
        return range_to_list<T_>(env, s->begin(), s->end());
    }
};

template<>
template <typename T_>
struct NativeToJavaTypeMapper<tr1::shared_ptr<const Set<T_> > >
{
    jobject operator() (JNIEnv * env, const tr1::shared_ptr<const Set<T_> > & s)
    {
        return range_to_set<T_>(env, s->begin(), s->end());
    }
};

I've spent a fair amount of the time fighting with make and the build system. It looks mostly sane now, though.

Dealing with exceptions has been a bit tricky, however, I think I have a good system to deal with it now, even though Ciaran tagged it as icky and ugly . That'd be the topic of the next part of the series.

The documentation is currently at http://dev.gentoo.org/~ferdy/paludis-jni/ . All of that has been accomplished in:

[ $ ~/git/paludis/jni(jni) ] git diff --shortstat trunk..
 65 files changed, 4371 insertions(+), 0 deletions(-)

Which is not a lot of code for what's exposed.

— ferdy

Leave a comment

After showing how is the structure of a regular Paludis class using JNI, next thing is showing part of the magic behind paludis_java.hh.

What's there is functions to convert Java types into Paludis (and C++ native) types and viceversa (this conversions exist only when needed, not for every single type). So, for instance, one of those conversions would be jboolean <--> bool and it looks like this:

inline jboolean to_java_boolean(bool b)
{
    return b ? JNI_TRUE : JNI_FALSE;
}

inline bool from_java_boolean(jboolean b)
{
    return b == JNI_TRUE;
}

That was easy, let me show you the dirty part. I call it dirty not because it is difficult or tricky to understand, but because it is a bit icky. It is the way we store the native Paludis pointers in Java classes:

inline jlong to_java_ptr(void * const ptr)
{
    return reinterpret_cast<jlong>(ptr);
}

template <typename T_>
inline T_ * from_java_ptr(jlong ptr)
{
    return reinterpret_cast<_ *>(ptr);
}

template <typename T_>
inline tr1::shared_ptr<T_> from_java_ptr_sptr(jlong ptr)
{
    return * reinterpret_cast<tr1::shared_ptr<T_> *>(ptr);
}

Although dirty, it is quite easy aswell. Something this bindings will be converting a lot is strings:

std::string
from_java_string(JNIEnv * const env, jstring s)
{
    const char * const c_s(env->GetStringUTFChars(s, 0));
    std::string result(c_s);
    env->ReleaseStringUTFChars(s, c_s);
    return result;
}

jstring
to_java_string(JNIEnv * const env, const std::string & s)
{
    return env->NewStringUTF(s.c_str());
}

And since C++ is such a nice language comparing and stringifying arbitrary types was just as easy (credits for this go to Mr. McCreesh):

template <typename T_>
jstring common_stringify(JNIEnv * const env, jlong ptr)
{
    return to_java_string(env, stringify(*from_java_ptr<T_>(ptr)));
}

template <typename T_>
jint common_compare(jlong lhs_ptr, jlong rhs_ptr)
{
    T_ * const lhs(from_java_ptr<T_>(lhs_ptr)), * const rhs(from_java_ptr<T_>(rhs_ptr));
    if (*lhs < *rhs)
        return -1;
    else if (*rhs < *lhs)
        return 1;
    else
        return 0;
}

This wasn't quite difficult and it certainly makes working with JNI easier. However, there is still stuff to do (actually, to show, since it is implemented and working in my git repository) like exception handling, converting arbitrary Paludis types and typesafe containers.

During the weekend I've written bindings for almost every core Paludis class and the patch is not that big:

[ $ ~/git/paludis(jni) ] git diff --shortstat trunk..
 30 files changed, 1975 insertions(+), 0 deletions(-)

Now it is time to stop the bindings for a while and start writing documentation, examples and integrating the bindings into the Paludis codebase properly.

Next part of the series will be about how I am converting arbitrary types and containers into Java types and typesafe collections respectively.

— ferdy

Leave a comment

Some days ago, the Wearer of the shiny hat started tinkering with JNI bindings for Paludis.

Even if we all think Java is perverse and should be avoided at all costs. Having used Java for the last four years and having studied part of the Java5 code I thought I could use everything I learnt and took the task of developing and maintaining the JNI bindings.

It has all been relatively easy. It's good that Ciaran started them; because his experience with the Ruby bindings gave the key to developing the Java bindings cleanly.

What we are currently doing is storing a C++ pointer in Java classes and passing it around in native methods. Then we delete it when Java's garbage collector decides it is good to call finalize.

So it mostly looks like this:

package paludis;
import paludis.Paludis;

public class Foo {
    private static Paludis _load_the_frickin_paludis_library = new Paludis();
    private long _ptr;

    private static native long _construct_string(String s);
    private static native void _destruct(long ptr);

    private static native boolean _is_foo(long ptr);

    protected void finalize() throws Throwable {
        _destruct(_ptr);
        super.finalize();
    }

    public Foo(String s) {
        _ptr = _construct_string(s);
    }

    public boolean isFoo() {
        return _is_foo(_ptr);
    }
}

Then the C++ part looks like this:

/* vim: set sw=4 sts=4 et foldmethod=syntax : */

#include "foo.hh"
#include "paludis_java.hh"
#include <jni.h>
#include <paludis/foo.hh>

using namespace paludis;
using namespace paludis::java;

JNIEXPORT jlong JNICALL
Java_paludis_Foo__1construct_1string(JNIEnv * env, jclass, jstring s)
{
    return to_java_ptr(new Foo(from_java_string(env, s)));
}

JNIEXPORT void JNICALL
Java_paludis_Foo__1destruct(JNIEnv *, jclass, jlong ptr)
{
    delete from_java_ptr<Foo>(ptr);
}

JNIEXPORT jboolean JNICALL
Java_paludis_Foo__1is_1foo(JNIEnv *, jclass, jlong ptr)
{
    return to_java_boolean(from_java_ptr<Foo>(ptr)->is_foo());
}

Here, all the magic is happening behind the scenes in the paludis_java.hh header which I'll show in a future post of the Paludis meets Java series.

— ferdy

Leave a comment

So, I finally dropped lighttpd/fcgi and switched back to Apache 2.2. Trac is running with mod_python now. I think I’ve tracked down the main cause of our load/memory issues – googlebot. It was indexing all 4000 some revisions of our svn repository via the trac browser… bad stuff, but it should be blocked from that by our robots.txt now.

Also, I’ve added WebDav for Subversion now. Paludis is at http://svn.pioto.org/paludis, and the same is true for most of the other repos I host as well. Now those who have evil company/school firewalls and proxies can get through.

So, been a while, but here’s something I just threw together. It’s still a little rough around the edges, but it seems to be more robust than tpb for my system, at least, and it doesn’t require me to make /dev/nvram available.

It uses /sys/class/backlight/thinkpad_screen/actual_brightness (from the thinkpad_acpi module in the 2.6.22 kernel) to figure out the screen brightness, and /proc/acpi/ibm/volume to figure out the volume and mute states, also provided by the thinkpad_acpi module (I’ll switch to using sysfs for this when it becomes available).

It also requires the ruby-xosd package, which you can get from my overlay: svn://svn.pioto.org/pioto-overlay

For lazy paludis users, just add this to /etc/paludis/repositories/pioto-overlay.conf:

format = ebuild
location = /var/paludis/repositories/pioto-overlay
sync = svn://svn.pioto.org/pioto-overlay
master_repository = gentoo
names_cache = /var/cache/paludis/names
write_cache = /var/cache/paludis/metadata

You can get the current version of this script from subversion at: http://svn.pioto.org/rbtpb/trunk/

It is often surprising how some people can really lose control of a situation.

Take this example http://dev.gentoo.org/~ferdy/tmp/seemant-moron.log . It is actually sad having to deal with that kind of thing. I asked three times for an apology and (surprise surprise) I didn't get one. One publicly in IRC, another one in a /query and a third one by mail.

Besides wondering who is the one acting like a moron, I really wonder why people have the feeling that Gentoo is oh so nice and friendly.

No love.

— ferdy

PS: Despite of his attitude, I still can't be bothered with personal issues. So if he wanted me to have an issue with him, I'm thankful he failed.

Leave a comment

I didn't really want to go there because it meant lots of car (4100 Km in a week) and not enough time to enjoy and getting to know each place we went.

In the end, well, I went. And I'm quite happy I did it. It is a very interesting country where the first thing you learn is:

Hurry kills...
    and lazyness re-kills. — Anyone in Morocco

We crossed the Atlas twice. Some of the views up there are just amazing.

We also went to the dunes in Merzouga. If you ever consider going to Morocco, you can't miss a couple of nights in 'Atlas du Sable' with Ali's crew, those guys make it very comfortable and enjoyable.

We also visited M'Hamid and that part of Morocco, that part was a bit harder emotionally. But it is still enlightening and a very interesting place to visit.

Last place was Marrakech. I loved it so much that I'll have to go again because two nights is not enough for this city.

Only complains are: Too many hours of 'car stuck because of sand' and not enough time to get to know any place. but all in all it's been a wonderful trip and a very nice experience. Their way of living and thinking is just too different from anything in Europe.

We took hundreds of photos, some of them are on my picasaweb album: http://picasaweb.google.com/fpereda/MoroccoTour07 .

If you ever have the oportunity to go to Morocco, go for it.

— ferdy

Leave a comment

Today, after a lot of polishing, I have finally committed Python Hooks to trunk.

If you are not familiar with Paludis Hooks, please read Hooks guide first.

A .py hook is much like .hook hook, but written in Python and with full access to the current Paludis environment through Python bindings. For each hook it can handle it must, at minimum, define a function named hook_run_$HOOK which accepts exactly two positional arguments: the current Environment used by Paludis, and the additional Hook environment variables represented by a Python dictionary. It may also define the hook_depend_$HOOK and hook_after_$HOOK functions which must take exactly one argument, the Hook environment, and return a list of strings. For example:

def hook_run_install_all_post(env, hook_env):
    from paludis import *

    print "* Checking for monkeys..."

    if list(env.package_database.query(Query.Package("nice/monkey"))):
        print "Found a monkey!"
    else:
        print "No monkeys found"

def hook_depend_install_all_post(hook_env):
    # we need to run after the Paludis standard eselect_env_update hook
    return ["eselect_env_update"]

def hook_after_install_all_post(hook_env):
    # if checking for rabbits or squirrels, do those first
    return ["check_for_rabbits", "check_for_squirrels"]

Leave a comment

Google Summer of Code is over, but, hopefully, the projects are not - at least I am not going to stop working on mine. I must say that SoC was very successful for me, I have learned plenty of new things and my dream of contributing to the Paludis itself came true. I must also thank all of the folks at #paludis, especially Ciaran McCreesh, my mentor Saleem Abdulrasool and rest of the Gentoo SoC staff for all the help.

By the way, it seems like a good time to start thinking about a project for SoC 2008 :)

Leave a comment

As Python 2.5 is now unmasked in Gentoo I decided to try it with Boost.Python, more specifically with Python bindings for Paludis. Everything seems to work, but, of course, boost needs to be rebuilt first.

Leave a comment

Lesson of the day: check the return code of all Unix calls, even those that can't possibly fail. Because that way, you catch that you're calling close() on a file descriptor twice straight away, rather than when that file descriptor gets reused in another thread. Also, don't assume that read() returning <= 0 means it's returning EOF rather than an error, because that way you catch when a bogus repeated close() in another thread is clobbering your FD, rather than reading in partial output and assuming it's ok.

Alternate lesson of the day: C's lack of exceptions sucks.

The Cokehabit strikes again!

Following his previous attempts to sneakily make Paludis disappear by having it deleted from Wikipedia using a supposedly anonymous account, our good friend George Prowse has repeated his naughtiness using his real account. His claimed reasons for deletion are laughable:

• "It is unfinished software". Well yes. Like almost all other software.
• "Used by a very small section of the community (mostly developers)". An amusing claim. Clearly untrue, as one could tell by looking at IRC, bug reports, mailing lists and forum posts.
• "There are other Portage replacements and add-ons that are not in Wikipedia." Doubly silly. There are lots of things that aren't in Wikipedia -- should articles referring to things related to them be deleted too? And there are other Portage replacements and add-ons that are in Wikipedia, including some that have been broken for years and thus don't have any users at all.

I shall be amused to see how badly Wikipedia handles this...

So, I’ve been digging into why Windows XP can’t deal with a system clock set in UTC. Apparently, it even keeps its internal time as UTC, but it still won’t deal with a system clock set to non-local time.

I found this article from an MSDN blog, which gives a decent summary of the reasons why. Basically, it would break support for some pre Windows NT/XP things. Well, XP and Vista already break enough other things compatability-wise that this shouldn’t be such an issue now.

I also found this nice article on all the reasons it should be changed.

Apparently, a registry key does exist that once allowed people to use a system clock set to UTC, but it isn’t documented, and most of the newer Windows code hasn’t been updated to support it. No word on if Vista does a better job at this. Has anyone else had luck getting windows to cope with a UTC system clock?

To answer a question in #paludis: given something like this:

template <typename T_>
struct Odd
{
    struct Thingie
    {
        T_ value;
    };
};

template <typename T_>
bool operator== (const typename Odd<T_>::Thingie & l, const typename Odd<T_>::Thingie & r)
{
    return l.value == r.value;
}

You can explicitly instantiate it like this:

template bool operator== <char> (const Odd<char>::Thingie &, const Odd<char>::Thingie &);

Note the explicit <char>. It's slightly fiddly...

At first glance, POSIX threads (pthreads) appear to support different thread priorities within a given process. This would, for example, allow us to have threads that precalculate cacheable values and so on if there is idle time available without disrupting the main program. The pthread.h header includes pthread_setschedprio and pthread_setschedparam. Unfortunately, both are useless on Linux since SCHED_OTHER's permitted priorities range from 0 to 0, and other schedulers are only usable by privileged real-time processes. So much for that.

One might think to look at how various higher level thread libraries handle things. However, Boost threads don't yet support priorities at all, and Glib's g_thread_set_priority does nothing on Linux.

There is a solution, albeit one that is horribly unportable and relies upon behaviour that directly contradicts POSIX. According to POSIX, niceness is a per-process property, with each thread within a process sharing the same nice value. With the old 2.4 kernel Linux threads, Linux simply ignores this, assigns each thread its own PID and lets setpriority operate on a per-thread basis. With 2.6 kernel NPTL, threads share the PID of the initial process, but also have their own special thread ID known as a TID obtainable via gettid().

Alas, gettid() isn't wrapped by Glibc, so it has to be called via syscall(). Putting this together, we get the following way of reducing a thread's priority:

#ifdef __linux__
    setpriority(PRIO_PROCESS, syscall(SYS_gettid), std::max(19, getpriority(PRIO_PROCESS, 0) + 10));
#endif

Unportable and ugly, yes, but surprisingly effective. The following dirty hack, which has several synchronisation bugs that aren't fixable without obfuscating the point I'm trying to make, shows things clearly:

#include <pthread.h>
#include <sched.h>
#include <utility>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <syscall.h>
#include <iostream>
#include <sstream>

namespace
{
    bool finish(false);

    void * count(void *)
    {
        std::stringstream s("nothing");
        for (unsigned x(0) ; x < 1000000 ; ++x)
        {
            if (finish)
                break;
            s.str("");
            s << x;
        }
        finish = true;

        std::cout << "I counted up to " << s.str() << std::endl;
    }

    void * nice_count(void *)
    {
#ifdef __linux__
        setpriority(PRIO_PROCESS, syscall(SYS_gettid),
                std::max(19, getpriority(PRIO_PROCESS, 0) + 10));
#endif
        count(0);
    }
}

int main(int, char *[])
{
    pthread_t t1, t2;
    pthread_create(&t1, 0, &count, 0);
    pthread_create(&t2, 0, &nice_count, 0);

    pthread_join(t1, 0);
    pthread_join(t2, 0);
}

On a single-way box:

I counted up to 999999
I counted up to 44371

On an SMP box:

I counted up to 999999
I counted up to 957915

On a single-way box run using nice -n19:

I counted up to 999999
I counted up to 975303

Not perfect, but a lot better than nothing at all.

The Wikipedia article on Paludis has been proposed for deletion. Normally this would be somewhere between irrelevant and mildly entertaining. However, a closer look turns this into a highly amusing commentary on the state of Gentoo afairs.

The article was not marked for deletion by an established Wikipedophile. It wasn't even marked for deletion by an unestablished Wikipedophile. No, it was marked for deletion by a non-logged-in user. This isn't necessarily suspicious on its own -- some people don't see the need to create an account. On the other hand, the contributions from that IP address at the time of deletion are slightly interesting:

• First unlabelled contribution, marking Paludis for deletion
• Second unlabelled contribution, removing a link to Paludis

A little while later, another user reverts the deletion and explains their position on the article talk page. The anonymous user promptly pops back up and unreverts the deletion and leaves the following claim on the talk page:

It is a minor project and is used my an extreme minority, mostly developers or the project itself and until it gains some weight it should not be in wikipedia

-- Someone using the IP address 81.79.237.61

Which is certainly an interesting claim... A quick glance at bug reports, mailing list, the Gentoo forums, various wiki contributors and IRC discussions shows that it's blatantly untrue -- so why would someone who doesn't even log in make such bold claims? And why would that user then go and remove yet another link to Paludis from another wiki article?

Perhaps there's something suspicious going on here...

Googling the IP finds a rather bizarre Gentoo forums post written by one Cokehabbit. Mmm. And then there's this:

-!- cokehabit [n=root@user-514fed3d.l4.c4.dsl.pol.co.uk]
-!-  ircname  : root
-!-  server   : irc.freenode.net [http://freenode.net/]
-!- End of WHOIS

Strange, that host resolves to 81.79.237.61, the IP being used to do the edits... It looks like our anonymous contributor isn't quite so anonymous after all. No, it's the entirely nonymous and not in the least bit nocent George Prowse, a.k.a. Cokehabit, who has his very own Wikipedia account. But why isn't he using it here? Could it be because he's known to be extremely biased on the issue, having already spent huge amounts of effort trying to derail the project in other ways?

Yes, it appears that his latest cunning plan is trying to get Paludis deleted from Wikipedia. Unfortunately, he seems to have forgotten that Wikipedia is not Canada.

This really is rather pathetic. It's even more pathetic than his previous round of miserable propaganda, where he repeatedly claimed that kloeri left Gentoo because of me, spb and eroyf -- a claim that was met with much laughter and derision by all those whose heads are not stuck so far up amne's arse that they can almost see the Harring's feet. But rather than retracting his claims when told by eroyf why kloeri really left, our good friend George resorted to claiming that, I quote:

you chose to be part of the ciaranm group, they lie all the time and whether or not you actually lied, you were part of massive lies by proxy

-- George Prowse, a.k.a. 'Cokehabit', known user of IP address 81.79.237.61

Whilst not quite so incoherent as Jory's latest, it does strike me as rather amusing that someone would make such implications whilst themselves blaming me for kloeri leaving and going around using easily-detectable sockpuppets on Wikipedia in order to try to get an article deleted... Please, George, next time at least have the good grace to create yourself a proper fake account or something.

As for the article itself... I'd merely like to point out that cokehabit claiming "it's not finished" (the only true claim made so far) as a reason for removing an article on software he doesn't like (not that he's ever used it) is a bit like Ted Haggard trying to get an article on evolution deleted because the theory "is not complete"...

Well, my work on adding Manifest-2 generation support to Paludis is nearly done. Look for the final commit tomorrow afternoon.

So, now I need another project to pick up. Suggestions?

See this post to the gentoo-dev mailing list for the nitty-gritty.

Hmm, looks like either Typo is being stupid, or Planet is…

I’d like to blame planet, but I’ll look into my config a bit later on.

Paludis 0.24.5 has been released:

• More ldconfig oddities fixed.
• The eselect news module is no longer shipped as part of Paludis.

Paludis 0.24.4 has been released:

• The ldconfig check introduced in 0.24.3 would incorrectly detect a BSD ldconfig on some GNU systems. This is now fixed.

Paludis 0.24.3 has been released:

• Bug fix: -* in keywords.conf now works.
• Non-GNU ldconfig is now supported.
• Various minor bug fixes and docs updates.

This is purely a bug fix release. We're doing this because 0.26 isn't going to be ready for a while yet.

Good to see there's a distribution with even more screwed up management than Gentoo:

lxnay>   i'm the sabayon linux chief arch
lxnay>   Just wanna know the implications of integrating paludis
ciaranm> you use binaries, right?
ciaranm> paludis doesn't do binaries currently, and when it does it will use a different format
ciaranm> that's probably the big issue...
ciaranm> other than that there's nothing major. with 0.26 we'll even be able to do special
         distribution-specific stuff for you... you'd just need to give us a .conf file for your 
         distro and compile with --with-default-distribution=sabayon
lxnay>   well if you devs are interested in sabayon, just contact me
ciaranm> lxnay: we don't care about sabayon, in that none of us use it. but we have no objection
         to supporting it, if there's demand from users
lxnay>   bah
lxnay>   die alone then
-!- lxnay has left #paludis

On the plus side, that's one less VM to manage.

I’m currently working on Manifest2 support for paludis (both validating and generating). It should be done Real Soon Now. If you’re interested, pay attention to Ticket 253.

Yesterday I finally got around to migrating the Trac page for Paludis from using SQLite to MySQL. Hopefully this will help with all the system load and locking issues we’ve seen recently.

I'm starting this blog over again... I was having issues with the old one, which was running off of sqlite. Now I'm using mysql, and I hope this will help the overall availability and system load.

Today's "annoying C++ standard library weirdness":

#include <list>
#include <functional>
#include <string>

struct Blah;

class Comparator :
    public std::binary_function<std::string, std::string, bool>
{
    private:
        Comparator(const Comparator &);
        Comparator & operator= (const Comparator &);

        // ... and presumably a load of member data

    public:
        Comparator(const Blah &);
        ~Comparator();

        bool operator() (const std::string &, const std::string &) const;
};

void f(const Blah & blah)
{
    std::list<std::string> l;
    Comparator c(blah);
    l.sort(c);
}

For those not entirely familiar with C++:

• std::list is sorted through a member, not the standard algorithm std::sort, because std::sort requires random access iterators to give log-linear performance. A different log-linear algorithm exists for sorting lists.
• C++ library routines involving sorting or comparisons usually accept an optional comparator argument. This allows comparisons to be performed using something other than operator<. This is extremely useful when several meaningful orderings exist for a class, or when no generally meaninful ordering exists at all.
• Unimplemented private methods that would otherwise be compiler-synthesised are used to indicate 'not permitted'. (C++0x may include a prohibited access specifier to make this idiom unnecessary.)

Unfortunately, the above code won't compile -- it will give errors about Comparator's copy constructor being private. This is highly annoying, because there's no technical reason it shouldn't work. Instead, it's down to how the standard specifies the list sort member template:

namespace std {
    template <class T, class Allocator = allocator<T> >
    class list {
    public:
        // ... other stuff snipped

        void sort();
        template <class Compare> void sort(Compare comp);

        // ... other stuff snipped
    };
}

ISO/IEC 14882, Second Edition: Programming Languages -- C++

Note how Compare is passed by value (that is, a copy is made). You might think we could cheat:

    Comparator c(blah);
    l.sort<const Comparator &>(c);

Unfortunately, that fails with a similar error. It's because the GCC implementation does the following:

    __counter->merge(__carry, __comp);

Where __comp is our comparator and merge is the list member template, which, alas, has signature:

template <class Compare> void merge(list<T,Allocator>& x, Compare comp);

Interestingly, the standard requires only that the comparator provides a "strict weak ordering". So far as I can see, at no place does it explicitly require that the copy operation be available. One could claim that it implies it by passing by value rather than const reference; however, a counter-argument would be that the standard does not explicitly require (although it does hint) a strict weak ordering to be const, and therefore const reference passing is not an option (and non-const reference passing would rule out passing a temporary).

The obvious solution is to make the comparator copy-constructible, for example by storing associated (presumably cache) data in a std::tr1::shared_ptr rather than as a direct member variable. This is mildly inconvenient, but not the end of the world. The less obvious but much quicker solution?

    Comparator c(blah);
    l.sort(std::tr1::cref(c));

A fairly regular question in #paludis is "which C++ books should I buy?". Another question commonly asked in #paludis is "what's GoF?". Because I'm highly lazy:

Learning

• If you can already code, work your way through the next section in order. If you can't, don't learn C++ as your first language.

General reference

• "The C++ Programming Language", 3rd Ed, Bjarne Stroustrup. (We call this TC++PL)
• "The C++ Standard Library: A Tutorial and Reference", 1st Ed, Nicolai M. Josuttis. (TC++SL) This is one of two candidates for "best written technical book I've read".
• "The C++ Standard Library Extensions: A Tutorial and Reference", 1st Ed, Pete Becker. (TC++SLE) This covers the upcoming TR1, which is not fully supported in most compilers yet.
• "C++ Templates: The Complete Guide", 1st Ed, David Vandevoorde and Nicolai M. Josuttis.

Design and Usage

• "Effective C++: 55 Specific Ways to Improve Your Programs and Designs", 3rd Ed, Scott Meyers. (EffC++)
• "Design Patterns: Elements of Reusable Object-Oriented Software", 1st Ed, Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides. (GoF, for Gang of Four, because the title is way too long) Not a C++ book per se, although it uses pre-Standard C++ and Smalltalk for code examples.
• "Modern C++ Design: Generic Programming and Design Patterns Applied", 1st Ed, Andrei Alexandrescu. (MC++D) Note that using any technique mentioned in this book is liable to get you kicked in the face, but being aware of said techniques is useful.
• "Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library", 1st Ed. (EffSTL)
• "More Effective C++: 35 New Ways to Improve Your Programs and Designs", 1st Ed, Scott Meyers. (MEffC++) This one's a lot more in depth and much more design-oriented than EffC++. Some of the material is slightly dated, but the underlying techniques are worth knowing.
• "The Design and Evolution of C++", 1st Ed, Bjarne Stroustrup. Covers up to part way through the standardisation process -- many useful explanations of why things are the way they are.
• "Inside the C++ Object Model", 1st Ed., Stanley B. Lippman. Answers questions like exactly what your compiler (probably) does when calling a virtual function, and how inheritance and up/down casts work at the machine level. Doesn't tell you anything you need to know to write working code, but knowing what's going on at the machine level can help when designing classes.

Non-C++ Books

• "Mastering Algorithms with Perl", 1st Ed, Jon Orwant, Jarkko Hietaniemi, John Macdonald. This is a really good data structures and algorithms book, ruined only by its use of Perl. If you can get your head around Perl's unreadable syntax, it's a good read.
• "Mastering Regular Expressions", 2nd Ed, Jeffrey E. F. Friedl. The other candidate for "best written technical book"; once you've read this, you'll understand why we don't use regular expressions on the library side for Paludis.
• "Advanced Programming in the Unix Environment", 2nd Ed, W. Richard Stevens, Stephen A. Rago.

Stuff I haven't read properly that others consider Good

• "Accelerated C++: Practical Programming by Example", 1st Ed, Andrew Koenig, Barbara Moo. Supposedly a good "learning C++" book.
• "Exceptional C++", 1st Ed, Herb Sutter. I've so far only glanced at Sutter's books, but I've seen a couple of his lectures and he appears to know what he's talking about...
• "Numerical Recipes in C++: The Art of Scientific Computing", 2nd Ed, William H. Press, Saul A. Teukolsky, William Vetterling, Brian P. Flannery.
• "Patterns for Parallel Programming", 1st Ed, Timothy G. Mattson, Beverly A. Sanders, Berna L. Massingill. I want to know whether this is any good before I buy it.
• "Standard C++ IO Streams and Locales: Advanced Programmers Guide and Reference", 1st Ed, Angelika Langer, Klaus Kreft. The definitive book on how C++ IOStreams work, and required reading if you ever have the misfortune of having to know anything about the implementation side of things.

The editions are important! You don't want to get earlier editions of some of these books. Earlier editions of "The C++ Programming Language", for example, refer to pre-Standard C++.

Interesting side-note: most of the above C++ books are published by Addison Wesley. None of the C++ books are published by O'Reilly. O'Reilly's C++ books really really suck. This may come as a surprise to people from other programming language backgrounds.

A simple Python program to look at, and two questions:

class Person:
    def __init__  (self, first_name, surname):
        self.__first_name = first_name
        self.__surname = surname

    def first_name (self):
        self.__first_name
    
    def surname (self):
        self.__surname

if __name__ == "__main__":
    fred_smith = Person("Fred", "Smith")
    fred_bloggs = Person("Fred", "Bloggs")

    if fred_smith.first_name == fred_bloggs.first_name:
        print "Same"
    else:
        print "Different"
    
    fred_smith.surname = "Bloggs"

    if fred_smith.surname == fred_bloggs.surname:
        print "Same"
    else:
        print "Different"

Without running the program, what output does it produce? What point am I making here?

Paludis provides a way of checking whether it's safe to uninstall a particular package. The --with-dependencies option for --uninstall will include any packages depending upon a target (recursively) in the uninstall list. If a package is safe to uninstall, no extra packages will be removed:

$ paludis --uninstall --with-dependencies --pretend hilite

These packages will be uninstalled:

* app-misc/hilite-1.5::installed

But if a package is still in use, extra packages will show up:

$ paludis --uninstall --with-dependencies --pretend imlib2

These packages will be uninstalled:

* media-gfx/feh-1.3.4::installed
* media-libs/giblib-1.2.4::installed
* x11-themes/fluxbox-styles-fluxmod-20050128-r1::installed
* x11-wm/fluxbox-1.0_rc3-r493700:0::installed
* media-libs/imlib2-1.3.0::installed

Currently, Paludis only uses this behaviour if explicitly told to. Using the same logic, it could easily be made to refuse to uninstall a package that is depended upon unless the user explicitly passes, say, --permit-unsafe-uninstalls. The question is, should it?

C++ has a feature known as covariant return types. The standard says:

The return type of an overriding function shall be either identical to the return type of the overridden function or covariant with the classes of the functions. If a function D::f overrides a function B::f, the return types of the functions are covariant if they satisfy the following criteria:

• both are pointers to classes or references to classes
• the class in the return type of B::f is the same class as the class in the return type of D::f, or is an unambiguous and accessible direct or indirect base class of the class in the return type of D::f
• both pointers or references have the same cv-qualification and the class type in the return type of D::f has the same cv-qualification as or less cv-qualification than the class type in the return type of B::f.

If the return type of D::f differs from the return type of B::f, the class type in the return type of D::f shall be complete at the point of declaration of D::f or shall be the class type D. When the overriding function is called as the final overrider of the overridden function, its result is converted to the type returned by the (statically chosen) overridden function (5.2.2).

ISO/IEC 14882, Second Edition: Programming Languages -- C++

What this means is that the following is legal:

struct Animal
{
};

struct Cow :
    Animal
{
};

struct AnimalMaker
{
    virtual Animal * make_animal() = 0;
    virtual AnimalMaker * clone() = 0;
};

struct CowMaker :
    AnimalMaker
{
    virtual Cow * make_animal();   // Implements Animal::make_animal
    virtual CowMaker * clone();    // Implements Animal::clone
};

Unfortunately, this trick doesn't work with anything except plain old pointers or references -- methods returning boost::shared_ptr or std::tr1::shared_ptr can't make use of this:

#include <tr1/memory>

struct Animal
{
};

struct Cow :
    Animal
{
};

struct AnimalMaker
{
    virtual std::tr1::shared_ptr<Animal> make_animal() = 0;
    virtual std::tr1::shared_ptr<AnimalMaker< clone() = 0;
};

struct CowMaker :
    AnimalMaker
{
    virtual std::tr1::shared_ptr<Cow> make_animal();  // Error: invalid covariant return type
    virtual std::tr1::shared_ptr<CowMaker> clone();   // Error: invalid covariant return type
};

One can achieve a similar effect as follows:

#include <tr1/memory>

struct Animal
{
};

struct Cow :
    Animal
{
};

struct AnimalMaker
{
    virtual std::tr1::shared_ptr<Animal> make_animal() = 0;
    virtual std::tr1::shared_ptr<AnimalMaker> clone() = 0;
};

struct CowMaker :
    AnimalMaker
{
    std::tr1::shared_ptr<Cow> make_cow();

    virtual std::tr1::shared_ptr<Animal> make_animal()
    {
        return make_cow();
    }

    std::tr1::shared_ptr<CowMaker> clone_cow_maker();

    virtual std::tr1::shared_ptr<AnimalMaker> clone()
    {
        return clone_cow_maker();
    }
};

But this requires users of the derived class to use a different function if they need the derived class. It would not be legal to rename make_cow and clone_cow_maker to clone and make_animal and have two methods with the same name for each, since they would be treated as overrides with invalid covariant return types.

For C++0x there are two open proposals to add explicit virtual function overrides: N2108 and N1826. Depending upon how they progress, it might be possible to implement covariant shared_ptr returns by implementing the virtual from the base and adding in a second, explicitly new function.