Aller au contenu | Aller au menu | Aller à la recherche


samedi, décembre 3 2011

celery and django-celery ebuilds for Gentoo

I'm currently working on a secret project using django and celery, and I needed ebuilds for celery since there are none in official portage tree. I started with some ebuilds found on quantumsummers overlay, but they were really outdated.

So I wrote some new ebuilds for celery, djang-celery and related dependencies. I only tested them on my server but I think they should work for you. They are available on my overlay:


I recently got a Samsung Series 9 from Intel. The laptop is really great, but some features like keyboard backlight where not working correctly. The driver responsible for that is samsung-laptop (drivers/platform/x86/samsung-laptop.c).

So, having some experience about x86 platform drivers, I took a shoot and tried to hack the driver to make it work. This driver does not use ACPI or WMI but poke directly the BIOS instead using an interface called SABI (Samsung ABI ?). So the hard part was to understand how this interface work, and of course, without any kind of documentation. I may describe later how I did it, but it was fun :).

So, if you have a samsung laptop, you should try the new samsung-laptop driver ! The code is not yet merged in mainline, but its available on github:

The code was tested on 3 different models, and I'm looking for more feedback.

vendredi, avril 15 2011

euscan: how to find new upstream versions ?

What is euscan ?


euscan is available in portage as a dev package (app-portage/euscan-9999). This tool allow to check if a given package/ebuild has new upstream versions or not. It will use different heuristic to scan upstream and grab new versions and related urls.

This tool was designed to mimic debian's uscan, but there is a major difference between the two: uscan uses a specific "watch" file that describe how it should scan packages, while euscan uses only what can already be found in ebuilds. Of course, we could later add some informations in metadata.xml to help euscan do its job more efficiently.

euscan heuristics are described in the "How does-it works ?" section.


$ euscan amatch

  • dev-ruby/amatch-0.2.7 [gentoo]

Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild Repository: gentoo Homepage: Description: Approximate Matching Extension for Ruby

  • SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem'
  • Using:

Upstream Version: 0.2.8

$ euscan rsyslog

  • app-admin/rsyslog-5.8.5 [gentoo]

Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild Repository: gentoo Homepage: Description: An enhanced multi-threaded syslogd with database support and more.

  • SRC_URI is ''
  • Scanning:${PV}.tar.gz
  • Scanning:
  • Generating version from 5.8.5
  • Brute forcing:${PV}.tar.gz
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]
  • Trying: ... [ ok ]
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]
  • Trying: ... [ ok ]
  • Trying: ... [ ok ]
  • Trying: ... [ ok ]
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]
  • Trying: ... [ !! ]

Upstream Version: 5.9.1
Upstream Version: 5.9.0
Upstream Version: 5.9.3
Upstream Version: 5.9.2

Hidden settings

You can configure some settings using the command line, but the file of the euscan package contains more settings, including blacklists and default settings.

Maybe we should add the ability to use /etc/euscan.conf and ~/.config/euscan/euscan.conf to override these settings.

euscan-www: euscan as a service

euscan-www is a web application that aggregates euscan results. For example there is an instance of euscan-www that monitors gentoo-x86 + some official overlays currently hosted at .

euscan-www uses django and provides some custom commands to feed the database. You can use euscan-www on you system tree, or preferably you can use a local tree to avoid messing with your system.


Like any django web app, just start by editing and then run these two commands.

$ python syncdb $ python migrate

Now your instance is ready, you can just run this command to browse it. If you want to host it publicly you should use a real webserver.

$ python runserver

Creating a local tree

Create a local tree with all that portage (and layman would need). There is an example in euscanwww/scripts/local-tree/. See to know what env variables you need to run any portage related command in this local tree.

Scanning process

The scanning process is done by You should read carefully this script, and adapt it to your needs. For example it uses gparallel to launch multiple process at a time, and you should adapt that to your number of cpu and network bandwith.

Once your is ok, just run it.

$ sh

Custom Django management commands

euscan-www povides some new management commands, here is a short description of these commands. Use "help" or read to get more informations.


List packages stored in database.


Scan the portage tree and store new packages and versions in the database.

Scan metadata and looks for homepage, maintainers and herds.


Scan upstream package. The prefered way to use this script it to first launch euscan on some packages, store the result of the file, and feed this command with the result.


Update statistics and rrd files.


If you deleted your rrd files, this script will use the database to regen them.

How does it work ?

euscan has different heuristics to scan upstream and provides multiple "handlers". First, here is a description of the generic handler.

Scanning directories

The first thing to do is to scan directories. It's also what uscan do, but it uses a file that describe what url and regexp to use to match packages.

euscan uses SRC_URI and tries to find the current version (or part of this version) in the resolved SRC_URI and generate a regexp from that.

For example for app-accessibility/dash-4.10.1, SRC_URI is: mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2 euscan will scan pages based on this template:${0}.${1}/dasher-${PV}.tar.bz2

Then, from that, it will scan the top-most directory that doesn't depend on the version, and try to go deeper from here.

Brute force

Like when scanning directories, a template of SRC_URI is built. Then euscan generate next possible version numbers, and tries to download the url generated from the template and the new version number.

For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6: SRC_URI is '' Template is${PV}.tar.gz Generate version from 0.6: 0.7, 0.8, 0.10, ... Try new urls:, etc..


euscan uses blacklist for multiple purposes.


For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4 is good example because it's a package which version will always be 3.4 (Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).


Some packages are dead, but SRC_URI refers to sources that are still being updated, for example: sys-kernel/xbox-sources that uses the same sources as vanilla-sources but is not updated the same way.


For urls that are not browsable. mirror://gentoo/ is a good example: it's both stupid to scan it and very long/expensive.


Disable brute force on those packages and urls. Most of the time it's because upstream is broken and will answer HTTP 200 even if the file doesn't exist.


Don't respect robots.txt for these domains (sourcefourge, berlios,

Site handlers


A site handler that uses the Pecl/PEAR rest API (


This one uses rubygems's json API (


Uses pypy's XML rpc API.

More ?

An euscan instance is available at, it scans gentoo-x86 and some overlays every weeks. The source code of euscan is available at 

dimanche, novembre 28 2010

eeepc-wmi's back in the game

Last week, I had an Eeepc 1005PE on my desk and I was finally able to work on eeepc-wmi. I added touchpad led support, fixed conflict with eeepc-laptop and rfkill support for wlan, bluetooth and 3g.

Now, please switch back to eeepc-wmi and remove acpi_osi=(Linux|!Windows 2009):

Why a new eeepc platform driver ?

The eeepc-laptop module use an ACPI device (ASUS010) which have usually the same interface in all eeepc. For example the WLDS will always control the wlan card.

But on newer eeepc, Asus introduced Windows 7 support. Models with windows 7 support have a new WMI interface. And when an OS is detected as Windows 7, the old ASUS010 interface is disabled.

OSI is an ACPI method provided by the operating system that can be invoked by ACPI BIOS code. It is used by BIOS developers to detect which operating system is running. The Linux kernel returns true when OSI is invoked with any known Windows OS string.

So, Linux is identified as Windows 7, ASUS010 device is disable, eeepc-laptop can't be loaded.

Why acpi blacklisting is not a solution ? (or acpi_osi="!Windows 2009")

A quick workaround is to boot with acpi_osi="!Windows 2009" (and sometime acpi_osi="Linux") but:

From Len Brown, maintainer of Linux ACPI subsystem

   Not necessarily the right fix. We have gone to a lot of trouble to discourage BIOS vendors from depending on the ill-defined OSI, so I hesitate to invoke it -- even for a workaround.

Anyway, no that thw wmi interface is here, we can't bet that the old interface won't be maintained and won't expose all features.

What models are affected ?

All Eeepc with windows 7 support (>= 1000).


Currently, a lot of distributions are using acpi_osi="Linux" with eeepc-wmi blacklisted or worse, loaded *with* eeepc-laptop ! They did that because eeepc-wmi was either unavailable or not finished (this is mainly because acpi4asus project was lacking hardware to develop the driver, let's skip that, but donations are still more than welcome !).

What next ?

In latest kernel, eeepc-wmi should support all important features:

  • backlight
  • hotkeys
  • cpufv (S.H.E.)
  • rfkill (wlan, bluetooth, 3g)
  • touchpad led
  • debugfs (to test not yet supported features without patching the kernel).

So if you're a distrbution developper, please switch back to eeepc-wmi, and remove that ugly acpi_osi=. eeepc-laptop is really not done to work on windows 7 aware Eeepcs.

If you're just a user, and you think eeepc-wmi is lacking something, please open a bug report ( or kernel bugzilla, but CC me ) or use the mailing list to send patchs.

lundi, octobre 25 2010

QtTvDB 0.2, Series Watcher 0.1.1, Lugdulo'V 0.3 and QCommandLine 0.2 released !


Library to use XML API with Qt. You'll need an API key to use this software, you can get one from Used by Series Watcher.

New in 0.2.0:

  • Guest list is now a QStringList

Website: QtTvDB

Series Watcher

Qt clone of SeriesFinal, not only for Maemo but for all platforms supported by Qt. It's a TV series browser and tracker application. Its goal is to help you manage the TV shows you watch regularly and keep track of the episodes you have seen so far. The shows and episodes can be retrieved automatically by using the “TheTVDB API”.

Series Watcher is a free software. It's available on multiple platforms: Windows, Mac-OS X, Linux, Maemo. We're also working on Symbian and iOS ports.

Website: Series Watcher


Find a bike quickly in your city, with map and geolocalisation support ! Currently available for 110 cities in 14 countries, but you're free to write your own plugins, and if your city is not supported, send us a bug report ! To see the list of supported bike sharing network, check our plugins page.

New in 0.3:

  • Symbian port
  • Small bug fixes (Montreal plugin, Plugins Dialog, etc..)
  • Offline mode on Maemo 5

Website: Lugdulo'V

Symbian port

Lugdulo'V source code has been ported to Symbian, but there is no binary available at the moment. It should be available through OVI store very soon.


Command line parser for Qt (like getopt). Features include options, switchs, params and automatic --version/--help generation.

New in 0.2:

  • Fixed some parsing bugs

Website: QCommandLine

lundi, août 2 2010

QEMU: adaptive Tight, ZRLE and ZYWRLE

Adaptive Tight

I recently added an infrastructure allowing to know the average update frequency for a given region on the screen. Basically, I store last 10 updates times for each 64x64 regions, and only count updates times that happened in the lasts 5 seconds.

Then, I plugged this infrastructure with tight, and adaptive tight is born. The main difference with a normal tight encoding, is that all updates are lossless by default, but if a region is updated a lot, then it'll switch to jpeg encoding.


ZRLE and ZYRLE are now supported by QEMU ! ZRLE is a basic run length encoding compressed with zlib, so it's lossless. ZYWRLE is a little more complex, and is well described here: . ZYWRLE is a lossy encoding.

Bandwidth Charts

Here are some results comparing ZRLE, ZYWRLE, and adaptive tight, while playing the big buck bunny trailler.



ZYWRLE have very good results, but the main issue is that it's a lossy encoding, and all the screen is lossy if you use zywrle. But that won't be an issue for long with the new adaptive infrastructure :).

Bellow are some screenshots of the different encodings.


Tight 5, Tight 8, Tight NoJpeg

bunny-tight5.png bunny-tight8.png bunny-tight.png



bunny-zywrle2.png bunny-zywrle5.png bunny-zywrle8.png

Static Images

Tight 5, ZYWRLE 5, ZYWRLE 8

kde-tight5.png kde-zywrle5.png kde-zywrle8.png

lundi, juin 14 2010

QEMU: Tight PNG and some profiling developer kanaka recently contacted me. noVNC is a HTML-5 VNC client, that means it's (almost) all in javascript, and only runs in a web browser. He needed an encoding that could be lossless and would allow a fast render in a web browser. We decided to try to add PNG to tight encoding.

Patchs can be found in my git tree, but nothing is standardized, nor merged in upstream QEMU,

Here was the first results with the non-threaded VNC server:


As we can see, PNG really slows down the guest, and the bandwidth is low only because a lot of frames are skipped and/or delayed.

So, I tried with the threaded VNC server.


And already way better, but a lot of frames were skipped, again.

Then, kanaka tried to disable png fitlers, and here is the result:


Without filters, the video was a lot smoother, and it's mainly why the bandwidth is higher: more frames were sent !

During these tests, I also wanted to profile QEMU. I tried callgrind, oprofile, gprof, but none of them did what I want (callgrind is actually a great tool, but it was really too slow with QEMU). I finally tried perf (/usr/src/linux/tools/perf/), and it's really great ! No overhead, and really simple to use :

 perf record -f -g -p <pid of vnc thread>
 perf report

jeudi, juin 3 2010

QEMU: Tight JPEG compression

I finished adding Tight JPEG subencoding to QEMU (patchs are comming soon). Here are the first results:

Kubuntu boot


Big buck bunny 33sec trailer


I used the non-threaded VNC server for these graph, and I cheated a little with the charts to make them overlap correctly,

I'll try to benchmark the thread and non-threaded server with jpeg as soon as I can get another machine to test that.

vendredi, mai 28 2010

QEMU: Threaded VNC Server results

Another subject of the GSoC was:

VNC server coroutine

Our builtin VNC server is implemented as a combination of a pure state machine to allow scalability. The VNC protocol is very complex though and this implementation is difficult to follow as a result. This project would convert to the existing server to be implemented as a cooperatively scheduled thread in the style of a coroutine. Time permitting, a pure coroutine implementation using ucontext could be implemented. Also, given sufficient progress, one could pursue running the VNC server thread in parallel with the QEMU IO thread in order to achieve better overall scalability.

Last week I decided to try to do that. My implementation use a worker thread and a queue of encoding jobs. The main thread produce encoding jobs and queue them, then the worker thread consume them as soon as possible. Resize (vnc_dpy_resize) and Copy (vnc_dpy_cpy) are still synchroneous at the moment. There is also a lock to block encoding process while updating the display to avoid screen corruptions.

The code can be found on my git tree and will be cleaned and sent soon to qemu-dev mailing list.

Here are some preliminary results. I created a simple image with Suse Studio which just boot and halt.

VNC Sync VS Async

I also made some test while playing the Big Buck Bunny Trailer, the 33sec video takes between 37 and 40 sec to play with vnc ! With the threaded vnc server, it takes between 34 and 35 seconds. But videos are something ticky, and I've got another project for them: adding some kind of mpeg subencoding to VNC :)

mardi, mai 18 2010

QEMU: some charts on tight zlib and hextile

Again, some charts to show some tight results.

I recorded the Kubuntu boot process (showing kpackagekit and a notification at the end). I used tcpdump to record the data and a small python script to convert tcpdump output into .csv files. You can find this small script here:



You can see that tight is slightly better than zlib for this workload, but remember that it currently doesn't support gradient filter and jpeg encoding. There is just one thing to remenber: don't use hextile !

mardi, mai 11 2010

GsoC 2010 QEMU: First (dumb) tight benchmarks

For my first (and probably last) summer of code, I've been selected to work on QEMU VNC server encodings. QEMU's built-in VNC server only supports the Hextile encoding. Hextile only uses tile-based compression. Other encodings are more popular such as Tight and ZRLE that use zlib-based compression. These encodings significantly reduce the bandwidth required by VNC.

I started a basic tight implementation (mostly stolen from tightvnc/tigervnc/libvncserver, which all share the same tight.c). But the code have been rewritten to match QEMU coding style, and I'll continue to clean it and improve it.

So, QEMU tight encoding currently support full color rectangles (compressed with zlib), and solid color rectangles (a rectangle with only one color). And here is some preliminary results. I used kubuntu livecd splashscreen, keeping the vnc client open from seconds 29 to 22.



  • initial rectangle 16384 bytes
  • following rectangles (avg) 139 bytes
  • total 17323 bytes


  • initial rectangle 13247 bytes
  • following rectangles (avg) 130 bytes
  • total 13247 bytes


  • initial rectangle 10150 bytes
  • following rectangles (avg) 90 bytes
  • total 11345 bytes

Keep in mind that this is not a real benchmark, but tight appears to be the winner for this workload, but zlib is not far and probably eats less cpu.

vendredi, janvier 22 2010

lighttpd configuration: variables and includes

A good way to factorise lighttpd configuration files is to use variables and includes. For example, let's say that you have a lot of vhost, each need specific php configuration.

First, create a _php.conf file:

var.phpconf += " -d session.save_path=" + var.statedir + "/sessions"
fastcgi.server = (
        ".php" =>(
                "localhost" =>(
                        "bin-path" => "/usr/bin/php-cgi " + var.phpconf,
                        "socket" => var.rundir + "/php.socket"

Then in your main configuration file:

var.statedir = "/var/lib/lighttpd/"
$HTTP["host"] =~ "" {
 var.phpconf = " -d magic_quotes_gpc=Off "
 var.rundir = "/var/run/lighttpd/" 
 include "_php.conf"
$HTTP["host"] =~ "" {
 var.phpconf = " -d magic_quotes_gpc=On "
 var.rundir = "/var/run/lighttpd/"
 include "_php.conf"

Using that trick, you can do a lot more. For example, here is my configuration file for

# {{{ variables     =  "iksaif"
var.basedir  = "/home/iksaif/public_html"
var.logdir   = "/var/log/lighttpd/" +
var.statedir = "/var/lib/lighttpd/" +
var.rundir   = "/var/run/lighttpd/" +
var.username = "iksaif"
var.groupname= "iksaif"
var.docroot  = var.basedir
var.port     = 8081
# }}}
include "_base.conf"
include "_php.conf" migration to lighttp

This week I migrated (and,, etc...) from apache to lighttpd !

It used to be like described in my previous post:

Now, the architecture is more like that:

lighttpd-misc --- git
                    \- redmine
                    \- lighttpd-iksaif -
                    \- apache          - svn
                                           \- trac
                                           \- finestown
                   \- lighttpd-zordania
                   \- lighttpd-zorddev
                   \- lighttpd-uffs    - blog
                                           \- git
                                           \- redmine

Is not as performant, but it's a lot more flexible. I had to keep apache for some svn repository, because there is no dav-svn module for lighttpd.

I learned a lot, so I'll post some lighttpd configuration examples that could help you one day.

jeudi, octobre 8 2009

Let's Work

Voila maintenant deux semaines que je bosse sur Virtual Browser à commonIT et tout ce passe bien, je suis sur des projs sympa.

A côté de ça, j'ai quand même trouvé un peu de temps pour bosser sur acpi4asus et en particulier pour android-x86. Android-x86, c'est un port d'android pour les "netbooks" et en particulier les EeePC.

J'ai donc backporté eeepc-laptop pour le kernel utilisé par android (un 2.6.29), et là je fais quelques tests avant d'envoyer ça upstream (cwhuang m'ayant gentillement donné un accès en écriture à Pour l'instant, j'ai juste ajouté un patch pour régler la keymap.

Mais bon, le projet en est à ses débuts, alors il y'a vraiment de quoi faire =)

dimanche, septembre 13 2009

Le code et ses raisons : goto en C

Just un post bookmark vers le blog d'rz0 et en particulier un article sur les goto

lundi, août 10 2009

Playing with my EOS400D






samedi, juin 13 2009

Poke-a-hole, let's align structs

Just readed . That's a cool tool !

You can find pahole here: :

git clone git://
cd pahole
mkdir build
cmake ..

I tested it on ikmalloc

$ pahole
struct chunk {
        struct chunk *             main_next;            /*     0     8 */
        struct chunk *             main_prev;            /*     8     8 */
        struct chunk *             free_next;            /*    16     8 */
        struct chunk *             free_prev;            /*    24     8 */
        size_t                     size;                 /*    32     8 */
        size_t                     type;                 /*    40     8 */
        size_t                     asked_size;           /*    48     8 */
        size_t                     alignment;            /*    56     8 */
        /* --- cacheline 1 boundary (64 bytes) --- */
        /* size: 64, cachelines: 1, members: 8 */

Ok, everything is aligned =).

$ pahole fs/nandfs/nandfs.o --reorganize --show_reorg_steps -C nandfs_io_ops
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
        /* XXX 4 bytes hole, try to pack */
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
        bool                       direct;               /*    40     1 */
        /* size: 48, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 4 */
        /* padding: 7 */
        /* last cacheline: 48 bytes */
/* Moving 'direct' from after 'retlen' to after 'ofs' */
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
        bool                       direct;               /*    20     1 */
        /* XXX 3 bytes hole, try to pack */
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
        /* size: 40, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 3 */
        /* last cacheline: 40 bytes */
/* Final reorganized struct: */
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
        bool                       direct;               /*    20     1 */
        /* XXX 3 bytes hole, try to pack */
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
        /* size: 40, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 3 */
        /* last cacheline: 40 bytes */
};   /* saved 8 bytes! */

Wow, cool, 8 byte saved :P

vendredi, mai 8 2009

Securing a shared Web Server

Typical users use shared hosting for their blog, etc ... On this kind of offer, the hoster have multiple clients on the same hardware. As it is impossible to be sure that clients applications are not vulnerable, we should at least be sure that if one is vulnerable, it won't bother the others. We will show you how a bad web server configuration could lead to disaster, and how to secure such a server.

The pdf version can be found at

Lire la suite...

dimanche, mai 3 2009

C Hacks

Some Links:

mercredi, mars 18 2009

Coccinelle: Patchs sémantiques

Je viens de découvrir coccinelle, et c'est vraiment un outil sympa.

Imaginez que vous voulez remplacer tout les expr ? 1 : 0 par des !!expr. Et bah avec coccinelle rien de plus simple :


@@ expression cond; @@
- cond ? 1 : 0
+ !!cond
@@ expression cond; @@
- (!!cond)
+ !!cond


int main(void)
  int val;
  val = val ? 1 : 0;
  printf("%d", val);
  printf("%d", val ? 1 : 0);
  printf("%d", (val ? 1 : 0));

$ spatch -sp_file test.cocci test.c

--- test.c      2009-03-18 11:06:17.000000000 +0100
+++ /tmp/cocci-output-14620-bafaae-test.c       2009-03-18 11:21:17.000000000 +0100
@@ -2,8 +2,8 @@ int main(void)
   int val;
-  val = val ? 1 : 0;
+  val = !!val;
   printf("%d", val);
-  printf("%d", val ? 1 : 0);
-  printf("%d", (val ? 1 : 0));
+  printf("%d", !!val);
+  printf("%d", !!val);

Donc vachement pratique pour changer des trucs quand un search&replace ne suffit pas ! Plus d'informations:

- page 1 de 2