/var/log/iksaif

http://github.com/kanaka/noVNC/ developer kanaka recently contacted me. noVNC is a HTML-5 VNC client, that means it's (almost) all in javascript, and only runs in a web browser. He needed an encoding that could be lossless and would allow a fast render in a web browser. We decided to try to add PNG to tight encoding.

Patchs can be found in my git tree, but nothing is standardized, nor merged in upstream QEMU,

Here was the first results with the non-threaded VNC server:

As we can see, PNG really slows down the guest, and the bandwidth is low only because a lot of frames are skipped and/or delayed.

So, I tried with the threaded VNC server.

And already way better, but a lot of frames were skipped, again.

Then, kanaka tried to disable png fitlers, and here is the result:

Without filters, the video was a lot smoother, and it's mainly why the bandwidth is higher: more frames were sent !

During these tests, I also wanted to profile QEMU. I tried callgrind, oprofile, gprof, but none of them did what I want (callgrind is actually a great tool, but it was really too slow with QEMU). I finally tried perf (/usr/src/linux/tools/perf/), and it's really great ! No overhead, and really simple to use :

 perf record -f -g -p <pid of vnc thread>
 perf report

Another subject of the GSoC was:

VNC server coroutine

Our builtin VNC server is implemented as a combination of a pure state machine to allow scalability. The VNC protocol is very complex though and this implementation is difficult to follow as a result. This project would convert to the existing server to be implemented as a cooperatively scheduled thread in the style of a coroutine. Time permitting, a pure coroutine implementation using ucontext could be implemented. Also, given sufficient progress, one could pursue running the VNC server thread in parallel with the QEMU IO thread in order to achieve better overall scalability.

Last week I decided to try to do that. My implementation use a worker thread and a queue of encoding jobs. The main thread produce encoding jobs and queue them, then the worker thread consume them as soon as possible. Resize (vnc_dpy_resize) and Copy (vnc_dpy_cpy) are still synchroneous at the moment. There is also a lock to block encoding process while updating the display to avoid screen corruptions.

The code can be found on my git tree and will be cleaned and sent soon to qemu-dev mailing list.

Here are some preliminary results. I created a simple image with Suse Studio which just boot and halt.

I also made some test while playing the Big Buck Bunny Trailer, the 33sec video takes between 37 and 40 sec to play with vnc ! With the threaded vnc server, it takes between 34 and 35 seconds. But videos are something ticky, and I've got another project for them: adding some kind of mpeg subencoding to VNC

For my first (and probably last) summer of code, I've been selected to work on QEMU VNC server encodings. QEMU's built-in VNC server only supports the Hextile encoding. Hextile only uses tile-based compression. Other encodings are more popular such as Tight and ZRLE that use zlib-based compression. These encodings significantly reduce the bandwidth required by VNC.

I started a basic tight implementation (mostly stolen from tightvnc/tigervnc/libvncserver, which all share the same tight.c). But the code have been rewritten to match QEMU coding style, and I'll continue to clean it and improve it.

So, QEMU tight encoding currently support full color rectangles (compressed with zlib), and solid color rectangles (a rectangle with only one color). And here is some preliminary results. I used kubuntu livecd splashscreen, keeping the vnc client open from seconds 29 to 22.

hextile

• initial rectangle 16384 bytes
• following rectangles (avg) 139 bytes
• total 17323 bytes

zlib

• initial rectangle 13247 bytes
• following rectangles (avg) 130 bytes
• total 13247 bytes

tight

• initial rectangle 10150 bytes
• following rectangles (avg) 90 bytes
• total 11345 bytes

Keep in mind that this is not a real benchmark, but tight appears to be the winner for this workload, but zlib is not far and probably eats less cpu.

This week I migrated iksaif.net (and zordania.com, finestown.com, etc...) from apache to lighttpd !

It used to be like described in my previous post: http://xf.iksaif.net/blog/index.php?post/2009/05/08/Securing-a-shared-Web-Server

Now, the architecture is more like that:

lighttpd-misc --- git
                    \- redmine
                    \- lighttpd-iksaif - xf.iksaif.net
                    \- apache          - svn
                                           \- trac
                                           \- finestown
                   \- lighttpd-zordania
                   \- lighttpd-zorddev
                   \- lighttpd-uffs    - blog
                                           \- git
                                           \- redmine

Is not as performant, but it's a lot more flexible. I had to keep apache for some svn repository, because there is no dav-svn module for lighttpd.

I learned a lot, so I'll post some lighttpd configuration examples that could help you one day.

Just un post bookmark vers le blog d'rz0 et en particulier un article sur les goto

Just readed . That's a cool tool !

You can find pahole here: http://git.kernel.org/?p=linux/kernel/git/acme/pahole.git :

git clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/pahole.git
cd pahole
mkdir build
cmake ..
make

I tested it on ikmalloc

$ pahole libikmalloc.so
 
struct chunk {
        struct chunk *             main_next;            /*     0     8 */
        struct chunk *             main_prev;            /*     8     8 */
        struct chunk *             free_next;            /*    16     8 */
        struct chunk *             free_prev;            /*    24     8 */
        size_t                     size;                 /*    32     8 */
        size_t                     type;                 /*    40     8 */
        size_t                     asked_size;           /*    48     8 */
        size_t                     alignment;            /*    56     8 */
        /* --- cacheline 1 boundary (64 bytes) --- */
 
        /* size: 64, cachelines: 1, members: 8 */
};

Ok, everything is aligned =).

$ pahole fs/nandfs/nandfs.o --reorganize --show_reorg_steps -C nandfs_io_ops
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
 
        /* XXX 4 bytes hole, try to pack */
 
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
        bool                       direct;               /*    40     1 */
 
        /* size: 48, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 4 */
        /* padding: 7 */
        /* last cacheline: 48 bytes */
};
 
/* Moving 'direct' from after 'retlen' to after 'ofs' */
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
        bool                       direct;               /*    20     1 */
 
        /* XXX 3 bytes hole, try to pack */
 
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
 
        /* size: 40, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 3 */
        /* last cacheline: 40 bytes */
}
 
/* Final reorganized struct: */
struct nandfs_io_ops {
        struct nandfs_object *     obj;                  /*     0     8 */
        uint8_t *                  buf;                  /*     8     8 */
        uint32_t                   ofs;                  /*    16     4 */
        bool                       direct;               /*    20     1 */
 
        /* XXX 3 bytes hole, try to pack */
 
        size_t                     len;                  /*    24     8 */
        size_t                     retlen;               /*    32     8 */
 
        /* size: 40, cachelines: 1, members: 6 */
        /* sum members: 37, holes: 1, sum holes: 3 */
        /* last cacheline: 40 bytes */
};   /* saved 8 bytes! */

Wow, cool, 8 byte saved :P

Some Links:

• http://graphics.stanford.edu/~seand...
• http://aggregate.org/MAGIC/

UFFS Website is now open at http://uffs.org/. You'll find news, documentation, etc ...

NandFS git tree will also probably be opened this week at git.iksaif.net. I'll post some mails on linux mailing-lists.

J'ai fait quelques modifications à mon malloc, qui s'appelle d'ailleurs maintenant ikmalloc. Les sources sont disponibles sur git.iksaif.net

Pour les récuperer: git clone git://git.iksaif.net//ikmalloc.git

Améliorations:

• Support de mmap
• Redéfinition de memalign, valloc, posix_memalign
• Les tailles allouées ne sont plus uniquement des puissances de sizeof (struct chunk)

Les prochaines versions apporterons les modifications suivantes:

• Utilisation de mmap ou brk suivant la taille à allouer
• Support de munmap et mremap
• Amélioration de la compatibilité avec ptmalloc
• Malloc thread-safe (il y'a actuellement des problèmes avec les programmes multi-threadés).

Cet article s'adresse à ceux qui auraient une (grosse) base de donnée, et qui aimeraient disposer d'un moteur de recherche rapide et performant. Une première solution est d'utiliser les index full text et de chercher directement dans la base en faisant des requêtes SQL. Cette solutions présente des avantages certains, mais aussi les désavantages suivants:

• La taille de l'index peut rapidement devenir très importante
• L'index peut ralentir les insertions (les insertions lockent la table sous MySQL !)
• Les fonctions mises à dispositions par MySQL (ou PostgreSQL) en matìere de recherche full-text sont assez limitées.
• Il faut refaire soit même le moteur de recherche qui va créer les requètes SQL

Sphinx

Sphinx

A Propos

Sphinx est un moteur de recherche full-text, distribué sous GPL v2. (Une licence commerciale est disponible)

En gros, c'est un moteur de recherche à part entière, crée pour fournir des fonctions de recherches rapides et pertinentes pour d'autres applications. Sphinx a été fait en particulier pour s'intégrer facilement avec des base de données SQL et des languages de scripts (typiquement, le couple PHP/MySQL). Pour l'instant les sources de données peuvent être des bases de donnée MySQL ou PostgreSQL, mais aussi des flux XML.

Sphinx signifie SQL Phrase Inde_x__

Programmes

• indexer: un utilitiaire pour créer les index
• search: a simple utilitaire (de test) pour rechercher en ligne de commandes
• searchd: un service pour faire des recherches en utilisant sphinxapi
• sphinxapi: un ensemple d'acpi pour les languages les plus utilisés PHP, Python, Java, Perl et Ruby.

Avantages

• Indexation assez rapide (sur mes machines, à peu près 2000 docs/sec)
• Support de plusieurs langues, y compris CJK
• Il est facile de mettre le moteur de recherche sur une (voir plusieurs) machine différente et ainsi mieux répartir la charge
• API PHP, Python, Java, Perl, et Ruby

Articles

Les articles en italique ne sont pas encore publiés

• Configuration de base
• Indexation
  • Indexation basique
  • Utilisation des range queries
  • Diminuer la charge du processus d'indexation: utilisation d'un index "delta"
  • Gestion de plusieurs langues
• Tests avec l'utilitaire search
• Utilisation de l'API PHP
• Utilisation de l'API Python