A frog around the world

Follow up on Debian & Clang

A week after the Euro LLVM conference, I am glad to share the results I presented and discussed during the event.

1) Parallel building infrastructure using clang instead of gcc

Based on the work that Alexander Pashaliyski and I did during the GSoC 2012, we are now able to automatically rebuild each package uploaded in Unstable transparently. Results are published here:
http://buildd-clang.debian.net/
This provides a quick and easy way to any Debian contributors or upstream packaged in Debian to check how the package behaves with clang.
For Debian Developer and maintainer, a per maintainer view is also provided. This will list all packages maintained by a Debian packager. My page as example

Here is an example of the list of package:

And an example of package failing:

At time of writing, most of the packages in unstable have been rebuilt (16 % are remaining). However, due to resource constraints, only amd64 packages are built. Contact me if you have resources to share for other architectures.

Thanks to Paul Wise, the PTS (Debian package tracking system) proposes a link to the clang build logs (failing or not).

A repository will be automatically updated with the results of these builds (once I find the time to get into buildd).

2) clang-built repository
Thanks to the Amazon Cloud, with our new DPL (Debian Project Leader), Lucas Nussbaum, we rebuilt unstable last April and published it at the following address:

deb http://clang.debian.net/repository-2013-04-07/ unstable-clang main

This allows some cool stuff like:

$ echo "deb http://clang.debian.net/repository-2013-04-07/ unstable-clang main">>/etc/apt/sources.list
$ apt-get update
$ apt-get install coreutils/unstable-clang
$ ls
$ awk

Note that this repository was only one-shoot and does not get updated.

Quality of the binary produced (performance and size) can be tested thanks to this repository. Feedbacks are welcome!

Original post blogged on b2evolution.

A New Identi.ca/Twitter feed: Debian uploads

After the Debian new packages, removal and bugs feeds (see the previous blog posts), I also plugged a feed with the last uploads in the archive:
Debian uploads on Identi.ca
Debian uploads on Twitter

As a reminder, here is the list.
For identi.ca:
Debian uploads
Debian NEW queue
Debian bugs
Debian removed packages

For Twitter:
Debian uploads
Debian NEW queue
Debian bugs
Debian removed packages

Original post blogged on b2evolution.

LLVM Debian/Ubuntu nightly packages

Lately, I have been working on providing nightly packages of the whole LLVM toolchain.
With the help of folks from Intel, Google and Apple, I am happy to announce the publication of these packages:
http://llvm.org/apt/

Built through a Jenkins instance (http://llvm-jenkins.debian.net/), packages for Debian wheezy and Unstable and Ubuntu quantal, precise and raring are created twice a day.

3.2 and 3.3 llvm-toolchain packages are currently waiting in the Debian NEW queue.

More information on the LLVM blog.

Original post blogged on b2evolution.

Rebuild of Debian using clang 3.2

After the studies of the rebuild of the Debian archive using clang 2.9 & 3.0 and 3.1, the results of the 3.2 rebuild have been published on http://clang.debian.net.

The percentage of failure is the same as clang 3.1: 12.1% of failure. That means that on 18264 packages, 2204 failed to be built with clang (it was 17710/2137 with the version 3.1).

The fact that the percentage is the same can be explained by at least two reasons:
First, some upstreams packages have been fixed to be built correctly with clang.
Second, the new warnings + -Werror (example: -Wsometimes-uninitialized) introduced in clang 3.2 triggered some new build failures.

To conclude, I think that the 3.2 release confirms the turning point of the 3.1. In term of support of the C and C++ standard, clang has now reached an equivalent quality to gcc (with better detection of errors and an extended set of warnings in -Wall).
Now, from the perspective of the Debian rebuilds, the decrease of number of failures will come from the upstream developers improving their codes (exemple of the error non-void function should return a value) or the Debian Developer/maintainer fixing the programming errors.

Original post blogged on b2evolution.

Some more cool stuff with LLVM/Clang

One of the drawback of C is the hard memory management.

It is because of this aspect that Clang provides a way to detect, at runtime, memory errors. Called Address Sanitizer, it allows, while the program is running, to keep track of the memory and detect typical errors (out-of-bound accesses, use of a variable after a free, etc).

# With the following packages (version 3.2-1~exp3)
$ sudo apt-get install clang clang-3.2 compiler-rt -t experimental

Taking the simple following example, even if the errors are obvious, compilers will accept this code.

#include <stdlib.h>
int main() {
    char *x = (char*)malloc(10 * sizeof(char*));
    free(x);
    return x[5];
}

Built and run with:

$ clang -O1 -g -fsanitize=address -fno-omit-frame-pointer foo.c -o foo
$ ./foo &> memoryDebug.log

The previous command will generate a log file. Log which can be post processed with the asan_symbolize command.

$ asan_symbolize memoryDebug.log

which will give:

=================================================================
==21368== ERROR: AddressSanitizer: heap-use-after-free on address 0x7fb22e547f45 at pc 0x408c44 bp 0x7ffff60c10b0 sp 0x7ffff60c10a8
READ of size 1 at 0x7fb22e547f45 thread T0
    #0 0x408c43 in main /tmp/foo.c:5
    #1 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
0x7fb22e547f45 is located 5 bytes inside of 80-byte region [0x7fb22e547f40,0x7fb22e547f90)
freed by thread T0 here:
    #0 0x408c90 in __interceptor_free ??:0
    #1 0x408c0a in main /tmp/foo.c:4
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
previously allocated by thread T0 here:
    #0 0x408d50 in __interceptor_malloc ??:0
    #1 0x408bff in main /tmp/foo.c:3
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
Shadow byte and word:
  0x1ff645ca8fe8: fd
  0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
More shadow bytes:
  0x1ff645ca8fc8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd0: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fe0: fa fa fa fa fa fa fa fa
=>0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff0: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff8: fa fa fa fa fa fa fa fa
  0x1ff645ca9000: fa fa fa fa fa fa fa fa
  0x1ff645ca9008: fa fa fa fa fa fa fa fa
Stats: 0M malloced (0M for red zones) by 1 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 1 calls
Stats: 0M really freed by 0 calls
Stats: 0M (128 full pages) mmaped in 1 calls
  mmaps   by size class: 8:2047;
  mallocs by size class: 8:1;
  frees   by size class: 8:1;
  rfrees  by size class:
Stats: malloc large: 0 small slow: 1
==21368== ABORTING

The main advantage compare to valgrind is that asan is supposed to be way more faster.

Threads can be also tricking to develop.
With the following example from stolen from upstream:

#include <pthread.h>
int Global;
void *Thread1(void *x) {
   Global = 42;
   return x;
}

int main() {
   pthread_t t;
   pthread_create(&t, NULL, Thread1, NULL);
   Global = 43;
   pthread_join(t, NULL);
   return Global;
}

$ clang -fsanitize=thread -g -O1 foo2.c -fPIE -pie -o foo
$ ./foo
==================
WARNING: ThreadSanitizer: data race (pid=21416)
  Write of size 4 at 0x7f2f1a214a50 by thread 1:
    #0 Thread1 /tmp/foo2.c:4 (exe+0x00000000f850)

  Previous write of size 4 at 0x7f2f1a214a50 by main thread:
    #0 main /tmp/foo2.c:10 (exe+0x00000000f8a4)

  Thread 1 (tid=21417, running) created at:
    #0 pthread_create ??:0 (exe+0x00000001267e)
    #1 main /tmp/foo2.c:9 (exe+0x00000000f894)

==================
ThreadSanitizer: reported 1 warnings

Note that clang also provides scan-build, a static analyzer for memory issues. Not as powerful as the Address Sanitizer (it only works on a file), it provides some excellent reports. See the Wouter's blog post on this subject or the automatic report of scan-build on Scilab.

Original post blogged on b2evolution.

Mini Debconf 2012 - videos and feedbacks

A bit more than two weeks after the Mini Debconf in Paris, I am glad to say that the videos of the event are finally published (the sound is not very good for the 4 first presentations, sorry about that).
They will be also available on the new IRILL website with a video player when ready.
All slides are also available on the page of the event.

I believe that there is a consensus about the quality of the event. We had around 150 people attending to the event, many interesting and various talks.
As usual, it was nice to meet some old and new friends from Debian.

Group picture by Frederic Lehobey

For those who wonder, I am confident there will be a 2013 Parisian Mini Debconf.

Various feedbacks about the event:
Lucas Nussbaum
Stefano Zacchiroli
Vincent Untz
Raphael Hertzog
Pietro Abate
Logilab (Julien Cristau)
The 'official' Debian news

And, once more, many thanks to the sponsors!

Original post blogged on b2evolution.

Mini Debconf & merchandising

Next week end, November 24th and 25th, the mini Debconf will take place at EPITA in Paris.

Mehdi Dogguy and I worked hard on the organization and on building a great program for this 2012 conference. This mini debconf will cover many subjects like Gnome (both as upstream and downstream), the Release team, how the Linux is packaged in Debian, etc.
The keynote will be "Free software and Debian, 20 years after" by Roberto Di Cosmo.

Also, some merchandising will be proposed during the event by the association Debian France. Here is a quick list:

Polos

Long-sleeved polo
40 euros - 10 pieces

Sleeveless polo
25 euros - 100 pieces

Thanks to Tanguy Ortolo for taking take of the order.

Buff

Buff Debian
18 euros - 75 pieces.

It is described as Original Multifunctional Headwear. Real life example in video.
I love them!

Finally, we will sell two kinds of Debian branded Sticker Portable:

Sticker Debian
1 euro - 100 pieces.

They can be also ordered on the it2l website. Thanks to Jérôme Lemaire for providing these great products.

If these products are successful, we will probably produce more for FOSDEM!

For more information about the conference:
The official website
Wiki page for subscription

Original post blogged on b2evolution.

libc++: New C++ standard library in Debian

Thanks to the work of Andrej Belym, Debian has now a new C++ standard library. This work has been done in the context of the Google Summer of Code 2012.
Available in Debian Experimental, this new packages provides both the runtime libraries (libc++abi1) and the C++ headers (libc++-dev).

With this library, it is possible to build a C++-based program without any dependency on libstdc++.

For example, as detailed in README.Debian, with the (amazing) C++ code:

// foo.cpp
#include <iostream>

int main() {
std::cout < < "plop" << std ::endl;
return 0;
}

with clang++ (or g++) will give:

$ clang++ -o plop foo.cpp
$ ldd plop |grep c++
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15791cb000)

Using libc++, it will drop the dependency on libstdc++ to use lib++

$ clang++ -stdlib=libc++ -o plop foo.cpp
$ ldd plop |grep c++
libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f87464df000)

For the record, it is not that trivial to do with g++. The command being:

g++ -nostdlib -lc++ -lc++abi -std=c++11 -o plop /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o foo.cpp /usr/lib/x86_64-linux-gnu/crtn.o -isystem/usr/include/c++/v1 -lc -lgcc_s

Besides this change, libc++ provides a support of C++ 11, considered as "correct" against the C++11 standard by upstream.

// bar.cpp
#include <chrono>
int main() {
return 0;
}

clang++ -stdlib=libc++ -o bar bar.cpp

More information:

Original post blogged on b2evolution.