Sylvestre's blog

A week after the Euro LLVM conference, I am glad to share the results I presented and discussed during the event.

1) Parallel building infrastructure using clang instead of gcc

Based on the work that Alexander Pashaliyski and I did during the GSoC 2012, we are now able to automatically rebuild each package uploaded in Unstable transparently. Results are published here:
http://buildd-clang.debian.net/
This provides a quick and easy way to any Debian contributors or upstream packaged in Debian to check how the package behaves with clang.
For Debian Developer and maintainer, a per maintainer view is also provided. This will list all packages maintained by a Debian packager. My page as example

Here is an example of the list of package:

And an example of package failing:

At time of writing, most of the packages in unstable have been rebuilt (16 % are remaining). However, due to resource constraints, only amd64 packages are built. Contact me if you have resources to share for other architectures.

Thanks to Paul Wise, the PTS (Debian package tracking system) proposes a link to the clang build logs (failing or not).

A repository will be automatically updated with the results of these builds (once I find the time to get into buildd).

2) clang-built repository
Thanks to the Amazon Cloud, with our new DPL (Debian Project Leader), Lucas Nussbaum, we rebuilt unstable last April and published it at the following address:

deb http://clang.debian.net/repository-2013-04-07/ unstable-clang main

This allows some cool stuff like:

$ echo "deb http://clang.debian.net/repository-2013-04-07/ unstable-clang main">>/etc/apt/sources.list
$ apt-get update
$ apt-get install coreutils/unstable-clang
$ ls
$ awk

Note that this repository was only one-shoot and does not get updated.

Quality of the binary produced (performance and size) can be tested thanks to this repository. Feedbacks are welcome!

Lately, I have been working on providing nightly packages of the whole LLVM toolchain.
With the help of folks from Intel, Google and Apple, I am happy to announce the publication of these packages:
http://llvm.org/apt/

Built through a Jenkins instance (http://llvm-jenkins.debian.net/), packages for Debian wheezy and Unstable and Ubuntu quantal, precise and raring are created twice a day.

3.2 and 3.3 llvm-toolchain packages are currently waiting in the Debian NEW queue.

More information on the LLVM blog.

After the studies of the rebuild of the Debian archive using clang 2.9 & 3.0 and 3.1, the results of the 3.2 rebuild have been published on http://clang.debian.net.

The percentage of failure is the same as clang 3.1: 12.1% of failure. That means that on 18264 packages, 2204 failed to be built with clang (it was 17710/2137 with the version 3.1).

The fact that the percentage is the same can be explained by at least two reasons:
First, some upstreams packages have been fixed to be built correctly with clang.
Second, the new warnings + -Werror (example: -Wsometimes-uninitialized) introduced in clang 3.2 triggered some new build failures.

To conclude, I think that the 3.2 release confirms the turning point of the 3.1. In term of support of the C and C++ standard, clang has now reached an equivalent quality to gcc (with better detection of errors and an extended set of warnings in -Wall).
Now, from the perspective of the Debian rebuilds, the decrease of number of failures will come from the upstream developers improving their codes (exemple of the error non-void function should return a value) or the Debian Developer/maintainer fixing the programming errors.

One of the drawback of C is the hard memory management.

It is because of this aspect that Clang provides a way to detect, at runtime, memory errors. Called Address Sanitizer, it allows, while the program is running, to keep track of the memory and detect typical errors (out-of-bound accesses, use of a variable after a free, etc).

# With the following packages (version 3.2-1~exp3)
$ sudo apt-get install clang clang-3.2 compiler-rt -t experimental

Taking the simple following example, even if the errors are obvious, compilers will accept this code.

#include <stdlib.h>
int main() {
    char *x = (char*)malloc(10 * sizeof(char*));
    free(x);
    return x[5];
}

Built and run with:

$ clang -O1 -g -fsanitize=address -fno-omit-frame-pointer foo.c -o foo
$ ./foo &> memoryDebug.log

The previous command will generate a log file. Log which can be post processed with the asan_symbolize command.

$ asan_symbolize memoryDebug.log

which will give:

=================================================================
==21368== ERROR: AddressSanitizer: heap-use-after-free on address 0x7fb22e547f45 at pc 0x408c44 bp 0x7ffff60c10b0 sp 0x7ffff60c10a8
READ of size 1 at 0x7fb22e547f45 thread T0
    #0 0x408c43 in main /tmp/foo.c:5
    #1 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
0x7fb22e547f45 is located 5 bytes inside of 80-byte region [0x7fb22e547f40,0x7fb22e547f90)
freed by thread T0 here:
    #0 0x408c90 in __interceptor_free ??:0
    #1 0x408c0a in main /tmp/foo.c:4
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
previously allocated by thread T0 here:
    #0 0x408d50 in __interceptor_malloc ??:0
    #1 0x408bff in main /tmp/foo.c:3
    #2 0x7fb22d99f6ac in __libc_start_main /home/adconrad/eglibc-2.16/csu/libc-start.c:227
Shadow byte and word:
  0x1ff645ca8fe8: fd
  0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
More shadow bytes:
  0x1ff645ca8fc8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd0: fa fa fa fa fa fa fa fa
  0x1ff645ca8fd8: fa fa fa fa fa fa fa fa
  0x1ff645ca8fe0: fa fa fa fa fa fa fa fa
=>0x1ff645ca8fe8: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff0: fd fd fd fd fd fd fd fd
  0x1ff645ca8ff8: fa fa fa fa fa fa fa fa
  0x1ff645ca9000: fa fa fa fa fa fa fa fa
  0x1ff645ca9008: fa fa fa fa fa fa fa fa
Stats: 0M malloced (0M for red zones) by 1 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 1 calls
Stats: 0M really freed by 0 calls
Stats: 0M (128 full pages) mmaped in 1 calls
  mmaps   by size class: 8:2047;
  mallocs by size class: 8:1;
  frees   by size class: 8:1;
  rfrees  by size class:
Stats: malloc large: 0 small slow: 1
==21368== ABORTING

The main advantage compare to valgrind is that asan is supposed to be way more faster.

Threads can be also tricking to develop.
With the following example from stolen from upstream:

#include <pthread.h>
int Global;
void *Thread1(void *x) {
   Global = 42;
   return x;
}

int main() {
   pthread_t t;
   pthread_create(&t, NULL, Thread1, NULL);
   Global = 43;
   pthread_join(t, NULL);
   return Global;
}

$ clang -fsanitize=thread -g -O1 foo2.c -fPIE -pie -o foo
$ ./foo
==================
WARNING: ThreadSanitizer: data race (pid=21416)
  Write of size 4 at 0x7f2f1a214a50 by thread 1:
    #0 Thread1 /tmp/foo2.c:4 (exe+0x00000000f850)

  Previous write of size 4 at 0x7f2f1a214a50 by main thread:
    #0 main /tmp/foo2.c:10 (exe+0x00000000f8a4)

  Thread 1 (tid=21417, running) created at:
    #0 pthread_create ??:0 (exe+0x00000001267e)
    #1 main /tmp/foo2.c:9 (exe+0x00000000f894)

==================
ThreadSanitizer: reported 1 warnings

Note that clang also provides scan-build, a static analyzer for memory issues. Not as powerful as the Address Sanitizer (it only works on a file), it provides some excellent reports. See the Wouter's blog post on this subject or the automatic report of scan-build on Scilab.

Thanks to the work of Andrej Belym, Debian has now a new C++ standard library. This work has been done in the context of the Google Summer of Code 2012.
Available in Debian Experimental, this new packages provides both the runtime libraries (libc++abi1) and the C++ headers (libc++-dev).

With this library, it is possible to build a C++-based program without any dependency on libstdc++.

For example, as detailed in README.Debian, with the (amazing) C++ code:

// foo.cpp
#include <iostream>

int main() {
std::cout < < "plop" << std ::endl;
return 0;
}

with clang++ (or g++) will give:

$ clang++ -o plop foo.cpp
$ ldd plop |grep c++
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15791cb000)

Using libc++, it will drop the dependency on libstdc++ to use lib++

$ clang++ -stdlib=libc++ -o plop foo.cpp
$ ldd plop |grep c++
libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f87464df000)

For the record, it is not that trivial to do with g++. The command being:

g++ -nostdlib -lc++ -lc++abi -std=c++11 -o plop /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o foo.cpp /usr/lib/x86_64-linux-gnu/crtn.o -isystem/usr/include/c++/v1 -lc -lgcc_s

Besides this change, libc++ provides a support of C++ 11, considered as "correct" against the C++11 standard by upstream.

// bar.cpp
#include <chrono>
int main() {
return 0;
}

clang++ -stdlib=libc++ -o bar bar.cpp

More information:

• libc++ website
• libc++abi website
• libc++abi Specification