Sylvestre's blog

iwyu (Include What You Use) has just been accepted in Debian.

This tool, based on LLVM/Clang, checks the headers of a C/C++ file. It will detect which include directives are useless, what could be improve and add comments to explain why #include are there.

Let's take this C file:

#include <stdio.h>
#include <malloc.h>
#include <limits.h>

int foo(void) {
fprintf(stderr, "an error\n");
return INT_MAX;
}


$ iwyu foo.c

will return:

foo.c should remove these lines:
- #include <malloc.h> // lines 2-2

The full include-list for foo.c:
#include <limits.h> // for INT_MAX
#include <stdio.h> // for fprintf, stderr
---


Package on Debian

Following a thread on the Clang mailing list, I did a quick analysis of the optimization flags used to build C or C++ code.

I used a rebuild result from last mid July of all Debian native packages (meaning that I don't have the log of full Java, Python or other packages without native code) and wrote a small script to count all their occurrences.
By default, deb packages uses the command "dpkg-buildflags --get CFLAGS" which will set the optimization level to -O2. However, many packages and upstream build systems do not respect such flags.

On 10320 binary-only packages (in total, Debian has around 19000), the results are the following:

Since a graphic talks more than numbers (note that the Y axis is logarithmic):

Disclaimer:
Some packages are using several optimization flags during the same build.
Some packages are not showing the full command line (see jessie release goal: verbose build logs)
The results might be approximate since I am basically grepping "-OX " on the log files.

The LLVM toolchain version 3.3 has been released a couple months ago.
Here are now the result of the rebuild of Debian archive using this version of the compiler.
Like the previous releases, we are at a bit less than 12% of packages failing (2188 packages on a total of 18854).
More warnings / errors detections have been added to the software (for example: like this defect of the C++ standard or the detection of unused linker option) causing more build failures, but, in the mean time, we fixed some issues in the Debian packages...

As usual, the following image shows clearly the evolution of the build failures over time.

As stated in my blog post for the release 3.2, this rebuilds prove that Clang is ready for production in term of support of the C, C++ and Objective C languages.

With the performance results showed by Chandler Carruth from Google at the last Euro LLVM summit (see this video from 5:40), I believe that it is now time to report and fix the bugs in the upstream packages.

I also presented this work (video) at the Debconf 13 last week in Vaumarcus (Switzerland) and I will be also presenting this work at the Linux Plumbers Conference, New Orleans.

With Léo Cavaille (as part of his GSOC) and Paul Tagliamonte, we are also working on providing a better automatic rebuild infrastructure for clang-built packages (and other static analyzers). More in the next few weeks.

Finally, I would like to thank folks at AWS for the Debian credit and David Suarez for helping on with the Ruby segfaults.

In parallel of my work on the LLVM Debian/Ubuntu nightly packages, I uploaded a couple months ago the same packages for the stable releases of the LLVM toolchain. By toolchain, this means LLVM + Clang + Clang extra tools + compiler-rt + polly + LLDB.
Previously, they were packaged as separate source packages. Making the maintenance harder and, especially, limiting the packaging of LLDB and polly.

Currently, we have in the archive:

• LLVM toolchain 3.2 (note that LLDB is disabled from this release).

• LLVM toolchain 3.3

• LLVM toolchain snapshot (3.4)

For the 3.2 release, I am planning to focus on RC bug fixes.
The 3.3 and snapshot releases will get the feature improvements.
llvm-default will point to the 3.3 release once it builds on all architectures and it reached testing aka Jessie (probably during summer).

After setting an automatic code coverage tool, I just plugged an automatic scan-build on the LLVM toolchain trunk:

http://buildd-clang.debian.net/scan-build/

Note that:

• polly is currently not analyzed. I had to disable the build until the new version of cloog is available.

• compiler-rt is not analyzed because it does not respect the CC/CXX argument to use the clang built locally [1]. Not sure it is a bug here.

This work is done through the Debian/LLVM jenkins instance.
The report is updated twice a day.
The scan-build/clang used to produce the report is the one published on http://llvm.org/apt/. That means that a new feature/fix done on scan-build will appear only about a day after on report. As a side effect, it tests automatically the packages distributed on llvm.org/apt/.

For a project of this size, the number of reported errors / warnings is pretty low. At time of writing, only 329 errors are reported and many of them are in generated code (AttrDump.inc and LLDBWrapPython.cpp for example) or in unit tests.

[1]
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/Release/bin/clang -fno-exceptions -fPIC -funwind-tables -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/include -Wall -Werror -O3 -fomit-frame-pointer -m64 -fPIE -fno-builtin -gline-tables-only -fno-rtti -DASAN_FLEXIBLE_MAPPING_AND_OFFSET=1 -c -o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/tools/clang/runtime/compiler-rt/clang_linux/asan-x86_64/x86_64/SubDir.lib__sanitizer_common/sanitizer_symbolizer_itanium.o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_itanium.cc