« An update on rust/coreutils | Debian rebuild with clang 10 + some patches » |
Debian running on Rust coreutils
tldr: Rust/coreutils ( https://github.com/uutils/coreutils/ ) is now available in Debian, good enough to boot a Debian with GNOME, install the top 1000 packages, build Firefox, the Linux Kernel and LLVM/Clang. Even if I wrote more than 100 patches to achieve that, it will probably be a bumpy ride for many other use cases.
It is also a terrific project to learn Rust. See the list of good first bugs.
Even if I see Rust code every day at Mozilla, I was looking for an actual personal project (i.e. this isn't a Mozilla project) to learn Rust during the various COVID lockdowns.
I started contributing to the alternative Coreutils developed in Rust. The project aims at proposing a drop-in replacement of the C-based GNU Coreutils, and I wanted to evaluate if this could be used to run a regular Debian. Similar to what I have done with clang.debian.net a few years ago (rebuilding the Debian archive using clang instead of gcc).
I expect that most of the readers know what is the Coreutils. It is a set of programs performing simple operations (copy/move file, change permissions/ownership, etc). Even if some commands are from the 70s, they are at the base of Linux, Unix and macOS. While different implementations can be found, they are trying to remain compatible in terms of arguments, options, etc. This implementation of Coreutils isn’t different!
If you want to learn more about the history of Unix, I recommend this great Corecursive podcast with Brian Kernighan.
While a lot of people contributed to this project, much was left to be done:
- missing programs to be implemented. See https://github.com/uutils/coreutils#utilities
- missing options in the various programs
- code not following latest Rust best practices
- lack of consistency in the code base (ex: functions with too many arguments)
- lack of tests/low code coverage
- Lots of failures when running the GNU Coreutils testsuite (141 tests pass on 613) - Some trivial to fix, some others harder… A good way to start on a Rust project!
To start easy, I defined 4 goals for this work:
- Package Coreutils in Debian/Ubuntu
- Boot a Debian system with a Rust-based coreutils
- Install the top 1000 packages in Debian - including GNOME
- Build Firefox, the Linux Kernel and LLV/Clang
Packaging of Coreutils in Debian
Packaging in Debian isn't a trivial or even simple task. It requires uploading independently all the dependencies in the archive. Rust, with its new ecosystem and small crates, is making this task significantly harder.
The package is called rust-coreutils - https://tracker.debian.org/pkg/rust-coreutils
For Debian/Ubuntu users, to have an idea of the complexity of packaging such applications, just run debtree --build-dep rust-coreutils | dot -Tsvg > coreutils.svg
(should be around 1M).
Since it isn't production ready, the rust-coreutils is installable in parallel with coreutils. This package does NOT replace the GNU/coreutils files (yet?), the new files are installed in /usr/lib/cargo/bin/
.
They can be used with:
export PATH=/usr/lib/cargo/bin/:$PATH
Or, uglier, overriding the files with the new ones.
Booting Debian with rust-coreutils
To achieve this, because I knew I would likely break the image a few times, I created a new project to quickly install a full Debian with PXE and preseed.
The project is available here:
https://github.com/opencollab/qemu-debian-install-pxe-preseed/
A script to create the full qemu image: build_qemu_debian_image.sh
A second script to boot on the newly created image: boot.sh
Then, building and installing coreutils on the system (yeah, it is ugly - don’t do that at home):
apt install rust-coreutils
cd /usr/lib/cargo/bin/
for f in *; do
cp -f $f /usr/bin/
done
First surprise, unlike the old init.d init system, as systemd is not relying on a series of scripts (it is mostly written in C), replacing the coreutils did not have an impact. Therefore, I didn't experience any issue during the boot process
Installing the most popular Debian packages
Debian packages rely a lot on post-install scripts (stored in /var/lib/dpkg/info/*
) to finalize and configure packages. They are (almost?) all using /bin/sh
(or /bin/bash)
to perform these actions. They intensively call coreutils applications.
For example, in /var/lib/dpkg/info/exim4-base.postinst
, we can find:
install -d -oDebian-exim -gadm -m2750 /var/log/exim4
With an ugly script, we can test the installations of the 1000 most popular packages one by one.
Running this, some classes of issues in Rust/coreutils could be easily identified.
Implementing missing options
A significant number of problems could be easily identified as a lack of support for some options.
Here is a list of most of the fixes I had to implement to make this plan work:
- Implement ln --relative: https://github.com/uutils/coreutils/pull/1540
update-initramfs: Generating /boot/initrd.img-4.19.0-12-amd64
ln: error: Unrecognized option: 'r'
- Implement cp --archive: https://github.com/uutils/coreutils/pull/1580
cp: error: Option 'archive' not yet implemented.
- Implement cp --no-dereference: https://github.com/uutils/coreutils/pull/1528
cp: error: Option 'dereference' not yet implemented.
- Implement
sync -fs
: https://github.com/uutils/coreutils/pull/1639
- Implement
install --owner & --group
: https://github.com/uutils/coreutils/pull/1641
- Implement
mktemp -t
https://github.com/uutils/coreutils/pull/1677 - ...
Different behavior
Most of the programs behaved as expected. Here is a list of differences:
- install doesn't support using /dev/null as source file
Setting up libreoffice-common (1:6.1.5-3+deb10u6) ...
A limitation of rust itself https://github.com/rust-lang/rust/issues/79390
install: error: install: cannot install ‘/dev/null’ to ‘/etc/apparmor.d/local/usr.lib.libreoffice.program.oosplash’: the source path is not an existing regular file
mkdir --parent
is sometime used (instead of --parents) - https://github.com/uutils/coreutils/commit/dbc716546b352c8b81af1952915137049ed12239
- "install foo.txt bar.txt" didn't work - https://github.com/uutils/coreutils/pull/1644
install: error: target ‘bar.txt’ is not a directory
install -d
won't fail if a directory already exists - https://github.com/uutils/coreutils/pull/1643
ls non-existing-file
wasn't returning an error code. "if ls" was used by some scripts - https://github.com/uutils/coreutils/pull/1654
- One of the issue was that
install -d /tmp/foo
could not be executed twice (while the second run with the gnu version wouldn't trigger an error if already existing) - https://github.com/uutils/coreutils/pull/1641 - ...
Compile Firefox, Clang and the Linux Kernel
Build systems can vary significantly one from the other.
To verify their usage of coreutils, I built these three major projects
Firefox
As Firefox relies mostly on Python as a build system, it went smoothly. I didn’t encounter any issue.
The only unrelated issue that I noticed working on it was apt-key was broken because the script relied on a buggy option of mktemp.
Linux Kernel
I identified only two issues compared to GNU Coreutils:
- The chown command on a non-existing symlink target doesn’t fail on the GNU version, the Rust one was triggering an error.
https://github.com/uutils/coreutils/pull/1694 - Linux kernel
ln -fsn ../../x86/boot/bzImage ./arch/x86_64/boot/bzImage
ln: error: Unrecognized option: 'n'
LLVM/Clang
The llvm toolchain relies on Cmake. Just like for Firefox, I didn’t face any issue.
Comparing with GNU coreutils using its testsuite
Recently, James Robson added a new test to run the GNU testsuite on the Rust/coreutils.
compared to 546 test passing with the GNU version. Even if a bunch of errors are just different outputs, it demonstrates that there is still a long road ahead.
# TOTAL: 611
# PASS: 144
# SKIP: 86
# XFAIL: 0
# FAIL: 342
# XPASS: 0
# ERROR: 39
Next steps & contribute
First, we will need more motivated contributors to work on this project. Many features remain to be implemented, optimizations to be done (e.g. decreasing the memory usage), etc.
I started to create a list of good first bugs for newcomers:
https://github.com/uutils/coreutils/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+first+bug%22
I will update this list of there is some interest for this project.
Helping improve the support of the GNU coreutils testsuite would be a huge step while being a great way to learn Rust!
Then, once it is in a better state, we will be able to make it a reliable alternative in Debian/Ubuntu to the GNU/Coreutils.
This might be also interesting for other folks who prefer a BSD license over a GPL.
3 commentaires

On the one hand, I’d absolutely love using well written utilities that I spend 50% of my terminal time on.
I bet they would add colors and what not.
On the other, I have to wonder if the people writing the new stuff understand things in and out enough to be able to cover every last case. I mean if my cp is silently doing data corruption in some edge case, that won’t be worth the productivity I gained from using better developed new tools

Interesting project!
Can you elaborate, why the Rust package is so much bigger than the C package?

@Jorg because crates are directly shipped into the final binary. No work has been done to minimize the size.
There are some options to do it: https://github.com/johnthagen/min-sized-rust
Ce post a 10 réactions en attente de modération...