GSOC 2020

Hello everyone. In this post, I will be walking through my whole work during Google Summer Of Code (GSOC) 2020.

Normally, because of the huge amount of work done, this blog post will not be the shortest. Well, one would be hard-pressed to call it a “blogpost” and it might be fit to be called a report.

The rest of this post is structured as follows: First I will introduce my GSOC 2020 task, the motivation behind it how I expected it to be VS how it ended up being like. Next, I will List all of the patches I submitted to relibc, alongside with description of what did each patch/patch-set achieve. I tried hard to keep track of all the patches I submitted, but I wouldn’t be surprised if I ended up missing to mention one or two of them. Last but not least, You will find a comprehensive step-by-step walkthrough for setting up an environment that is similar to the one I already have.

Introduction

Relibc is a libc implementation for X86_64 as well as ARM64. It supports both Linux kernel and redox kernel. Unlike most of the currently available libc implementations, relibc is written mainly in Rust, that makes it relatively easy to write secure code with a very small additional footprint thanks to zero-cost abstraction.

Gentoo is an open-source Linux based operating system. The other distinctive feature of Gentoo is its support for openrc as a service manager, portage as its package manager, and the fact that everything is source distributed. Aside from that, you are free to shape it the way you would like it to be.

The main objective of my work was to improve relibc support to Linux, mainly by improving the toolchain compatibility for relibc. The ultimate goal was to provide a fully working x86_64-*-linux-relibc Gentoo stage3 transparent enough for end-users where they don’t have trouble with almost any package they install via Gentoo.

For the purposes of this post, I will try to make huge distention between ld which is part of Binutils and ld.so which part of relibc. There has been lots of problem with both ld and ld.so. And I thought it might be worth mentioning that they are different for the sake of avoiding any sort of confusion.

Patches submitted to relibc and friends.

In this section, I will list all the patches I submitted to relibc repo as well as the repositories I created to during this project. At least at the time of writing this post, I had some modification made to multiple other code repositories (like gcc) but there were no patches submitted to these repositories yet. All the modification will be discussed in the next section.

Create relibc Gentoo overlay

This overlay contains building script for relibc on Gentoo. I never allowed it to fully install and replace my libc implementation but should you do so, First, you would need to get rid of your current libc implementation, yet maintain working toolchain for emerging. This is usually achieved through the bootstrapping process described in the next section.

Implement transparent ld.so relibc

Up until this stage, the default relibc loader was not capable of working with Linux execve(). It expected the target program to be loaded as a second argument as follows ld.so [OPTION] executable [args]. That way of loading is not what execve() use. This patch allows programs linked against relibc to run normally on Linux machines as in executable [args] and ld.so will be aware that it is invoked indirectly.

The motivation behind this patch was to try to get gcc to compile a simple program written C where the output binary is: a) linked against relibc. b) working!

Patches required to Cross Compile libstdc++ against relibc

libstdc++ is the equivalent of libc for C++. It is essential for preparing Binutils and Gcc stage 2. I attempted compiling libstdc++ and there was an issue with missing header files features.h, max_align_t was missing, ptrdiff_t was defined twice. This patch fixed the missing features.h issue and the other compiler time problems.

Once compilation was a success I tried generating simple C++ based hello world program.

I faced an issue regarding resolving weak symbols. ld.so ignored all symbols of type STB_WEAK. This patch adds support for STB_WEAK symbols while giving STB_STRONG higher priority than them.

Once symbols were loaded correctly, I noticed that there were crashes during cout. It turned out that C++ makes huge use of init, fini, init_array, fini_array. And this patch set makes that possible.

Right at this stage, it was possible to compile C/C++ binaries and have them linked against relibc and loaded with relibc’s own ld.so.

support gdb for relibc binaries

Up until the moment where I had to figure out the init, fini, init_array, finit_array problem. It was possible to debug anything (which happened to be always ld.so) with gdb. Problems happened when ld.so step passes bug the program seg-fault after that. (like for example when cout object was not initialized due to init_array not being run). The problem is that gdb cannot map executables on disk to virtual memory unless: a) It is either statically linked binary start to finish. b) It literally instructs gdb on how to figure that mapping out.

And since we are case b, I had to implement that “instruct gdb” thing. And as a matter of fact, there is no one way to instruct gdb but rather two approaches. The first is known as the standard debugging interface usually, exposed via a global structure that is part of the ELF header, and the second is a probe-based, and it not standard but it is much faster and flexible.

Both protocols are thoroughly described in Glibc rtld debugging interface documentation. Before I discovered that documentation, I thought that the reason gdb failed to load symbols or debugging info was that relibc Loader makes anonymous mapping for all loaded objects and copies data into them instead of doing file mapping. That proved to be wrong when I tried implementing, which motivated me to dig deeper until I found out about the real reason.

This patch was the first step in the implementation of the protocol. It extends ld.so transparency by making it even aware when the kernel loads executables with execve(), this way it doesn’t have to be loaded twice. The reason this patch is important in this context is that gdb can identify the symbols and source files for binaries loaded via execve().

The real implementation of the debugging interface can be found in this patch set. Without this patch, it is always possible for gdb to do assembly instruction based single-stepping (into or over). But what this patch achieves is allowing gdb to do source debugging as well.

Prepare pass 2 Binutils

The goal here was to be able to compile Binutils and have them linked against relibc. This patch implements some of the missing header files that were required for Binutils. namely sys/user.h and sys/procfs.h.

After that, it was clear that relibc had poor support for wide-character functions. This patch set implemented four of these functions: towlower, towupper, wcsncasecmp, and wcscasecmp.

Next, we had a patch set that fixed regressions in ld.so, and another one that fixed bugs in printf implementation.

Prepare Pass 2 GCC

Getting gcc pass2 to work was extremely complex. Not only did it revealed many bugs in relibc and ld.so, but also it abused lots of gnu extensions and non-standard libc features that had to be implemented in relibc for things to work.

This patch set is a good example of the non-standard libc features required by gcc. First is the c_addr data type which is equivalent to void *. Second is the ungetc function. relibc did have an ungetc function. As per C specifications, the user is only guaranteed 1 ungetc to succeed. But gcc did arbitrarily many ungetc and expected all of them to succeed. So I had to implement that into relibc IO stack.

Next patch set was more bug fixes in our ld.so implementation. One bug was that LD_LIBRARY_PATH (which was using during gcc compilation) did, in fact, overwrite current ld.so search path. The other bug was that the thread-local variable errno was accessed using access function, before Thread Local Storage (TLS) being initialized. I fixed that by using system calls directly. But that approach turned out to not be compatible with both redox and Linux, so this patch fixes it for both operating systems by introducing OS-specific solution for both redox and Linux where we avoid using errno in both cases.

This patch modified some header files so that they are compliant with -fpermissive flag which is used in the gcc compilation process.

This patch set fixes issues in inner_scanf and all the related functions: The main issue was that inner_scanf processes its input (whether it is a file or a buffer) as a dyn Read object, but it always has Lookahead of 1. Unfortunately dyn Read cannot seek back, thus ftell() will always be wrong right after any of the scanf functions family. This patchset implemented an abstraction over both files and memory buffers that supports lookahead() and commit() operation where lookahead() will get the next character without modifying the current position in file while commit() operation will modify the current position in file to be the current position of lookahead().

Gcc assumed that sys/select.h will be included by default when sys/types.h is included. In relibc that is particularly an issue because sys/types.h is a raw header file that is used to compile some of the pure C code (that doesn’t depend on rust) while sys/select.h is generated by cbindgen at a very late stage. This patch set fixes that issue so that all tools become happy by creating an internal wrapper for sys/types.h.

While compiling gcc using relibc, there was a stage where xgcc is generated and linked against relibc and used to compile the rest of the compiler. But the issue was that I was using the debug version of relibc which is significantly slower than the release version. When I tried switching to the release version, ld.so crashed in a very early stage. The issue was that init, finit handlers were written in assembly, and they were not ABI compatible with rust. So this patch replaces the assembly code with some rust that should be ABI compatible with C and Rust.

This patch set implements elf.h for relibc and also implements the posix struct flock.

The problem that this patch solved was that we always added the base address, and we assumed that all addresses we access are relative but this is not the case in case of -fno-pie binaries. The issue is that all addresses were base + offset. so if we added the base again it will, of course, generate the wrong address.

This patch set dealt with an extremely complicated memory corruption issue. The problem is that ld.so and libc.so have identical copies of the same libc as part of them including rust stdlib and rust memory allocator. The allocator itself is based on dlmalloc. Both copies operate on the same heap but without sharing internal data structures. This patch fixes this issue via bookkeeping allocated memory metadata.

This patch properly implements dlopen, dlclose and dlsym. These three functions were originally required for loading gcc link-time optimization in ld.

This patch set improves ld.so stability boost its performance by preventing libraries that have been previously loaded from being loaded once again thus saving huge memory and cpu time.

This patch as well as this patch set fixses regressions in ld.so and improve its stability.

Gentoo from scratch

The goal of this section is to bootstrap Gentoo working environment for relibc.

Set up testing chroot environment

This step is optional yet highly recommended. In this step we will be creating a chroot environment to work in so that you would not ruin your operating system should anything go wrong.

First download a no multilib gentoo stage 3 from here. Next, create a folder that will hold your chrooted environment and extract the stage3 in that folder while preserving permissions, attributes, group and user id:

$ cd ~
$ mkdir chroot_me
$ cd chroot_me
# tar xpvf ../stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner

Next, configure the freshly extracted chroot environment as follows

# mirrorselect -i -o >> etc/portage/make.conf
# mkdir --parents etc/portage/repos.conf
# cp --dereference /etc/resolv.conf etc/
$

Next use the following script to chroot into the working environment:

#!/bin/sh
mount --types proc /proc chroot_me/proc
mount --rbind /sys chroot_me/sys
mount --make-rslave chroot_me/sys
mount --rbind /dev chroot_me/dev
mount --make-rslave chroot_me/dev
chroot chroot_me /bin/bash
umount -R chroot_me/proc
umount -R chroot_me/sys
umount -R chroot_me/dev

Once chrooted you need to sync portage, update the system, configure timezone and locales as follow:

# emerge-webrsync
# emerge --sync
# emerge --ask --verbose --update --deep --newuse @world
# echo "Europe/Berlin" > /etc/timezone
# emerge --config sys-libs/timezone-data
# echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
# echo "en_US ISO-8859-1" >> /etc/locale.gen
# locale-gen
# env-update && source /etc/profile

Last but not least we would need to create a non-root user with sudo privilege

# emerge --ask app-admin/sudo lzip vim git
# useradd -m -G users,wheel,audio -s /bin/bash oddcoder
# echo "%wheel         ALL = (ALL) NOPASSWD: ALL" >> /etc/sudoers
# su - oddcoder
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

With that being done we should have a Gentoo Glibc based environment to work on. This environment should allow protecting our real operating system yet utilize 100% of the hardware (CPU/ram) which is not always the case when using virtualization.

The rest of this guidehttp://archive.clfs.org/clfs-museum/3.0.0-SYSTEMD/x86_64-64/ is heavily inspired by both LFS, CLFS and MUSL ELFS.

Create Working directories and environment variables

$ cd ~
$ mkdir -p lfs/tools/lib
$ mkdir -p lfs/sources
$ export LFS=/home/oddcoder/lfs
$ cat > ${LFS}/.bashrc << "EOF"
set +h
umask 022
LFS=/home/oddcoder/lfs
HOME=/home/oddcoder/lfs
PS1='lfs-\u:\w\$ '
LC_ALL=POSIX
PATH=${LFS}/tools/bin:/bin:/usr/bin
LFS_TGT=$(uname -m)-lfs-linux-relibc
LFS_HOST="$(echo $MACHTYPE | sed "s/$(echo $MACHTYPE | cut -d- -f2)/cross/")"
source $HOME/.cargo/env
export LFS LC_ALL PATH LFS_TGT LFS_HOST
EOF
$ source ${LFS}/.bashrc
# ln -sv ${LFS}/tools /
# ln -sv /tools/lib /tools/lib64
# chown oddcoder:oddcoder /tools/

The folder /home/oddcoder/lfs will hold the final relibc toolchain while /home/oddcoder/lfs/tools will hold the cross compiler toolchain (host = glibc target = relibc).

as per Musl ELFS:

The set +h command turns off bash’s hash function. Hashing is ordinarily a useful feature—bash uses a hash table to remember the full path of executable files to avoid searching the PATH time and again to find the same executable. However, the new tools should be used as soon as they are installed. By switching off the hash function, the shell will always search the PATH when a program is to be run. Setting the user file-creation mask umask to 022 ensures that newly created files and directories are only writable by their owner, but are readable and executable by anyone (assuming default modes are used by the open(2) system call, new files will end up with permission mode 644 and directories with mode 755).

Build Binutils Pass 1

$ cd $(LFS)/sources
$ wget https://sourceware.org/pub/binutils/releases/binutils-2.33.1.tar.lz
$ tar xvf binutils-2.33.1.tar.lz
$ cd binutils-2.33.1
$ mkdir build_pass_1
$ cd build_pass_1
$ ../configure                            \
  --prefix=/tools                         \
  --disable-multilib                      \
  --with-sysroot=$LFS                     \
  --with-lib-path=/tools/lib              \
  --target=$LFS_TGT                       \
  --disable-nls                           \
  --disable-werror
make
make install

Build Gcc Pass 1

$ cd ~/sources
$ wget ftp://ftp.gnu.org/gnu/gcc/gcc-9.3.0/gcc-9.3.0.tar.xz
$ tar xvf gcc-9.3.0.tar.xz
$ cd gcc-9.3.0
$ ./contrib/download_prerequisites
$ # TODO put patches
$ for file in gcc/config/{linux,i386/linux{,64}}.h
> do
> cp -uv $file{,.orig}
> sed -e 's@/lib\(64\)\?\(32\)\?/ld@/tools&@g' \
>   -e 's@/usr@/tools@g' $file.orig > $file
> echo '
> #undef STANDARD_STARTFILE_PREFIX_1
> #undef STANDARD_STARTFILE_PREFIX_2
> #define STANDARD_STARTFILE_PREFIX_1 "/tools/lib/"
> #define STANDARD_STARTFILE_PREFIX_2 ""' >> $file
> touch $file.orig
> done
$ sed -e '/m64=/s/lib64/lib/'   -i.orig gcc/config/i386/t-linux64
$ mkdir build_pass_1
$ cd build_pass_1
$ ../configure                                                  \
        --target=$LFS_TGT                                       \
        --host=$LFS_HOST                                        \
        --build=$LFS_HOST                                       \
        --prefix=/tools                                         \
        --with-sysroot=$LFS                                     \
        --with-newlib --without-headers                         \
        --with-local-prefix=/tools/                             \
        --with-native-system-header-dir=/tools/include          \
        --disable-nls --disable-shared --disable-multilib       \
        --disable-decimal-float --disable-threads               \
        --disable-libatomic --disable-libgomp                   \
        --disable-libquadmath --disable-libssp                  \
        --disable-libvtv --disable-libstdcxx                    \
        --enable-languages=c,c++
$ make
$ make install

Build Linux headers

$ cd ~/sources
$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.4.tar.xz
$ tar xvf linux-5.8.4.tar.xz
$ cd linux-5.8.4
$ make mrproper
$ make headers
$ cp -rv usr/include/* /tools/include

Build relibc

$ cd ~/sources
$ git clone https://gitlab.redox-os.org/redox-os/relibc.git
$ cd relibc
$ git submodule update --init --recursive
$ cd relibc
$ TODO apply patches
$ make gentoo-install LIB=lib TMP=/tools

Build relibc based Libstdc++

$ cd ~/sources/gcc-9.3.0
$ mkdir build-libstdc++
$ cd build-libstdc++
$ ../libstdc++-v3/configure             \
    --host=$LFS_TGT                     \
    --prefix=/tools                     \
    --disable-multilib                  \
    --disable-nls                       \
    --disable-libstdcxx-threads         \
    --disable-libstdcxx-pch             \
    --with-gxx-include-dir=/tools/$LFS_TGT/include/c++/9.3.0
$ make 
$ make install

Build Binutils Pass 2

$ cd ~/sources/binutils-2.33.1
$ mkdir build_pass_2
$ cd build_pass_2
$ CC=$LFS_TGT-gcc                       \
AR=$LFS_TGT-ar                          \
LD=$LFS_TGT-ld                          \
RANLIB=$LFS_TGT-ranlib                  \
../configure                            \
    --prefix=/tools                     \
    --disable-nls                       \
    --disable-werror                    \
    --with-lib-path=/tools/lib          \
    --with-sysroot
$ make
$ make install

Building gcc Pass 2

$ cd ~/sources/gcc-9.3.0
$ mkdir build_pass_2
$ cd build_pass_2
$ CC=$LFS_TGT-gcc                                                  \
$ CXX=$LFS_TGT-g++                                                 \
 AR=$LFS_TGT-ar                                                    \
 RANLIB=$LFS_TGT-ranlib                                            \
 ../configure                                                      \
    --with-lib-path=/cross_tools/lib                               \
    --prefix=/cross_tools                                          \
    --with-local-prefix=/cross_tools                               \
    --with-native-system-header-dir=/tools/include                 \
    --enable-languages=c,c++                                       \
    --disable-libstdcxx-pch                                        \
    --disable-multilib                                             \
    --disable-bootstrap                                            \
    --disable-libgomp                                              \
    --disable-nls                                                  \
    --disable-libatomic                                            \
    --disable-libstdcxx                                            \
    --with-gxx-include-dir=/tools/$LFS_TGT/include/c++/9.3.0       \
    --disable-symvers
$ echo "MAKEINFO = :" >> Makefile
$ make
$ make install

Notes and future plans

While writing this post I noticed that unfortunately, Binutils pass 2 doesn’t successfully build due to problem with R_X86_64_COPY relocation, upon investigation I found that also gdb support was broken for some reason which made the job twice as hard. As such, It was not possible to verify gcc build pass 2 on a clean environment. However, it is worth mentioning that at some point in time I did manage to have everything in this post working.

This leads us to the project’s future plan. Currently, Most of my work to relibc ld was hand tested with no solid test suite. So I am planning next to create a Gentoo based test suite where the whole project gets compiled from scratch and recompiled using itself. and that should serve as a good test for relibc.

Updated:

Comments