Tag : Build Engineering
High sky, plump horses, and Container Dieting
By Mario (Manseok) ChoIntroduction
Most Linux distributions, such as Ubuntu, RedHat, and CentOS, use glibc as the system's standard C library. When you install a library package, such as OpenSSL, with apt on Ubuntu or rpm (yum) on the RedHat line, it is dynamically linked with glibc by default.
GNU (Gnu) is an operating system and includes a wide range of computer software. GNU is open source, developed and maintained by the Free Software Foundation (FSF). Examples of things created by GNU include compilers and development tools such as GCC, G++, and Make. GNU uses glibc as its standard C library. glibc uses the GNU Lesser General Public License.
musl is a Linux standard C library distributed under the MIT license. Its developer is Rich Felker, and while glibc uses dynamic linking, musl aims to implement a standard C library that conforms to POSIX standards using static linking. It also implements non-standard features of Linux, BSD, and glibc.
Differences between glibc and musl in the Linux environment
When you install a package on Linux, it uses glibc by default. If you've ever built a C/C++ program using gcc, you've most likely done a glibc-based dynamic link build. However, in addition to this common glibc dynamic build, you can also do a MUSL-based dynamic/static build.
There are the following differences between
*-linux-gnuand*-linux-musl.| Build targets | Standard C libraries | Linking method |----------------|-------------------|----------------| |
*-linux-gnu| glibc | dynamic linking | |*-linux-musl| musl | dynamic/static linking |Consider the case of building an executable with Rust. When you install Rust on a Linux environment using
rustup,*-linux-gnuis selected as the default target.If you don't specify any other options, Rust will build the binary with the
*-linux-gnutarget and dynamically link it with glibc. To run a binary built in this way, you must have glibc installed in your Linux environment for it to work. If the binary relies on external libraries such as OpenSSL (if it is dynamically linked), you will also need to install those libraries via a package manager such as apt. If you want to run these dynamically linked binaries as a regular user, you can bundle them into a package like a DEB or RPM that describes the dependencies on external libraries. The package manager will then automatically find and install the appropriate dependent libraries. However, if you're using a library that isn't registered with the package manager, or even the same library, there are subtle compatibility issues between the installed version and the version you used to develop it, there's a chance that the binary you build won't run as intended.If you specify the
*-linux-musltarget, Rust will statically link with musl when building the binary. If you rely on external libraries like OpenSSL, it will also statically link those as well, embedding them all into the binary. This means that you end up with all of these libraries inside a single binary file in Rust. This static binary can run on any Linux environment, as long as it matches the CPU architecture and the set of system calls provided by the Linux kernel. This makes it easier to distribute binaries because you only need to pass a single binary to run it, rather than using a package like a DEB or RPM.If this makes deploying binaries so easy, why isn't the
*-linux-musltarget the default for Linux environments?The reason is that using MUSL makes build preparation somewhat more complicated. This is because if a developer-created binary package uses
*-linux-musland also relies on external libraries, those external libraries must also be statically linked with musl instead of dynamically linked with glibc. This means that all dependent libraries, as well as the main body of the program you want to build using the compiler for musl, must be built as static links from source code.Fortunately, you don't have to build everything from scratch if it's a commonly used external library in Rust. By utilizing a Docker image that bundles frequently used libraries with the Rust compiler/gcc, you can easily create a musl-based static build. (In the command examples that follow, I'll arbitrarily use the
<distro>#prompt to distinguish the container environment for each Linux distribution).$ docker run -it --name ubuntu ubuntu:22.04 bash ubuntu# apt update && apt install -y curl gcc vimLet's configure a dynamic link, glibc, and a static link, musl, in the Rust language environment, which is commonly used for development. First, install Rust on your Ubuntu environment.
ubuntu# curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ubuntu# source $HOME/.cargo/envLet's compare dynamic and static linking using Rust's default example, "Hello World" output.
First, let's build "Hello World" using glibc.
ubuntu# cd ubuntu# cargo new --bin hello && cd $_ Created binary (application) `hello` package ubuntu# cargo build --release Compiling hello v0.1.0 (/root/hello) Finished release [optimized] target(s) in 0.35sLet's use the
lddcommand to verify that the library is configured as a dynamic link in the glibc environment. We can see thatlinux-vdso,libgcc_s,libc, etc. are configured as dynamic links.ubuntu# ldd target/release/hello linux-vdso.so.1 (0x00007fffe87df000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdce9c3f000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdce9a17000) /lib64/ld-linux-x86-64.so.2 (0x00007fdce9cc2000)So let's change the RUST target configuration with a MUSL static link.
ubuntu# rustup target add x86_64-unknown-linux-musl info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl' info: installing component 'rust-std' for 'x86_64-unknown-linux-musl' 34.7 MiB / 34.7 MiB (100%) 8.6 MiB/s in 4s ETA: 0s ubuntu# rustup show Default host: x86_64-unknown-linux-gnu rustup home: /root/.rustup installed targets for active toolchain -------------------------------------- x86_64-unknown-linux-gnu x86_64-unknown-linux-musl active toolchain ---------------- stable-x86_64-unknown-linux-gnu (default) rustc 1.72.0 (5680fa18f 2023-08-23) ubuntu#Let's build "Hello World" to verify that static links are configured correctly.
ubuntu# cargo build --release --target=x86_64-unknown-linux-musl Compiling hello v0.1.0 (/root/hello) Finished release [optimized] target(s) in 0.37s ubuntu# ldd target/x86_64-unknown-linux-musl/release/hello statically linkedYou can see that "Hello World" is configured as a static link using the musl environment.
Now let's run "Hello World" built with both dynamic and static links by copying the binaries on CentOS and Alpine environments. CentOS 8 uses glibc dynamic linking and Alpine Linux uses musl static linking.
CentOS Container Environment
$ docker run -it --name centos centos:centos8 bash centos#Alpine Container Environment
The Alpine distribution uses musl by default rather than glic.
$ docker run -it --rm alpine:3.18 alpine#Let's copy 'Hello World' into a glibc environment and a musl environment to see the behavior.
$ docker cp ubuntu:/root/hello/target/x86_64-unknown-linux-musl/release/hello . $ docker cp hello centos:/root/ $ docker cp hello alpine:/root/Let's check the behavior on centOS.
centos# ./hello Hello, world!Let's check the behavior on alpine.
alpine# ./hello Hello, world!Comparing glibc and musl using the Rust application 'slice'
Let's take the Rust application 'slice' and compare the container images created with glibc and musl.
The Rust implementation of 'slice', like Python's 'slice', is publicly available on the GitHub repository https://github.com/ChanTsune/slice. 'slice' is a tool that prints the contents of a file from the front or back, like 'head' or 'tail'. For example, the command below will print lines 10 through 20 from 'file.txt'.
$ slice 10:20 file.txtWhen you build 'slice' in a Rust environment and create a container to use it, you can use it like this
$ docker run -i --rm -v `pwd`:`pwd` -w `pwd` sliceLet's build a container using glibc in the Ubuntu 22.04 environment.
FROM rust:latest as builder WORKDIR /work RUN git clone https://github.com/ChanTsune/slice /work/. RUN cargo build --release RUN strip /work/target/release/slice -o /slice FROM ubuntu:22.04 COPY --from=builder /slice /usr/local/bin/ ENTRYPOINT ["slice"]This time, we'll create a container image based on Ubuntu 22.04 using musl static links.
FROM rust:latest as builder RUN rustup target add "$(uname -m)"-unknown-linux-musl WORKDIR /work RUN git clone https://github.com/ChanTsune/slice /work/. RUN cargo build --release --target "$(uname -m)"-unknown-linux-musl RUN strip /work/target/"$(uname -m)"-unknown-linux-musl/release/slice -o /slice FROM ubuntu:22.04 COPY --from=builder /slice /usr/local/bin/ ENTRYPOINT ["slice"]Let's create a container image based on theAlpine distribution using a musl static link.
FROM rust:latest as builder RUN rustup target add "$(uname -m)"-unknown-linux-musl WORKDIR /work RUN git clone https://github.com/ChanTsune/slice /work/. RUN cargo build --release --target "$(uname -m)"-unknown-linux-musl RUN strip /work/target/"$(uname -m)"-unknown-linux-musl/release/slice -o /slice FROM alpine COPY --from=builder /slice / ENTRYPOINT ["slice"]If we compare the size of a glibc container image and a musl container image on Ubuntu 22.04 and a musl container image on Alpine, we can see that the container image with musl is smaller.
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE slice distroless-musl d38a74f8568a 11 seconds ago 3.52MB slice alpine-musl e3abb5f0aace 39 seconds ago 8.4MB slice ubuntu22.04-musl 467edd130e79 About a minute ago 78.9MB slice ubuntu22.04-glibc 09fe5ad40d56 3 minutes ago 78.8MBIn the Ubuntu environment, using glibc or musl doesn't make much difference in the size of the container image, but in the Alpine distribution, you can see that the container image size is reduced by about a tenth. This shows that by utilizing Alpine Linux with static builds, we can make our container images lightweight and reduce deployment time.
Conclusion
Using static links in programs that use standard C libraries can simplify the process of deploying Linux binaries. It also reduces the size of the container image compared to dynamic links, and makes deployment convenient regardless of the distribution. When you replace glibc with musl, you benefit not only from the difference in container image size, but also from features newly supported by musl, such as mDNS (a multicast-DNS-based zero config system) and NUMA clusters. Furthermore, if you use distroless, which is distributed by Google to better utilize musl, as your default container image, you can deploy and take advantage of smaller container images.
This post is automatically translated from Korean
20 September 2023
aiomonitor-ng: Debugging tool for complex asyncio applications
By Joongi KimAs program complexity grows, software developers need robust debugging tools. The optimal debugging method involves pinpointing a reliable way to replicate an issue within a development setting conducive to free experimentation, followed by the creation of automated tests. However, when the reproduction scenario is overly complex or involves bugs that sporadically appear in production environments, detailed logging becomes the alternative to comprehend the issue retrospectively. In this post, we presents the 'aiomonitor-ng', designed to simplify the debugging of intricate asyncio programs.
Debugging asyncio applications has its own difficulties. In Python, the stack trace is commonly used for debugging, revealing the program's location at the time of an exception. However, with asyncio's concurrent execution of multiple coroutine tasks, each with its own stack, it's crucial to examine not just the stack of the coroutine where the exception occurred but also those of 'related' coroutines to pinpoint if the error stemmed from another task. This issue intensifies when an external library implicitly generates a coroutine that invokes my code. Moreover, certain bugs, like coroutine task explosions that only manifest in production, or silent terminations of ongoing coroutine tasks, are particularly elusive in development settings, as they don't produce clear exceptions and are only detectable through post-incident logs.
aiomonitor is a production-grade live debugging tool created by the asyncio core developers. Wrapping asyncio-based code within a monitor object allows you to initiate a telnet session to a pre-set TCP port outside the process while the code is active. Through simple commands, you can inspect the list of coroutine tasks running in the event loop and the status of individual stacks. Backend.AI has integrated aiomonitor, assigning a unique debugging telnet port to each service process. (For security purposes, only local connections are permitted.) This integration has significantly aided in troubleshooting production-specific issues. Nonetheless, pinpointing the cause of a coroutine task's failure due to an external library, not specific to Backend.AI's code, remains a challenge when using aiomonitor at the time of the problem's occurrence.
We have developed an enhanced version named aiomonitor-ng, where "ng" signifies next-generation. This version includes the following additions and enhancements:
- Task creation tracker: For all running coroutine tasks, the momentary stack trace is preserved for each job that created the coroutine task (
asyncio.create_task()) to allow the entire chain of task creation to be tracked (ps,wherecommand). - Task termination tracker: Recently terminated coroutine tasks can be preserved and viewed up to a maximum of N, especially when one job cancels (
Task.cancel()) another job. The momentary stack trace of the cancellation trigger is also preserved to enable tracking of the entire cancellation chain (ps-terminated,where-terminatedcommand). - Persistent task marker: By default, to prevent memory leaks, recently terminated jobs are tracked up to a maximum of N. However, if specific jobs that must continue running throughout the application's lifespan are marked with a decorator, those jobs always preserve their termination logs, regardless of the history limit. They also provide a filtering function as an additional option in the termination log query command (
aiomonitor.task.preserve_termination_logdecorator). - Sophisticated terminal UI: We improved command-line processing, which was previously composed of a simple REPL (read-evaluate-print loop) based on handcrafted command parsing. We rewrote the aiomonitor server-side implementation to use Click and prompt_toolkit. We also developed a Telnet client that natively operates with asyncio to provide argument autocomplete, such as command and task ID.
Here are some screenshots of the actual usage:
We have successfully resolved resource leaks and performance issues stemming from excessive coroutine task creation in the grpcio library through callbacks. Additionally, we addressed problems where tasks that monitor events produced by the docker daemon would silently stop due to specific input message patterns. This was preventing the outcomes of container creation or deletion tasks from being reported, leading to system crashes.
We anticipate that developers working not only on Lablup but also on various Python asyncio applications will find aiomonitor-ng useful for debugging purposes in the future.
aiomonitor-ng can be installed via PyPI using the command
pip install aiomonitor-ng, and it is open-sourced on my GitHub account for anyone to use and contribute.28 November 2022
- Task creation tracker: For all running coroutine tasks, the momentary stack trace is preserved for each job that created the coroutine task (
