Backend.AI's core engine utilizes many open-source software components and is itself being developed as open source. When enterprise customers encounter inconveniences or bugs while using Backend.AI, we provide issue tracking and support through customer support and technical support channels. However, those using the open-source version can also directly contribute to the project.

There are mainly two ways to contribute: creating an issue that explains in detail what problem exists or what improvement ideas you have, and making a pull request to directly contribute code changes. In this post, we'd like to introduce a few things that are good to know in advance for more effective and faster communication with the development team during the contribution process.

Introduction to GitHub Repositories

As seen in the previous post Backend.AI Open Source Contribution Guide, Backend.AI was originally developed with repositories divided into the Backend.AI meta-repository and several sub-components.

However, from version "22.06", Backend.AI has changed to a mono-repository using Pants.

This transition in the development workflow has greatly helped in resolving package compatibility issues that often occur across multiple individual components, creating a more convenient development environment.

Pants is a fast, scalable, and user-friendly build system.

If you want to submit an issue, the first place to look is the Backend.AI repository. The repository named Backend.AI integrates multiple packages using Pants. This repository is not only for project management but also contains code that actually performs functions. All issues related to Backend.AI's server and Client SDK are managed here, and links to other projects are provided through the README.

When creating a new issue, two default templates are provided: bug report and feature request. However, it's not strictly necessary to follow these templates. Considering the complexity of Backend.AI and its various usage environments, following these templates when writing content makes it easier to share context for problem identification.

Introduction to Mono-repository

From version "22.06", Backend.AI has changed to a mono-repository using Pants. A mono-repository is a project with an integrated code base that shares the basic dependencies, data models, features, tooling, and processes of multiple projects. It operates the repository by integrating multiple projects that were previously used into a single project.

Introduction to Pants

Backend.AI is installed using Pants as a build system. For more details about Pants, please check the following link Pants - Getting started.

Relationship between Backend.AI Components

Figure 1. Relationship structure between major Backend.AI components

Figure 1 is a diagram showing the relationship between the major components of Backend.AI.

Figure 2. Major component structure of Backend.AI and examples of execution methods

Figure 2 is a diagram showing the major component structure of Backend.AI, and shows the location of the source code of the components and execution commands.

Most of Backend.AI's components are managed in the Backend.AI repository, and the source code is located in the src/ai/backend/ subdirectory. Briefly, summarizing what each component does by directory:

• src/ai/backend/manager (Manager): Core service that monitors computational resources of the entire cluster, handles session scheduling, provides user authentication and APIs for session execution
• src/ai/backend/agent (Agent): Service installed on compute nodes to manage and control containers
• src/ai/backend/common (Common): Library of functions and data formats commonly or frequently used across multiple server-side components
• src/ai/backend/client (Client SDK for Python): Official command-line interface and library providing API wrapper functions and classes for Python
• src/ai/backend/storage (Storage Proxy): Service that allows user web browsers or Client SDK to directly perform large-volume I/O from network storage
• src/ai/backend/web (Web Server): HTTP service that provides routing for Web UI and SPA (single-page app) implementation and web session-based user authentication
• src/ai/backend/webui (Web UI & Desktop App): Web component-based implementation of the actual UI that users interact with. Also supports Electron-based desktop app builds. Also includes a local lightweight version of the app proxy that allows users to directly access application ports running inside containers.

Backend.AI Version Management Method

Backend.AI has major releases every 6 months (March and September each year), with post-release support provided for about 1 year. Therefore, the version number follows the CalVer format in the form of YY.0M.micro (e.g., 20.09.14, 21.03.8). However, due to the version number normalization of the Python packaging system, the version of the wheel package is in the format YY.MM.micro without zero-padding in the month part (e.g., 20.9.14, 21.3.8). Some detailed components with version update cycles different from the main release cycle follow the general SemVer format.

Essential Packages to Install Before Development

Before installing Backend.AI, you need to install Docker, Docker Compose v2, etc. first. When installing Backend.AI using the scripts/install-dev.sh script in the repository, it checks for the installation of Docker, Docker Compose v2, etc., and guides you through the installation process. If Python, pyenv, Docker, npm are not installed, you need to install the essential packages as follows. For Python, please install it using the system package's Python3. Then, you need to install pyenv and pyenv-virtualenv.

$ curl https://pyenv.run | bash

Then, you can install Docker and Docker Compose v2 as follows:

MacOS

For MacOS, Docker Desktop on Mac automatically installs Docker and Docker Compose v2.

Ubuntu, Debian, CentOS, Fedora Core, and other Linux environments

For Ubuntu, Debian, CentOS, Fedora Core, you can automatically install Docker and Docker Compose v2 using the following script:

$ sudo curl -fsSL https://get.docker.io | bash

After installing Docker, if you get a unix:///var/run/docker.sock access permission error when running without sudo, like this:

$ docker ps
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied

If such a permission problem exists, set the permissions using the following command:

$ sudo usermod -aG docker $(whoami)
$ sudo chown root:docker /var/run/docker.sock

After that, reboot and run docker run hello-world to confirm that it runs normally.

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:94323f3e5e09a8b9515d74337010375a456c909543e1ff1538f5116d38ab3989
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/

Instead of changing the group ownership of /var/run/docker.sock with chown, changing the permissions of the /var/run/docker.sock file to 666 allows other users in the group to access it without rebooting.

sudo chmod 666 /var/run/docker.sock

However, setting the permissions of the /var/run/docker.sock file to 666 creates a security vulnerability.

You can check if Docker Compose v2 is installed as follows:

$ sudo docker compose version
Docker Compose version v2.28.1

If nvm is not installed, you should install nvm as shown in the following link nvm - install & Update Script.

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

After installing nvm, install the latest LTS version of Node.js and set it up for use.

$ nvm install --lts
$ nvm use --lts

How to Install the Development Environment

To actually contribute code, you need to write a pull request, and unless it's a simple typo correction or documentation-related contribution, you need to directly modify the code and run it, so it's essential to set up your own development environment. Backend.AI has a structure where multiple components work together, so installation is not complete just by cloning one repository and creating a Python virtual environment with an editable install^[1]. At a minimum, you need to set up and run manager, agent, storage-proxy, webserver, and wsproxy to check the functioning GUI, and for the CLI environment, you need to install the client SDK separately. Also, Redis, PostgreSQL, and etcd servers need to be run together for manager operation and communication with the agent.

If you have installed the essential packages introduced earlier and want to install multiple components of Backend.AI, you can install them using the scripts/install-dev.sh script in the repository. This script does the following:

• Checks for the installation of pyenv, Python, Docker, npm, etc., and guides the installation method
• Installs all of these various components in their respective directories
  • At this time, components such as accelerator-cuda, which are necessary for the operation of other components, are additionally installed in an editable state.
• Adds database/etcd fixtures including basic port settings and example authentication keys that each component can look at each other
• Creates and runs PostgreSQL, Redis, etcd services using Docker Compose under the name "halfstack"

When the install-dev script execution is successfully completed, it outputs commands to run service daemons such as manager and agent, and basic configured example account information. Following the instructions, use terminal multiplexers like tmux, screen, or multiple tab features of terminal apps to run service daemons in separate shells, and confirm that the hello world example works. Then you're ready to develop and test Backend.AI.

Currently, this method only supports Intel (amd64/x86_64) and ARM-based macOS and Ubuntu/Debian/CentOS/Fedora and Linux environments where Docker Compose can be installed.

Usually, when you first use this install-dev script, it often stops due to various errors or pre-check failures and needs to be run again. In this case, you can easily perform the deletion procedure using the scripts/delete-dev.sh script.

Installing and Uninstalling Backend.AI

Using these install-dev and delete-dev scripts, you can freely install and uninstall Backend.AI. First, clone the Backend.AI repository.

$ git clone https://github.com/lablup/backend.ai 

Then install Backend.AI.

$ cd backend.ai
$ ./scripts/install-dev.sh 

After the installation is complete, please take note of the result content that appears on the screen.

If you want to uninstall Backend.AI, run the scripts/delete-dev.sh script from the location where you cloned the Backend.AI repository.

$ cd backend.ai
$ ./scripts/delete-dev.sh 

Things to Know Before Contributing

As with most projects managed in distributed version control systems, to contribute to Backend.AI, code work should be based on the latest commit of the main branch of the original remote repository, and if conflicts occur, they should be resolved before requesting a review. If you've forked the original repository, the current forked original repository and the actual original repository need to be synchronized.

Before explaining the method, please refer to the following terminology to help understanding:

• Original remote repository (upstream): The original Backend.AI repository. All major commit contents are reflected here.
• Forked original repository (origin): The Backend.AI repository copied to "your" account via GitHub. (Note: Original remote repository != Forked original repository)
• Code copy (local working copy): The forked repository currently downloaded to your local machine

Git command branch notation

• main: The main branch of the current local working copy
• origin/main: The main branch of the repository (origin) from which I cloned to create my local working copy
• upstream/main: The main branch belonging to the separately added upstream remote repository

Workflow concepts

• At the time of forking, origin/main is created
• When you clone the forked repository, main is created on your work computer
• Create a new topic branch from main and proceed with work
• When you upload this work branch to origin and create a PR, GitHub automatically points to the original repository of the fork
• At this point, to synchronize changes to the main of the original repository during work, follow the procedure below

The method of synchronization is as follows:

• step1: Add the original remote repository as a name called upstream

$ git remote add upstream https://github.com/lablup/backend.ai

• step2: Fetch the latest commits of the main branch of the original remote repository to the code copy (local working copy)

$ git fetch upstream

• step3: Bring the latest commit reflection history of the main branch of the original remote repository to origin (the code copy (local working copy) of the original repository you forked)

$ git switch main && git merge --ff upstream/main

• step4: Reflect the changes in the code copy (local working copy) made in steps 1 ~ 3 to origin (the remote repository of the original repository you forked)

$ git push origin main

Now upstream/main and origin/main are synchronized through main.

• step5: Reflect the latest updates to my branch that I'm working on

$ git switch topic
$ git merge main

When performing this process, if a history branch is created between origin/main and upstream/main and step 5 is performed incorrectly, it can become extremely difficult to recover. Also, when the CI tools used by Backend.AI test PRs, they are set to find common ancestor commits to see the differences between upstream/main and origin/topic, but if you reuse the main name for the topic branch, these tools will not work properly. If possible, think of always giving a new name when creating a new branch.

How to Write a Pull Request

To send a specific bug patch or feature implementation as a PR, you first need to upload it to GitHub. There are several methods, but the following is recommended:

• Fork the repository on the GitHub repository page. (If you have direct commit permissions, it's recommended to create a branch directly without forking.)
• In your local working copy, use git remote to point to that forked repository.
  • Following convention, it's good to name Lablup's original repository as upstream and the newly created forked repository as origin.
  • If you installed with install-dev first instead of cloning after forking, the original repository will be origin, so you need to rename the remote.
• Create a new branch.
  • For branch names, prepend fix/ for bug fixes or feature/ for feature additions or improvements, and summarize the topic in kebab-case. (e.g., feature/additional-cluster-env-vars, fix/memory-leak-in-stats) Other prefixes like docs/, refactor/ are also used.
  • It's possible to write a PR by directly modifying the main branch, but during PR review and modification periods, if additional changes occur on the main branch, you'll have to rebase or merge every time you synchronize with the upstream repository, which is more troublesome. Having a separate branch allows you to rebase and merge when you want.
• Commit changes to that branch.
  • Commit messages should follow the conventional commit style as much as possible. Like branch names, use title prefixes such as fix:, feat:, refactor:, docs:, release:, and for Backend.AI specifically, setup: for dependency-related commits, repo: for cases like gitignore updates or repository directory structure changes. You can also indicate affected components in parentheses. (e.g., fix(scripts/install-dev): Update for v21.03 release)
  • Commit messages should be written in English.
• Push the branch and write the PR.
  • For PRs with separate issues, you should write the issue number in the PR body. If you want to reference an issue in the repository, look at the number in the issue link like https://github.com/lablup/backend.ai/issues/401 and write it in the format #401, and GitHub will automatically link it.
  • There's no specific format required for the PR body, but it's good to write what problem it's solving, what principle it's written on, or what tools or libraries were used, and why those choices were made.
  • PR titles and bodies can be written in English or Korean.
  • When you create a PR, you'll see various automated inspection tools in action. In particular, you must sign (register your GitHub username) the CLA (contributor license agreement) for the review to proceed.
  • You must pass all basic coding style and coding rule checks for each language. (For Python code, flake8, mypy, etc.)
  • In repositories with a changes directory and towncrier check, when you create a PR and receive its number, create a file named changes/<PR number>.<modification type> and write a one-line English sentence summarizing the changes in Markdown syntax. (For relatively simple content or if there's a separate existing issue, this content can also serve as the PR body.) Modification types include fix, feature, breaking, misc, deprecation, doc, and parts that differ by project are defined in each repository's pyproject.toml. You can refer to files like CHANGELOG.md or CHANGES.md to see how existing messages were written.
• Proceed with the review process.
  • When completed, the reviewer usually organizes the commit log in a squash-merge form to create a single commit for merging.
  • Therefore, don't feel burdened about making frequent small modification commits during the review process, and feel free to make commits whenever you think of something.

It's even better to use tools like GitHub CLI, SourceTree, GitKraken along with git commands.

Summary

We've looked at Backend.AI's overall component structure and repository structure, how to install the development environment, and how to write pull requests. I hope this guide has helped you take one step closer to Backend.AI's source code.

---

[1]: An "editable" installation refers to a method of installing a Python package to directly look at the source directory, allowing changes to be immediately reflected when importing the package just by modifying the source directory without editing inside the site-packages directory.