Lablup

 ![](https://cdn.lablup.com/lablup_intel_6f980e65b3.jpg) 

**Seoul —** Lablup announces support for Intel® Gaudi® 2 & Intel® Gaudi® 3 AI Accelerators* in Backend.AI at Supercomputing 2024. Adding support for Intel's AI accelerators to the list of AI accelerator vendors already supported by Backend.AI, including NVIDIA, Rebellions, FuriosaAI, AMD, and others, Lablup is able to offer its customers the most AI accelerators and GPUs on the market, making the Backend.AI platform more competitive and giving customers more choice.

*As of November 2024, Backend.AI supports the Intel® Gaudi® 2 AI accelerator. 

*Support for Intel® Gaudi® 3 AI Accelerators is planned on the first half of 2025.


Lablup and Intel have worked closely to unlock the power of Intel® Gaudi® 2 & 3 Platform and make it available in Backend.AI, and as the result of this collaboration, we are pleased to announce Backend.AI now supports the Intel® Gaudi® 2 and Intel® Gaudi® 3 AI Accelerators.

### Powerful container orchestration with Sokovan™ by Backend.AI
Sokovan™ is a standalone open-source container orchestrator highly optimized for latest hardware acceleration technologies used in multi-tenant and multi-node scaling scenarios. With fully customizable job scheduling and node allocation policies, it accommodates a hybrid mix of interactive, batch, and service workloads in a single cluster without compromising the performance of the AI.

### Get the most out of the AI Accelerators, reach your maximum possibilities.
In complex business, promising AI performance and manageability is the key to success. Intel® Gaudi® 3 AI accelerator, the latest release from Intel, offers powerful AI performance and features. Lablup Backend.AI, a Platform-as-a-Service, offers a wide range of features which are optimized for enterprise-grade AI environments.

### Innovation deeply integrated with Intel® Gaudi® 2 & 3 Platform
Customers who have already adopted Intel® Gaudi® 2 AI Accelerators or Intel® Gaudi® 3 AI Accelerators in their business environments, as well as those who will adopt Intel® Gaudi® 2 & 3 Platform in the future, will benefit from the wide range of features that Lablup Backend.AI supports for Intel® Gaudi®. Check out the following examples made possible with Backend.AI with Intel® Gaudi® 2 & 3 Platform.

#### Card-level accelerator allocation
Maximize Intel® Gaudi® 2 & 3 AI Accelerators cluster workload by lending users the actual number of accelerators intended to. For example, customers can run and train models on their existing preferred platform, then serve on Intel® Gaudi® 2 & 3 Platforms, or vice versa.
#### External storage allocation 
Get the most of out the integrated storage solutions in terms of performance. Utilize vendor-specific filesystem acceleration features without user intervention. Backend.AI supports major, widely used platforms such as Dell PowerScale, VAST Data, WEKA, NetApp, etc.,
#### Multi-scale workloads
Whatever your environment is, from single-card AI workload which can run small models, to multi-node multi-card AI workload which can run gigantic models, Backend.AI ensure its best performance. At this point as Nov.1, Backend.AI is ready to run Single-card AI workloads and Single-node, Multi-card AI workloads. Multi-node, Multi-card AI workload support will be finalized this year. 
#### Inference statistics management
Monitor up-to-date, detailed metrics about the performance provided by your AI framework. Backend.AI makes inference statistics management easy, not only showing the information from the hardware, but also on software so that administrator can deep-dive into the metrics.
#### Rule-based inference replica auto scaling 
Let the system self-optimize the resource usage.  With varied user traffic to inference workloads based on a combination of hardware and software performance metrics, administrators do not need to manually control remaining resources.

*Currently in development (Targeting Dec. 2024)

#### NUMA-aware resource allocation 
Achieve the maximum bare-metal performance by eliminating inter-CPU and PCIe bus overheads within a single node, when there are multiple CPU sockets and multiple accelerators for each socket. 

#### User-based, Project-based storage quota management 
Budget-efficient, easy data-space management by limiting data storage quota per user or single project. 

#### Hugepage memory allocation support 
Minimize the CPU overheads when using accelerators by using larger memory pages but fewer in number to reduce address translation overheads. This support will be finalized this year. 

#### ... and much more
Lablup continues to communicate with Intel, expanding the possibility of Backend.AI. Many more are still in development and will be announced soon. Elevate what your cluster can do with Backand.AI and Intel® Gaudi® 3 AI Accelerators.

### Getting most out of your Intel® Gaudi® 3 AI Accelerators.
 ![](https://cdn.lablup.com/2024_11_06_141836_9836bea9c4.png) 
Backend.AI is designed to bring out the maximum performance Intel® Gaudi® 3 AI Accelerators are capable of. Built on the high-efficiency Intel® Gaudi® platform with proven MLPerf benchmark performance, Intel® Gaudi® 3 AI accelerators are built to handle demanding training and inference. Support AI applications like Large Language Models, Multi-Modal Models and Enterprise RAG in your data center or in the cloud—from node to mega cluster, all running on the Ethernet infrastructure you likely already own. Whether you need a single accelerator or thousands, Intel® Gaudi® 3 can play a pivotal role in your AI success.


### Do these with superior user interface.
Unlike other systems, Backend.AI is designed for system administrators to control their system easily as possible. Our consumer-grade user interface makes administrators manage their system within a few clicks and types. Backend.AI WebUI is widely used by our proven customers, and they love what they can do without opening the Command Line Interface. 

### Make your Intel® Gaudi® 2 & 3 Platform 'manageable' with Backend.AI, unleash your performance.
We are making AI services efficient, scalable, and accessible to scientists, researchers, DevOps, enterprises, and AI enthusiasts. Lablup and Intel are working closely together to enable the success of Generative AI and deep learning-based services that are popular today. 
With our proven technology, Backend.AI provides hardware-level integration with Intel® Gaudi® 2& 3 Platform for the best effort.


### About Intel® Gaudi® 3 AI accelerator
Intel® Gaudi® 3 AI accelerator is driving improved deep learning price-performance and operational efficiency for training and running state-of-the-art models, from the largest language and multi-modal models to more basic computer vision and NLP models. Designed for efficient scalability—whether in the cloud or in your data center, Intel® Gaudi® 3 AI Accelerators bring the AI industry the choice it needs—now more than ever.
To learn more about Intel® Gaudi® 3, visit [intel.com/gaudi3](https://intel.com/gaudi3)

### About Lablup Backend.AI
Backend.AI supports a wide range of GPUs and AI accelerators on the market to achieve maximum efficiency of its performance and provides a user interface to make everything easy. This allows customers to efficiently build, train, and deliver AI models, from the smallest to the largest language models, significantly reducing the cost and complexity of developing and operating services. Backend.AI is the key to unlock the full potential of Generative AI and Accelerated Computing, transforming your business with cutting-edge technology.
To learn more about Backend.AI®, visit [backend.ai](https://www.backend.ai/)


Lablup x Intel Announce Support for Intel® Gaudi® 2 & 3 Platforms in Backend.AI

Uncharted AI: The Age of AI

Yonggeun Kwon

Finetuning Domain adaptive language model with FastTrack

Jihyun Kang


# Introduction

Imagine a scenario where you need to train an AI for research purposes and produce results. Our job would simply be to wait for the AI to correctly learn the data we've taught it. However, if we assume we're creating a service that 'utilizes' AI, things get more complicated. Every factor becomes a concern, from how to apply various models to the system to what criteria to use for scaling under load conditions. We can't carelessly modify the production environment where users exist to get answers to these concerns. If an accident occurs while expanding or reducing the production environment, terrible things could happen. If something terrible does happen, we'll need time to recover from it, and we can't expect the same patience from consumers using our service as we would from researchers waiting for model training. Besides engineering difficulties, there are also cost challenges. Obviously, there's a cost to serving models, and users are incurring expenses even at the moment of training models as resources are being consumed.

However, there's no need to worry. Many well-made models already exist in the world, and in many cases, it's sufficient for us to take these models and serve them. As those interested in our solution may already know, Backend.AI already supports various features you need when serving models. It's possible to increase or decrease services according to traffic, and to serve various models tailored to users' preferences.

But the Backend.AI team doesn't stop here. We have enhanced the model service provided from Backend.AI version 23.09 and improved it to easily serve various models. Through this post, we'll explore how to serve various models easily and conveniently.

This post introduces features that allow you to serve various types of models more conveniently. Since we've already given an explanation about model service when releasing the 23.09 version update, we'll skip the detailed explanation. If you're unfamiliar with Backend.AI's model service, we recommend reading the following post first: [Backend.AI Model Service Preview](https://www.backend.ai/blog/2023-05-30-backend.AI-model-service-sneak-peek)

# Existing Method

| Requirement | Existing Method | Model Variant |
|-------------|-----------------|---------------|
| Writing model definition file (model-definition.yaml) | O | X |
| Uploading model definition file to model folder | O | X |
| Model metadata needed | O | △ (Some can be received automatically) |

Backend.AI model service required a model definition file (model-definition.yaml) that contains commands to be executed when serving the model in a certain format, in addition to the model metadata needed to run. The service execution order was as follows: Write the model definition file, upload it to the model type folder so it can be read, and when starting the model service, mount the model folder. Then, an API server that automatically transfers the end user's input to the model according to the model definition file and sends back the response value would be executed. However, this method had the disadvantage of having to access the file every time the model definition file needed to be modified. Also, it was cumbersome to write different model definition files each time the model changed because the model path was already set in the model definition file.

The Model Variant introduced this time is a feature that allows you to serve models immediately by inputting a few configuration values or without any input at all, using only model metadata without a model definition file. Model Variant supports command, vLLM, and NIM (NVIDIA Inference Microservice) methods. The methods of serving and verifying model service execution are as follows.

Basically, model service requires metadata of the model to be served. Download the model you want to serve from Hugging Face, where you can easily access model metadata. In this example, we used the Llama-2-7b-hf model and Calm3-22b-chat model from Hugging Face. For how to upload model metadata to the model folder, refer to the "Preparing Model Storage" section in the previous post.

# Automatically Serving Model from Built Image (Command Method)

The first introduced command method is a form where the command part that executes to serve the model in the model definition file is included in the execution image. After specifying the command to execute in the CMD environment variable, build the image and execute it immediately without any other input when actually serving the model. The command method does not support what's called a Health check, which verifies if the service is running properly. Therefore, it's more suitable for immediately setting up and checking a service as a prototype rather than performing large-scale services. The execution method is as follows:

1. On the start screen, select `Llama-2-7b-hf` in the Model Storage To Mount item to mount the model folder containing the model metadata corresponding to the model service to be served, and select Predefined Image Command in the Inference Runtime Variant item.

> Activate the Open To Public switch button if you want to provide model service accessible without a separate token.

![모델-서비스-시작화면-모델-메타데이터-마운트-및-CMD-선택](https://cdn.lablup.com/cmd_model_variant_selected_in_model_service_launcher_1_279931a66c.png)

2. Select the environment to serve. Here, we use `vllm:0.5.0` and allocate CPU 4 Core, Memory 16 GiB, NVIDIA CUDA GPU 10 fGPU as resources.

![모델-서비스-시작화면-실행환경-선택-및-자원할당](https://cdn.lablup.com/cmd_model_variant_selected_in_model_service_launcher_2_01fb14385f.png) 

3. Finally, select the cluster size and click the start button. The cluster size is set to single node, single container.

![모델-서비스-시작-화면-클러스터-크기-선택-및-시작](https://cdn.lablup.com/cmd_model_variant_selected_in_model_service_launcher_3_d401885607.png) 

If the service has been successfully launched, the service status will change to `HEALTHY` and the endpoint address will appear.

![모델-서비스-상세-화면](https://cdn.lablup.com/cmd_model_variant_status_healthy_be1df76a76.png) 


## Verifying the Service

If the service has been launched normally, check the service model name with the `cURL` command:

```sh
curl https://cmd-model-service.asia03.app.backend.ai/v1/models \
-H "Content-Type: application/json"
```

![모델명-확인하기](https://cdn.lablup.com/cmd_model_variant_serving_model_check_e485a7a941.png) 

Now, let's send input to the service with the `cURL` command and check the response:

> For model services run with CMD, the model name is already defined in the image, so after checking the model name, you must enter the model name as the value of the `model` key when sending a request.

```sh
curl https://cmd-model-service.asia03.app.backend.ai/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "image-model",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0}'
```

![모델-서비스-요청-결과-화면](https://cdn.lablup.com/cmd_model_variant_serving_result_9c48dc19e6.png) 

# Serving Models in vLLM Mode

The vLLM mode is similar to the command method introduced earlier, but various options entered when running vLLM can be written as environment variables. The execution method is as follows:

## How to Run

1. On the start screen, mount the model folder for the model service to be served and select `vLLM` in the Inference Runtime Variant item.

![모델-서비스-시작-화면-모델-메타데이터-마운트-및-vLLM-선택](https://cdn.lablup.com/vllm_model_variant_selected_in_model_service_launcher_1_68087bc66c.png) 


2. Select the environment to serve. As with the command method explained earlier, select `vllm:0.5.0`, and (although you can set the resources the same) this time we'll allocate CPU 16 Core, Memory 64 GiB, NVIDIA CUDA GPU 10 fGPU.

![모델-서비스-시작-화면-실행환경-선택-및-자원-할당](https://cdn.lablup.com/vllm_model_variant_selected_in_model_service_launcher_2_851a94e52b.png) 

3. Finally, select the cluster size and enter the environment variable `BACKEND_MODEL_NAME`. This value corresponds to the `--model-name` option in vLLM and becomes the model value specified by the user when sending a request to the service.
![모델-서비스-시작-화면-실행환경-선택-및-자원-할당](https://cdn.lablup.com/vllm_model_variant_selected_in_model_service_launcher_3_updated_15b63bb66e.png) 

Likewise, if the service has been successfully launched, the service status will change to `HEALTHY`, and the endpoint address where the service is launched will appear.

![모델-서비스-상세-화면](https://cdn.lablup.com/vllm_model_variant_status_healthy_c0f91fb42f.png) 

## Verifying the Service

Let's send input to the service with the `cURL` command and check the response value. At this time, enter the model value as the `BACKEND_MODEL_NAME` value you set earlier. Once the input is complete, click the `START` button to create the service.

```sh
curl https://vllm-calm3-22b-chat.asia03.app.backend.ai/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "vllm-model",
"prompt": "初めて会う日本人ビジネスマンに渡す最高の挨拶は何でしょうか？",
"max_tokens":  200,
"temperature": 0
}'
```

![모델-서비스-요청-결과-화면](https://cdn.lablup.com/vllm_model_variant_result_2_ad74f0309e.png) 

# Serving Models in NIM Mode

> To run NIM, you need an API key issued from an account that can access NGC's NIM model registry.
> For how to obtain the key value, please refer to the following content: [NVIDIA Docs Hub : How to get NGC API Key](https://docs.nvidia.com/ai-enterprise/deployment-guide-spark-rapids-accelerator/0.1.0/appendix-ngc.html)

The NIM (NVIDIA Inference Microservice) mode is also similar to the command mode, but it must be run with an image that has NVIDIA's NIM-supporting model server built-in. Also, when loading the model, the NGC API key value is needed. Assuming everything is ready, let's start the model service.

## How to Run

1. On the start screen, select an empty model type folder to cache the metadata to be received from NIM, and select `NIM` in the Inference Runtime Variant item.

![모델-서비스-시작-화면-모델-폴더-마운트-및-NIM-선택](https://cdn.lablup.com/nim_model_variant_selected_in_model_service_launcher_1_efc31faaf0.png) 

2. Select the environment to serve. Here, we use `ngc-nim:1.0.0-llama3.8b` and set to allocate CPU 8 Core, Memory 32 GiB, NVIDIA CUDA GPU 15 fGPU as resources.

![모델-서비스-시작-화면-실행환경-선택-및-자원-할당](https://cdn.lablup.com/nim_model_variant_selected_in_model_service_launcher_2_ee2be391e0.png) 

3. Finally, select the cluster size and enter the default path `/models` for the environment variable `HF_HOME`. Then enter `NGC_API_KEY` and input the issued key value. Once the input is complete, click the `CREATE` button to create the service.

![모델-서비스-시작-화면-클러스터-크기-선택-환경변수-입력-및-시작](https://cdn.lablup.com/nim_model_variant_selected_in_model_service_launcher_3_updated_bfbe08607d.png) 

> When using NIM, it may take some time for the first execution as it receives model metadata from the repository. You can check the progress by viewing the container logs for the routing session in service on the session page.
> ![모델-서비스에-대응하는-라우팅-세션](https://cdn.lablup.com/nim_model_variant_session_row_dd01dac411.png) 
> ![NIM-에서-데이터를-받고-있는-로그가-띄워진-컨테이너-로그-화면](https://cdn.lablup.com/nim_model_variant_nim_cache_cb99e937e1.png) 

Like the command and vLLM modes, if the service has been successfully launched, the service status will change to `HEALTHY`. Let's input the content to send to the service using the endpoint address where the service is launched as follows, and check the response value.

## Verifying the Service

```python
from openai import OpenAI

client = OpenAI(
  base_url = "https://nim-model-service.asia03.app.backend.ai/v1",
  api_key = "$YOUR_NGC_API_KEY"
)

completion = client.chat.completions.create(
  model="meta/llama3-8b-instruct",
  messages=[
      {        
        "role":"user", 
        "content":"Hello! How are you?"
      },
      {
        "role":"assistant",
        "content":"Hi! I am quite well, how can I help you today?"
      },
      {
        "role":"user",
        "content":"Can you write me a song?"
      }],
  temperature=0.5,
  top_p=1,
  max_tokens=1024,
  stream=True
)

for chunk in completion:
  if chunk.choices[0].delta.content is not None:
    print(chunk.choices[0].delta.content, end="")
```

![모델-서비스-요청-결과-화면](https://cdn.lablup.com/nim_model_variant_serving_result_ed05acc7d8.png) 

# Conclusion

The Model Variant feature will be of great help to researchers and companies aiming to provide actual services with already trained models. Based on a powerful resource management system and support for various AI accelerators such as NVIDIA GPU, AMD ROCm, TPU, Graphcore IPU, Furiosa Warboy, Rebellions ATOM, Hyperaccel LPU, etc., Backend.AI now provides an integrated environment that can easily deploy services beyond simply training models.
Try serving your desired AI model anytime with Backend.AI!

Model Variant: Easily Serving Various Model Services

Daehyun Sung

Backend.AI Open Source Contribution Guide (Jul. 2024)

Jeongkyu
Shin

> This article originally appeared in [Engineers and Scientists for Change](https://www.esckorea.org/SnEnews/?idx=11476297&bmode=view), May 2022.

![](https://cdn.lablup.com/Neuromorphic_Computing_e0063e6c53.jpeg) 

The field of artificial neural networks has been gaining serious attention for nearly a decade now. In that short time, along with the advancements in deep learning, the field has been solving countless problems at an astonishing pace. It is considered the most promising approach for achieving artificial intelligence.

At the cutting edge, news about hyperscale deep learning models and their implementations have been garnering attention. In April 2022, news about NVIDIA's new H100 GPU flooded the headlines. AMD's high-performance computing GPUs like the MI series, along with Intel's new Ponte Vecchio GPU, are expected to drastically improve AI and blockchain mining acceleration, creating a new battleground for hyperscale AI.

Amidst the buzz around hyperscale AI, there was one piece of news that didn't receive much public interest: Intel's announcement of the Loihi 2 chip in October last year<a name="note_1" href="#footnote_1">[1]</a>. This news comes with a fascinating history and technical background. While AI training and service acceleration chips are proliferating, let's explore the science behind this intriguing tech news.

# Can we code intelligence with deep learning?

Since 2013, when deep learning took off with matrix computation acceleration using GPUs, the field has begun exploring various possibilities empowered by the scale of computation. Starting with the AlphaGo shock in 2016, deep learning has gradually expanded its scope beyond research into practical applications. The transformer model architecture<a name="note_2" href="#footnote_2">[2]</a>, proposed in 2017 and widely adopted from 2018, introduced the concepts of attention and self-attention, greatly improving the process of deep learning models creating their own memory structures. The transformer architecture has since been used in a wide range of deep learning models, particularly excelling in the language processing and image processing domains, where data is abundant. Transformers have enabled deep learning models to solve various problems that previously seemed intractable.

This seemingly omnipotent model has led the trend of scaling up deep learning models since 2018. The size of a deep learning model is determined by the number of its parameters, which are the connection information between the perceptrons that make up the model, corresponding to the synaptic connections of actual neurons. More connections allow the deep learning model to differentiate and judge more complex inputs. As deep learning models become more complex and massive, the number of parameters grows exponentially. Until 2019, the number of parameters in deep learning models increased roughly 3 to 5 times each year, but since 2019, it has been increasing more than tenfold annually. The massive deep learning models that have emerged in the past 2-3 years are sometimes referred to as "hyperscale AI". Well-known hyperscale deep learning models in the language processing domain include OpenAI's GPT-3 and Google's LaMDA. For these huge models, such as GPT-3, the system cost for training the model (the cost of a single training run on the cloud without purchasing equipment) is estimated to be at least around 5 billion KRW (approximately 4 million USD)<a name="note_3" href="#footnote_3">[3]</a>.

Hyperscale models are solving problems that were previously difficult or impossible to solve. They discover new gravitational lenses<a name="note_4" href="#footnote_4">[4]</a> and unwarp the distortions caused by lenses<a name="note_5" href="#footnote_5">[5]</a> to unravel the mysteries of the universe. They predict protein folding structures in much shorter times and at lower costs than previous methods<a name="note_6" href="#footnote_6">[6]</a> and discover new drugs<a name="note_7" href="#footnote_7">[7]</a>. They even solve problems like StarCraft II strategic simulations<a name="note_8" href="#footnote_8">[8]</a>, which require understanding flows over long periods of time.

As these models solve various problems, naturally, some questions arise. Is this approach of pouring in massive amounts of resources to create deep learning models sustainable? And can we code "intelligence" using this method?

To answer these two questions, let's quickly understand deep neural networks and today's topic, neuromorphic computing.

# Deep Neural Networks: Origins and Differences 

Deep learning is actually an abbreviation. The full term is Deep Neural Network (DNN), or more elaborately, Artificial Neural Network with deep layers. The theory of artificial neural networks has its roots in mathematically imitating the electrical properties of neurons. It began by mathematically imitating the electrical properties of neurons along with the plasticity<a name="end_1" href="#endnote_1">[e1]</a> of strengthening or weakening the connections between neurons during information processing, and then simplifying it. The artificial neural network model is a mathematical model that consists of a perceptron<a name="note_9" href="#footnote_9">[9]</a>, which is an extreme simplification of the activation process based on the connections between neurons; an activation function that simplifies the firing process of neurons into a function of signal input to neurons, excluding time dependency; and weights as parameters representing the strength of connections between neurons.

Although artificial neural network theory has its roots in the characteristics of actual neural networks, there is a fundamental difference: the presence or absence of dynamics that determine behavior over time. In real neural networks, various outcomes are determined by the dynamics between neurons. Neurons have their own dynamic characteristics when stimulated from the outside, and they have plasticity that physically strengthens or weakens accordingly. For example, neurons that are continuously used and connected together when making a certain decision are activated at similar times when receiving an input signal. We can observe that the axons corresponding to the connections between neurons activated at "temporally" similar times become physically thicker. In contrast, general artificial neural networks simulate plasticity using backpropagation theory instead of dynamics. Backpropagation theory is a method to simplify the calculation of strengthening the weights of connections between perceptrons that were used to make a correct decision. In artificial neural networks, the process of processing input information is instantaneous using the weights between perceptrons. Since the process of input information leading to output information is not calculated as a function of time, there is no dynamic element.

There are various other differences besides dynamics. These differences are usually the result of introducing assumptions that are impossible in biological neural networks in order to overcome the limitations of artificial neural network theory in the 1990s. One example is the use of ReLU<a name="end_2" href="#endnote_2">[e2]</a> as an activation function. Ordinary neurons have thresholds and weight limits. Infinite activation values are physically impossible. So mathematical models also used functions that well represent thresholds and weight limits as activation functions. However, as deep artificial neural networks became deeper, researchers discovered that the training of artificial neural networks stopped progressing.<a name="end_3" href="#endnote_3">[e3]</a> The ReLU activation function, although physically impossible, can have infinite weights mathematically<a name="end_4" href="#endnote_4">[e4]</a>. By introducing ReLU into deep artificial neural networks, new training became possible, and the difference from biological neural networks grew.

Artificial neural networks that do not need to consider dynamics can be transformed into a sequence of matrix operations, enabling incredibly fast computation. However, they have become very different from the neural networks seen in biology. So, are deep learning models and the neurological processes occurring in our brains now on completely different foundations?

# Learning from Nature Again: Dynamics of Neural Networks

Individual neurons communicate signals in various ways. Some are electrical signals, and some are chemical signals. The electrical signal characteristics within single neurons were interpreted and formulated very early<a name="note_10" href="#footnote_10">[10]</a> and became the theoretical basis for perceptrons. The problem was that the formula was too complex to calculate dynamics without simplification. Later, various mathematical models were proposed to approximate the electrical responses over time with reduced computational burden, and various single neuron simulators based on these models have been released. A representative simulator is NEURON<a name="note_11" href="#footnote_11">[11]</a>.

As mentioned earlier, simulating dynamics requires an enormous amount of computation. At some point, we are entering an era of abundant computational power. What would happen if we connected these single neuron simulations based on the overflowing computational power?

There are two approaches, in terms of algorithms and hardware, to solve the problem of computational speed due to the enormous amount of computation and create dynamics-based artificial neural networks. The algorithmic approach is the spiking neural network (SNN), which attempts to create a dynamics-based artificial neural network by introducing spike-based plasticity that occurs in actual neurons. The hardware approach is neuromorphic computing, which has been in full swing since 2012. It involves implementing an artificial neural network by creating physical objects corresponding to neurons. Computers are still too slow to solve the enormous amount of computation involved in dynamics simulation with general-purpose computation. To address this, the idea is to create dedicated devices that either make objects with mathematical properties corresponding to neurons at the circuit level or hardcode computations. Recently, there has been an integration of not distinguishing between neuromorphic computing and SNN, and referring to the implementation of SNN at the device level as neuromorphic computing. Both approaches are attempts to simulate the dynamic characteristics that traditional artificial neural network theory did not use, to discover new phenomena or possibilities of deep learning.

One of the companies making strides in the field of neuromorphic computing is Intel. In the fall of 2017, Intel unveiled the Loihi<a name="end_5" href="#endnote_5">[e5]</a> chip, a research neuromorphic chip containing approximately 130,000 neurons and 130 million synapses. They ported existing DNN-based algorithms onto the Loihi chip, performed various comparative tests<a name="note_12" href="#footnote_12">[12]</a>, and interestingly, showed that similar results to DNN can be obtained using SNN as well.

Intel then connected multiple Loihi chips to create massive SNN systems. Nahuku implemented 4.1 billion synapses, and the Pohoiki Springs neuromorphic supercomputer<a name="note_13" href="#footnote_13">[13]</a> implemented approximately 101 million neurons and 100 billion synapses based on 768 Loihi chips. In this process, Intel developed a software stack to implement SNN on Loihi. As a result, last fall, along with Loihi 2, they released the Lava open-source software framework<a name="note_14" href="#footnote_14">[14]</a> for developing neuromorphic applications.

It was expected that DNN and SNN would show similar results. From a physics perspective, the process of artificial neural networks inferring various problems is ultimately defining an ultra-high-dimensional discontinuous state space based on information and projecting new information onto that space. Both DNN and SNN have the characteristic of being able to define ultra-high-dimensional discontinuous state spaces. Through evolution, biology has physically created the characteristic of adapting to information, and humanity has invented artificial neural network theory through biomimetics and developed deep learning.

# Always Finding Answers, as Always

So far, we have learned that networks that imitate neurons at the dynamics level can also obtain results similar to what we expected from deep learning. Then, a question arises: if the results are similar, is there a need to use SNN and neuromorphic computing? The examples introduced today are just a tiny fraction of various attempts. Research is ongoing on how SNN and neuromorphic computing produce different results from existing approaches. There are also results showing that SNN performs better, especially in robotics and sensors, and studies suggesting that reflecting dynamic characteristics would be more powerful for inferring causality. There are even attempts to simulate the chemical signals occurring at synapses<a name="note_15" href="#footnote_15">[15]</a>. This is because, in addition to the connection structure of neural networks, there may be elements in the individual components that make up neural networks that evoke the emergence of intelligence that we are not yet aware of. However, this may not be a sufficient answer to why SNN is used.

Let's ask the two questions posed at the beginning of the article again. Is this approach of pouring in massive amounts of resources to create deep learning models sustainable? And can we code "intelligence" using this method? Could neuromorphic computing be the answer? It could be, or it might not be.

The reason why DNN and SNN each show high performance and results is ultimately because there is an information optimization theory that we do not yet know at the foundation of both implementations. If we come to understand that, we may be able to implement AI in a different way. It could be one path to finding an answer to the first question: "Is this approach of pouring in massive amounts of resources to create deep learning models sustainable?" Neuromorphic computing and SNN allow us to examine this problem from a new perspective.

And it could also be the answer to the second question. We always carry a question in our hearts: 'Who are we?' The approach of neuromorphic computing and SNN is the most easily understandable method when we physically approach this fundamental philosophical question. Because it explains using a system that we already know (although we don't yet know its framework).

Various fields, including neuromorphic computing, are challenging to answer the above two questions. One of them is quantum computing. Next time, let's take the opportunity to read an article together about quantum computing and deep learning.

**References**
- <a name="footnote_1" href="#note_1">[1]</a> https://www.anandtech.com/show/16960/intel-loihi-2-intel-4nm-4
- <a name="footnote_2" href="#note_2">[2]</a> https://arxiv.org/abs/1706.03762
- <a name="footnote_3" href="#note_3">[3]</a> https://lambdalabs.com/blog/demystifying-gpt-3
- <a name="footnote_4" href="#note_4">[4]</a> https://iopscience.iop.org/article/10.3847/1538-4357/abd62b
- <a name="footnote_5" href="#note_5">[5]</a> https://academic.oup.com/mnras/article-abstract/504/2/1825/6219095 
- <a name="footnote_6" href="#note_6">[6]</a> https://www.nature.com/articles/s41586-021-03819-2
- <a name="footnote_7" href="#note_7">[7]</a> https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full
- <a name="footnote_8" href="#note_8">[8]</a> https://www.deepmind.com/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii
- <a name="footnote_9" href="#note_9">[9]</a> https://doi.apa.org/doi/10.1037/h0042519
- <a name="footnote_10" href="#note_10">[10]</a> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1392413
- <a name="footnote_11" href="#note_11">[11]</a> https://neuron.yale.edu/neuron
- <a name="footnote_12" href="#note_12">[12]</a> https://ieeexplore.ieee.org/document/8259423
- <a name="footnote_13" href="#note_13">[13]</a> https://arxiv.org/abs/2004.12691
- <a name="footnote_14" href="#note_14">[14]</a> https://www.intel.com/content/www/us/en/newsroom/news/intel-unveils-neuromorphic-loihi-2-lava-software.html
- <a name="footnote_15" href="#note_15">[15]</a> https://www.ibm.com/blogs/research/2016/12/the-brains-architecture-efficiency-on-a-chip/

**Endnotes**
- <a name="endnote_1" href="#end_1">[e1]</a> Plasticity is the ability to adapt and change one's characteristics in response to changes in the external environment or stimuli.
- <a name="endnote_2" href="#end_2">[e2]</a> It stands for Rectified Linear Unit. It is an activation function that takes the shape of y=x for values greater than 0, which means y can continue to increase as x increases.
- <a name="endnote_3" href="#end_3">[e3]</a> This is known as the Vanishing Gradient problem.
- <a name="endnote_4" href="#end_4">[e4]</a> Neurons cannot produce outputs beyond the physical limits of the cell, no matter how much larger the input they receive. It's like not being able to pass an unlimited current through a wire. ReLU is a function where the output increases linearly and indefinitely in proportion to the input.
- <a name="endnote_5" href="#end_5">[e5]</a> Intel is using various Hawaiian place names as codenames for their neuromorphic chips and systems.

Learning from Nature Again: Neuromorphic Computing and Deep Learning

Jeongseok Kang

From the now-classic AlexNet to various large language models (LLMs) that are garnering a lot of attention these days, we train and evaluate various models to suit our needs. However, realistically, it's difficult for us to gauge when the training will end until we run the model multiple times and gain experience.

**Backend.AI**'s excellent scheduling minimizes GPU idle time and allows model training to run even while we sleep. Then, what if we could receive the results of a model that finished training while we were asleep? In this article, we'll cover how to receive model training results as messages using the new feature of FastTrack and Slack.

> This article is based on the [Backend.AI FastTrack](https://www.backend.ai/ko/product/fasttrack) version 24.03.3.

# Before We Start

This article does not cover how to create a Slack App and Bot. For detailed information, we recommend referring to the [official documentation](https://api.slack.com/tutorials/tracks/getting-a-token).

# Creating a Pipeline

Let's create a pipeline for model training. A pipeline is a unit of work used in FastTrack. Each pipeline can be expressed as a collection of tasks, the minimum execution unit. Multiple tasks included in a single pipeline can have interdependencies, and they are executed sequentially according to these dependencies. Resource allocation can be set for each task, allowing flexible resource management.

When an execution command is sent to a pipeline, it is executed by replicating the exact state at that point, and this unit is called a pipeline job. Multiple pipeline jobs can be run from a single pipeline, and each pipeline job is generated from a single pipeline.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_01_1e7a9f9bcc.png"/>

 Create Pipeline button


Click the Create Pipeline button ("+") at the top of the pipeline list.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_02_e6d7004df2.png"/>

 Creating a Pipeline


You can specify the pipeline's name, description, location of the data store to use, environment variables to be applied commonly across the pipeline, and the method of pipeline initialization. Enter the name "slack-pipeline-0", and then click the "Create" button at the bottom to create the pipeline.

# Creating Tasks

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_03_47d8a90ee4.png"/>

 Dragging a Task


You can see that the new pipeline has been created. Now let's add some tasks. From the task template list (Task templates) at the top, drag and drop the "Custom Task" block onto the workspace below.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_04_d136041eea.png"/>

 Entering the Task's Actions


A task details window appears on the right where you can enter the task's specifics. You can give it a name like `model-training-task` to indicate its role, and set it to use the `pytorch:1.11-py38-cuda11.3` image for model training. Since actual model training can take a long time, for this example, we'll have it execute the following simple commands:

```shell
# Pause for 3 seconds to increase the execution time.
sleep 3
# Create a `result.txt` file in the pipeline-dedicated folder. Assume this is the accuracy of the trained model.
echo "0.$RANDOM" > /pipeline/outputs/result.txt
```

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_05_6ee1550de7.png"/>

 Creating a Task (1)


Finally, enter the resource allocation for the task, and then click the "Save" button at the bottom to create the task.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_06_32b7134630.png"/>

 Dragging Another Task


You can see that the `model-training-task` has been created in the workspace. This time, to create a task that reads the value from the `result.txt` file saved earlier and sends a Slack notification, drag another "Custom Task" block into the workspace below.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_07_ae06c67ede.png"/>

 Entering the Task-level Environment Variable `SLACK_TOKEN`


For this task, set the name to `slack-alarm-task`, and enter the following script to send a notification to Slack:

```shell
pip install slack-sdk
python -c '
import os
from pathlib import Path
from slack_sdk import WebClient
SLACK_BOT_TOKEN = os.environ.get("SLACK_TOKEN")
JOB_ID = os.environ.get("BACKENDAI_PIPELINE_JOB_ID")
def main():
 result = Path("/pipeline/input1/result.txt").read_text()
 client = WebClient(token=SLACK_BOT_TOKEN)
 client.chat_postMessage(
 channel="#notification",
 text="Pipeline job({}) finished with accuracy {}".format(JOB_ID, result),
 )
if __name__ == "__main__":
 main()
'
```

The code above uses two environment variables: `SLACK_TOKEN` and `BACKENDAI_PIPELINE_JOB_ID`. Environment variables in the `BACKENDAI_*` format are values automatically added by the **Backend.AI** and FastTrack systems, where `BACKENDAI_PIPELINE_JOB_ID` represents the unique identifier of the pipeline job in which each task is running.

The other environment variable, `SLACK_TOKEN`, is a task-level environment variable. This feature allows you to manage and change various values without modifying the code.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_08_1773bf037e.png"/>

 Creating a Task (2)


After allocating appropriate resources for the `slack-alarm-task`, click the "Save" button at the bottom to create the task.

# Adding Task Dependencies

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_09_0e7fb291fe.png"/>

 Adding Task Dependencies


Now there are two tasks (`model-training-task` and `slack-alarm-task`) in the workspace. Since `slack-alarm-task` should be executed after `model-training-task` completes, we need to add a dependency between the two tasks. Drag the mouse from the bottom of the task that should run first (`model-training-task`) to the top of the task that should run later (`slack-alarm-task`).

# Running the Pipeline

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_10_0eadb2825f.png"/>

 Running the Pipeline (1)


You can see an arrow connecting from `model-training-task` to `slack-alarm-task`, indicating that the dependency has been added. Now, to run the pipeline, click the "Run" button in the top right.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_11_146b8b7ef7.png"/>

 Running the Pipeline (2)



Before running the pipeline, you can review a brief summary of it. After confirming the presence of the two tasks, click the "Run" button at the bottom.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_12_1b7b772e3c.png"/>

 Running the Pipeline (3)


The pipeline was successfully run, and a pipeline job was created. Click "OK" at the bottom to view the pipeline job information.

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_13_fe1f9c4b43.png"/>

 Pipeline Job


The pipeline job was created successfully. You can see that the model training (`model-training-task`) has completed, and `slack-alarm-task` is running.

# Receiving Slack Notification

<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_slack_01_f1f1864953.jpg"/>

 Slack Notification (1)


<img style="margin: auto;" src="https://cdn.lablup.com/blog_fasttrack_slack_02_38a389156e.jpg"/>

 Slack Notification (2)


You can see that the pipeline job execution results have been delivered to the user via Slack. Now we can sleep soundly.

FastTrack Guide: Receiving Notifications for Model Training Results

Gyubong Lee

Deconstructing a working Raft implementation - 2

24.03, the first release of Backend.AI in 2024, has been released. This update brings significant improvements to the UI and user experience, making it even more functional to install and operate. It includes the following updates since 23. 9, which include

# Backend.AI Core & WebUI

- We've made installing Backend.AI even easier with the addition of a TUI-based installer, which automates the download process and makes it easy for users to install and get started with Backend.AI.

![](https://cdn.lablup.com/backendai_TUI_installer_2feb143fc1.png)
- New: Added trash functionality to vfolders. Files in unused vFolders are now moved to the trash instead of deleting them completely, and are then deleted through a complete deletion process. New: Added an argument value to indicate the state of the vfolder.
- New: Added Backend.AI Model Store, where you can now store, search, and easily utilize various machine learning and deep learning models.
- Added metadata for indexing to vfolders to utilize indexes instead of full directory scans for queries.
- Improved system resource utilization by introducing a limit policy for session creation based on the number of pending sessions and requested resource slots. This new resource policy option helps filter and set the maximum value for resource presets and custom resource sliders in the Session Launcher.
- We've added dark themes to the WebUI, so users can now choose from a variety of options to suit their personal preferences.

![](https://cdn.lablup.com/Web_UI_dark_theme_397145c15d.png)
- Improved screen tuning for unaligned line breaks, whitespace, and extruded announcements in the WebUI, as well as stability improvements such as session name validation.
- The Session Launcher for Model Serving also limits UI input so that only as much input is available as the allocated resources.
- Added an `allowAppDownloadPanel` argument to hide the WebUI app download panel in the config.toml file for different UI user options. 

Backend.AI is constantly evolving to provide a more powerful and user-friendly experience while supporting a variety of environments in the ever-changing AI ecosystem. We look forward to seeing what's next!
Make your AI accessible with Backend.AI!

24.03: Release Update

Deconstructing a working Raft implementation - 1

Sujin Kim

Lablup commemorated the 2024 GTC event by hosting a special event. Participants created images similar to the given image using the LLM model provided by Lablup, and among those who scored high, an NVIDIA RTX 4090 graphics card was awarded through lottery. 🫢 In this post, we aim to highlight the subscription feature of GraphQL, which was used in the leaderboard page of the event, allowing participants to monitor their scores in real time.

<img style="margin: auto;" src="https://cdn.lablup.com/GTC_24_event_main_page_c675fa1692.png"/>

 GTC24 event page


## What is a Subscription?

It is a mechanism that allows the client to query data in response to a server side **event stream**. In cases where data changes in real time, for example when implementing real-time logs or chat applications, updates can be immediately reflected when pushed from the server.


Subscription sends data only when the required information changes on the server. Therefore, in the case where data changes are not frequent, Subscription can reduce data traffic, which can also lead to cost savings.

A similar concept is setting the `fetchPolicy` of GraphQL's Query to network-only to always get the latest results, but it’s different from the features of subscriptions. This ensures the latest data by always requesting the server whenever the client needs data. However, network costs accompany each request. Thus, while it is okay to set fetchPolicy to network-only to guarantee the latest results whenever a button is clicked, if it is used to retrieve data where updates are frequent like a stock trading window, network costs would be significant.

## How to Use

### Defining Subscription

The usage is similar to Query, just use the keyword `subscription`.

```graphql
 const leaderboardSubscriptions = graphql`
 subscription Ranking_leaderboardSubscription {
 leaderboard {
 submissions {
 id
 name
 score
 imageUrl
 }
 lastUpdatedAt
 }
 }
 `;
```

When an event occurs in the `leaderboard` stream, a notification is sent to the application, and the client can get the updated result.

Then the following result can be obtained.

```json
leaderboard: {
	submissions: [
		{
 	"id": "76293167-e369-4610-b7ac-4c0f6aa8f699",
	 "name": "test",
 	"score": 0.5910864472389221,
	 "imageUrl": "<IMAGE_URL>"
		},
 ],
	lastUpdatedAt: 1710176566.493705
}
```

### subscribe

To display real-time rankings, when entering the relevant page, call subscribe, and when moving to other pages, call dispose to unsubscribe using `useEffect`.

```ts
import { useEffect } from 'react';
import { requestSubscription } from 'react-relay';

useEffect(() => {
 const subscriptionConfig = {
 subscription: leaderboardSubscriptions,
 variables: {},
 onNext: (response: any) => {
 setLeaderboard(response.leaderboard.submissions); // 미리 정의된 state
 },
 onError: (error: any) => {
 console.error('Leaderboard subscription error', error);
 },
 };
 const { dispose } = requestSubscription(
 RelayEnvironment, // refer 'How to Configure' below
 subscriptionConfig,
 );
 return () => {
 dispose();
 };
}, []); // Executing this part only when the component is mounted or unmounted by setting an empty dependency array
```

`requestSubscription`

- Provides a `Disposable` object as a return value.
- This `Disposable` object includes a `dispose method to cancel the subscription.

`onNext`

- As data is updated through subscription, it updates the pre-defined state to display real-time rankings.
- In addition to `onNext`, `onError`, there are various configurations such as onCompleted called when the subscription ends and `updater` to update the in-memory relay storage based on server response. For detailed descriptions, refer to [this link](https://relay.dev/docs/v10.1.3/subscriptions/).

`dispose`

- A cleanup function is returned in the `useEffect` hook and the dispose method is called to end the subscription when the component is unmounted.

## How to set up (+Relay)

According to the [Relay documentation](https://relay.dev/docs/guided-tour/updating-data/graphql-subscriptions/#configuring-the-network-layer), GraphQL subscriptions communicate with [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API), and you can set up a network using [graphql-ws](https://github.com/enisdenjo/graphql-ws). (There is also a way to use [subscriptions-transport-ws](https://github.com/apollographql/subscriptions-transport-ws), but it's deprecated, so we'll pass on that).

```ts
import { ExecutionResult, Sink, createClient } from 'graphql-ws';
import {
 Environment,
 Network,
 RecordSource,
 Store,
 SubscribeFunction,
 RelayFeatureFlags,
 FetchFunction,
 Observable,
 GraphQLResponse,
} from 'relay-runtime';
import { RelayObservable } from 'relay-runtime/lib/network/RelayObservable';
import { createClient } from 'graphql-ws';

const wsClient = createClient({
 url: GRAPHQL_SUBSCRIPTION_ENDPOINT,
 connectionParams: () => {
 return {
 mode: 'cors',
 credentials: 'include',
 };
 },
});

const subscribeFn: SubscribeFunction = (operation, variables) => {
 return Observable.create((sink: Sink<ExecutionResult<GraphQLResponse>>) => {
 if (!operation.text) {
 return sink.error(new Error('Operation text cannot be empty'));
 }
 return wsClient.subscribe(
 {
 operationName: operation.name,
 query: operation.text,
 variables,
 },
 sink,
 );
 }) as RelayObservable<GraphQLResponse>;
};

// Export a singleton instance of Relay Environment
// configured with our network function:
export const createRelayEnvironment = () => {
 return new Environment({
 network: Network.create(fetchFn, subscribeFn),
 store: new Store(new RecordSource()),
 });
};

export const RelayEnvironment = createRelayEnvironment();
```

`wsClient`

- For url, enter the websocket URL of the GraphQL server.
- credentials can be set via `connectionParams`.

subscribeFn

- Defines the subscription behavior of the Observable.
- Validate the query string in `if (!operation.text) { ... }` and if it is invalid, raise an error and abort the execution.
- Finally, the `return wsClient.subscribe( ... )` code actually subscribes to the subscription using the WebSocket client and passes the payload of the GraphQL operation to the sink (i.e., the Observer).
- In short, this function is responsible for handling the GraphQL subscription request and pushing the result to the Observable stream whenever a subscription event occurs.

`createRelayEnvironment`

- Create and return a new Relay Environment.
- A Relay environment is a container that manages other high-level Relay objects, network layer, cache, etc.
- We have assigned functions to `fetchFn` to handle GraphQL query/mutation requests and `subscribeFn` to handle subscription requests.
- To create a Relay Store to store and manage cache data, we used the `RecordSource` store.

`RelayEnvironment`

- The `createRelayEnvironment` function is called to initialize the RelayEnvironment and export it for later import and use elsewhere.
- This configured `RelayEnvironment` is mainly used by `QueryRenderer`, `useLazyLoadQuery`, `commitMutation`, etc.

## CORS error

Initially, I read the `config.toml` file used on the server side to set the websocket URL of the GraphQL server and set the address. However, I kept getting CORS errors and Unauthorized every time I sent a request. So I did a lot of shoveling around, and with the help of my colleague, I was able to solve it. (Thank you so much 🥹🙏)

The solution is to use `http-proxy-middleware` to set up `setupProxy`!

As you can see in the [create-react-app manual](https://create-react-app.dev/docs/proxying-api-requests-in-development#configuring-the-proxy-manually), you can set up a `setupProxy` to proxy requests from your development server to a specific path on your real server, usually to prevent CORS issues in development environments where the frontend and backend are separated, or to proxy requests from your development server to a specific path on your real server.

The code looks like this

```js title="/src/setupProxy.js"
const { createProxyMiddleware } = require('http-proxy-middleware');

module.exports = function (app) {
 app.use(
 createProxyMiddleware('/graphql', {
 target: 'http://127.0.0.1:9220',
 changeOrigin: true,
 followRedirects: true,
 ws: true,
 }),
 );
};
```

`createProxyMiddleware('/graphql', { ... })`

- Sets the middleware to handle all HTTP requests originating from '/graphql'.

`target: 'http://127.0.0.1:9220'`

- Set the address of the server to which proxied requests will be forwarded. Here we set it to port 9220.

`changeOrigin: true`

- Change the host header of the request to the host of the target. Use this to work around CORS issues.

`followRedirects: true`

- This setting causes the proxy to follow redirects when the server sends a redirect response to a request.

`ws: true`

- This setting enables the WebSocket proxy. The websocket connection between the client and server is also passed through this proxy, which we set to `true` for subscribe.

## Leaderboard page

After a lot of digging, we've finally finished the leaderboard page! 🎉 A big thank you to everyone who participated. 🙇🏻‍♀️

![](https://cdn.lablup.com/gtc24_leaderboard_page_a633011356.png)

## Conclusion

Using GraphQL subscriptions, we were able to implement features like real-time rankings. Although I struggled with how to set it up because of CORS, it was not difficult to use because it is not much different from writing a query.

I think the biggest advantages of subscriptions are real-time updates and efficiency.
Because it receives data from the server in real time, users always see the latest status, and because it only gets updates when the data it needs changes, it can minimize server requests for data that doesn't change often.

However, it is **complex** as it requires an implementation of websockets or similar real-time protocols, as well as logic to manage the connection state between the client and server. Although not covered in this article, subscription requires additional work on the server side. And because it requires a real-time connection, it can consume server resources and client resources.

Therefore, which method is more cost or performance efficient depends on many factors, including the nature of your application, the frequency of data updates, and the number of concurrent users, so use your best judgment.

## references

- https://relay.dev/docs/v10.1.3/subscriptions/
- https://relay.dev/docs/guided-tour/updating-data/graphql-subscriptions/#configuring-the-network-layer
- https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
- https://github.com/enisdenjo/graphql-ws
- https://github.com/apollographql/subscriptions-transport-ws
- https://graphql.org/blog/subscriptions-in-graphql-and-relay
- https://create-react-app.dev/docs/proxying-api-requests-in-development

> This post is automatically translated from Korean

2024 GTC Event Live Rankings: How to Utilize GraphQL Subscription

[Featured articles] Scale entanglement

 ![](https://cdn.lablup.com/Lablup_GTC_24_a1eb755914.png) 

Greetings from Lablup! We are thrilled to announce our participation in the upcoming NVIDIA GTC 2024 conference, taking place from March 18th to 21st in San Jose, California. As a Silver Sponsor, Lablup is gearing up to showcase our cutting-edge AI technologies and products at this premier event, which is making a comeback as an in-person gathering after a five-year hiatus.

# About GTC 2024

GTC is the world's largest AI conference, hosted by NVIDIA. With over 300,000 attendees expected to join both online and in-person, this year's event promises an unparalleled opportunity to explore the latest AI tech trends. From the highly anticipated keynote by NVIDIA CEO Jensen Huang to more than 900 sessions, 300+ exhibits, and 20+ technical workshops covering generative AI and beyond, GTC 2024 is set to be a game-changer for anyone interested in the future of AI.

# Lablup at GTC 2024

At GTC, Lablup will be running an exhibition booth (#1233) where we will demonstrate Backend.AI Enterprise, the only NVIDIA DGX-Ready software in the APAC region. Backend.AI is an AI infrastructure management platform that maximizes the performance of NVIDIA DGX systems and other GPU infrastructures while enhancing usability.

We will also be introducing FastTrack, our MLOps solution that streamlines and automates the entire development process for generative AI models. Prepare to be amazed by our demo showcasing how FastTrack can automatically fine-tune foundation models for various industries and transform them into chatbots and other practical applications.

# Sessions at GTC

[Lablup will be presenting two sessions at GTC.](https://www.nvidia.com/gtc/sponsors/?search=lablup#/exhibitor/1694022458813001tZJB)

The first session, titled **"Idea to Crowd: Manipulating Local LLMs at Scale,"** will delve into the techniques and use cases for fine-tuning and operating local LLMs across various scales, from personal GPUs to large-scale data centers. We will share how we optimize resource usage through quantization and lightweight techniques, and illustrate the expansion process of personalized LLMs through concrete examples.

Our second session, **"Personalized Generative AI,"** will explore how to effortlessly run and personalize generative AI models on small-scale hardware such as personal GPUs, PCs, or home servers. We will introduce automated methods for operating and fine-tuning generative AI in compact form factors, offering a glimpse into a future where personalized AI assistants become an integral part of our daily lives.

# Hope to meet you soon!

We've given you a sneak peek into the exciting technologies and vision Lablup will be presenting at GTC 2024. If you're attending the event in San Jose this March, be sure to visit our **booth (#1233)** to experience the latest AI tech firsthand and engage with the Lablup team.

For those joining online, our session presentations will provide valuable insights into the present and future of local LLMs and personalized generative AI. Lablup remains committed to pushing the boundaries of AI technology, making it more accessible and user-friendly for businesses and individuals alike.
 
Don't miss this incredible opportunity to witness the power of AI and its potential to revolutionize our world. Join us at GTC 2024 and let's embark on this exciting journey together. See you there!

Meet Lablup at NVIDIA GTC 2024: Pushing the Frontiers of AI Technology

Byeongjo Kim

# Overview

I applied to the Open Source Contribution Academy (hereinafter referred to as the Contribution Academy) hosted by OpenUp, and worked as a fall intern at Lablup Inc. (hereinafter referred to as Lablup) from November to December for 8 weeks. Afterwards, I extended it for an additional 8 weeks from January to February, working a total of 16 weeks.

After being discharged from the military, I have written about the experiences I had while working at Lablup, my first company as a developer.

# Motivation for Applying

Even before the Contribution Academy, I was interested in Lablup, and coincidentally, I had an opportunity to contribute through the Contribution Academy.

During the Contribution Academy period, I worked on resolving issues and refactoring the webui of Backend.AI.

While participating in the Contribution Academy, I felt a lot of affection, interest, and enjoyment towards Backend.AI, and I began to think that I wanted to continue contributing after the program ended.

Lablup happened to provide an opportunity to work there in conjunction with the Contribution Academy, so I applied without hesitation.

# Onboarding

For the first 3 weeks of the internship, I underwent an onboarding process.

I went through implementing a RealTime Chat, setting up the Backend.AI environment, and then the Pebble Seminar in that order.

 ## RealTime Chat

This was the first assignment to become familiar with the core side of Backend.AI's code. I implemented a real-time chat app using Python, utilizing the aiohttp, aioredis, and asyncio libraries.

Since there was no condition to store the chat contents, I used the in-memory database redis.

I made it so that when a user enters a chat room, they subscribe to that chat room, and when a user enters a message, it publishes the entered message to the subscribed users, meaning the other users in the same chat room.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_01_f466f06469.png"/>

 RealTime Chat in action


While I was able to handle Python at a basic level from preparing for coding tests, I had no experience using libraries like aiohttp, asyncio, and aioredis, so it took me some time to understand and grasp the concepts.

However, this assignment helped me a lot in understanding the core side of Backend.AI's code, and it was good to be able to study new libraries.

## Setting up the Backend.AI environment

Since I had already installed Backend.AI during the Contribution Academy, setting up the environment during the internship period wasn't too difficult.

However, I was well aware that installing Backend.AI is not easy, as I had encountered many errors and failures while trying to install it during the Contribution Academy, and the other person doing the internship with me also had a lot of difficulties during the installation process.

Since I had already experienced those failures and knew the solutions, I was able to help, and we were able to install it quickly and move on to other tasks.

While setting up the environment, I also configured a virtual machine and VPN, and set up the environment on a virtual machine as well, so that I could work even if there were problems on my local machine. After setting up the configuration on the virtual machine, I mainly used the local for development during the subsequent work, and the virtual machine as a test server. The company's VM Farm, which allows for easy management and configuration of virtual machines, made it great for setting up development and testing environments.

## Pebble Seminar

After completing the RealTime Chat and setting up the Backend.AI environment, I prepared a short seminar based on understanding the structure and code of Backend.AI. I was tasked with presenting on GraphQL and Relay, which are used in the Backend.AI WebUI.

While I had experience with GraphQL, I felt that my knowledge was lacking for presenting in front of others, and Relay was a new library to me, so I was quite worried about preparing for the Pebble Seminar and read through many documents to prepare. First, I familiarized myself with the concepts by reading the official documentation for GraphQL and Relay, and then analyzed the Backend.AI code one by one to understand how they were applied and functioning in Backend.AI.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_02_25c7f67c06.png"/>

 Pebble Seminar preparation materials


By analyzing the code while preparing for the Pebble Seminar, I naturally came to understand the code running in the WebUI, and this greatly helped me in finding and resolving issues during the subsequent work.

# Resolving Backend.AI issues and implementing features

After completing the onboarding, I finally joined the frontend team and started resolving Backend.AI issues and implementing features. I had a coffee chat with the frontend lead to define the categories of work for this internship period:

1. Creating a Table Column Setting component
2. Researching E2E Testing
3. Daily tasks

During the 8-week internship period from November to December, I created a total of 19 Pull Requests, of which 18 were merged, and 1 is still under review. Since I had experience finding and assigning issues during the Contribution Academy, I had less difficulty with it, and because I enjoyed resolving issues, I was able to resolve more issues than others.

## Feature Addition PRs
### 1. Implementing Table Columns Setting

https://github.com/lablup/backend.ai-webui/pull/2071

This was one of the issues I aimed to work on during the internship period. It was the only component that I conceived and implemented from scratch during the fall internship, rather than refactoring an existing component. Before implementing this feature, I thought it was a simple task that I could finish quickly, but things turned out differently from my expectations.

First, I realized that I had been thinking too simplistically about creating new components. Even though I had designed and considered the props to be received before creating components in the past, through this issue, I felt that when creating a new component, I should invest more time and effort while considering scalability. I also realized that I should pay more attention to how other sites are designed and what features are applied.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_03_b4d7b20042.png"/>

 Table Columns Setting


### 2. Adding service endpoint and owner columns to the Model Serving page table

https://github.com/lablup/backend.ai-webui/pull/2047

Previously, when creating a model service, users had to go to the detail page to check the endpoint, which is a frequently used feature. So there was a request to add the endpoint to the table column. Additionally, since the admin account can see services of users in the same group, there was a suggestion to have a column showing the service owner. Since the GraphQL fields for retrieving this data were already implemented, I added the fields to the query to fetch the endpoint and service owner data, and then added columns to the table to display the data. The owner column is only shown for admin accounts.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_04_cd838ef9b2.png"/>

 Implementation view. Screen for admin account (left) and user account (right)


### 3. Disabling log button for sessions in CANCELLED state

https://github.com/lablup/backend.ai-webui/pull/2045

The CANCELLED state means that the container has never been created or failed to be created. Previously, the log button was enabled even for sessions in the CANCELLED state, and if a user clicked the log button, the agent could not find the container information, resulting in a 500 error. In this PR, I made it so that the log button is disabled for sessions in the CANCELLED state, preventing users from clicking it.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_05_973936500c.png"/>

 Session in TERMINATED state (session 1) and CANCELLED state (session 2)


### 4. Testing and creating a custom hook for dark mode

https://github.com/lablup/backend.ai-webui/pull/2120

Before implementing dark mode, I found components with hardcoded colors and implemented a custom hook named useThemeMode for applying dark mode. When creating the custom hook, I tried to use the useLocalStorageState hook from ahooks, but contrary to my expectations that it would automatically manage states with the same key value, I found that they operated independently. To handle states with the same key value automatically updating when the value changes, I added a custom hook named useLocalStorageGlobalState, and then used that to create the useThemeMode custom hook for setting the dark mode.

## Bug fix PR

### 1. Allowing signup without invitation token

https://github.com/lablup/backend.ai-webui/pull/2046

In the config.toml, when the allowSignupWithoutConfirmation option is set to true, users can sign up without an invitation token. However, when a user clicked the sign up button, an error occurred because the token value was undefined. In this PR, I modified it so that if allowSignupWithoutConfirmation is true, the token variable is not used. Additionally, previously users could modify other input values while the core was processing the data after clicking the sign up button, and the previous data remained when the dialog was closed and reopened. In this PR, I made it so that other input values cannot be entered while data is being processed, and the previous input values are cleared when the dialog is closed.

### 2. Displaying the correct screen for the selected sub-tab on the user management page

https://github.com/lablup/backend.ai-webui/pull/2055

On the user management page, there are sub-tabs for displaying active users and deactivated users. However, if a user navigated to another page and then returned to the user management page, even though the sub-tab was set to inactive, the screen displayed the list of active users, causing confusion. This PR resolved that issue by remembering the current sub-tab when navigating to another page, and displaying the appropriate screen for that sub-tab when returning to the user management page.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_06_c206b10b14.png"/>

 Before (left) and after (right) the fix


# Extending the internship

As I resolved issues, the 8-week period flew by, and it was time to wrap up the fall internship.

Working at Lablup as my first job after being discharged from the military was an important period for me. During the internship at Lablup, I was able to experience my strengths and weaknesses, what skills I needed to further prepare, and how other developers work. The 2-month period felt very short, and since I had enjoyed working so much during that time, I wanted to continue working. So I expressed my desire to extend the internship to the lead, and we agreed to extend it for another 8 weeks until February. During the fall internship, I had thought a lot about my weaknesses, but I couldn't find my strengths. So, I started with these 3 personal goals:

1. Find my strengths during this period
2. Read the documentation whenever I have time
3. Work even harder, leaving no regrets

# Resolving issues and implementing features during the extended period

The work during the extended period did not differ much from before. Without the onboarding process and installation, I could focus more on resolving issues.

## Feature Addition PRs
### 1. Refactoring ErrorLogList

https://github.com/lablup/backend.ai-webui/pull/2131

I refactored the ErrorLog List, which was previously implemented using Lit elements, to React. This feature was the most satisfying issue for me, as I personally use it frequently after the refactoring.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_07_11ad8297d0.png"/>

 Before (left) and after (right) refactoring


During the refactoring, new Search and Error filter features were added.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_08_fb17ed6a93.png"/>

 Added Search feature (left) and Filter feature (right)


### 2. Modal drag functionality

https://github.com/lablup/backend.ai-webui/pull/2179

I used the React-draggable library to add functionality for modals to be dragged. By adding the Draggable prop to a modal, it can be applied to any modal that requires dragging.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_09_e0ee20954a.png"/>

 Draggable Modal


By clicking the icon on the left side of the modal title and moving the mouse, the modal can be moved to the desired position on the screen.

Currently, it is applied to the modal for viewing user information on the user management page and the modal for changing user settings, where you can check it.

<iframe title="'일하는 개미의 개발..'에서 업로드한 동영상" width="640" height="360" src="https://play-tv.kakao.com/embed/player/cliplink/444997096?service=player_share" allowfullscreen frameborder="0" scrolling="no" allow="autoplay; fullscreen; encrypted-media"></iframe>

While it is not being used in many places yet, I think this PR will be useful as more components and features are added.

## Bug fix PR
### 1. Modifying Vfolder invitation permissions

https://github.com/lablup/backend.ai-webui/pull/2143

There was an issue where the user permissions for group vfolders were not being updated. When trying to modify the permissions, the items were not displayed or selectable properly in the select. Previously, the items were being displayed using option tags, but I changed it to use mwc-list-item to display the items and modified the overflow option to resolve this issue.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_10_bf182d8b57.png"/>

Before (left) and after (right) the PR


### 2. ResourceGroupSelect extending outside the card

https://github.com/lablup/backend.ai-webui/pull/2166

There was an issue where the ResourceGroupSelect value would be displayed outside the card if it was too large.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_11_e1038e9aec.png"/>

Symptoms of the issue


To resolve this issue, I set the max-width CSS on the Select component so that it cannot exceed the width of the card.

Additionally, in this PR, I added a Search feature to the Select component, for which I used the useControllableValue hook from ahooks. The useControllableValue hook helps manage props from either the parent or itself. While it was a simple PR, it took me longer than expected since it was my first time using useControllableValue. I was able to resolve this issue with the help of the lead and another intern.

### 3. Key pair list not showing when clicking the generate & manage key pair button on the summary page

https://github.com/lablup/backend.ai-webui/pull/2194

On the summary page, there are buttons for "Generate New Key Pair" and "Manage Key Pairs." However, when clicking these buttons, instead of showing the key pair list, it simply navigated to the user management page, displaying the user list.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_12_ace27a911e.png"/>

"Generate New Key Pair" and "Manage Key Pairs" buttons on the summary page


<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_13_5e99f8c028.png"/>

When clicking the "Generate New Key Pair" button (left) and when clicking "Manage Key Pairs" (right)


While this issue was not critical, I resolved it because I had experienced a lot of confusion when I first used Backend.AI and didn't fully understand the key pair feature.

After resolving this issue, I could confirm that the key pair list was displayed on the screen as intended.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_14_429c2582f9.png"/>

After resolving the issue, when clicking the "Generate New Key Pair" button (left) and when clicking "Manage Key Pairs" (right)


# Completing the Internship

Thanks to the Contribution Academy, which started from a friend's recommendation after being discharged from the military, I was able to contribute at Lablup for an extended period. Since I had no previous internship or project experience at other companies, this was a very important period for me as I was starting anew after being discharged. It was great that I could experience my strengths and weaknesses, the skills I lack, and the culture of an open-source company at Lablup. How many companies make you want to go to work every day with their horizontal structure, free atmosphere, pleasant working environment, and good equipment? Although I worked at Lablup for 4 months, I felt like I wanted to go to work every day, and if it was Lablup, I could work at a company while doing interesting and desirable work for a long time. Over the 4-month period, I also developed a fondness for Backend.AI, the service provided by Lablup, and I plan to attend the conference hosted by Lablup every year whenever possible to see its advancements and technologies.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_15_45032e1593.jpeg"/>

Lablup Office


> This post was also published on the author's personal blog.
> https://gee05053.tistory.com/32
> This post is automatically translated from **Korean**

2023 Winter Intern in Lablup

Sergey Leksikov

> - [Part 1. Introduction to LLMs and Tool Interaction](/blog/2024-01-28-tool-llms-part-1)
> - [Part 2. Backend.AI Gorilla LLM model serving](/blog/2024-01-29-tool-llms-part-2)
> - Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM

# Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM

Previously, in Part 1 we talked about Tool LLM and their usage. Part 2 demonstrated how to run Gorilla LLM on Backend.AI. In the Part 3, there will be talk about the case when there are no GPU available, but we still want to get help and assistance regarding our API.

Suppose we have Backend.AI, and we want to get information about Backend.AI REST API and Functional API in more interactive way via question answering style. The example of REST API can be described in this documentation: https://docs.backend.ai/en/latest/manager/rest-reference/index.html
<img style="margin: auto;" src="https://cdn.lablup.com/Figure1_part3_672d0913df.png"/>

 Figure 1. Backend.AI REST API Documentation


In addition, Backend.AI REST API documentation can be exported into **openapi.json** format:
<img style="margin: auto;" src="https://cdn.lablup.com/Figure2_part3_4301cfdc46.png"/>

 Figure 2. Backend.AI openai.json


Another source of BackendAI API is functional API defined in Backend.AI Client. We want to know how to interact with Backend.AI and which parts of code are responsible. The client code repository is responsible with managing and interacting with cloud and computing environment: 

Steps to make a Question Answering API system

1. Let’s setup Backend.AI Client locally from https://github.com/lablup/backend.ai/tree/main/src/ai/backend/client on our local PC environment and create a new directory `bai-dev/src/ai/backend/client/gpt_api_client`
<img style="margin: auto;" src="https://cdn.lablup.com/Figure3_part3_786b65a096.png"/>

 Figure 3. The directory location of gpt_api_client


2. At vector_data directory let’s create two sub directories data1/ which will store a REST api documentation: **openapi.json** and `data2/` will store selected B.AI Client files over which we want to do an API Question Answering.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure4_part3_a112e9b5f9.png"/>

 Figure 4. Overview of data directories with openapi.json and client function code files


3. Let’s install python library LlamaIndex library. Pip install llama-index.
Note LlamaIndex is not related to Meta LLaMA language model. LlamaIndex is about data structures and methods for efficient processing and storing documents for retrieval. 

4. Let’s convert our api and code files into an embedded vector and store them in a Vector Database with LLamaIndex. Let’s use Jupyter Notebook interactive environment which is also integrated in out VSCode on a local PC.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure5_part3_b2e80a9191.png"/>

 Figure 5. Jupyter Notebook interactive environment. Loading openapi.json from data/ directory. Then asking questions from query engine over a vector index.


5. Vectorize `data2/` directory with our code functions
<img style="margin: auto;" src="https://cdn.lablup.com/Figure6_part3_988ad5b585.png"/>

 Figure 6. Load data2/ directory with code files from B.AI Client. Then vectorize them into index and create a question answering engine.


We can save both indexes using python Pickle or Joblib libraries which are commonly used for storing and serializing objects to later load them into system. **joblib.dump(index, "rest_api_index.joblib")** and **joblib.dump(index, "functional_index.joblib")**

6. Jupyter Notebook environment already provides to us ability to ask questions and get response in interactive way. Additionally, we can load the saved vectorized indexes on FastAPI server and answer questions over the web. In previous Part 2, we set computational session with Gorilla LLM. From the previous demo we still have a computational session with a FastAPI server.

7. Let’s transfer the files **rest_api_index.joblib** and **functional_index.joblib** to api_helper/ vFolder at Backend.AI Cloud session

8. At file **server.py** load the vector indexes and define the query engines.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure7_part3_cb02dc1748.png"/>

 Figure 7. server.py definition of index files and query engine.


9. For each query engine we specify an FastAPI endpoint.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure8_part3_dad39fb820.png"/>

 Figure 8. Code snippets for REST and Functional API retrieval


10. Test server response from your local PC using curl command. When a server gets queried on a specific endpoint, it will get an answer from a user.
```sh
curl -X POST -H "Content-Type: application/json" -d '{"instruction":"Create a new session"}' http://127.0.0.1:8000/rest_api
```
<img style="margin: auto;" src="https://cdn.lablup.com/Figure9_part3_a0a03c3a13.png"/>

 Figure 9. Command line response from curl command. Example 1


```sh
curl -X POST -H "Content-Type: application/json" -d '{"instruction":"Create a new session"}' http://127.0.0.1:8000/functional
```
<img style="margin: auto;" src="https://cdn.lablup.com/Figure10_part3_6a35792213.png"/>

 Figure 10. Command line response from curl command. Example 2


In addition, we can make a web app which receives user input, sends to corresponding endpoint, and receives the answer.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure11_part3_0b36446b3d.png"/>

 Figure 11. A web app prototype for Question Answering over Backend.AI REST and Functional API. Example 1


<img style="margin: auto;" src="https://cdn.lablup.com/Figure12_part3_48aef2d6ec.png"/>

 Figure 12. A web app prototype for Question Answering over Backend.AI REST and Functional API. Example 2


## Conclusion
In Part 3, we demonstrated how to locally create a Question-Answering system using open-source python library LLamaIndex which helped to convert our documents and Backend.AI code into vector form. The question answering can be done in interactive way in a Jupyter Notebook environment which Visual Studio Code supports with plugins. Furthermore, we decided to move those vector indexes to a Backend.AI Cloud environment where a Gorilla LLM API tuned model is server. Then an API Question-Answering web app was implemented to assist users over network.

## Reference:
- LLama Index. https://docs.llamaindex.ai/en/stable/

Demo video for Backend.AI API Helper and Gorilla LLM: 
<iframe width="640" height="360" src="https://www.youtube.com/embed/ggaw6ijxGL0?si=ZgRFGtyGg1oLh8Au" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

> - [Part 1. Introduction to LLMs and Tool Interaction](/blog/2024-01-28-tool-llms-part-1)
> - [Part 2. Backend.AI Gorilla LLM model serving](/blog/2024-01-29-tool-llms-part-2)
> - Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM

Backend.AI Meets Tool LLMs : Revolutionizing AI Interaction with Tools - Part 3

> - [Part 1. Introduction to LLMs and Tool Interaction](/blog/2024-01-28-tool-llms-part-1)
> - Part 2. Backend.AI Gorilla LLM model serving
> - [Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM](/blog/2024-01-30-tool-llms-part-3)

# Part 2. Backend.AI Gorilla LLM model serving

Previously, we talked about the Tool LLM capabilities and usage. In this article, there will be a step-by-step demonstration of how to run the Gorilla LLM model on the Backend.AI Cloud while using Backend.AI Desktop app.

<img style="margin: auto;" src="https://cdn.lablup.com/Figure1_part2_dabfc01542.png"/>

 Figure 1. A Backend.AI Desktop app installed on MacOs


1. Press a start button to make a session creation menu appear.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure2_part2_96d42c949d.png"/>

 Figure 2. New session start interactive screen


2. Select *NGC-Pytorch 23.07* image

3. Attach a vFolder which is a working directory containing the model files. For example: *api_helper/* directory name.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure3_part2_0b854d81e9.png"/>

 Figure 3. Attaching vFolder screen


4. Select the resource amount 128 GB RAM and 5 fGPU
<img style="margin: auto;" src="https://cdn.lablup.com/Figure4_part2_fe7aa72175.png"/>

 Figure 4. Resource selection screen


5. Select a Visual Studio Code Desktop environment 

<img style="margin: auto;" src="https://cdn.lablup.com/Figure5_part2_ad4ca84601.png"/>
 Figure 5. IDE environment selection screen


6. At `/home/work/api_helper/` directory create a **server.py** file

7. Create a **requirements.txt** file
<img style="margin: auto;" src="https://cdn.lablup.com/Figure6_part2_0ba3c54ee6.png"/>

 Figure 6. Content of requirements.txt file


To install requirements run the command: `pip install -r requirements.txt`
<img style="margin: auto;" src="https://cdn.lablup.com/Figure7_part2_024d2ce91a.png"/>

 Figure 7. Executing install requirements command


8. Create a **server.py** and define using transformers library the tokenizer and model loader.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure8_part2_00ebe46cd8.png"/>

 Figure 8. Code snippet of server.py


9. Define server IP address and port number
<img style="margin: auto;" src="https://cdn.lablup.com/Figure9_part2_c15140debf.png"/>

 Figure 9. Definition of server IP address and port number


10. To run the model type: **python server.py**
<img style="margin: auto;" src="https://cdn.lablup.com/Figure10_part2_4651d19db6.png"/>

 Figure 10. Starting a server.py


11. Accessing the created server

 VSCode automatically creates a port tunneling session from your device to a Backend.AI Cloud server. You may see the server status by accessing the localhost address and the request will be tunneled to a Backend.AI Cloud. In addition, you may define other custom endpoints according your needs.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure11_part2_2c34fa72a1.png"/>

 Figure 11. The server run log

<img style="margin: auto;" src="https://cdn.lablup.com/Figure12_part2_fa41d8cd06.png"/>

 Figure 12. VSCode Port Forwarding configuration

<img style="margin: auto;" src="https://cdn.lablup.com/Figure13_part2_109ff20416.png"/>

 Figure 13. Accessing the root of a server


 Up to this point, we create a computation session on Backend.AI Cloud, attached an api_helper/ vFolder directory with requirements.txt file and server.py. Then we started our FastAPI server where the Gorilla LLM is gets downloaded from HuggingFace repository and loaded into computation session memory with inference/ api .endpoint

12. API Inference testing
 To test the API inference of Gorilla LLM you may create a curl request from your local computer command line:
```sh
curl -X POST -H "Content-Type: application/json" -d '{"text":"Object detection on a photo. <<<api_domain>>>:"}' http://127.0.0.1:8000/inference
```
<img style="margin: auto;" src="https://cdn.lablup.com/Figrue14_part2_9f02402b1c.png"/>

 Figure 14. An example of curl request

<img style="margin: auto;" src="https://cdn.lablup.com/Figure15_part2_a2d8e4f2d7.png"/>

 Figure 15. The GPU workload on a server after receiving the request

<img style="margin: auto;" src="https://cdn.lablup.com/Figrue16_part2_2c99f02e17.png"/>

 Figure 16. The server logs of receiving the request and printing the result


13. Defining UI web app. You may use any web technology to make a UI app which can display the result in a better way. For example, you may use html and JavaScript files and place them in static directory under root of server.py Then define an endpoint for a web app.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure17_part2_fa32fd5cc4.png"/>

 Figure 17. Example of adding an html web app to a FastAPI server


14. Gorilla LLM Web App prototype - an API tuned Large Language Model for API question answering and code generation.
<img style="margin: auto;" src="https://cdn.lablup.com/Figure18_part2_b4733d5db5.png"/>

 Figure 18. Gorilla LLM web app prototype. Example 1

<img style="margin: auto;" src="https://cdn.lablup.com/Figure19_part2_a726a83d94.png"/>

 Figure 19. Gorilla LLM web app prototype. Example 2


## Conclusion
Despite some difficulties of Gorilla LLM serving, LLM tuned on own API has a large potential and promises. Since, the model can provided the most recent results with more accurate parameters and function calls than commercial large models and be useful in tasks such as question answering over API, code autocomplete, API code executions.

## Limitations and difficulties:
While trying to server the Gorilla LLM model there were following issues to consider:
- Model may generate response in not expected format
- Model may generate result different for same questions
- Parsing and rendering LLM response
- Eliminating the duplicate sentences and lines

> - [Part 1. Introduction to LLMs and Tool Interaction](/blog/2024-01-28-tool-llms-part-1)
> - Part 2. Backend.AI Gorilla LLM model serving
> - [Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM](/blog/2024-01-30-tool-llms-part-3)

Backend.AI Meets Tool LLMs : Revolutionizing AI Interaction with Tools - Part 2

> - Part 1. Introduction to LLMs and Tool Interaction
> - [Part 2. Backend.AI Gorilla LLM model serving](/blog/2024-01-29-tool-llms-part-2)
> - [Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM](/blog/2024-01-30-tool-llms-part-3)

# Part 1. Introduction to LLMs and Tool Interaction

 What if future AI technology capabilities were available now? Probably while you are on the way home from your workplace, you could ask an AI Assistant to turn on the air-conditioner in the home before your arrival. At same time you are planning the vacation and after having few options you ask an AI model to do hotel booking on your behalf. As the model books your trip, you receive a notification from a cloud provider about your deep learning model's training progress. You ask the AI Assistant to run another session with another set of parameters for the experiment while targeting specific values for performance accuracy. How be such a futuristic scenario realized in the present days?

 This kind of interaction of LLM with real world could be possible via Application Programmatic Interfaces (API). The specific Tool Large-Language Model (LLM) fine-tuned on APIs dataset can respond user’s query with specific API and that API can invoke a program or functions to make a real-world impact. Large Language Models (LLM) are rising in popularity due to their outstanding capabilities of generating text in context while also having reasoning capability for problem solving. Text model utilization ranges from text generating, editing they as well become useful as a copilot for a programmer. How else can LLMs extend their usage beyond their text-generating capabilities?

 With Tool LLM, we are stepping into an era where AI in addition to understanding our requests, the AI can act on those requests using a universe of online tools. Tool LLM are pushing the boundaries of what AI can do with tools via functional and REST APIs.

 GPT-4 is currently the state-of-the-art among LLMs, topping most AI benchmarks. Consider this scenario, a GPT-4 model is being asked to transcribe the audio file into text of another language. However, when prompted to use specific APIs, GPT-4 may hallucinate and suggest non-existent APIs or provide incorrect arguments. As consequence causing function execution failure and not achieving objectives of user specified task.

 Besides issues with hallucinations and inaccuracies, API documentation and versions are constantly changing. The retraining general purpose LLM is costly and not practical to keep the LLM models updated with constantly changing documentations. Tool LLMs provides a solution to the hallucination issues of general large models, enabling interaction with the physical world via programmatic interfaces. Tool LLM are much smaller, making it feasible to periodically be retrained with recent data. In addition, API documentation Retriever module can be added into model serving pipeline to help supplement the model with the most recent API documentation which is relevant to user’s input query.

 To overcome these challenges, researchers have recently proposed two notable open-source methods for enhancing LLMs tool use abilities such as Gorilla LLM and ToolLLaMA, each having its own advantages and specific use cases. Moreover, those models can be prepared for inference serving on Backend.AI Cloud.

## What is Tool LLM?

 Tool LLM is an LLM which was trained on a dataset with user query and API request with relevant context information such as API code usage and API description documentation. The response from such LLM can be executed as a code. The code execution implies that the LLM can interact with various online services and tools. Such as Cloud Computing Providers, Kubernetes machine learning and Deep Learning libraries and repositories such as HuggingFace, TorchHub, TensorFlowHub.

 The main advantage of such Tool LLM is ability to accurately generate an API response to user query which can be executed to obtain the results.

## Understanding the Types of API

 An Application Programming Interface (API) is a crucial element in modern computing, serving as a set of rules and protocols for how different software applications or hardware systems can communicate and interact. 

 Functional APIs are designed to be invoked through function calls within a programming environment. For instance, machine learning and deep learning libraries like HuggingFace and TensorFlow offer various models that can be loaded into memory and utilized through Functional API calls. These APIs are integral in executing specific functions and operations within the software.

 This capability of LLM to generate a code related to an API extends their utility far beyond basic text generation and processing. Tool LLMs can seamlessly integrate with diverse online services and tools, ranging from cloud computing platforms to advanced machine learning libraries. Furthermore, their application is not limited to human queries; they can also be integrated into systems where they interact with other programs or AI agents. This versatility positions Tool LLMs as vital components in complex systems and infrastructures, enhancing their potential for real-world applications.

 In the following sections, we'll delve into how Tool LLM were trained and how they are operated. After that two specific research examples will be covered such as Gorilla LLM and ToolLLaMA.

## Tool LLM Training and Inference Workflow

 Tool LLM training involves several steps which includes setting api database, creating a training dataset, model training and inference.

 The API Database includes descriptions and relevant code samples. To generate a Self-Instruct training dataset there is a need to pre-process API database samples into {Input User Query-API Output} pairs. ChatGPT can help with automatically generating such dataset by covering various scenarios and query complexities which humans might ask. From specific cases to general and abstract cases. After Self-Instruct dataset is generated the model is trained to make accurate prediction in terms of API given user input query.

 For Tool LLM inference, it's crucial that the LLM not only responds with accurate argument parameters but also uses the latest API documentation. Thus, API Document Retriever is used which helps to keep the model with the most recent API changes.
<img style="margin: auto;" src="https://cdn.lablup.com/overview_workflow_of_Tool_LLM_806e1506f1.png"/>

 Figure 1. An overview workflow of Tool LLM training and inference over API instuction dataset


## Case Studies: Gorilla LLM and ToolLLaMA
### Gorilla
 Gorilla, a fine-tuned LLaMA 7 billion-based model that outperforms GPT-4 in writing API calls. The notable aspects of Gorilla are:
- It addresses the limitations of current LLMs in generating accurate input arguments for APIs and their tendency to hallucinate incorrect API usage.
- Gorilla integrates with a document API retriever, allowing it to adapt to real-time changes in documentation, a significant advantage considering how frequently APIs get updated.
- The authors have developed a dataset called APIBench to evaluate the model's abilities, which includes APIs from HuggingFace, TorchHub, and TensorHub totaling 1600+ APIs.
- Gorilla seems to mitigate hallucination issues and improves the reliability of LLM outputs. Also, Gorilla got updated and extended to work with Cloud providers such as AWS, GCP and managing Kubernetes clusters.

### ToolLLaMA
 ToolLLaMA is a model which was fine-tuned on ToolBench an instruction-tuning dataset for tool based on RapidAPI repository. There are following keypoints of ToolLLaMA:
- ToolBench covers an impressive range of over 16,000 real-world APIs, offering diverse instruction sets and solution paths.
- The paper proposes a novel Depth-First Search-Based Decision Tree algorithm (DFSDT) to enhance the reasoning capabilities of LLMs such as multiple tool usage and multi-step reasoning.
- Finetuned ToolLLAMA on ToolBench matches the performance of ChatGPT and demonstrates the generalization abilities in out-of distribution datasets like APIBench.

Both papers are significant in pushing the boundaries of LLM’s capabilities in real-world tool use by navigating and utilizing a vast array of APIs. This advancement is crucial for practical applications. Below is a comparative summary table provided.
<img style="margin: auto;" src="https://cdn.lablup.com/comparative_API_tuned_LLM_36a8cc1125.png"/>

 Figure 2. A comparative table between two API tuned LLM


## Synergy between Backend.AI and ToolLLM

 The training or model serving of LLM requires a significant computer resource, especially since there is a huge demand for Graphic Processing Units (GPU) with high capacity for RAM and computational speed.

 Backend.AI offers a scalable foundation for building, training, and serving diverse models. Backend.AI includes scaling on demand feature for model inference with adding external node for serving and Load Balance to optimize the workload. Backend.AI has vLLM and TensorRT server which can be used for high performance inference of LLMs. In addition, there is a well-designed user-friendly interface and pipeline maker FastTrack tool to create computing environment sessions of various complexities.

## Conclusion

 The futuristic scenario which can be realized at present day where various AI Assistants and Agents interact with various devices and services are possible through API and Tool LLM specifically fine-tuned on such interactions. Gorilla LLM and ToolLLaMA offer a good opportunity to incorporate them in complex tasks. The workflow of how they trained and served is easy to comprehend. Gorilla LLM could be recommended to use for Machine Learning and cloud administration tasks. While ToolLLaMA for more general API usage, multi-tool, and multi-step cases.

 There is also an advantage of training your own model on your own API documentation or code to have a LLM model which understands your code. Such LLM can be helpful at assisting or interacting with users who want to get the relevant information. 

 The Backend.AI can effectively to be a backbone for model training and providing scalable model serving while offering a simplistic GUI. How to set up such models and step by step guide will be explained in other parts.

## Commonly asked questions:
> - **Q:** What is source of hallucinations and LLM limitations and how it solved in Tool LLM?
> - **A:** GPT-4, like other Large Language Models, faces limitations such as hallucinations and inaccuracies, which are primarily due to its training on extensive yet potentially outdated or inaccurate datasets from the internet. These 'hallucinations' refer to instances where the model confidently produces information that's either factually incorrect or not based in reality, a challenge stemming from the nature of its purely text-based training data and not directly from its size or lack of interaction with the physical world. To address these issues, Tool LLMs are being developed with a focus on specialization and frequent updates. They are fine-tuned on specific datasets, like API documentation, enabling direct interaction with real-world systems through programmatic interfaces for more accurate and current information. The retraining frequency of Tool LLMs varies, depending on the application and the pace of change in the relevant field, with updates potentially needed monthly, quarterly, or bi-annually to keep the model up-to-date with the latest trends and information.

> - **Q:** What are example pairs of user Query and API?
> - **A:** The example pairs are provided below.
> > - **User Query:** "Summarize this article about space exploration."
> > - **API Output:** HuggingFace.summarize(text="Article text here", model="facebook/bart-large-cnn")
>
> > - **User Query:** "What is the sentiment of this customer review?"
> > - **API Output:** HuggingFace.analyze_sentiment(text="Customer review text", model="distilbert-base-uncased-finetuned-sst-2-english")
>
> > - **User Query:** "Identify the objects in this photo."
> > - **API Output:** HuggingFace.image_recognition(image_file="path/to/photo.jpg", model="google/vit-base-patch16-224")
>
> > - **User Query:** "Convert this speech recording to text."
> > - **API Output:** HuggingFace.speech_to_text(audio_file="path/to/recording.wav", model="facebook/wav2vec2-base-960h")

> - **Q:** How do the GorillaLLM and ToolLLaMA papers differ in their approach to utilizing API documentation during the training and inference of their models?
> - **A:** GorillaLLM appends relevant API documentation during training and offers two inference modes, while ToolLLaMA employs Sentence-BERT for fine-tuning embeddings in the API domain. GorillaLLM uses BM25 and GPT-Retriever from LLamaIndex for documentation retrieval, whereas ToolLLaMA uses Sentence-BERT for a similar purpose.

> - **Q:** How frequently should small API models be retrained, and what role does the API Retriever play in handling changes in API documentation?
> - **A:** Training small API models annually is reasonable, but monthly retraining for API changes isn't practical. The API Retriever, using up-to-date documentation, can mitigate the need for frequent retraining. Evaluating and benchmarking fine-tuned API models and RAG methods is essential for effectiveness.

> - **Q:** What is the difference between ToolLLM and RAG systems, and how do they function in the context of LLMs?
> - **A:** ToolLLM is a model fine-tuned on API documentation, focusing on incorporating knowledge. RAG systems, on the other hand, are algorithms for data chunking, storage, search, re-ranking, and synthesis. They can work independently or in combination to enhance LLM efficiency, especially in handling context limits and knowledge updates.

## Reference:
- Gorilla: Large Language Model Connected with Massive APIs. https://gorilla.cs.berkeley.edu/
- ToolLLM: Facilitating Large Language Models To Master 16000+ Real-World APIs. https://github.com/OpenBMB/ToolBench

> - Part 1. Introduction to LLMs and Tool Interaction
> - [Part 2. Backend.AI Gorilla LLM model serving](/blog/2024-01-29-tool-llms-part-2)
> - [Part 3. Making own API Retriever and Question Answering system with few lines of code locally without training and serving LLM](/blog/2024-01-30-tool-llms-part-3)

Backend.AI Meets Tool LLMs : Revolutionizing AI Interaction with Tools - Part 1

> * [Deconstructing a working Raft implementation - 2](/blog/2024-05-29-deepdive-raft-2)
> * [Deconstructing a working Raft implementation - 1](/blog/2024-03-29-deepdive-raft)
> * Introduce raftify: High-level Raft framework created with focusing on scalability

Hello, I've been working on introducing Raft to the Backend.AI manager processes at Lablup since last year.

Here's a rough breakdown of the related tasks I'm working on.

> 1. Introducing Raft to the Backend.AI manager process and making it a leader-follower structure.
> 2. Replacing the existing distributed lock-based GlobalTimer with a Raft-based global timer, and ensuring that a specific task is performed exactly once in the cluster.
> 3. Embedding a global, shareable state store in the manager process and synchronizing it appropriately.

In this post, I'll introduce the Raft framework I've been shoveling over the past year to accomplish this task, some of the issues I've encountered while developing it, and walk through a raftify example code that implements a distributed key-value store in less than 300 lines of code in total.

# Introducing raftify

[raftify](https://github.com/lablup/raftify) is a Raft framework developed with a focus on extensibility so that it can be easily integrated with any server application.

Among the Raft implementations utilized in production, [tikv's raft-rs implementation](https://github.com/tikv/raft-rs) was developed on top of raftify, using LMDB as stable storage and gRPC as the network layer.

# Writing binding of the Raft module

I decided that building and maintaining a reliable Raft implementation from the ground up would be a significant burden, so I decided to write a Python binding for the Raft module first.

So I initially thought I'd try writing a Python binding using [gopy](https://github.com/go-python/gopy), which is the most starred Raft implementation on GitHub, [hashicorp/raft](https://github.com/hashicorp/raft).

However, gopy didn't support the binding of `goroutine`, and it didn't support the latest Python version.

Then, on the advice of a senior developer in the company, I learned about a Rust implementation called [tikv/raft-rs](https://github.com/tikv/raft-rs) and [PyO3](https://github.com/PyO3/pyo3), which inspired me to try writing a Python binding for tikv/raft-rs using PyO3.

# rraft-py

Thus, I decided to develop a Python binding for Raft modules named [rraft-py](https://github.com/lablup/rraft-py) which is combined with *Rust*, *Raft*, and *Py*

My first concern in developing rraft-py was to make the semantics of the rust code and the python code as close to a 1:1 match as possible.

To achieve a 1:1 match, I needed to bypass some of the details of Rust's syntax.

My main concern at the time was how to expose Rust references to the Python side, which you can see in my [PyCon KR presentation](https://www.backend.ai/video/pycon-kr-23-pyO3-distributed-binding) if you're interested.

The result is rraft-py, a port of over 10,000 lines of integration test code from raft-rs, which has become a fairly reliable implementation of Raft bindings that can be used directly in Python.

Currently, raftify is in the process of being completely rewritten in Rust, and rraft-py is no longer used, but it was a great experience to write my first PyO3 bindings and try out the APIs of a Raft implementation.

# riteraft-py

After developing riteraft-py and porting over 10,000 lines of integration tests from raft-rs and even the [multiple-mem-node example](https://github.com/lablup/rraft-py/blob/main/example/multi_mem_node/main.py) to python code to get it working, my only thought was that I still didn't know where to start.

The raft-rs really only provided the Raft implementation itself and I had no idea how to integrate it into my application.

While browsing GitHub, I came across a high-level Rust implementation based on tikv/raft-rs called [riteraft](https://github.com/ritelabs/riteraft) in an issue called [How to use this lib?](https://github.com/tikv/raft-rs/issues/402), and it was much more intuitive to figure out how to use. So I decided to develop [riteraft-py](https://github.com/lablup/riteraft-py) with the goal of mimicking its behavior in Python and integrating it at the application level.

The job of riteraft is to integrate this Raft implementation directly with Raft modules and logs, state machines, and network layers, but the problem was that it didn't work very well, apart from being intuitive to use.

[Leader election not happening when the leader is dead](https://github.com/lablup/riteraft-py/pull/10), [data replication not happening in certain scenarios](https://github.com/lablup/riteraft-py/issues/1), [panic when the commit count goes over 255](https://github.com/lablup/riteraft-py/issues/8), etc... All sorts of miscellaneous issues had to be resolved.

Even after resolving all of these issues and getting the cluster to look like it was working, the issues kept coming: it would seem to be working fine, but then certain failures would cause catastrophic issues, such as cluster inconsistency or log synchronization locking up.

Each time we encountered an issue, we needed to be able to dig into the technical details of RAFT-RS and understand them, which ended up being a process of ripping apart RAFT-RS's code and understanding it piece by piece.

# raftify

While troubleshooting the issue, I decided to use a different abstraction than riteraft and implemented many changes, including a CLI module for debugging node and cluster state, which led me to rename the library to [raftify](https://github.com/lablup/raftify).

When I first started developing the library, the goal was to make it compatible with any Python application, hence the name raftify, which means to raftify.

I am no longer developing the Python implementation, but you can find it on [its branch](https://github.com/lablup/raftify/tree/python-legacy).

# raftify written in Rust

Developed in Python on top of rraft-py, raftify ended up working well, but the crude test harness written in a multi-process structure was hard to test in CI, easily broke cluster consistency, and got out of control at the slightest hint of code complexity.

As a result, we decided to completely rewrite raftify's internal logic in Rust and expose only the high-level interface of the Raft package in Python.

Once completely rewritten in Rust, raftify was single-threaded, integration testable, and could be tested in CI, which helped eliminate the fear of making code changes.

# raftify example code

In this section, we'll create a simple distributed key-value store using raftify.

For the full source code, see [this link](https://github.com/lablup/raftify/blob/main/examples/memstore/dynamic-members/src/main.rs).

## Define the state machine

The first thing we need to do is define the log entries and state machine that we will use in our key-value store.

For the sake of this article, we'll just define a simple `Insert` type command that defines a value as a log entry.

> 💡 Disclaimer: This article does not explain the Rust language syntax and the theoretical background of Raft.

```rust
#[derive(Clone, Debug, Serialize, Deserialize)]
pub enum LogEntry {
 Insert { key: u64, value: String },
}
```

Let's define a state machine of type `HashMap` as shown below.

```rust
#[derive(Clone, Debug)]
pub struct HashStore(pub Arc<RwLock<HashMap<u64, String>>>);
```

Then we need to define `encode` and `decode` methods to indicate how we want to serialize and deserialize these data structures. You can use the *bincode crate* to define these as simply as below.

```rust
impl AbstractLogEntry for LogEntry {
 fn encode(&self) -> Result<Vec<u8>> {
 serialize(self).map_err(|e| e.into())
 }

 fn decode(bytes: &[u8]) -> Result<LogEntry> {
 let log_entry: LogEntry = deserialize(bytes)?;
 Ok(log_entry)
 }
}

impl AbstractStateMachine for HashStore {
 fn encode(&self) -> Result<Vec<u8>> {
 serialize(&self.0.read().unwrap().clone()).map_err(|e| e.into())
 }

 fn decode(bytes: &[u8]) -> Result<Self> {
 let db: HashMap<u64, String> = deserialize(bytes)?;
 Ok(Self(Arc::new(RwLock::new(db))))
 }
}
```

Finally, we need to define three methods in the HashStore that will be used by raftify's internal code.

Define `apply`, a method that will be called when a new log entry is applied to the HashStore, `snapshot`, which will be called when saving the current state of the HashStore as a snapshot, and `restore`, which will be called when restoring the state of the HashStore via a snapshot byte slice, as shown below.

```rust
#[async_trait]
impl AbstractStateMachine for HashStore {
 async fn apply(&mut self, data: Vec<u8>) -> Result<Vec<u8>> {
 let log_entry: LogEntry = LogEntry::decode(&data)?;
 match log_entry {
 LogEntry::Insert { ref key, ref value } => {
 let mut db = self.0.write().unwrap();
 log::info!("Inserted: ({}, {})", key, value);
 db.insert(*key, value.clone());
 }
 };
 Ok(data)
 }

 async fn snapshot(&self) -> Result<Vec<u8>> {
 Ok(serialize(&self.0.read().unwrap().clone())?)
 }

 async fn restore(&mut self, snapshot: Vec<u8>) -> Result<()> {
 let new: HashMap<u64, String> = deserialize(&snapshot[..]).unwrap();
 let mut db = self.0.write().unwrap();
 let _ = std::mem::replace(&mut *db, new);
 Ok(())
 }
}
```

## Define the web server API

Let's define the web server API that will be used in our example. We will use this API to access the Raft object on the node and manipulate the HashStore.

For our example, we'll use the *actix-web crate* and define it as shown below.

The *put* command can be implemented by calling the `propose` method on the RaftNode of the Raft object. We can do this by encoding the `Insert` type LogEntry we defined earlier and passing it as an argument to the `RaftNode::propose` method.

The *get* command can be implemented by returning the value corresponding to the id from the HashMap stored in memory.

```rust
#[get("/put/{id}/{value}")]
async fn put(data: web::Data<(HashStore, Raft)>, path: web::Path<(u64, String)>) -> impl Responder {
 let log_entry = LogEntry::Insert {
 key: path.0,
 value: path.1.clone(),
 };
 data.1.raft_node.propose(log_entry.encode().unwrap()).await;

 "OK".to_string()
}

#[get("/get/{id}")]
async fn get(data: web::Data<(HashStore, Raft)>, path: web::Path<u64>) -> impl Responder {
 let id = path.into_inner();

 let response = data.0.get(id);
 format!("{:?}", response)
}

let web_server = tokio::spawn(
 HttpServer::new(move || {
 App::new()
 .app_data(web::Data::new((store.clone(), raft.clone())))
 .service(put)
 .service(get)
 })
 .bind(addr)
 .unwrap()
 .run(),
);
```

## Bootstrap a Raft cluster

Next, let's bootstrap a cluster of RaftNodes.

If the `--peer-addr` argument is given, send a join request to the cluster to get a new `node_id` via peer-addr and bootstrap a new cluster if this argument is not given.

### Leader

In this example, we'll stick with the `node_id` of the leader node for intuitive understanding. This means that when we call `Raft::bootstrap_cluster`, we can create a Raft object of the leader node by passing 1 to the `node_id`. After that, we can call the `Raft::run` method and the RaftNode will run.

### Followers

As shown below, after calling `Raft::request_id` to receive a `ClusterJoinTicket` that can be used to join the cluster, you will call `Raft::new_follower` with the specified `node_id` to create a follower Raft node object, call the `Raft::run` method to have the RaftNode run, and join the cluster via the `Raft::join` method.

```rust
let (raft, raft_handle) = match peer_addr {
 Some(peer_addr) => {
 log::info!("Running in Follower mode");

 let ticket = Raft::request_id(raft_addr, peer_addr.clone(), logger.clone()).await.unwrap();
 let node_id = ticket.reserved_id;

 let raft = Raft::new_follower(
 node_id,
 raft_addr,
 store.clone(),
 cfg,
 None,
 logger.clone(),
 )?;

 let handle = tokio::spawn(raft.clone().run());
 raft.join(ticket).await;
 (raft, handle)
 }
 None => {
 log::info!("Bootstrap a Raft Cluster");
 let node_id = 1;
 let raft = Raft::bootstrap_cluster(
 node_id,
 raft_addr,
 store.clone(),
 cfg,
 None,
 logger.clone(),
 )?;
 let handle = tokio::spawn(raft.clone().run());
 (raft, handle)
 }
};

let _ = tokio::try_join!(raft_handle)?;
```

You can now bootstrap a Raft cluster of three nodes in the terminal as shown below.

```sh
$ ./target/debug/memstore --raft-addr=127.0.0.1:60061 --web-server=127.0.0.1:8001
$ ./target/debug/memstore --raft-addr=127.0.0.1:60062 --peer-addr=127.0.0.1:60061 --web-server=127.0.0.1:8002
$ ./target/debug/memstore --raft-addr=127.0.0.1:60063 --peer-addr=127.0.0.1:60061 --web-server=127.0.0.1:8003
```

## Test

We can now try out the key-value store we defined through the `actix-web` server API via the `curl` command.

```sh
❯ curl http://localhost:8001/put/1/test
OK

❯ curl http://localhost:8001/get/1
Some("test")
```

If you're interested in learning more, you can check out the [raftify repository](https://github.com/lablup/raftify) for instructions on how to use the [CLI module](https://github.com/lablup/raftify#debugging) to help with debugging, example code for `RaftServiceClient`, and more.

# Summary

raftify is an experimental framework that aims to make it easier for anyone to integrate Raft modules that are otherwise hard to access by the normal developer.

It was developed to introduce a leader-follower structure to Backend.AI manager processes, but as I've shown in this post, it could be used in a variety of places where you need an HA structure, such as creating your own simple distributed key-value store with short source code.

If you're intrigued by the inner workings of the `tikv/raft-rs` implementation, stay tuned for my next post where I'll be analyzing what happens inside the source code line by line in a few scenarios.

> This post is automatically translated from Korean

Introducing Raftify: High-level Raft framework created with focusing on scalability

High availability (HA) has become an indispensable concept when talking about modern applications. High availability is the ability of an IT system to remain nearly 100% accessible and reliable at all times by eliminating or minimizing downtime[^1]. Backend.AI, which is developed and serviced by Rableup, also employs various methods to maintain high availability.

<img src="https://cdn.lablup.com/backendai_architecture_7b94e46789.svg">
 Backend.AI architecture
</img>

## Background

Backend.AI consists of many different components, including managers and agents, storage proxies, and web servers. Each of these components runs as multiple processes in a distributed environment to increase reliability, especially the manager, which is responsible for scheduling session execution and many core functions of Backend.AI. Currently, the manager has an Active-Active HA structure that ensures high availability through load balancing.

One of the many features of the Backend.AI Manager is event handling. Backend.AI raises various events, such as `AgentStartedEvent` and `DoScheduleEvent`, to track the lifecycle of agents and sessions and provide optimal scheduling. For example, when a Backend.AI Agent process runs, it generates an `AgentStartedEvent`, and the Backend.AI Manager process receives this event and performs a specific action (`schedule()`). Backend.AI Manager also raises a `DoScheduleEvent` internally to ensure periodic scheduling. This is where the problem arises. If you are running multiple Backend.AI Manager processes for high availability, having each process raise an event with its own timer adds unnecessary load and can cause the health of the entire system to be unreliable. The Backend.AI Manager implements a [GlobalTimer](https://github.com/lablup/backend.ai/blob/2f90d03c4477eda8e0beeabb7fe4b067c56dae09/src/ai/backend/common/distributed.py#L19-L43) to ensure that only one manager process generates events within the same system. The `GlobalTimer` uses distributed locks to ensure mutual exclusivity between processes and to ensure that only one process generates events.

```python title="backend.ai/src/ai/backend/common/distributed.py (2f90d03)"
@preserve_termination_log
async def generate_tick(self) -> None:
 try:
 await asyncio.sleep(self.initial_delay)
 if self._stopped:
 return
 while True:
 try:
 async with self._dist_lock:
 if self._stopped:
 return
 await self._event_producer.produce_event(self._event_factory())
 if self._stopped:
 return
 await asyncio.sleep(self.interval)
 except asyncio.TimeoutError: # timeout raised from etcd lock
 if self._stopped:
 return
 log.warn("timeout raised while trying to acquire lock. retrying...")
 except asyncio.CancelledError:
 pass
```

Currently, Backend.AI provides an interface for distributed locks, [AbstractDistributedLock] (https://github.com/lablup/backend.ai/blob/2f90d03c4477eda8e0beeabb7fe4b067c56dae09/src/ai/backend/common/lock.py#L33-L44), and we have developed and are using [`FileLock`] (https://github.com/lablup/backend.ai/blob/2f90d03c4477eda8e0beeabb7fe4b067c56dae09/src/ai/backend/common/lock.py#L47-L142), [`EtcdLock`] (https://github.com/lablup/backend.ai/blob/2f90d03c4477eda8e0beeabb7fe4b067c56dae09/src/ai/backend/common/lock.py#L145-L190) based on the [etcd concurrency API] (https://etcd.io/docs/v3.5/dev-guide/api_concurrency_reference_v3/), and [`RedisLock`] (https://github.com/lablup/backend.ai/blob/2f90d03c4477eda8e0beeabb7fe4b067c56dae09/src/ai/backend/common/lock.py#L193-L248) based on [Redis Lock] (https://redis.io/docs/manual/patterns/distributed-locks/) as actual implementations.

> [etcd](https://etcd.io/) is a distributed, open-source key-value store used to store and manage critical information needed to keep distributed systems running[^2], most notably in [Kubernetes](https://kubernetes.io/).

```python title="backend.ai/src/ai/backend/common/lock.py (2f90d03)"
class AbstractDistributedLock(metaclass=abc.ABCMeta):
 def __init__(self, *, lifetime: Optional[float] = None) -> None:
 assert lifetime is None or lifetime >= 0.0
 self._lifetime = lifetime

 @abc.abstractmethod
 async def __aenter__(self) -> Any:
 raise NotImplementedError

 @abc.abstractmethod
 async def __aexit__(self, *exc_info) -> Optional[bool]:
 raise NotImplementedError
```

## Requirements

The `GlobalTimer` does a good job of controlling event generation on a per-process basis in a distributed environment. However, requirements are always changing and the software needs to change with them. This time, the added requirement was to implement a rate limit for requests. With the current load balancing scheme, we can't guarantee that every request is handled by the same manager, which can lead to the following problems because the state of each manager is not shared.

```
1. Set the counters for both managers to 0 and the request count limit to 1.
2. The first request is received by manager 1.
3. Increase the counter on manager 1 by 1. (C1: 0 -> 1)
4. The counter reaches the maximum allowed number of requests and the next request is rejected.
5. Manager 2 receives the second request due to load balancing.
6. The counter on manager 2 has not reached the maximum allowed number of times because it is still 0. (C2: 0)
7. Manager 2 processes the request.
8. The request count limit didn't work!
```

Therefore, the following [issue](https://github.com/lablup/backend.ai/issues/415) has been proposed to discuss ways to improve these limitations.

<img src="https://cdn.lablup.com/distributed_timer_issue_a7018bb974.png">
 Issue suggesting improvements to distributed timers (lablup/backend.ai#415)
</img>

 

To delegate global state management to a single manager process, represented by a leader, we investigated consensus algorithms and decided to use the [Raft Consensus Algorithm](https://raft.github.io/) (hereafter Raft), which is used in projects such as etcd, which is used as a repository in Kubernetes (https://kubernetes.io/docs/concepts/overview/components/#etcd), and which we believe has been well validated.

## Raft consensus algorithm

The Raft algorithm was proposed in "In Search of an Understandable Consensus Algorithm"[^3] submitted to USENIX in 2014. It was created to improve upon Paxos[^4][^5], the leading algorithm at the time, which was difficult to understand and implement in practice due to its complex consensus process, hence the title.

> But our most important goal — and most difficult challenge — was **understandability**.
>
> - In Search of an Understandable Consensus Algorithm

 

A Raft cluster typically consists of five nodes, because a maximum of two nodes can fail and still satisfy a `quorum` to maintain the system.
Each node in a cluster has one of three states: leader, follower, or candidate. In general, there can be at most one leader in each cluster, with the rest of the nodes being followers.

> Glossary #1
> - quorum: The minimum number of people required to make a decision. (N/2+1)

<img src="https://cdn.lablup.com/raft_server_states_d2b1a7b53c.png">
 State transition diagram of a Raft node (Source: In Search of an Understandable Consensus Algorithm)
</img>

The Raft algorithm delegates all power to an elected leader and makes the flow of logs unidirectional, making it easier to understand the overall picture. The Raft algorithm has the following characteristics

> Glossary #2
> - term: The generation of the current leader or candidate. Incremented by 1 each time a leader election begins.
> - index: Refers to the location of a specific value in the log.
> - commit: Indicates that a specific value from the log was applied to the state machine.
> - commitIndex: Highest index that successfully commits

> - Election Safety: Each term has a maximum of one leader.
> - Leader Append-Only: Readers cannot overwrite or delete logs, they can only add new ones.
> - Log Matching: If two logs have values with the same index and term, all values up to that index are the same.
> - Leader Completeness: If a value is committed to the log in a particular term, all subsequent generations of readers are guaranteed to have that value.
> - State Machine Safety: If one server applies a log value from a particular index to its state machine, another server cannot apply a different value from the same index.

Using the above features, Raft divides the entire consensus process into three independent parts.

> - Leader election: If the existing leader is not working, a new leader must be elected.
> - Log replication: The leader replicates the request logs it receives from clients to other nodes. The other nodes unconditionally accept the leader's logs.
> - Safety: When one server applies a log value from a particular index to its state machine, another server cannot apply a different value from the same index.

In this article, we'll discuss the different states a Raft node can be in, and implement the leader election process in code.

### Follower
Followers do not send requests themselves, but only receive and respond to requests from the leader or candidate. The Behavior Spec for a Follower proposed in the paper and the code written based on it is shown below.

- Handle RPC requests from leaders and candidates.
```python title="aioraft/raft.py"
async def on_append_entries(
 self,
 *,
 term: int,
 leader_id: RaftId,
 prev_log_index: int,
 prev_log_term: int,
 entries: Iterable[raft_pb2.Log],
 leader_commit: int,
) -> Tuple[int, bool]:
 await self._reset_timeout()
 if term < (current_term := self.current_term):
 return (current_term, False)
 await self._synchronize_term(term)
 return (self.current_term, True)

async def on_request_vote(
 self,
 *,
 term: int,
 candidate_id: RaftId,
 last_log_index: int,
 last_log_term: int,
) -> Tuple[int, bool]:
 await self._reset_timeout()
 async with self._vote_request_lock:
 if term < (current_term := self.current_term):
 return (current_term, False)
 await self._synchronize_term(term)

 async with self._vote_lock:
 if self.voted_for in [None, candidate_id]:
 self._voted_for = candidate_id
 return (self.current_term, True)
 return (self.current_term, False)

async def _synchronize_term(self, term: int) -> None:
 if term > self.current_term:
 self._current_term.set(term)
 await self._change_state(RaftState.FOLLOWER)
 async with self._vote_lock:
 self._voted_for = None
```

- If you don't receive any requests from leaders or candidates for a period of time, you'll be placed in candidate status.
```python title="aioraft/raft.py"
async def _wait_for_election_timeout(self, interval: float = 1.0 / 30) -> None:
 while self._elapsed_time < self._election_timeout:
 await asyncio.sleep(interval)
 self._elapsed_time += interval
 await self._change_state(RaftState.CANDIDATE)
```

> Leaders must periodically announce their presence by sending heartbeat messages to their followers. If a follower does not receive any messages for a certain amount of time (`election_timeout`), it assumes that the cluster is leaderless and starts an election by becoming a candidate to become the new leader.

### Candidate
The candidate's behavior statement and implementation code is as follows
- Become a follower when you receive the `AppendEntries` RPC request from the new leader (see `on_append_etries()` for followers).
- Start the election with the following procedure
 - Increase term by 1. (term += 1)
 - Vote for yourself.
 - Initialize the election timeout.
 - Send a `RequestVote` RPC request to the other nodes.
```python
async def _start_election(self) -> None:
 self._current_term.increase()
 async with self._vote_lock:
 self._voted_for = self.id

 current_term = self.current_term

 terms, grants = zip(
 *await asyncio.gather(
 *[
 asyncio.create_task(
 self._client.request_vote(
 to=server,
 term=current_term,
 candidate_id=self.id,
 last_log_index=0,
 last_log_term=0,
 ),
 )
 for server in self._configuration
 ]
 )
 )
```
- If you receive votes from a majority of nodes, you are the leader.
```python
 for term in terms:
 if term > current_term:
 await self._synchronize_term(term)
 break
 else:
 if sum(grants) + 1 >= self.quorum:
 await self._change_state(RaftState.LEADER)
```
- If the election timeout occurs, start a new election.
```python
case RaftState.CANDIDATE:
 while self.__state is RaftState.CANDIDATE:
 await self._start_election()
 await self._reset_election_timeout()
 await self._initialize_volatile_state()
 if self.has_leadership():
 await self._initialize_leader_volatile_state()
 break
 await asyncio.sleep(self.__election_timeout)
```

### Leader
- Send the first heartbeat message (an empty `AppendEntries` request) immediately after the election. Send heartbeat messages periodically thereafter.
```python
async def _publish_heartbeat(self) -> None:
 if not self.has_leadership():
 return
 terms, successes = zip(
 *await asyncio.gather(
 *[
 asyncio.create_task(
 self._client.append_entries(
 to=server,
 term=self.current_term,
 leader_id=self.id,
 prev_log_index=0,
 prev_log_term=0,
 entries=(),
 leader_commit=self._commit_index,
 ),
 )
 for server in self._configuration
 ]
 )
 )
 for term in terms:
 if term > self.current_term:
 await self._synchronize_term(term)
 break
```
- When it receives a request from a client, it adds a value to the log. After applying that value to the state machine, send a response to the request.
- If the follower has a log value with an index greater than the value the leader is tracking (nextIndex), replicate the log to the follower starting at nextIndex.
 - If successful, update the leader's nextIndex and matchIndex.
 - If it fails due to an inconsistency, it decrements the leader's nextIndex and tries again.
- If the value (N) below exists, update the commitIndex to that value.
 - The majority of matchIndexes are greater than or equal to N (matchIndex >= N)
 - The term of the Nth log is the same as the current term

> The leader manages a nextIndex and a matchIndex for each follower.
> - nextIndex: The next index that should be sent to each follower.
> - matchIndex: the highest index that was successfully replicated to each follower

## Conclusion

In this article, we've briefly covered the Raft algorithm and written code to perform a leader election. The remaining two features (log replication, membership changes) will face a variety of challenges in actual implementation, including timing issues. If you're interested in learning more about the Raft algorithm, we recommend reading the author's (Diego Ongaro) PhD thesis (CONSENSUS: BRIDGING THEORY AND PRACTICE)[^6].

Finally, let's end by checking out how ChatGPT describes the Raft algorithm.
<img src="https://cdn.lablup.com/raft_by_chatgpt_7f4e271db8.png">
 Raft algorithm explained by ChatGPT (Source: OpenAI ChatGPT 3.5)
</img>

> This article is based on the code in [lablup/aioraft-ng](https://github.com/lablup/aioraft-ng).
> Please also pay attention to [lablup/raftify](https://github.com/lablup/raftify), the next generation Raft project currently under development at Lablup.

[^1]: https://www.redhat.com/ko/topics/linux/what-is-high-availability
[^2]: https://etcd.io/
[^3]: https://web.stanford.edu/~ouster/cgi-bin/papers/raft-atc14
[^4]: https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf
[^5]: https://lamport.azurewebsites.net/pubs/paxos-simple.pdf
[^6]: https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf

Raft Consensus algorithm for Backend.AI: Leader election

Dongjin Park

#Overview

I applied to CUop, a collaboration between universities specializing in science and technology, and worked as an intern at Lablup for 8 weeks.

I wrote about my experiences through onboarding, developing Backend.AI, and attending PyCon.

# Motivation for applying

I first learned about Lablup through a PyCon presentation session I stumbled upon. I could tell that the company had a lot of technically deep and passionate members. I applied to Lablup because I was interested in Python and asynchronous programming.

# onboarding

During the first two weeks, we conducted an onboarding process.

We went through the Realtime Web Chat implementation, Backend.AI development environment, and code base seminar.

## Realtime Web Chat

This is an assignment to familiarize yourself with Python asyncio. You will develop a real-time chat app using aiohttp, an asynchronous web framework, and redis, an in-memory database. The task also includes setting up docker compose to build python and redis at once. For more information, see the [Github Readme](https://github.com/pderer/webchat).

To broadcast the messages through redis, we used Redis pub/sub. pub/sub acts as a platform that delivers messages without storing them. Since we didn't have a requirement to store messages, we used Redis pub/sub. We also registered the Redis pub/sub process as a task using asyncio.create_task() to run it in an event loop.
![Realtime Web Chat Launch Screen](https://cdn.lablup.com/Realtime_webchat_0f87103fe6.png) 
Realtime Web Chat Launch Screen

I was able to understand the basic behavior of asyncio. I was able to ask questions and solve the difficult parts. I think Lablup has a great environment for interns and junior developers to grow because they can freely ask questions through Microsoft Teams.

## Build the Backend.AI development environment

I installed Backend.AI on a VM Farm and a local VM and tried to run it myself. I read the official documentation, but the process was not smooth. I encountered various errors, which I solved by sharing with my fellow interns and asking questions on Teams.

> 💡 A VM farm is an environment where virtual machines are managed and run. Lablup has its own VM Farm.

This was my first experience of developing with a VM Farm and VSCode connected via SSH. To develop Backend.AI, I need to run multiple processes (Manager, Agent, Storage proxy, Web server, etc.) and Docker Containers. If I use a laptop, it's a bit of a battery drain. However, with VM Farm, you can develop Backend.AI lightly because all you need is an SSH connection. In fact, I was able to develop for a long time using VM Farm when I was out of the office and couldn't charge my laptop.

## Code Base Seminar

After looking at the code, focusing on the difficult parts of Backend.AI, I prepared a seminar based on my understanding. I was in charge of presenting the Manager, Agent, and GraphQL parts.

Since Backend.AI is open source, the official documentation is well written. I studied the architecture of Backend.AI by looking at the official documentation to understand the overall structure and asking the employees directly if I had any questions. Since the topic of the presentation was session/kernel creation and scheduling control of Backend.AI Manager, I studied the manager code and analyzed the logs of the manager process.
 ![A sequence diagram I drew in preparation for a seminar presentation.](https://cdn.lablup.com/Sequence_Diagram_27d4a0128e.png) 
A sequence diagram I drew in preparation for a seminar presentation.

Analyzing the logs, I found a bug that caused the session state to change from Preparing back to Pulling. It was rewarding to analyze the logs one by one. It was difficult to analyze the logs in order because of the asynchronous code base, but drawing a call graph and a sequence diagram was very helpful.

# Develop Backend.AI

After onboarding, I started working on Backend.AI. I worked on GitHub issues and volunteered to help or found and resolved issues myself.


I created 9 pull requests from the [Backend.AI repository](https://github.com/lablup/backend.ai/pulls?q=is%3Apr+is%3Aclosed+author%3Apderer) repository and 2 pull requests from the [Backend.AI-WebUI repository](https://github.com/lablup/backend.ai-webui/pulls?q=is%3Apr+is%3Aclosed+author%3Apderer), and they all merged!

I chose high-priority issues that I was confident in addressing. I wanted to make as many contributions as I could in the short time frame of two months.

## First PR

https://github.com/lablup/backend.ai/pull/1395

I created a PR to fix a bug I found while preparing for a seminar. It was an easy PR to fix the API parameter. However, it was a good experience to learn about branch name convention, commit convention, and news fragment, and to experience the CI (Continuous Integration) process with GitHub Actions and to do some Git-related shoveling beforehand.

> 💡 A news fragment is a one-sentence Markdown description of what the branch created by the PR is trying to do. You want to keep it simple and clear so that if you see this PR again in the future, you'll know what it was trying to do.

## PRs for vfolder

When I heard that Teams had an open issue for an intern, I jumped at the chance to apply. I had to learn a new concept called vfolder, but I knew it would be important to get to know the product.

### PR (1)

https://github.com/lablup/backend.ai/pull/1397

Only admin can create a project type vfolder. It should be possible to create a vfolder regardless of the max_vfolder_count of the keypair resource policy, but if the user type vfolder exceeds the max_vfolder_count, the project type vfolder cannot be created. At first, I was confused by the terminology, but by analyzing the code and asking questions, I was able to interpret the terminology.

### PR (2)

https://github.com/lablup/backend.ai/pull/1400

Fixed new bugs discovered while addressing PR (1).

### PR (3)

https://github.com/lablup/backend.ai/pull/1417

I found an issue related to the PR (1) issue. DB migration and GraphQL were new to me, but I wanted to try them out, so I supported them. I used a DB migration tool called [Alembic](https://alembic.sqlalchemy.org/en/latest/), studied GraphQL schema concepts, and modified my query and mutation code to support backward compatibility. I tried to use cURL to test the modified code, but GraphQL has a much longer request format than the REST API, which was cumbersome. I wrote the test code by asking questions to interns and employees who are familiar with GraphQL. I wrote python code to test the modified query and mutation in the form of CLI to make testing easier.

## WSProxy related PRs

Supported an issue in Teams. There was a bug in the WebUI where it was not possible to delete a session if the wsproxy address of the resource group was invalid. I also wanted to get some experience with WebUI development.

### PR (1)

https://github.com/lablup/backend.ai/pull/1423

I read through the WebUI code to troubleshoot the issue, but I couldn't quite grasp the concept of wsproxy. I realized that wsproxy has v1 and v2, but it was not easy to understand the difference between the two, so I asked the employees. The main difference between v1 and v2 is the path of the traffic: v1 goes through the manager to communicate with the container, while v2 can communicate directly with the container without going through the manager, which is faster. Once I understood what wsproxy does and the difference between v1 and v2, it was easier to understand how the code worked, and I realized that a lot of people didn't know the difference. I realized that questions that seemed easy to ask might not have been asked in the organization.

### PR (2)

https://github.com/lablup/backend.ai-webui/pull/1819

I also modified the webui code to fix the issue. To fix the JavaScript code, I learned about callback functions, promise objects, and async/await. I handled errors so that they didn't affect other logic, and defined duplicate code as functions to eliminate duplication of code.

### PR (3)

https://github.com/lablup/backend.ai-webui/pull/1833

However, since the WebUI needs to be backwards compatible with Backend.AI 22.09, we had a review from our CEO that it should also handle HTTP Status 404, so we made it handle both 404 and 500.

### PR (4)

https://github.com/lablup/backend.ai/pull/1466

However, after the code was merged, a bug occurred: when setting up v1 wsproxy, the return value for wsproxy-version disappeared. This happened because I was modifying the core code and didn't handle all the branches. I fixed the code in a hurry, but it was a simple mistake, and I realized that I should write test code to prevent such mistakes.

## PRs related to Manager

https://github.com/lablup/backend.ai/pull/1444

In preparation for the seminar, I came across an issue with manager that I had been studying. With my internship coming to an end, I thought I could contribute by resolving the issue on the code I knew best.

This PR has been heavily modified by code reviews. Initially, we designed scheduler health check and scheduler trigger as the same API. After receiving code reviews, I split the two functions into different APIs to separate responsibilities. We originally stored health information only for the schedule function, but we also stored health information for the prepare function and the scale_services function because we needed to create the trigger API after storing health information for the three functions that run periodically according to the scheduler's global timer to get a complete picture of the scheduler's health. We also changed the design so that we can store the scheduler's health based on the manager ID because there may be multiple manager processes.

The scheduler's storage was also reviewed. Initially, we looked at the code in the existing manager state API to store manager state in etcd and did the same for scheduler state. etcd is good for storing configuration information that needs to be kept consistent, but it is slow to write. redis is a volatile database, but it performs well even with many reads/writes. Since the scheduler state is read/written periodically and does not need to be consistent, we switched to storing it in redis.

## Agent-related PRs

https://github.com/lablup/backend.ai/pull/1472

Now that I had a good understanding of the Manager part of Backend.AI, I wanted to understand another important component: the Agent. I came across an issue about the Agent, so I took a look.

While Backend.AI was running, there was a bug where the internal state of the Agent did not match the state of the actual working container. As a result, when creating a session, an InsufficientResource Error was thrown during the resource allocation process, even though there were actually enough resources. When the error occurred, we needed to improve the logging to understand what went wrong during the resource allocation process.

It took a while to figure out the resource allocation process. The concurrency issues were difficult, and it took a lot of Q&A with the CTO to get a general idea of the flow and what to log.

A few weeks after my internship ended, the CTO merged it with more than 10 commits, adding refactorings and test code. What impressed me was that he wrote test code to reproduce the error. I had to go through a complex process (see PR) to reproduce the error myself, which took a lot of development time. I could see the difference in productivity here. Of course, I thought about writing test code, but I thought the implementation would be too complicated, and I thought that writing test code would be the end of my internship. In the future, I shouldn't be so intimidated by writing test code and just try it out and learn as I go. Also, it seems that the refactorings were done with a focus on code readability. The functions in the part I modified were too long and not very readable, but after refactoring, the functions were shorter and the logging was cleaner. I realized that I shouldn't stop at implementation, but try to make good code.

# Attending PyCon

On August 12 and 13, Lablup will have a booth at PyCon. Companies that sponsor PyCon are given the opportunity to have a booth. I was an intern, but I wanted to participate in the booth activities and listen to some of the talks. The company had some PyCon tickets left over, so I was able to participate.

At the Lablup booth, we had an event where we challenged Llama2 to write a 10-line pyramid at a prompt. The challenge wasn't that easy, and it was important to explain it in a way that Llama2 could understand. Two lucky people who submitted the correct answer were drawn to win a Nintendo Switch and a Logitech mouse. My role at the booth was to help direct PyCon attendees to the event, and since there were a lot of employees at PyCon, I was free to attend any talks I wanted to hear. Since Rableup is an open source company, we encourage people to contribute to open source and participate in conferences. In fact, four of our presenters attended PyCon, so we value participating in conferences.

 ![Lablup Inc. booth](https://cdn.lablup.com/pycon23_booth_6059bf7e37.jpeg) 
Lablup Inc. booth

During the RustPython session, a tool called ruff was introduced as a replacement for the python lint tools flake8 and isort. Since ruff is composed of Rust, it is 100x faster than flake8. At Backend.AI, we were using flake8 and isort for lint, but after our CTO reviewed ruff, I watched him adopt ruff for the Backend.AI project right on the stairs of COEX. I realized that he was really good at the process involved in coding, applying new tools to the project in a short time and even revising the official documentation. I realized that I want to be a proficient developer someday. After PyCon, I read the updated official documentation, applied ruff to the Backend.AI development environment, and experienced linting 100x faster. If I hadn't participated in PyCon, I wouldn't have gotten up to speed on these great tools so quickly. I hope to continue participating in developer conferences in the future.
![Group photo with members of Lablup Inc.](https://cdn.lablup.com/pycon23_all_dee19bbc26.jpeg) 
Group photo with members of Lablup Inc.

# End of internship

During my internship, I tried to get as much experience as possible, and I wanted to contribute a lot. In the end, I was able to experience a lot because I tried to contribute a lot. I was quick to volunteer for issues that came up in Teams, so I was able to understand the core components of Backend.AI: vfolder, wsproxy, web-ui, manager, and agent. I also learned new concepts like DB Migration, GraphQL, and etcd. Although it was a bit physically demanding to attend the conference from morning to evening on the weekend, it was fun to listen to more than 10 presentation sessions, get inspired, and meet various people through booth activities.

During my internship, I think I actively asked questions about things I didn't understand, which helped me to solve issues quickly. I think the reason why I was able to ask a lot of questions was because there was a culture of horizontal rabble-rousing, and there were many people who were kind enough to answer my questions, so I was able to actively ask questions. I would like to take this opportunity to thank the members for their support.

I was able to experience a variety of things, including asynchronous programming experience, GitHub collaboration, presenting English seminars, and attending conferences. I feel like I've grown a lot as a developer through this program. I recommend the Lablup internship to anyone who is thirsty for growth.

> This post is automatically translated from Korean

2023 Summer Intern in Lablup

Kyujin Cho

Backend.AI version 23.09 has been officially released to the public.
We covered Model Service, a key feature in version 23.09, in our previous [Sneak Peek: Backend.AI Model Service preview article](https://www.backend.ai/blog/2023-05-30-backend.AI-model-service-sneak-peek).
Since then, we have added a variety of new features, including GUI support, authentication token history management, and more, and we are going to walk you through them in a tutorial format to make it easy to understand the Backend.AI Model Service.
In this tutorial post, we will show you how to use the Backend.AI Model Service to run GPT-NeoX models on top of Triton Inference Server.
Triton Inference Server is an open source model inference framework from NVIDIA that enables easy HTTP and gRPC[1](#footnote_1) delivery of its TritonRT, FasterTransformer, and TritonRT-LLM models, as well as PyTorch, TensorFlow, vLLM, and many others.

## Create a Model VFolder
1. Navigate to the Data & Folders tab. Click the "New Folder" button to open the VFolder creation dialog.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_30_48_PM_f6b4c7a4b9.jpg) 
2. Create a new model folder. It does not matter how you name the folder, but make sure to set the "Usage" at the bottom to "Model". Once you have specified all the values, click the "Create" button at the bottom. Your model VFolder has now been created.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_31_12_PM_7782dc5224.png) 

## FasterTransformer Format Model Conversion
1. Navigate to the "Sessions" tab. Click the "Start" button to open the session creation dialog.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_33_29_PM_7e2bf0d6f2.jpg)
2. Select `ngc-pytorch` for "Running Environment" and `23.07` for "Version". Once you have made your selections, click the arrow icon in the lower right corner.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_34_17_PM_c3b42b5e27.png) 
3. The window to select the VFolder to mount in the session. To load the model, select the VFolder you just created under the "Model storage folder to mount" section. Once you have made your selections, click the arrow icon in the lower right corner.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_36_18_PM_c9600865e0.png) 
4. A window to specify the amount of resources to be used by the model session. You should allocate at least 16 CPU cores and 128 GB of RAM to ensure smooth model conversion. Once you have made your selections, click the arrow icon in the lower right corner.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_37_16_PM_cf32bb2e21.png) 
5. After confirming that all settings have been applied correctly, click the "Start" button below to start the session.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_37_59_PM_be8660c450.png) 
6. Once the session is created, a popup will appear to select an app, as shown below. Click the "Console" app to access the terminal environment.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_38_30_PM_d9c42fb8fa.png) 
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_43_08_PM_96b88d44c1.jpg) 
7. Run the following shell script to download the GPT-NeoX 20B model and convert it to the FasterTransformer format. Note that where the script mentions `<VFolder name>`, you must replace it with the name of the model VFolder you created.
```sh
cd /home/work/<VFolder name>
pip install -U transformers bitsandbytes
git clone https://github.com/NVIDIA/FasterTransformer
git clone https://huggingface.co/ElutherAI/gpt-neox-20b
cd neo-gptx-20b
git lfs install
git lfs pull
```
> The GPT-NeoX 20B model requires at least 40GB of VRAM to run. If the physical GPU you are using has less VRAM than this and you need to split the model across multiple GPUs, adjust the number in the `-i_g` parameter to match the number of GPUs you are using.

```sh
cd /home/work/<VFolder name>
mkdir -p triton-deploy/gpt-neox-20b-ft
python ~/<VFolder name>/FasterTransformer/examples/pytorch/gptneox/utils/huggingface_gptneox_convert.py \
 -i /home/work/<VFolder name>/gpt-neox-20b \
 -o /home/work/<VFolder name>/triton-deploy/gpt-neox-20b-ft \
 -i_g 1 \
 -m_n GPT-NeoX-20B
```
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_55_43_PM_172b2e02a3.jpg) 

8. If you followed all the steps up to step 7, you should have the following folders under the VFolder.
```sh
work@main1[PRRLCIqu-session]:~/GPT-NeoX-Triton-FT$ ls -al
total 62
drwxr-xr-x 5 work work 11776 Oct 12 12:14 .
drwxr-xr-x 9 work work 4096 Oct 12 12:29 ..
drwxr-xr-x 14 work work 12800 Oct 12 11:24 FasterTransformer
drwxr-xr-x 3 work work 16896 Oct 12 10:18 gpt-neox-20b
drwxr-xr-x 3 work work 11776 Oct 12 11:56 triton-deploy
```
Now it's time to add the configuration file for Triton Inference Server. Create the file `triton-deploy/gpt-neox-20b-ft/config.pbtxt` and add the following contents.
> If you set the value of the `-i_g` parameter to anything other than 1 in step 7, you must modify the value of `tensor_para_size` in the settings below to match the value of `-i_g`.
```sh
name: "gpt-neox-20b-ft"
backend: "fastertransformer"
default_model_filename: "gpt-neox-20b-ft"
max_batch_size: 1024

model_transaction_policy {
 decoupled: False
}

input [
 {
 name: "input_ids"
 data_type: TYPE_UINT32
 dims: [ -1 ]
 },
 {
 name: "start_id"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "end_id"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "input_lengths"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 },
 {
 name: "request_output_len"
 data_type: TYPE_UINT32
 dims: [ -1 ]
 },
 {
 name: "runtime_top_k"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "runtime_top_p"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "beam_search_diversity_rate"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "temperature"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "len_penalty"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "repetition_penalty"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "random_seed"
 data_type: TYPE_UINT64
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "is_return_log_probs"
 data_type: TYPE_BOOL
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "beam_width"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "bad_words_list"
 data_type: TYPE_INT32
 dims: [ 2, -1 ]
 optional: true
 },
 {
 name: "stop_words_list"
 data_type: TYPE_INT32
 dims: [ 2, -1 ]
 optional: true
 },
 {
 name: "prompt_learning_task_name_ids"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "top_p_decay"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "top_p_min"
 data_type: TYPE_FP32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 },
 {
 name: "top_p_reset_ids"
 data_type: TYPE_UINT32
 dims: [ 1 ]
 reshape: { shape: [ ] }
 optional: true
 }
]
output [
 {
 name: "output_ids"
 data_type: TYPE_UINT32
 dims: [ -1, -1 ]
 },
 {
 name: "sequence_length"
 data_type: TYPE_UINT32
 dims: [ -1 ]
 },
 {
 name: "cum_log_probs"
 data_type: TYPE_FP32
 dims: [ -1 ]
 },
 {
 name: "output_log_probs"
 data_type: TYPE_FP32
 dims: [ -1, -1 ]
 }
]
instance_group [
 {
 count: 1
 kind: KIND_CPU
 }
]
parameters {
 key: "tensor_para_size"
 value: {
 string_value: "1"
 }
}
parameters {
 key: "pipeline_para_size"
 value: {
 string_value: "1"
 }
}
parameters {
 key: "data_type"
 value: {
 string_value: "fp16"
 }
}
parameters {
 key: "model_type"
 value: {
 string_value: "GPT-NeoX"
 }
}
parameters {
 key: "model_checkpoint_path"
 value: {
 string_value: "/models/triton-deploy/gpt-neox-20b-ft/1-gpu"
 }
}
parameters {
 key: "enable_custom_all_reduce"
 value: {
 string_value: "0"
 }
}
```
9. Finally, you need to add the Backend.AI Model Service definition file to the root of the VFolder, under `model-definition.yaml` (`model-definition.yml` is also acceptable). Let's take a closer look at the model definition file for running Triton Inference Server.
```yaml
models:
- name: "GPT-NeoX"
 model_path: "/models/triton-deploy"
...
```
This is where you specify the model name and the path to the model.
> The name and path you set here can be accessed by the model server process as the `BACKEND_MODEL_NAME` and `BACKEND_MODEL_PATH` environment variables, respectively.
> ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_10_54_11_PM_1ce66d14e0.png) 
```yaml
...
 service:
 start_command:
 - tritonserver
 - --model-repository=/models/triton-deploy
 - --disable-auto-complete-config
 - --log-verbose
 - "1"
...
```
This is the part that defines the command line syntax for starting the Model Server process.
```yaml
...
 port: 8000
...
```
This is where you fill in the port for API communication that the model server process exposes. If not specified, Triton Inference Server exposes port `8000` for HTTP API communication by default, so you will also write that port in the model definition file.
```yaml
...
 health_check:
 path: /v2/health/ready
 max_retries: 3
 max_wait_time: 5
 expected_status_code: 200
```
This is where you enable and set up the Health Check feature. If the Health Check feature is enabled, Backend.AI will continuously send HTTP GET requests to the path to verify that it returns an HTTP response code corresponding to the `expected_status_code` (can be omitted, defaults to `200`). If the model server does not respond, or returns an undefined response code, Backend.AI determines that the session is unhealthy and excludes it from the service. When a session is excluded from the service, it is not automatically terminated and the Model Service administrator must manually take the appropriate action by checking container logs, etc.
The Health Check feature can be disabled by omitting the syntax entirely. If you do this, Backend.AI will not check the health of the model server and will always assume it is in a healthy state.
The `max_wait_time` is the part that defines the API response timeout. It must be a number in seconds.
The `max_retries` is the number of times the request is retried before the model server is judged to be unhealthy. 
The finished model definition file looks like this.
```yaml
models:
- name: "GPT-NeoX"
 model_path: "/models/triton-deploy"
 service:
 start_command:
 - tritonserver
 - --model-repository=/models/triton-deploy
 - --disable-auto-complete-config
 - --log-verbose
 - "1"
 port: 8000
 health_check:
 path: /v2/health/ready
 max_retries: 3
 max_wait_time: 5
```
> More information about model definition files can be found in the [Backend.AI WebUI documentation](https://webui.docs.backend.ai/en/latest/model_serving/model_serving.html#creating-a-model-definition-file).

Now you're all set to run the Model Service.

## Create a Model Service
1. Navigate to the "Model Serving" tab. Click the "Start Service" button to open the Create Model Service window.
 ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_07_00_PM_c8cfaaffc9.jpg) 
 ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_11_31_PM_91d1100104.png) 
 Let's take a look at each section in a little more detail.
 - ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_08_25_PM_6a054b93bd.png) Service name: This is where you specify the name of the Model Service. The name of the Model Service can be used as a subdomain of the Model Service Endpoint (coming soon).
 - ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_12_20_PM_c403eb3d5b.png) Resource Group: This is the field to select the resource group where the Inference Session for the Model Service will be created.
 - ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_12_50_PM_22f7b84b81.png) Open your app to the outside world: When this feature is enabled, all API requests to the model server must be accompanied by an authentication header before they can be made. For more information about Model Service authentication, see the [Backend.AI WebUI documentation](https://webui.docs.backend.ai/en/latest/model_serving/model_serving.html#generating-tokens).
 - ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_14_43_PM_06d5c0cbc9.png) Desired number of routes: A field to specify the number of inference sessions the Model Server process runs in. Setting this value to a number greater than 1 creates multiple identical sessions and enables the round-robin load balancer feature, which distributes API requests evenly among these sessions. This value can be modified at any time after Model Service creation.
 - ![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_16_43_PM_be8f40c871.png) A panel that specifies the amount of resources for the inference session.
> The GPT-NeoX 20B model requires a minimum of 40 GB of vRAM to run.
> The relationship between fGPU units and vRAM in Backend.AI may apply differently depending on the settings of your Backend.AI. Consult with the administrator of your Backend.AI for more information.
If you have set all the values correctly, press the "OK" button to create the Model Service.
2. the Model Service has been created. If the Model Service is not yet ready for the model process in the reasoning session, the status will remain "PROVISIONING".
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_19_08_PM_13ca45792d.jpg) 
Click on the "INFERENCE" section of the "Sessions" tab and you'll see that an inference session has been created corresponding to the Model Service you created in 1.
![](https://cdn.lablup.com/image_7_df9dc80956.png) 
Model Service administrators can click the clipboard icon in the "Control" row to view logs related to the model server processes in an inference session.
![](https://cdn.lablup.com/image_784ac56fce.jpg) 
3. When the Model Server process is running normally, the status of the route at the bottom and the status at the top will both change to "HEALTHY", and the address to access the Model Service will appear under "Service Endpoints".
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_23_34_PM_42bb7108c3.png) 
You can now access the Triton Inference Server that ran the inference session through that address.
![](https://cdn.lablup.com/Screenshot_2023_10_12_at_11_28_04_PM_d80ded6c89.png) 

## Conclusion
In this article, you've learned how to start serving LLM models using the Backend.AI Model Service. The Model Service feature is available in Backend.AI's Cloud Beta. Start serving your own models today!

<a name="footnote_1">1</a>: Not supported by Backend.AI Model Service

> This post is automatically translated from Korean

Backend.AI Model Service Hands-on: Running GPT-NeoX

# 23.09: September 2023 update
In the second half of 2023, we released 23.09, a major release of [Backend.AI](https://backend.ai). In 23.09, we've significantly enhanced the development, fine-tuning, and operational automation of generative AI. We've automatically scaled and load-balanced AI models based on workload, expanded support for various GPUs/NPUs, and increased stability when managing a single node as well as 100-2000+ nodes.
The team is working hard to squeeze every last bit out of it. Here are the main improvements since the last [23.03 July update] (/posts/2023/07/31/Backend.AI-23.03-update).

## Backend.AI Core & UI
- The Backend.AI Model Service feature has been officially released. You can now use Backend.AI to more efficiently prepare environments for inference services as well as training of large models such as LLM. For more information, see the blog [Backend.AI Model Service sneak peek](/posts/2023/05/30/backend.AI-model-service-sneak-peek).
- Added the ability to sign in to Backend.AI using OpenID single sign-on (SSO).
- If your kernel image supports it, you can enable the sudo command without a password in your compute session.

- Support for Redis Sentinel without HAProxy. To test this, we added the `--configure-ha` setting to the `install-dev.sh` file.
- Added the ability to use the RPC channel between Backend.AI Manager and Agent for authenticated and encrypted communication.
- Improved the CLI logging feature of Backend.AI Manager.
- Fixed an issue where Manager could not make an RPC connection when Backend.AI Agent was placed under a NAT environment.
- The Raft algorithm library, riteraft-py, will be renamed and developed as [raftify](https://github.com/lablup/raftify).
- Support for the following new storage backends
 - VAST Data
 - KT Cloud NAS (Enterprise only)

## Backend.AI FastTrack
- Improved UI for supporting various heterogeneous accelerators.
- Deleting a VFolder now uses an independent unique ID value instead of the storage name.
- Upgraded Django version to 4.2.5 and Node.js version to 20.
- Added pipeline template feature to create pipelines in a preset form.
- If a folder dedicated to a pipeline is deleted, it will be marked as disabled on the FastTrack UI.
- Improved the process of deleting pipelines.
- Added a per-task (session) accessible BACKENDAI_PIPELINE_TASK_ID environment variable.
- Actual execution time per task (session) is now displayed.

## Contribution Academy
Especially in the past period, the following code contributions were made by junior developer mentees through the [2023 Open Source Contribution Academy](https://www.contribution.ac) organized by NIPA.
- Created a button to copy an SSH/SFTP connection example to the clipboard.
- Refactored several Lit elements of the existing WebUI to React.
- Wrote various test code.
- Found and fixed environment variable and message errors that were not working properly.

Backend.AI is constantly evolving to provide a more powerful and user-friendly experience while supporting various environments in the AI ecosystem. Stay tuned for more updates! 
Make your AI accessible with Backend.AI!

> This post is automatically translated from Korean

23.09: September 2023 Update

Mario (Manseok) Cho

## Introduction

Most Linux distributions, such as Ubuntu, RedHat, and CentOS, use glibc as the system's standard C library. When you install a library package, such as OpenSSL, with apt on Ubuntu or rpm (yum) on the RedHat line, it is dynamically linked with glibc by default.

GNU (Gnu) is an operating system and includes a wide range of computer software. GNU is open source, developed and maintained by the Free Software Foundation (FSF). Examples of things created by GNU include compilers and development tools such as GCC, G++, and Make. GNU uses glibc as its standard C library. glibc uses the GNU Lesser General Public License.

musl is a Linux standard C library distributed under the MIT license. Its developer is Rich Felker, and while glibc uses dynamic linking, musl aims to implement a standard C library that conforms to POSIX standards using static linking. It also implements non-standard features of Linux, BSD, and glibc.

## Differences between glibc and musl in the Linux environment 

When you install a package on Linux, it uses glibc by default.
If you've ever built a C/C++ program using gcc, you've most likely done a glibc-based dynamic link build.
However, in addition to this common glibc dynamic build, you can also do a MUSL-based dynamic/static build.

There are the following differences between `*-linux-gnu` and `*-linux-musl`.

| Build targets | Standard C libraries | Linking method
|----------------|-------------------|----------------|
| `*-linux-gnu` | glibc | dynamic linking |
| `*-linux-musl` | musl | dynamic/static linking |

Consider the case of building an executable with Rust.
When you install Rust on a Linux environment using `rustup`, `*-linux-gnu` is selected as the default target.

If you don't specify any other options, Rust will build the binary with the `*-linux-gnu` target and dynamically link it with glibc.
To run a binary built in this way, you must have glibc installed in your Linux environment for it to work.
If the binary relies on external libraries such as OpenSSL (if it is dynamically linked), you will also need to install those libraries via a package manager such as apt.
If you want to run these dynamically linked binaries as a regular user, you can bundle them into a package like a DEB or RPM that describes the dependencies on external libraries.
The package manager will then automatically find and install the appropriate dependent libraries.
However, if you're using a library that isn't registered with the package manager, or even the same library, there are subtle compatibility issues between the installed version and the version you used to develop it, there's a chance that the binary you build won't run as intended.

If you specify the `*-linux-musl` target, Rust will statically link with musl when building the binary.
If you rely on external libraries like OpenSSL, it will also statically link those as well, embedding them all into the binary.
This means that you end up with all of these libraries inside a single binary file in Rust.
This static binary can run on any Linux environment, as long as it matches the CPU architecture and the set of system calls provided by the Linux kernel.
This makes it easier to distribute binaries because you only need to pass a single binary to run it, rather than using a package like a DEB or RPM.

If this makes deploying binaries so easy, why isn't the `*-linux-musl` target the default for Linux environments?

The reason is that using MUSL makes build preparation somewhat more complicated.
This is because if a developer-created binary package uses `*-linux-musl` and also relies on external libraries, those external libraries must also be statically linked with musl instead of dynamically linked with glibc.
This means that all dependent libraries, as well as the main body of the program you want to build using the compiler for musl, must be built as static links from source code.

Fortunately, you don't have to build everything from scratch if it's a commonly used external library in Rust.
By utilizing a Docker image that bundles frequently used libraries with the Rust compiler/gcc, you can easily create a musl-based static build.
(In the command examples that follow, I'll arbitrarily use the `<distro>#` prompt to distinguish the container environment for each Linux distribution).

```console
$ docker run -it --name ubuntu ubuntu:22.04 bash
ubuntu# apt update && apt install -y curl gcc vim
```

Let's configure a dynamic link, glibc, and a static link, musl, in the Rust language environment, which is commonly used for development.
First, install Rust on your Ubuntu environment.

```console
ubuntu# curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
ubuntu# source $HOME/.cargo/env
```

Let's compare dynamic and static linking using Rust's default example, "Hello World" output.

First, let's build "Hello World" using glibc.

```console
ubuntu# cd
ubuntu# cargo new --bin hello && cd $_
 Created binary (application) `hello` package
ubuntu# cargo build --release
 Compiling hello v0.1.0 (/root/hello)
 Finished release [optimized] target(s) in 0.35s
```

Let's use the `ldd` command to verify that the library is configured as a dynamic link in the glibc environment.
We can see that `linux-vdso`, `libgcc_s`, `libc`, etc. are configured as dynamic links.

```console
ubuntu# ldd target/release/hello
 linux-vdso.so.1 (0x00007fffe87df000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdce9c3f000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdce9a17000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fdce9cc2000)
```


So let's change the RUST target configuration with a MUSL static link.

```console
ubuntu# rustup target add x86_64-unknown-linux-musl
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
 34.7 MiB / 34.7 MiB (100%) 8.6 MiB/s in 4s ETA: 0s

ubuntu# rustup show
Default host: x86_64-unknown-linux-gnu
rustup home: /root/.rustup

installed targets for active toolchain
--------------------------------------

x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.72.0 (5680fa18f 2023-08-23)

ubuntu# 
```

Let's build "Hello World" to verify that static links are configured correctly.

```console
ubuntu# cargo build --release --target=x86_64-unknown-linux-musl
 Compiling hello v0.1.0 (/root/hello)
 Finished release [optimized] target(s) in 0.37s

ubuntu# ldd target/x86_64-unknown-linux-musl/release/hello
statically linked
```

You can see that "Hello World" is configured as a static link using the musl environment.

Now let's run "Hello World" built with both dynamic and static links by copying the binaries on CentOS and Alpine environments.
CentOS 8 uses glibc dynamic linking and Alpine Linux uses musl static linking.


### CentOS Container Environment

```console
$ docker run -it --name centos centos:centos8 bash
centos#
```

### Alpine Container Environment 
The Alpine distribution uses musl by default rather than glic.

```console
$ docker run -it --rm alpine:3.18
alpine#
```

Let's copy 'Hello World' into a glibc environment and a musl environment to see the behavior.

```console
$ docker cp ubuntu:/root/hello/target/x86_64-unknown-linux-musl/release/hello .
$ docker cp hello centos:/root/
$ docker cp hello alpine:/root/
```

Let's check the behavior on centOS. 
```console
centos# ./hello
Hello, world!
```

Let's check the behavior on alpine. 
```console
alpine# ./hello
Hello, world!
```


## Comparing glibc and musl using the Rust application 'slice' 

Let's take the Rust application 'slice' and compare the container images created with glibc and musl. 

The Rust implementation of 'slice', like Python's 'slice', is publicly available on the GitHub repository https://github.com/ChanTsune/slice. 'slice' is a tool that prints the contents of a file from the front or back, like 'head' or 'tail'.
For example, the command below will print lines 10 through 20 from 'file.txt'.

```console
$ slice 10:20 file.txt
```

When you build 'slice' in a Rust environment and create a container to use it, you can use it like this

```console
$ docker run -i --rm -v `pwd`:`pwd` -w `pwd` slice
```

Let's build a container using glibc in the Ubuntu 22.04 environment.



```dockerfile
FROM rust:latest as builder

WORKDIR /work
RUN git clone https://github.com/ChanTsune/slice /work/.
RUN cargo build --release
RUN strip /work/target/release/slice -o /slice

FROM ubuntu:22.04
COPY --from=builder /slice /usr/local/bin/

ENTRYPOINT ["slice"]
```

This time, we'll create a container image based on Ubuntu 22.04 using musl static links.


```dockerfile
FROM rust:latest as builder

RUN rustup target add "$(uname -m)"-unknown-linux-musl
WORKDIR /work
RUN git clone https://github.com/ChanTsune/slice /work/.
RUN cargo build --release --target "$(uname -m)"-unknown-linux-musl
RUN strip /work/target/"$(uname -m)"-unknown-linux-musl/release/slice -o /slice

FROM ubuntu:22.04
COPY --from=builder /slice /usr/local/bin/

ENTRYPOINT ["slice"]
```


Let's create a container image based on theAlpine distribution using a musl static link.



```dockerfile
FROM rust:latest as builder

RUN rustup target add "$(uname -m)"-unknown-linux-musl
WORKDIR /work
RUN git clone https://github.com/ChanTsune/slice /work/.
RUN cargo build --release --target "$(uname -m)"-unknown-linux-musl
RUN strip /work/target/"$(uname -m)"-unknown-linux-musl/release/slice -o /slice

FROM alpine
COPY --from=builder /slice /

ENTRYPOINT ["slice"]
```



If we compare the size of a glibc container image and a musl container image on Ubuntu 22.04 and a musl container image on Alpine, we can see that the container image with musl is smaller.


```console
$ docker images 
REPOSITORY TAG IMAGE ID CREATED SIZE
slice distroless-musl d38a74f8568a 11 seconds ago 3.52MB
slice alpine-musl e3abb5f0aace 39 seconds ago 8.4MB
slice ubuntu22.04-musl 467edd130e79 About a minute ago 78.9MB
slice ubuntu22.04-glibc 09fe5ad40d56 3 minutes ago 78.8MB
```


In the Ubuntu environment, using glibc or musl doesn't make much difference in the size of the container image, but in the Alpine distribution, you can see that the container image size is reduced by about a tenth.
This shows that by utilizing Alpine Linux with static builds, we can make our container images lightweight and reduce deployment time.

## Conclusion

Using static links in programs that use standard C libraries can simplify the process of deploying Linux binaries.
It also reduces the size of the container image compared to dynamic links, and makes deployment convenient regardless of the distribution.
When you replace glibc with musl, you benefit not only from the difference in container image size, but also from features newly supported by musl, such as mDNS (a multicast-DNS-based zero config system) and NUMA clusters.
Furthermore, if you use distroless, which is distributed by Google to better utilize musl, as your default container image, you can deploy and take advantage of smaller container images.

> This post is automatically translated from Korean

High sky and plump horses, and Container Dieting

# 23.03: July 2023 update
A wrap-up of the ongoing updates to Backend.AI 23.03 and 22.09. The development team is working hard to squeeze every last bit out.

Here are the most important changes in this update

- Enhanced storage manageability: Added per-user and per-project storage capacity management (quotas) with VFolder v3 architecture.
- Expanded NVIDIA compatibility: Support for CUDA v12 and NVIDIA H100 series.
- Extended hardware compatibility: Support for WARBOY accelerators from [FuriosaAI](https://www.furiosa.ai).

### Backend.AI Core & UI
* Supports CUDA v12 and NVIDIA H100 series.
* Supports the WARBOY accelerator, the first NPU from [FuriosaAI company](https://www.furiosa.ai).
* Added storage capacity management function (Quota) by user and project by applying VFolder v3 architecture.
  * However, it is limited to storage that supports Directory Quota.
* Fixed an error that caused multi-node cluster session creation to fail.
* Fixed an error where a compute session in the `PULLING` state was incorrectly labeled as `PREPARING`.
* Fixed an error in which the `CLONING` state was incorrectly displayed when cloning a data folder with the same name when multiple storage devices have the same folder.
* Improved the web terminal of a compute session to use zsh as the default shell if the zsh package is installed in the kernel image.
* Added the ability to know the health status of the (managed) storage proxy and event bus.

### Backend.AI FastTrack
* Added the ability to set `multi-node` cluster mode by task.
* Fixed an error where environment variables set in `.env` were not applied to the frontend.
* Fixed an error recognizing out-of-date when accessing with a mobile browser.
* Added a field to show the cause message when a task-specific error occurs.
* Fixed other editor-related issues.

Backend.AI is constantly evolving to provide a more powerful and user-friendly experience while supporting various environments in the ever-changing AI ecosystem. Stay tuned for more updates!   
Make your AI accessible with Backend.AI!

23.03: July 2023 Update

**Backend.AI** is a popular choice for developing these LLMs because of its ease of use in running large clusters and distributed processing. In fact, we get a lot of feedback and requests from customers, and today I'd like to share how we solved one of them.

On April 4, 2023, we received a report of an issue where an error occurs when running certain packages in the container environment provided by the [NGC Catalog](https://catalog.ngc.nvidia.com/containers)[^1] (NVIDIA GPU Cloud). The NGC Catalog is a list of containers[^2] with optimized environments for developing AI/ML, metaverse, and high-performance computing applications, and because it is operated and distributed directly by NVIDIA, it is highly trusted and considered the standard for CUDA environments in particular. Therefore, an issue with this environment represents a potential risk that many users will face in the future, and we have decided to address this issue as a high priority.

## Reproducing the problem

I first went through the process of reproducing the issue to determine the exact cause. In this case, I was running [ViperGPT](https://github.com/cvlab-columbia/viper)[^3] developed by Columbia University and encountered an error in a package called `bitsandbytes`. ViperGPT has a dependency on `bitsandbytes` as shown below.
```txt showLineNumbers title="requirements.txt"
accelerate==0.18.0
backoff==2.2.1
// highlight-next-line
bitsandbytes==0.38.1
cityscapesscripts==2.2.1
git+https://github.com/openai/CLIP.git
decord==0.6.0
dill==0.3.6
...
```

I was able to reproduce the problem by simply `importing` `bitsandbytes`.

> The execution environment used the [nvcr.io/nvidia/pytorch:22.05-py3](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) image.

```bash
$ pip install bitsandbytes # 0.37.1
$ python
>> import bitsandbytes
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA exception! Error code: OS call failed or operation not supported on this OS
CUDA exception! Error code: initialization error
CUDA SETUP: CUDA runtime path found: /home/work/data/miniconda3/envs/vipergpt/lib/libcudart.so
/home/work/data/miniconda3/envs/vipergpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
 warn(msg)
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /home/work/data/miniconda3/envs/vipergpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
/home/work/data/miniconda3/envs/vipergpt/lib/python3.10/site-packages/bitsandbytes/cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
 warn("The installed version of bitsandbytes was compiled without GPU support. "
```

The `bitsandbytes` traverses all the CUDA devices installed in the execution environment and checks their Compute Capability [^4]. We were supposed to check the number of CUDA devices installed in the execution environment using `libcuda.so` in the following way. We noticed that an error occurs when we call `cuDeviceGetCount()`[^5]. The error was 304 CUDA_ERROR_OPERATING_SYSTEM.

```python title="bitsandbytes/cuda_setup/main.py"
def get_compute_capabilities(cuda):
 """
 1. find libcuda.so library (GPU driver) (/usr/lib)
 init_device -> init variables -> call function by reference
 2. call extern C function to determine CC
 (https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__DEVICE__DEPRECATED.html)
 3. Check for CUDA errors
 https://stackoverflow.com/questions/14038589/what-is-the-canonical-way-to-check-for-errors-using-the-cuda-runtime-api
 # bits taken from https://gist.github.com/f0k/63a664160d016a491b2cbea15913d549
 """

 nGpus = ct.c_int()
 cc_major = ct.c_int()
 cc_minor = ct.c_int()

 device = ct.c_int()

 # highlight-next-line
 check_cuda_result(cuda, cuda.cuDeviceGetCount(ct.byref(nGpus)))
 ccs = []
 for i in range(nGpus.value):
 check_cuda_result(cuda, cuda.cuDeviceGet(ct.byref(device), i))
 ref_major = ct.byref(cc_major)
 ref_minor = ct.byref(cc_minor)
 # 2. call extern C function to determine CC
 check_cuda_result(cuda, cuda.cuDeviceComputeCapability(ref_major, ref_minor, device))
 ccs.append(f"{cc_major.value}.{cc_minor.value}")

 return ccs
```

## What is bitsandbytes?

Since the advent of [Transformer](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html), language models have shown high performance gains, and it has become a trend to increase the size of the model by stacking more Transformer blocks. This has led to a large number of GPU resources being required not only to train the model but also to service it. For example, to service GPT-3 with 175B parameters, eight 80GB A100 GPUs costing about $15,000 are required. This is a huge burden not only for individuals, but also for enterprises or research institutes, which is why there is a lot of research on lightweighting inference models for servicing.


 ![](https://cdn.lablup.com/230728_large_language_model_3205b7be28.png) 
Image source: [A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes (Hugging Face)](https://huggingface.co/blog/hf-bitsandbytes-integration)

[**bitsandbytes**](https://github.com/TimDettmers/bitsandbytes) has open-sourced [LLM.int8()](https://arxiv.org/abs/2208.07339)[^6], a work by Tim Dettmers, a PhD candidate at the University of Washington, with [Facebook AI Research](https://ai.meta.com) (now Meta AI). It has shown to reduce the size of the model while maintaining performance by applying a vector-wise quantization method that treats each vector independently when computing matrix products, and by using a mix of 8-bit and 16-bit techniques to minimize losses by representing important vectors in 16-bit. It has been merged into Hugging Face's Transformer implementation and is used in a variety of models including [Llama2] (https://github.com/facebookresearch/llama-recipes/blob/cd82118b74d2fd739bd6227af33b661d04a97406/requirements.txt#L6), [QLoRA] (https://github.com/artidoro/qlora/blob/6c6fc4653abd17ce550f48878a24c7bd8772e98a/requirements.txt#L1), [KoAlpaca] (https://github.com/Beomi/KoAlpaca/blob/4596f882957d286b4d60559b97dcf783822d23f5/webui/requirements.txt#L5), and [KULLM] (https://github.com/nlpai-lab/KULLM/blob/b7a78b62ed6cd9d83c51ad5a92a9dd40b9f35998/requirements.txt#L4).

## Identify the cause

Now that we've located and reproduced the problem, it's time to get to the bottom of it. I looked to see if there were any similar cases, but I couldn't find any. Also, `cuInit()` was called normally, making it even more difficult to pinpoint the cause.

```python
import ctypes

count = ctypes.c_int()

libcuda = ctypes.CDLL("libcuda.so")
libcuda.cuInit(0) # 0 (CUDA_SUCCESS)
libcuda.cuDeviceGetCount(ctypes.byref(count)) # 304 (CUDA_ERROR_OPERATING_SYSTEM)

libcudart = ctypes.CDLL("libcudart.so")
libcudart.cudaGetDeviceCount(ctypes.byref(count)) # 304 (CUDA_ERROR_OPERATING_SYSTEM)
```

I filed an issue on the GitHub repo ([TimDettmers/bitsandbytes#264](https://github.com/TimDettmers/bitsandbytes/issues/264)) for advice, and was told to update the package to the latest version and try again. After updating to version 0.38.0.post1, which was the latest at the time, I tested again, and the same problem occurred. I couldn't afford to lose too much time, so I decided to switch gears and remove the offending part.

 ![](https://cdn.lablup.com/axe_c24307c59a.jpeg) 
Image source: Greco-Roman Mythology in Comics (Ghana Publishers)

## Troubleshooting

My first approach was to use [CUDA-Python](https://github.com/NVIDIA/cuda-python)[^7]. CUDA-Python is the CUDA Python Low-Level Bindings package officially distributed by NVIDIA. I had used it before and found it useful, so I immediately thought of it and decided to install and test it.

```bash
$ pip install cuda-python
```

```python
from cuda import cuda
from cuda import cudart

cuda.cuInit(0) # (<CUresult.CUDA_SUCCESS: 0>,)
cudart.cudaGetDeviceCount() # (<cudaError_t.cudaSuccess: 0>, 1)
```

Fortunately, `cudart.cudaGetDeviceCount()` worked fine, and I proceeded to test integrating it into `bitsandbytes`. However, calling `torch.cuda.is_available()` after calling `cuda.cuInit(0)` resulted in an error. This was because I called `cudaGetDeviceCount()` inside `torch.cuda.is_available()`.

```python
from cuda import cuda, cudart

cuda.cuInit(0) # <CUresult.CUDA_SUCCESS: 0>,)
cuda.cudaGetDeviceCount() # (<cudaError_t.cudaSuccess: 0>, 1)

import bitsandbytes

# ...
# /opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:82: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:109.)
# return torch._C._cuda_getDeviceCount() > 0
# ...
```

The problem seemed to be back to square one. I took a breath and calmly reread the error log above. Then something caught my eye.

**torch._C._cuda_getDeviceCount() > 0**

Note that `bitsandbytes` was already using PyTorch internally, which means it had a dependency on PyTorch. To be precise, `bitsandbytes' had a [dependency](https://github.com/TimDettmers/bitsandbytes/blob/e229fbce66adde7c2a6bc58cbe7d57c1f4a0ba02/requirements.txt#L1) on [lion-pytorch](https://github.com/lucidrains/lion-pytorch), which had a [dependency](https://github.com/lucidrains/lion-pytorch/blob/66295196111875bde30c8060d4f0f76add98377f/setup.py#L19) on PyTorch. And PyTorch already had an interface to [CUDA functions](https://github.com/pytorch/pytorch/blob/cce2b7e3c95a7505b41bdfc53939d84d56e31260/torch/_C/__init__.pyi.in#L1529-L1530), which I decided to take advantage of this time.

Fortunately, all of the CUDA functions used by `bitsandbytes` existed in PyTorch. I made the following changes to the functions that were previously called via `libcuda.so` and `libcudart.so`.

<table>
 <thead>
 <tr>
 <th>libcuda/libcudart</th>
 <th>torch</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>libcuda.cuDeviceGetCount()</td>
 <td>torch.cuda.device_count()</td>
 </tr>
 <tr>
 <td>libcuda.cuDeviceGet()</td>
 <td>torch.cuda.device()</td>
 </tr>
 <tr>
 <td>libcuda.cuDeviceComputeCapability()</td>
 <td>torch.cuda.get_device_capability()</td>
 </tr>
 <tr>
 <td>libcudart.cudaRuntimeGetVersion()</td>
 <td>torch.version.cuda</td>
 </tr>
 </tbody>
</table>

After verifying that it worked after the change, I registered a PR in the GitHub repository ([TimDettmers/bitsandbytes#375](https://github.com/TimDettmers/bitsandbytes/pull/375)) to apply to the distribution package version.

## Postscript

On July 14, 2023, about two months after registering the PR, the patch was merged into the main branch and included in version [0.40.1](https://github.com/TimDettmers/bitsandbytes/blob/main/CHANGELOG.md#0401).

 ![](https://cdn.lablup.com/bitsandbytes_changelog_0_40_1_795b090d96.png) 
I was also able to get some feedback from the author, Tim Dettmers, whose thoughts and philosophy are evident in this short article.
 ![](https://cdn.lablup.com/timdettmers_approval_9f4db84e13.png) 
Through this opportunity, I was able to learn more about LLM's ecosystem. It was also the first time in a long time that I was able to feel the fun of open source activities. I think the appeal of open source activities is that we can collaborate beyond spatial constraints and learn from each other's ideas. We run an open source version of **Backend.AI** alongside an enterprise version. We will always strive to provide a better user experience and a better developer experience.

[^1]: NVIDIA GPU Cloud
[^2]: The NGC catalog hosts containers for AI/ML, metaverse, and HPC applications and are performance-optimized, tested, and ready to deploy on GPU-powered on-prem, cloud, and edge systems.
[^3]: [ViperGPT: Visual Inference via Python Execution for Reasoning](https://arxiv.org/abs/2303.08128), March 14, 2023.
[^4]: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability
[^5]: https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__DEVICE.html#group__CUDA__DEVICE_1g52b5ce05cb8c5fb6831b2c0ff2887c74
[^6]: [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339), November 10, 2022.
[^7]: https://developer.nvidia.com/cuda-python

> This post is automatically translated from Korean

Digging bitsandbytes issue

In this post, I'll share my experience as a developer at Lablup over the past 9 months.

## Table of Contents

- Motivation to apply
- From Intern to DevOps!
- rraft-py Development
- Open Source Contribution Academy Regional Sprint Backend.AI Mentoring
- Attending various conferences
- 2023 Open Source Contribution Academy
- Presenting at PyCon
- Conclusion

## Motivation to apply

Even before I joined Lablup, I knew that I wanted to have a career where I could continue to help others through the programs I develop, whether as a hobby or during work hours.

Open source was particularly appealing to me because it meant that not only could my code help others, but that they could freely modify and utilize it if they wanted to.

One of the things I realized after working on my own project, Arvis, for my graduation project, is that it's not really easy to keep a project going simply because it's something I love to do, as it keeps growing in size. I tried to plan and execute the project carefully from the beginning, but in the end, I realized that I underestimated the time and effort required to maintain the project.

In that regard, Lablup, which actively encourages and supports open source-related activities and even develops core parts of its source code as open source, was the company of my dreams.

## From Intern to DevOps!

The last three weeks of [My internship at OSSCA Lablup](https://jopemachine.github.io/2023/01/02/2022-Ossca-Lablup-Internship/)were spent studying and researching distributed systems, specifically implementing the Raft algorithm. Although my job title changed from intern to DevOps, I still felt like I was expanding on my internship learning, including Raft, to solve [issues I worked on during my internship](https://github.com/lablup/backend.ai/issues/415).

I've been involved in a variety of other activities that I'll mention below, but my main work at the company to date has been writing a Python binding of the Raft algorithm implementation to replace the existing distributed lock structure, including writing rraft-py, and thinking about how to integrate it with Backend.AI.

## [rraft-py](https://github.com/lablup/rraft-py) Development

rraft-py is a Python binding implementation of `tikv/raft-rs`, and you can read more about it in the [GitHub Readme](https://github.com/lablup/rraft-py) / [Wiki](https://github.com/lablup/rraft-py/wiki). I'll also be presenting some technical details on the topic in my PyCon 2023 KR talk next month, if you're interested.

For now, I'm going to focus on my **experience as a Lablup developer**, leaving aside the technical details of what I learned while developing rraft-py.

I had to think a lot about rraft-py because it was not just about fixing an issue in Backend.AI, but also about creating a separate project and integrating that project with Backend.AI.

Overall, there were several mile stones in the project, and I feel like I was able to move forward with the project with a little more stability after each mile stone. There was definitely a high sense of accomplishment each time, but there were also many times when I was frustrated because I realized later that the code I had initially written didn't work the way I intended. But Lablup allowed me the time to do these shoveling sessions, and I think I've gotten to where I am today because of the things I've learned that I would have otherwise dismissed as "shoveling".

 ![](https://cdn.lablup.com/2023_07_18_2_0fedfd92cb.png)
Results of running the rraft-py example code

There's still a long way to go to integrate rraft-py into Backend.AI, but the bottom line is that it's great to have the experience of thinking for yourself and making your own decisions as you continue to evolve your project, and for developers who like this kind of experience, Lablup could be one of the best options out there.

## [Open Source Contribution Academy Regional Sprints](https://www.oss.kr/notice/show/3d3945b6-d23e-4e42-94aa-d6d0243b60d8) Backend.AI Mentoring

While rraft-py development was my main focus, as it required more time than I had anticipated, I also had the opportunity to work on a variety of other projects.

One of the most memorable experiences was participating in the 1st Daegu Open Source Contribution Academy Regional Sprint as a Backend.AI mentor.

In fact, I participated as a mentor without a deep understanding of Backend.AI, and to make matters worse, the sprint period was only 2 days, so I was worried about many things.

In order to make sure that the mentees learn at least one thing and go home as satisfied as possible, I had to think about how to explain Backend.AI to those who don't know it at all, and how to build a development environment on different platforms (personally, I usually only develop on macOS + docker desktop environment, but some of the mentees were working on Windows environment, so I had to shovel while building the development environment). I had to think about a lot of things and prepare.

In conclusion, I was able to learn a lot more than I thought because I was unfamiliar with these processes, and the mentees followed along better than I thought, so I think it was a meaningful time for everyone to create more than one PR.

 ![](https://cdn.lablup.com/2023_07_18_3_2a66168649.jpg) 
The 1st Daegu Open Source Contribution Academy Regional Sprint


## Participation in various conferences

We had the opportunity to participate in various conferences and exhibitions such as AI Expo, AWS Summit, and Next Rise. It was great to learn how to explain Backend.AI to different types of people, and it was also interesting to see the different technologies of other companies.

 ![](https://cdn.lablup.com/2023_07_18_1_9b6f10270a.jpg) 

 AI EXPO KOREA 2023


## [2023 Open Source Contribution Academy](https://www.contribution.ac/2023-ossca)

As a company with an open source culture, Lablup actively participates in the Open Source Contribution Academy every year. This year, I also participated in the Open Source Contribution Academy, which encourages participation in various other projects besides the Backend.AI team, so I've been working on GlueSQL as a mentee.

I think this culture of freedom is very attractive to developers with a strong desire to grow.

(In addition to myself, there are two other people involved in other projects in the 2023 Contribution Academy).

## PyCon announcement

Based on my experience in developing rraft-py at my company, I was also given the opportunity to present at [2023 PyCon KR](https://2023.pycon.kr/session).

Personally, I'm a bit nervous because it's my first time presenting in public, but I'm doing my best to prepare. For those who are interested in presenting, I am looking forward to sharing not only the presentation materials but also the source code and work history through GitHub.

## Conclusion

Lablup is a company with a strong open source culture, encouraging participation in various open source and community-related events such as the Open Source Contribution Academy (https://www.oss.kr/contribution_academy) and PyCon, and giving developers the opportunity to take initiative in their work.

I hope to continue to participate, learn, grow, and contribute to open source activities of various nature at Lablup.

> This post is automatically translated from Korean

2023 Lablup DevOps Summer Retrospect

Sanghyeon Seo

# Learning from history

ChatGPT and other language giants that have gained traction in the last half-decade didn't just fall out of the sky out of nowhere. We've seen it many times throughout history, where a cumulative development of technology reaches an inflection point and rapidly transforms society. Sometimes, the path to that inflection point is strikingly similar for technologies that evolved in very different times and contexts.

---

Mankind has long wanted to fly - it's no coincidence that Civilization 6 has "Dream of Flying" as its theme song.

How did the Wright brothers realize their dream? In 1899, Wilbur Wright writes to the Smithsonian Institution and asks them to send him what is known about airplanes. After three months of reviewing the material he receives, Wilbur concludes that not much is known about flight, except that it's a problem. Plausible theories have turned out to be false and improbable theories have turned out to be true, he writes, so he can't believe anything he hasn't seen for himself.

What Wilbur wanted to know from his literature review was this: what do we need to know about flying? What of it is already known? What are the remaining problems that need to be solved? Surprisingly, Wilbur was able to answer all of these questions from his literature review, something his competitors were unable to do.

In a 1901 lecture, Wilbur summarized his conclusion: "There are three problems with flying. You need wings to make the airplane float, you need an engine to make the airplane go, and you need a way to control the airplane."[^1]

[^1]: [Some Aeronautical Experiments](https://www.wright-brothers.org/History_Wing/Wright_Story/Inventing_the_Airplane/Kitty_Hawk_in_a_Box/Some-Aeronuatical-Experiments-by-Wilbur-Wright.htm) (Wilbur Wright, 1901).

Wilbur saw that the wing problem and the engine problem had been solved to a certain extent, so he needed to solve the piloting problem. To solve the piloting problem, he needed an airplane, and to build an airplane, he needed to solve the piloting problem. Wilbur concluded that he could solve the problem of controlling an airplane by taking the problem of controlling a glider.

To test the glider, he needed high hills and strong winds, and they had to be sand dunes for the safety of the experimenters. In 1900, Wilbur requested data from the Weather Bureau to review the windiest places in the United States. The staff at the Kitty Hawk Weather Station wrote back that the beach next to the station was unobstructed and would be suitable for the experiment."[^2]

[^2]: [Letter from J. J. Dosher, Weather Bureau, to Wilbur Wright](https://www.weather.gov/mhx/FirstFlight), August 16, 1900.

The 1901 experiment was a disappointment: the wing didn't have enough lift. The Wright brothers had used data from the Auto Lilienthal to calculate the area of the wing, and they had become suspicious of its accuracy.

After analyzing their experimental data, they concluded that John Smeaton's value of the proportionality constant, which had been used without question for over 100 years, including by Otto Lilienthal, was incorrect.

In order to systematically analyze the lift of wings without the time-consuming and laborious glider experiments, the Wright brothers built a wind tunnel. Their analysis showed that the data from the Auto Lilienthal was correct, except for the incorrect value of the proportionality constant, but the wing used by the Auto Lilienthal was inefficient.

The 1902 glider that resulted from this analysis had a larger area to offset the revised value of Smithen's constant and a flatter shape to increase efficiency. (They changed the flatness of the wing from 1/12 to 1/24.) The new glider flew very well.

That's how the Wright brothers were able to make their historic first flight at Kitty Hawk in 1903.

![](https://cdn.lablup.com/first_flight_2aff237efc.jpg)

---

Humans have long wanted to talk to machines - countless science fiction novels and movies bear witness.

To create AI, OpenAI had to solve three problems. Computing infrastructure, models, and data. You can think of the computing infrastructure as the engine, the models as the wings, and the data as the controls.

To manage the compute infrastructure, OpenAI used Kubernetes, but it wasn't something they could just grab and go. When they hit 2,500 nodes, they ran into problems with the Linux kernel ARP cache overflowing,[^3] and when they hit 7,500 nodes, they had to fix a bug in Kubernetes to enable anti-affinity.[^4]

[^3]: [Scaling Kubernetes to 2,500 nodes](https://openai.com/research/scaling-kubernetes-to-2500-nodes), January 18, 2018.

[^4]: [Scaling Kubernetes to 7,500 nodes](https://openai.com/research/scaling-kubernetes-to-7500-nodes), January 25, 2021.

(Advertisement: Lablup's Backend.AI has already been used in practice on large clusters for AI training and inference, solving a number of scaling problems and implementing its own scheduling algorithms to support features like affinity and anti-affinity.)

The scaling law for AI is the equivalent of the lift equation for an airplane. Just as the lift equation describes the lift of a wing as the area of the wing, the lift coefficient, and the proportionality constant, the scaling law describes the loss of an AI model as the size of the model, the size of the data, and the power law exponent.

Just as the Wright brothers discovered that John Smeaton's constant of proportionality was 0.003, not 0.005, the power law exponent of scaling law was initially thought to be 0.73,[^5] but was actually found to be 0.50.[^6] The incorrect value was calculated because the learning rate was not adjusted for the size of the data.

[^5]: [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361), January 23, 2020.

[^6]: [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556), March 29, 2022.

OpenAI knew that control of the model was an important issue, so before we trained our first GPT, we were already working on reinforcement learning from human preferences,[^7] which we applied to the control of robots, reminiscent of the Wright brothers referencing bird flight for control of airplanes and first applying it to gliders instead of airplanes.

[^7]: [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741), June 12, 2017.

To apply this research to language models, human preference data was collected, resulting in InstructGPT.[^8] It's hard to know exactly because OpenAI hasn't published its research since GPT-4, but research is showing that it can learn not only from explicit feedback, but also from implicit feedback, such as users retrying to create or continuing and stopping conversations.[^9] If so, OpenAI could create a positive feedback loop where it improves its models to gather users, who then gather users to improve their models.

[^8]: [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155), March 4, 2022.

[^9]: [Rewarding Chatbots for Real-World Engagement with Millions of Users](https://arxiv.org/abs/2303.06135), March 10, 2023.

---

In this article, we've compared how humans went from flying to talking to machines, and we've seen some very similar patterns in the evolution of technology throughout history.

What other examples will we see in the future as AI technology advances and we work to make it more accessible to more people? Can Rableup and Backend.AI help accelerate that process, allowing people to experiment and realize what we've learned from history more quickly? We're in the middle of this inflection point.

> This post is automatically translated from Korean


Learning from history

A recap of the ongoing updates to Backend.AI 23.03 and 22.09. The development team is working hard to squeeze every last bit out.

Here are the most important changes in this update:

- Expanded hardware compatibility: Expanded hardware compatibility with support for ATOM accelerator idle checking and Dell EMC storage backends from [Rebeillons](https://rebellions.ai/).
- High-speed upload enhancements: Introduced SFTP functionality to support high-speed uploads to storage.
- Development Environment Enhancements: Enhanced the development environment by allowing sessions to be accessed in remote SSH mode from local Visual Studio Code.
- Increased manageability: Improved the user interface for administrators to make it easier to set up AI accelerators and manage resource groups.


## Backend.AI Core & UI
* Added support for idle state checking of ATOM accelerators.
* Introduced SFTP functionality to support high-speed uploads directly to storage.
* Added ability to force periodic password updates based on administrator settings.
* Added an upload-only session (SYSTEM) tab.
* Added Inference type to the allowed session types.
* Added the ability to connect to a session in remote SSH mode from local Visual Studio Code.
* Added support for uploading folders from Folder Explorer.
* Improved the display of the amount of shared memory allocated when creating a session.
* Added support for Dell EMC storage backend.
* Improved the accuracy of container memory usage measurement.
* Improved the ability to run multiple agents concurrently on a single compute node.
* Added project/resource group name filter for administrators.
* Added user interface for administrators to set various AI accelerators, including GPUs, in resource presets/policies.
* Added a user interface for administrators to display the allocation and current usage of various accelerators, including GPUs.
* Added a user interface for administrators to set the visibility of resource groups.
* Provided a user interface for administrators to view the idle-checks value per session.
* Added recursion option when uploading vfolders in the CLI, and improved relative path handling.
* Added a recursive option in the CLI to terminate sessions with dependencies on specific session termination at once.
* Added a new mock-accelerator plugin for developers, replacing the old cuda-mock plugin.
* Added status and statistics checking API for internal monitoring of the storage proxy for developers.

## Backend.AI FastTrack
* Improved searching for vfolders by name when adding pipeline modules.
* Added an indication to easily recognize success/failure after pipeline execution.

## Backend.AI Forklift
* Bug fixes and stability improvements.
* Support for deleting build job history.
* Supports pagination of the build task list.

Backend.AI is constantly evolving to support a variety of environments in the ever-changing AI ecosystem, while providing a more robust and user-friendly experience. Stay tuned to see what's next!

Make your AI accessible with Backend.AI!

> This post is automatically translated from Korean

23.03: May 2023 Update

# Introduction

As super-sized AI models flood the market, there is a growing concern about not only developing the models, but also how to deliver them "well" and "efficiently" to users.
Prior to Large Language Models (LLMs), the computing power of AI models was focused on training rather than inference, as the hardware requirements for attempting to make inferences with a trained model were much smaller than the computing power needed to train the model. Deployers of models could get enough power for inference from the NPU of a real user's end device (such as a smartphone). However, with the advent of LLMs, the tables were turned.

![](https://cdn.lablup.com/4_c3523b4fd6.png) 

Take Meta's [OPT 175b] (https://github.com/facebookresearch/metaseq) as an example: OPT-175b, as its name implies, has 175 billion parameters and requires roughly 320+ GB of GPU memory just to load them onto the GPU to perform inference tasks. That's a huge difference from the 4GB that pre-LLM image processing models used to require.  
With this change in AI model behavior, efficiently managing service resources has become paramount to keeping your service running reliably. In this article, we'll preview Backend.AI's upcoming model service feature, Backend.AI Model Service, and show you how it will allow you to efficiently run your AI model from training to serving with a single infrastructure.

# Backend.AI Model Service

![](https://cdn.lablup.com/Model_Service_Architecture_2a72d506ea.png) 

Backend.AI Model Service is a model serving system that runs on top of the existing Backend.AI solution. It takes Backend.AI's tried-and-true container management technology and container app delivery system, AppProxy[^1], to the next level, enabling both AI training and model service in one infrastructure without installing additional components and by simply upgrading the existing Backend.AI infrastructure. It also supports an auto-scaling feature that automatically scales up and down inference sessions based on per-session GPU usage, number of API calls, or time of day, allowing you to effectively manage AI resources used for inference.

# Inference Sessions

![](https://cdn.lablup.com/Inference_Sessions_b5e36f4c15.png) 

Inference sessions in Backend.AI are conceptually the same as traditional training sessions. You can use the same execution environment you've been using for training for inference sessions, or you can deploy a dedicated execution environment just for inference sessions. Inference sessions are volatile and stateless, so you can terminate them at any time if the session is not performing well. In this case, Backend.AI will attempt to recover the original state by creating a new inference session, while simultaneously forwarding inference requests to other living inference sessions to minimize downtime for the inference service.

# Model storage

Models to be served through Backend.AI are managed as "model storage" units. Model storage consists of model files, code for model services, and model definition files.

## Model definition file

The model definition file is where you define the information for running a service provider's model in the Backend.AI Model Service. The model definition file contains information about the model, the ports exposed by the model service, and a set of tasks that must be executed to run the model service. If your model service provides a health check feature that reports its own health, you can use that information to take action, such as excluding sessions from the service if they are in bad health.

```yaml
models:
  - name: "KoAlpaca-5.8B-model"
    model_path: "/models/KoAlpaca-5.8B"
    service:
      pre_start_actions:
        - action: run_command
          args:
            command: ["pip3", "install", "-r", "/models/requirements.txt"]
      start_command:
        - uvicorn
        - --app-dir
        - /models
        - chatbot-api:app
        - --port
        - "8000"
        - --host
        - "0.0.0.0"
      port: 8000
      health_check:
        path: /health
        max_retries: 10
```

Here is an example of a well-defined model definition file, which contains a set of steps to run the [KoAlpaca 5.8B model](https://huggingface.co/beomi/KoAlpaca-Polyglot-5.8B/) as a model service.

# Tutorial: Model Service with Backend.AI Model Service

In this tutorial, we'll actually use Backend.AI to service a [KoAlpaca 5.8B model](https://huggingface.co/beomi/KoAlpaca-Polyglot-5.8B/) quantized to 8 bits.

## Write the API server code

Write a simple API server to serve the model.

```python
import os
from typing import Any, List

from fastapi import FastAPI, Response
from fastapi.responses import RedirectResponse, StreamingResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
import numpy as np
from pydantic import BaseModel
import torch
from transformers import pipeline, AutoModelForCausalLM
import uvicorn

URL = "localhost:8000"
KOALPACA_MODEL = os.environ["BACKEND_MODEL_PATH"]

torch.set_printoptions(precision=6)

app = FastAPI()

model = AutoModelForCausalLM.from_pretrained(
    KOALPACA_MODEL,
    device_map="auto",
    load_in_8bit=True,
)


pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=KOALPACA_MODEL,
)


class Message(BaseModel):
    role: str
    content: str


class ChatRequest(BaseModel):
    messages: List[Message]


BASE_CONTEXTS = [
    Message(role="맥락", content="KoAlpaca(코알파카)는 EleutherAI에서 개발한 Polyglot-ko 라는 한국어 모델을 기반으로, 자연어 처리 연구자 Beomi가 개발한 모델입니다."),
    Message(role="맥락", content="ChatKoAlpaca(챗코알파카)는 KoAlpaca를 채팅형으로 만든 것입니다."),
    Message(role="명령어", content="친절한 AI 챗봇인 ChatKoAlpaca 로서 답변을 합니다."),
    Message(role="명령어", content="인사에는 짧고 간단한 친절한 인사로 답하고, 아래 대화에 간단하고 짧게 답해주세요."),
]


def preprocess_messages(messages: List[Message]) -> List[Message]:
    ...


def flatten_messages(messages: List[Message]) -> str:
    ...


def postprocess(answer: List[Any]) -> str:
    ...


@app.post("/api/chat")
async def chat(req: ChatRequest) -> StreamingResponse:
    messages = preprocess_messages(req.messages)
    conversation_history = flatten_messages(messages)
    ans = pipe(
        conversation_history,
        do_sample=True,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        return_full_text=False,
        eos_token_id=2,
    )
    msg = postprocess(ans)

    async def iterator():
        yield msg.strip().encode("utf-8")

    return StreamingResponse(iterator())


@app.get("/health")
async def health() -> Response:
    return JSONResponse(content={"healthy": True})


@app.exception_handler(404)
async def custom_404_handler(_, __):
    return RedirectResponse("/404.html")


app.mount(
    "/",
    StaticFiles(directory=os.path.join(KOALPACA_MODEL, "..", "chatbot-ui"), html=True),
    name="html",
)
```

## Create a model definition file

Create a model definition file for your API server.

```yaml
models:
  - name: "KoAlpaca-5.8B-model"
    model_path: "/models/KoAlpaca-Ployglot-5.8B"
    service:
      pre_start_actions:
        - action: run_command
          args:
            command: ["pip3", "install", "-r", "/models/requirements.txt"]
      start_command:
        - uvicorn
        - --app-dir
        - /models
        - chatbot-api:app
        - --port
        - "8000"
        - --host
        - "0.0.0.0"
      port: 8000
      health_check:
        path: /health
        max_retries: 10
```

> In a session of the model service, model storage is always mounted under the `/models` path.

## Prepare model storage

![](https://cdn.lablup.com/Model_Storage_0d407ef5ac.png) 

Add the model API server code you wrote, the model definition file, and the KoAlpaca model to your model storage.

## Create a model service

![](https://cdn.lablup.com/New_Model_Service_7d395df4c7.png)

With both the model file and the model definition file ready, you can now start the Backend.AI Model Service. The Model Service can be created using the `backend.ai service create` command in the Backend.AI CLI. The arguments accepted by `service create` are almost identical to the `backend.ai session create` command. After the image to use, you pass the ID of the model storage and the number of inference sessions to initially create.

![](https://cdn.lablup.com/Model_Service_Status_03d7db4b4c.png) 

Using `backend.ai service info`, you can check the status of the model service and the inference sessions belonging to the service. You can see that one inference session has been successfully created.

## Use the Reasoning API

![](https://cdn.lablup.com/Inference_API_39bb05ce77.png) 

You can use the `backend.ai service get-endpoint` command to see the inference endpoint of a created model service. The inference endpoint continues to have a unique value until a model service is created and removed. If a model service belongs to multiple inference sessions, AppProxy will distribute requests across the multiple inference sessions.

## Restricting access to the Reasoning API

![](https://cdn.lablup.com/API_Token_151e1468c1.png) 

If you want to restrict who can access the inference API, you can enable authentication for the inference API by starting the model service with the `--public` option removed. Authentication tokens can be issued with the `backend.ai service generate-token` command.

## Scaling inference sessions

![](https://cdn.lablup.com/Model_Serivce_Scaling_dda4ec0eb5.png) 

The `backend.ai service scale` command allows you to change the scale of inference sessions belonging to the model service.

# Closing thoughts

So far, we've learned about Backend.AI Model Service and how to actually deploy a model service with the Model Service feature. Backend.AI Model Service is targeted for general availability in Backend.AI 23.03. We're working hard to make the Model Service feature publicly available in the near future, so stay tuned.

---]

[^1]: Available from Backend.AI Enterprise.

> This post is automatically translated from Korean


Sneak Peek: Backend.AI Model Service Preview

# 23.03: 2023년 3월 업데이트

We're excited to announce version 23.03.0, the first major release of [Backend.AI](https://backend.ai) for 2023.
Some features will continue to be rolled out in subsequent updates.

Specifically in this update:
- Support for the 'inference' service with a new computation session type.
- Support for 'model' management with a new storage folder type.
- Support for managing storage capacity on a per-user and per-project basis.
- Significant improvements to FastTrack's pipeline versioning and UI.

## Backend.AI Core & UI (23.03)
* Added model management and inference session management capabilities.
  - More advanced inferential endpoint management and network routing layers will be added in subsequent updates.
* The codebase has been updated to be based on Python 3.11.
* Introduced React components to the frontend and leveraged Relay to introduce a faster and more responsive UI.
* Full support for cgroup v2 as an installation environment, starting with Ubuntu 22.04.
* Updated the vfolder structure to v3 for storage capacity management on a per-user and per-project basis.
* Kernel and sessions are now treated as separate database tables, and the state transition tracking process has been improved to work with less database load overall.
* Improved the way the agent displays the progress of the image download process when running a session.
* Improved the display of GPU usage per container in CUDA 11.7 and later environments.
* Scheduling priority can be specified by user and project within each resource group.
* Supports two-factor authentication (2FA) login based on one-time password (TOTP) to protect user accounts.
 ![](https://cdn.lablup.com/5_ec31924326.png) 
* Support for users to register their own SSH keypair for session access.
* Supports user interfaces for Graphcore IPUs and Rebellions ATOM devices.

## Backend.AI Forklift (23.03)
* Added Dockerfile templates and advanced editing capabilities.
* Support for creating container images for inference.
* Extended image management capabilities to work with the Harbor registry.

## Backend.AI FastTrack (23.03)
* Storage folder contents can be viewed directly from the FastTrack UI.
* Improved session state synchronisation with Core to event-based.
* You can set the maximum number of iterations for a pipeline schedule.
* If a task fails to execute, the pipeline job is automatically cancelled instead of waiting.
* Added pipeline versioning. You can track the shape history of your pipeline, and you can recall the contents at a specific point in time to continue working on it.
* You can modify pipelines in YAML format directly through the code editor.

## 개발 및 연구 프레임워크 지원
* Supports TensorFlow 2.12, PyTorch 1.13
* Support for NGC (NVIDIA GPU Cloud) TensorFlow 22.12 (tf2), NGC PyTorch 22.12, NGC Triton 22.08
* Added python-ff:23.01 image, which provides the same libraries and packages as Google Colab

In addition to what we've listed above, we've included many bug fixes and internal improvements.  
Stay tuned for more to come!

> This post is automatically translated from Korean

23.03: March 2023 Update

Jongmin Kim

> The content of this post is based on my own personal experience, which is not necessarily true in all cases, and opinions may vary.

## What is DevRel?
![](https://cdn.lablup.com/teaming_7ef724c5f8.jpg)

Image by <a href="https://pixabay.com/users/henning_w-327613/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=386673">Henning Westerkamp</a> from <a href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=386673">Pixabay</a>


As the name implies, DevRel's primary purpose is to create relationships with developers. This includes not only the developers who write the code, but also the planners, designers, and all the production people involved in creating services and products.
The specific role of a DevRel varies a lot depending on the nature of the organisation and the industry, but often the objectives converge to promote (product, technology) or recruit.

The primary roles of a DevRel are to
- Product promotion and **technical evangelism**
- Build, run, and support the **community**
- Gathering user input and **feeding back** into product development
- **Provide a variety of resources** to help users better use the product

Developer Advocate 은 DevRel 안에서도 **기술 전파** 및 **리소스 제공** 에 조금 더 초점을 맞추고 있는 역할이라고 볼 수 있습니다.

Developer Advocate can be seen as a role within DevRel that is a little more focused on skills **evangelism** and **resource provision**.

## Why DevRel?

In order to make their products and services known to the public, companies conduct marketing and public relations activities through various channels and events. These activities are called PR - Public Relations, which literally means creating a relationship with the public.
In companies that develop or service IT products, the main consumers are engineers (developers). And products aimed at engineers often need to provide more detailed and technical information, which requires different resources and strategies than general PR activities. DR - Developer Relations, as the name suggests.

The community is arguably the most important driver of an IT ecosystem, especially for open source products like Backend.AI.
Moreover, when new technologies of a similar kind come along, the size of the community is often a factor in choosing and adopting them, in addition to performance or maturity. One of the important roles of DevRel is to help and engage with these communities.

A great analogy I like to use to explain the importance of community is the relationship between an artist and their fan club.
An artist's primary role is to create music and perform, but it's through their fanbase that viral and secondary creations are born, and it's an important driver of their career and the entertainment business.
Nowadays, entertainment companies are well aware of the importance of fandom and support various activities.

 ![](https://cdn.lablup.com/kpop_fan_6c9ba87f2a.jpeg) 
Image created by <a href="https://www.midjourney.com/showcase/recent/">Midjourney</a>

Similarly, the DevRel role in an IT organisation is to support and communicate with the various communities and ensure that the product and organisation don't lose focus.

## Developer Advocate in Lablup

Lablup has been active in the Korean developer community for a long time by Wooyoung Yoo, and our CEO, Jeongkyu Shin, is also active in various developer communities, so we have already done a lot of DevRel activities for an organisation of our size. Backend.AI is a great piece of software, and it's an important tool, especially in this era of big AI. However, as a tool for AI learning and inference, it requires a lot of preparation and resources to set up, which can be daunting, especially if you're new to AI (which I am 😅).

Since Backend.AI does not have a large user base yet, we have been conducting various community activities such as the [Open Source Contribution Academy] (https://www.oss.kr/contribution_academy) in conjunction with the theme of open source culture. In order for our own community to grow, information needs to be produced and disseminated among users in addition to the one-way transmission of technology from creators to users. In order for such activities to take place, it is my main task to continue to create and improve various resources and exchange opportunities that anyone can easily use. I will continue to strive to communicate with you in various opportunities and ways.

> This post is automatically translated from Korean

Developer Advocate in Lablup

Jongeun Lee

Backend.AI's MLOps platform, FastTrack is using React 18. We will explore the differences between rushed and non-rushed renders enabled by the Concurrent Renderer in React 18.

The Concurrent feature in React, initially introduced as Async Rendering at JSConf Iceland in 2018, was not fully integrated into React 18 until the year 2022. As you might expect from this time period, the Concurrent Renderer is the biggest and most significant change in React 18. Even though the renderer has been updated, React developers can still run code written for versions before React 18 with minimal changes. It is possible to build user interfaces with React without knowledge of React's Concurrent Renderer. Understanding the Concurrent Renderer and its applications can simplify the complexities in your mind during React development, enabling you to create user interfaces(UI) that provide an enhanced user experience(UX). This article will not delve into the inner workings of the Concurrent Renderer. Instead, it will focus on defining what the Concurrent Renderer is and how it can transform the mindset of React developers, which is crucial for those creating applications using React.

To summarize the content of this article, here's what you need to know:

Because of the Concurrent Renderer,

* Component rendering can be interrupted.
* Parts of the tree can be rendered even when they are not visible on the screen.
* This allows React developers to distinguish between non-rush renders like never before.

> **“React components are abstractly pure functions.”** 
React components are actually created as JavaScript functions. (You can also create it as a class, although this is generally not advised in most situations.) A function generates an output based on the provided input. Changing the input can alter the output, hence it is necessary to execute the function again to generate a new result. (A pure function consistently returns the same output for identical inputs.)

> **What are the inputs and outputs of a React component?** 
The inputs of a React component are known as properties, or 'props', which the component receives as a function. The outputs are the React elements that are returned by the function.
 
> **Is state via hooks also an input?** 
'hooks' can be conceptually understood as inputs to a function. Similar to React props, they act as triggers that prompt a re-render when their values change, leading to variations in the output of our React component.

Now, back to the topic of rendering. 

## Component rendering can be interrupted.
The essence of Concurrent React lies in the ability to interrupt rendering, a feature unavailable before React 18(except in experimental form). In previous versions, when a React component began rendering, all JavaScript operations were blocked until the rendering completed. This means that if the rendering function takes a long time, the event handler function that handles the user's click cannot be executed until the element is returned. However, with React 18, rendering can now be interrupted.

```shell
const A = ({ count }) => {
 return (
 <div>
 {count}
 
 <C/>
 </div>
 );
};

const B = () => {
 const [text, setText] = useState("");
 return (
 <div>
 <input value={text} onChange={(e) => setText(e.target.value)} />
 <D/>
 </div>
 );
};

const C = () => {
 return C;
};

const D = ({ text }) => {
 verySlowFunction(text); //Consider this function that takes a few seconds to compute.
 return D;
};
```

In earlier versions of React 18, rendering component A necessitated the rendering of components B and C, and component D had to be rendered for B. No other JavaScript operations could be performed until A's return value, a React element, was returned. The component tree that A returned was rendered as a single block, and it was not possible to interrupt A's rendering once it had begun.

In Concurrent React, it is possible to interrupt the rendering process. Why is it necessary to interrupt rendering? You can think of the following:

* When the current render in progress is no longer valid(stale)
 * For example, consider the situation in the code above where A's `count` prop is rendering with a value of 1. Before this render completes, `count` changes to 2, resulting in a render request for A on day 2. Consequently, the rendering result from day 1 becomes obsolete as it does not reflect the most recent value. By halting the rendering of day 1 and promptly beginning the rendering of day 2, you can present the user with the most recent value quickly.
* When you have something you want to do before the ongoing render updates the screen you want to show.
 * When a user event occurs during rendering, it's possible to halt the ongoing render and give precedence to the event handler for an immediate response.

These are all cases where you're improving the UX by stopping rendering that component so it can do something else.

## It is possible to render sections of the tree that are not visible on the display.
Concurrent React enables you to render components corresponding to specific screen areas separately, in addition to what is currently visible on the screen. This feature allows the existing render to remain visible and functional while independently rendering a future screen update in advance, swapping it in once rendering is complete. Concerns may arise about rendering more than necessary and reducing usability. Yet, thanks to the Concurrent Renderer, this separate rendering process can be halted at any moment, ensuring it does not disrupt user interactions. Ultimately, this capability can enhance the user experience.

So far, we've seen two features of the Concurrent Renderer, and now we'll see how they are utilized to “distinguish between non-rush renders”.

## Distinguish between non-rush renders

> **Examples of rushed and non-rushed renders** 
>&nbsp; 
Consider the experience of visiting your website for the first time via a browser. What's the most urgent thing you need to do when you're faced with a white, blank screen? The most critical action is to display your site's content promptly. If the screen remains white for an extended period, users may not wait and leave. Therefore, it's essential to prioritize rendering the initial content quickly. 
>&nbsp; 
> 
> On the left sidebar of your homepage, there is a set of menus for navigation. If a user intends to select Menu A but accidentally selects Menu B, and then attempts to select Menu A again while Menu B is still loading, the screen for Menu B will complete rendering before the screen for Menu A appears. 
>&nbsp; 
> 
> If we consider such user pressed menu B and then pressed menu A immediately. it is more urgent to render the screen for Menu A than it is to render the screen for Menu B, because the screen for B is now invalid. 


As a React developer, you can inform React about non-rush renders by specifying which input changes that trigger a render are not pressing. The hooks that facilitate this for developers are `useDeferredValue` and `useTransition`. Both APIs, introduced in React 18, serve to defer non-rush rendering. We will examine these two hooks individually to grasp the distinctions between them.

## `useDeferredValue`: Separate using a changed input value
It is used by components that use a specific value and want to handle changes to that specific value in a non-rush manner.

```shell
function App() {
 const [text, setText] = useState('');
 return (
 <>
 <input value={text} onChange={e => setText(e.target.value)} />
 <SlowList text={text} />
 </>
 );
}
```
The example code above is one of the `useDeferredValue` examples from [beta.reactjs.org](beta.reactjs.org).

In this scenario, `text` serves as a state variable; thus, any changes to `text` will cause the `App` to re-render. The same `text` is also passed as a prop to both `<input>` and `<SlowList>`. Consequently, when `text` is modified, it initiates a re-render of `App`, and as part of this process, both `input` and `SlowList` will update to reflect the new `text`. However, if `SlowList` has a lengthy render time, the user's input will not appear until the rendering is fully completed, regardless of how fast the user types.

In this scenario, `input` represents the user's keyboard input, which is rendered quickly, while `SlowList` is a result of the user's input and is rendered more slowly than `input`. We can utilize `useDeferredValue` to generate a `deferredText`, which will be displayed during a non-rush render, with `text` initiating an rush render.

```shell
function App() {
 const [text, setText] = useState('');
 const deferredText = useDeferredValue(text);
 return (
 <>
 <input value={text} onChange={e => setText(e.target.value)} />
 <SlowList text={deferredText} />
 </>
 );
}
```

In this manner, when the `text` value changes, `deferredText` immediately retains the previous `text` value. Concurrently, `deferredText` undergoes a separate offscreen rendering with the new value. Only after this rendering is complete, both `text` and `deferredText` update to the latest value. The rendering of `deferredText` is not a rushed process and can be halted.

If there be successive non-rushed render requests for the same component, the initial non-rushed render will cease and commence rendering the most recent change, provided it has not concluded. For instance, with `text`, if a user inputs 'A' followed by 'B' in quick succession into an empty input field, the render for 'A' will initiate. If 'B' is entered before the rendering of 'A' concludes, the render for 'A' will stop, and the rendering for 'AB' will begin.

## `useTransition`: Separate using a function that changes the input
Previously, we discussed how both `useTransition` and `useDeferredValue` help manage non-urgent renderings. Now, let's explore the distinctions between the two and delve into `useTransition`.


> :warning: **CAUTION**
>
> To clarify the distinction, the example of `useDeferredValue` has been altered to demonstrate `useTransition`. It's important to note that `useTransition` is not compatible with `input`, as it necessitates synchronous updates. For an explanation of this limitation, refer to the [Troubleshooting section](https://react.dev/reference/react/useTransition#troubleshooting) on the `useDeferredValue` page at beta.reactjs.org.

```shell
function App() {
 const [text, setText] = useState("");
 const [isPending, startTransition] = useTransition();
 return (
 <>
 <button
 onClick={(e) => {
 startTransition(() => setText((v) => v + "a"));
 }}
 >
 a 키
 </button>
 <SlowList text={text} />
 </>
 );
}
```

### Different things:
If `useDeferredValue` utilizes its value, `text`, to specify a non-urgent render, then it employs `setText`, which alters the value and triggers a render. In cases where `text` is not available, understanding `setText` alone is sufficient.

It is not possible to instantly display the change in `text` as it occurs within `startTransition`. A distinct render will initiate for the updated `text`, but as it is a separate process, the actual screen render won't recognize the updated value, though it will be aware that the separate render is underway through `isPending`. The `useTransition` hook delays the change in `state`, and the `useDeferredValue` hook postpones certain renderings based on the altered `state`.

### Common things:
If multiple non-rush render requests for the same component are made through `startTransition`, the initial render—similar to `useDeferredValue`—will be canceled if it's still ongoing, and a new render with the latest value will commence.

## Wrapping
React 18's Concurrent Renderer introduces the ability to "distinguish between non-rush renders." Utilizing useTransition and useDeferredValue, it allows for updates to complex structures without compromising usability. Prior to React 18, achieving such seamless usability demanded significant development work. Now, with the streamlined process of "distinguishing between non-rush renders," developers can offer users a smooth user experience.

Concurrent React Changed Everything: Distinguishing Renders That Aren't Rushed

Sion Kang

## Introduction
My first encounter with 'Lablup' was in the summer of 2019. At that time, I attended a GDG Seoul 'Everyone's Toy Story' event because someone I knew was presenting there. I had the opportunity to sit in on a presentation by Lablup about their GPU virtualization tools for machine learning, and that piqued my interest. At that time, my fascination with machine learning was at its peak, and the technical depth of their presentation was impressive—it was my introduction to a company pioneering in this field.

 ![](https://cdn.lablup.com/202424596_37015a5e_8c90_4de4_af98_d9bb60047361_1389ca8bcb.png) 

Subsequently, I reconnected with the company at the 42 Seoul open source hackathon. This was an event focused on developing a product using a designated open source within a limited timeframe, where I became a part of the Backend.AI team. Despite it being three years since the initial presentation at GDG Seoul, the impression it left was so profound that the company's name was instantly familiar to me. Throughout the contest, the guidance provided by Jungkyu (CEO of Lablup) was instrumental, contributing significantly to our achievement being a second place at a hackathon.

In May 2022, I was working on a school-based internship at a company called SATREC INITIATIVE. As my internship was drawing to a close, I came across an announcement for Lablup's summer internship on Facebook. Recalling the valuable mentorship from Mr. Shin during a competition, and with a keen interest in the developer community and open source, I chose Lablup as my next destination.

At that period, I was engaged in a project named 42 World, which was a significant phase of learning and growth for me. In my interview with Lablup, I detailed my experiences with the 42 World project, particularly the challenges I faced while implementing a MonoRepo. Interestingly, Lablup had encountered similar issues with MonoRepo in Backend.AI, allowing us to exchange a sense of empathy as developers during our conversation.

After being accepted into the Lablup internship, I started my internship with four other interns. I joined the company about a week later than the others to give myself time to finish my existing internship and relocate. My first week was dedicated to orientation, familiarizing myself with Backend.AI, and acclimating to the company's culture. The onboarding documentation was comprehensive, contributing to a welcoming environment for newcomers to quickly adapt. A significant portion of my orientation involved installing and configuring Backend.AI. Since I began a week after the other interns, they were able to assist me, which made the orientation process relatively smooth.

## Getting to my work
During the second week, I began tackling the actual tasks. I opted to collaborate with the DevOps, Frontend, and Research teams. Each chapter leader provided me with a 'good-first-issue' to address. My choice was the DevOps team. My initial task involved refactoring the 'run' command, which initiates a session and runs the specified code. The goal was to integrate 'start', to launch the session, and 'exec', to execute the code, thereby minimizing redundant code.

 ![](https://cdn.lablup.com/202433613_91dd0878_8d6d_480a_b0ec_e3f637d7169b_6856417e93.png) 

I faced challenges with the first issue assigned to me as it required a thorough understanding of Backend.AI's repository structure, irrespective of the implementation difficulty. I realized that while it's important to know why an issue exists, it's also important to know exactly what Backend.AI is trying to accomplish and understand how the issue needs to be solved to achieve that goal so that I can solve it correctly.

After addressing the initial issue, I took on the task of testing a feature in development known as vfolder clone. My role was solely focused on DevOps, providing me with my inaugural experience with Backend.ai-webui, a Frontend chapter project. My involvement wasn't limited to just testing the vfolder clone; I also executed it personally, identifying areas for improvement and bugs, which I then documented as issues. This made me feel somewhat guilty, as it seemed I was generating work for other teams. However, the Frontend chapter's encouragement to contribute assuaged my concerns. This experience reaffirmed my understanding that open-source contributions extend beyond code; there are numerous ways to contribute.

 ![](https://cdn.lablup.com/202441600_3da0d632_abda_4953_ab42_c8c5379dbab2_82a74cd001.png) 

## Improving CI/CD

As I've always been interested in CI/CD, I was intrigued by the actions utilized in Backend.AI. At the time, Backend.AI could bypass CI with the `skip:ci` tag, but I noticed that neither `skip:ci` nor `skip:changelog` tags would work if the PR was labeled after creation, necessitating an extra commit. Since external contributors lack label permissions and Backend.AI is open source, resolving this seemed crucial. After exploring GitHub Actions, I discovered a trigger for labeling, which allowed me to address the issue. My proactive approach in finding and fixing problems independently, rather than through assigned tasks, was highly appreciated by the company. This experience deepened my interest in actions, leading to further enhancements. Noticing frequent omissions in assignments, I proposed the auto-auth-assign action for automation, which I had previously utilized. Subsequently, I considered automating the labeling process. Although new to the labeler action, I conducted multiple tests in a test repository before implementing it in Backend.AI, successfully automating the labeling of various systems integrated into a monolith. During this process, I identified the benefit of linking labels from issues to PRs. Unable to find an existing action, I took the initiative to create the auto-label-in-issue action, learning the GitHub API and action scripting in the process.

## Finishing my Internship
This internship has been a significant learning experience for me. Although it's my second internship, it's my first within an IT company, as my previous employer wasn't in this sector. What draws me to Lablup is its commitment to open-source products and the active contributions to the community. The company's horizontal structure allowed me to freely share my thoughts, making me wonder if a company could really operate so collaboratively. One of the greatest aspects of Lablup is the autonomy it offers, allowing you to pursue what you're passionate about rather than what you're obliged to do.

Having developed a strong understanding of the project, I was disappointed when the internship neared its conclusion. Fortunately, Lablup offered me the opportunity to extend my internship, which I accepted, and continued to work on action issues. As there are few developers specializing in this area, I recently presented on the subject at GDG Daejeon. This led to the amusing nickname "Action Mask" from my colleagues.

 ![](https://cdn.lablup.com/202441346_27e360f9_6397_4c79_960d_c47732f5d070_ea1a7af4ce.png) 

I have shared my internship experience with the Open Source Contribution Academy, where I performed well. I believe that my experience at the company was the foundation for that.

Recap of my Lablup Internship, Summer 2022

Introducing FastTrack, the MLOps Platform of Backend.AI. FastTrack allows you to organize each step of data preprocessing, training, validation, deployment, and inference into a single pipeline. FastTrack makes it easy for you to customize each step as you build your pipeline. In this article, we'll explain why you need an MLOps platform, how Backend.AI FastTrack came to be, and what makes itself really unique.

## Rise of MLOps Platforms
 ![](https://cdn.lablup.com/mlops_platform_c4731f2c5f.png) 
Over the past few years, the IT industry, as well as most industries undergoing digital transformation, has been working hard to adopt AI to make meaningful predictions from scattered data and respond to rapidly changing markets. In order to make a good use of AI in this process, it is necessary to respond to various stages such as model training and optimization, hardware introduction considering data I/O, model version management, etc. The concept of MLOps (Machine Learning Operations) emerged from this. If you are unfamiliar with the concept, we recommend that you skim our 'MLOps series' before reading this article.

## FastTrack: History
In 2019, we added the Backend.AI pipeline as a beta release to address the demand for DevOps pipelines. We developed and tested the ability to simplify the process of creating and managing complex pipelines, and to operate unidirectional pipelines that split into two or more paths in the middle. However, with the rise of the MLOps concept and the proliferation of various pipeline solutions such as AirFlow, MLFlow, and KubeFlow, we shifted our development direction to integrating and supporting open source pipeline tools instead of developing pipeline features as a full-fledged feature.

Meanwhile, AI development pipelines have became increasingly complexed, and became clear that open-source MLOps pipeline tools were unable to meet the diverse needs of the users. At this point, we decided to revive the pipeline feature of Backend.AI. During the process of revitalizing and prototyping the Backend.AI pipeline, we changed the direction of development to a MLOps pipeline solution that works with the Backend.AI cluster, but stands independently, so that we could directly address our user requests.

With such a colorful history, Lablup's AI/MLOps solution is called 'FastTrack'. This name came from the airport or logistics, a lane which expedites passenger or custom clearance. FastTrack became available with Backend.AI 22.09, and still being tested to meet our customer standards.

## FastTrack: What it is
FastTrack is a machine learning workflow platform that enables users to tailor multiple work units based on Backend.AI clusters and execute them as a Directed Acyclic Graph (DAG). Users can run sessions for each stage of the machine learning pipeline, linked through pre- and post-relationships, allowing them to integrate steps like data preprocessing, training, validation, deployment, monitoring, and optimization into a unified workflow as needed. This means users can more efficiently build and reuse models by structuring sessions into workflows and automatically scheduling them after each phase, rather than manually crafting them in a conventional Backend.AI cluster.
 ![](https://cdn.lablup.com/fasttrack_introduction_moving_4be00cb551.gif) 

## FastTrack: Structure and features
FastTrack categorizes workflow templates as pipelines, executes workflows to pipeline jobs, divides the units of work in a workflow into tasks, and the units of work that are to be executed into task instances. The flowchart following outlines the step-by-step progression of work within FastTrack.
 ![](https://cdn.lablup.com/fasttrack_architecture_f46ec19d8d.png) 

### Pipeline
A pipeline is a structured collection of data and tasks, represented by a Directed Acyclic Graph (DAG). In setting up an AI workflow, constructing a pipeline prompts FastTrack to create a specific folder in your Backend.AI cluster dedicated to pipelines. This setup facilitates the monitoring of training progress via artifacts. FastTrack streamlines the modification of task relationships with an intuitive drag-and-drop interface, allowing for immediate visual feedback in the form of a schematic flow and verification through a YAML file. Moreover, managing pipelines in YAML format allows for easy export, import, and sharing among users.

### Pipeline Job
Within the FastTrack GUI, the progress of job units is indicated by the color of the nodes associated with each unit. Similar to pipelines, the information and relationships of the task instances being configured in YAML are managed. Upon completion of all task instances, the pipeline job's status is displayed as either successful or failed.

### Task
A task is the smallest unit of execution in a pipeline that allows you to allocate resources by purpose. For example, sole task for model training can dedicate a lot of GPU resources, to use resources more efficiently, as opposed to preprocessing. You can also specify the execution environment. Based on the images supported by the Backend.AI cluster, you can use images such as TensorFlow, PyTorch, Python 3.x, NGC TensorFlow, NGC PyTorch, etc. without Docker building process. You can also mount virtual folders created by the Backend.AI cluster on a per-task basis as needed.

### Task Instance
 ![](https://cdn.lablup.com/fasttrack_task_instance_cfa320eaa2.png) 

Task instances are physical objects created when a pipeline job is created, based on the task informations that makes up the pipeline. Executing an AI workflow means that the task instances that make up the pipeline job are executed according to the specified preceding and following relationships. Task instances currently have a 1:1 correspondence with Sessions in the Backend.AI cluster, equating the state of a session with the state of a task instance, but we plan to expand beyond sessions to other units of execution in the near future.

## Wrap up
So far, we've covered MLOps with an introduction to FastTrack, the Backend.AI MLOps platform. The latest release of Backend.AI FastTrack is version 22.09 (in the time of Nov.2022). Our development plans include a range of user-friendly features, such as debugging pipelines, creating dependencies between pipelines, optimizing the usage of resources for tasks, and providing support for GitHub-based model and data repositories. True to Lablup's vision of empowering anyone to develop and use AI models from anywhere, FastTrack will simplify the process of building automated models. We look forward to your interest in our future endeavors.

Introducing FastTrack: Backend.AI MLOps Platform

Joongi 
Kim

As program complexity grows, software developers need robust debugging tools. The optimal debugging method involves pinpointing a reliable way to replicate an issue within a development setting conducive to free experimentation, followed by the creation of automated tests. However, when the reproduction scenario is overly complex or involves bugs that sporadically appear in production environments, detailed logging becomes the alternative to comprehend the issue retrospectively. In this post, we presents the 'aiomonitor-ng', designed to simplify the debugging of intricate asyncio programs.

Debugging asyncio applications has its own difficulties. In Python, the stack trace is commonly used for debugging, revealing the program's location at the time of an exception. However, with asyncio's concurrent execution of multiple coroutine tasks, each with its own stack, it's crucial to examine not just the stack of the coroutine where the exception occurred but also those of 'related' coroutines to pinpoint if the error stemmed from another task. This issue intensifies when an external library implicitly generates a coroutine that invokes my code. Moreover, certain bugs, like coroutine task explosions that only manifest in production, or silent terminations of ongoing coroutine tasks, are particularly elusive in development settings, as they don't produce clear exceptions and are only detectable through post-incident logs.

[aiomonitor](https://github.com/aio-libs/aiomonitor) is a production-grade live debugging tool created by the asyncio core developers. Wrapping asyncio-based code within a monitor object allows you to initiate a telnet session to a pre-set TCP port outside the process while the code is active. Through simple commands, you can inspect the list of coroutine tasks running in the event loop and the status of individual stacks. Backend.AI has integrated aiomonitor, assigning a unique debugging telnet port to each service process. (For security purposes, only local connections are permitted.) This integration has significantly aided in troubleshooting production-specific issues. Nonetheless, pinpointing the cause of a coroutine task's failure due to an external library, not specific to Backend.AI's code, remains a challenge when using aiomonitor at the time of the problem's occurrence.

We have developed an enhanced version named [aiomonitor-ng](https://github.com/achimnol/aiomonitor-ng), where "ng" signifies next-generation. This version includes the following additions and enhancements:

* Task creation tracker: For all running coroutine tasks, the momentary stack trace is preserved for each job that created the coroutine task (`asyncio.create_task()`) to allow the entire chain of task creation to be tracked (`ps`, `where` command).
* Task termination tracker: Recently terminated coroutine tasks can be preserved and viewed up to a maximum of N, especially when one job cancels (`Task.cancel()`) another job. The momentary stack trace of the cancellation trigger is also preserved to enable tracking of the entire cancellation chain (`ps-terminated`, `where-terminated` command).
* Persistent task marker: By default, to prevent memory leaks, recently terminated jobs are tracked up to a maximum of N. However, if specific jobs that must continue running throughout the application's lifespan are marked with a decorator, those jobs always preserve their termination logs, regardless of the history limit. They also provide a filtering function as an additional option in the termination log query command (`aiomonitor.task.preserve_termination_log` decorator).
* Sophisticated terminal UI: We improved command-line processing, which was previously composed of a simple REPL (read-evaluate-print loop) based on handcrafted command parsing. We rewrote the aiomonitor server-side implementation to use [Click](https://palletsprojects.com/projects/click/) and [prompt_toolkit](https://python-prompt-toolkit.readthedocs.io/en/master/). We also developed a Telnet client that natively operates with asyncio to provide argument autocomplete, such as command and task ID.

Here are some screenshots of the actual usage:
 ![](https://cdn.lablup.com/aiomonitor_ng_ps_b57dc89630.png "Coroutine task list view") 
 ![](https://cdn.lablup.com/aiomonitor_ng_task_creation_tracker_f066f43262.png "Tracking Coroutine task creation") 
 ![](https://cdn.lablup.com/aiomonitor_ng_task_termination_tracker_1bea7f6f5f.png "Tracking Coroutine task termination") 

We have successfully resolved resource leaks and performance issues stemming from excessive coroutine task creation in the grpcio library through callbacks. Additionally, we addressed problems where tasks that monitor events produced by the docker daemon would silently stop due to specific input message patterns. This was preventing the outcomes of container creation or deletion tasks from being reported, leading to system crashes.

We anticipate that developers working not only on Lablup but also on various Python asyncio applications will find aiomonitor-ng useful for debugging purposes in the future.

> aiomonitor-ng can be installed via [PyPI](https://pypi.org/project/aiomonitor-ng/) using the command `pip install aiomonitor-ng`, and it is open-sourced on [my GitHub account](https://github.com/achimnol/aiomonitor-ng) for anyone to use and contribute.

backend.ai

Blog

Top Stories