Skip to content

Installation

This page explains how to install the library. We assume that you have a basic understanding of the Python programming language and are familiar with machine learning models.

You should be comfortable with the Python programming language and know how to work with Machine Learning models. Using Docker is optional and not necessary.

The library can be installed in any of the following ways:

  • system environment
  • virtualenv
  • Docker image

If you opt for using Docker, you can get NVIDIA optimized Docker images for Python frameworks from the NVIDIA NGC Catalog.

To run model inference on NVIDIA GPU using the Docker runtime, we recommend that you install the NVIDIA Container Toolkit, which enables GPU acceleration for containers.

Prerequisites

Before installing the library, ensure that you meet the following requirements:

  • An operating system with glibc >= 2.35. Triton Inference Server and PyTriton have only been rigorously tested on Ubuntu 22.04. Other supported operating systems include Ubuntu Debian 11+, Rocky Linux 9+, and Red Hat Universal Base Image 9+.
  • to check your glibc version, run ldd --version
  • Python version >= 3.8
  • pip >=20.3`
  • libpython3.*.so available in the operating system (appropriate for Python version).

Install from pypi

You can install the package from pypi.org by running the following command:

pip install -U nvidia-pytriton

Triton Inference Server binaries

The Triton Inference Server binaries are installed as part of the PyTriton package.

Setting Up Python Environment

The Triton Inference Server is automatically run with your Python interpreter version. To use Triton binary you need to make sure that libpython3.*.so library can be linked during PyTriton start. Install and provide location to libpython3.*.so library in LD_LIBRARY_PATH before you will run PyTriton. Below we presented some options on how to prepare your Python environment to run PyTriton with common tools.

Upgrading pip version

You need to have pip version 20.3 or higher. To upgrade an older version of pip, run this command:

pip install -U pip

Using system interpreter

When you are running PyTriton on Ubuntu 22.04 install the desired Python interpreter and libpython3*so. library. 

# Install necessary packages
apt update -y
apt install -y software-properties-common

# Add repository with various Python versions
add-apt-repository ppa:deadsnakes/ppa -y

# Install Python 3.8
apt install -y python3.8 libpython3.8 python3.8-distutils python3-pip \
     build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev \
     libffi-dev curl libbz2-dev pkg-config make

# Install library for interpreter
python3.8 -m pip install nvidia-pytriton

Creating virtualenv using pyenv

In order to install different version replace the 3.8 with desired Python version in the example below:

# Install necessary packages
apt update -y
apt install -y python3 python3-distutils python-is-python3 git \
    build-essential libssl-dev zlib1g-dev \
    libbz2-dev libreadline-dev libsqlite3-dev curl \
    libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# Install pyenv
curl https://pyenv.run | bash

# Configure pyenv in current environment
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

# Install Python 3.8 with shared library support
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.8

# Create and activate virtualenv
pyenv virtualenv 3.8 venv
pyenv activate venv

# export the library path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pyenv virtualenv-prefix)/lib

# Install library for interpreter
pip install nvidia-pytriton

Creating virtualenv using venv

In order to install different version replace the 3.8 with desired Python version in the example below:

# Install necessary packages
apt update -y
apt install -y software-properties-common

# Add repository with various Python versions
add-apt-repository ppa:deadsnakes/ppa -y

# Install Python 3.8
apt install -y python3.8 libpython3.8 python3.8-distutils python3.8-venv python3.8-pip python-is-python3 \
     build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev \
     libffi-dev curl libbz2-dev pkg-config make

# Create and activate virtualenv
python3.8 -m venv /opt/venv
source /opt/venv/bin/activate

# Install library for interpreter
pip install nvidia-pytriton

Creating virtualenv using miniconda

In order to install different version replace the 3.8 with desired Python version in the example below:

# Install necessary packages
apt update -y
apt install -y python3 python3-distutils python-is-python3 curl

# Download miniconda
CONDA_VERSION=latest
TARGET_MACHINE=x86_64
curl "https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-${TARGET_MACHINE}.sh" --output miniconda.sh

# Install miniconda and add to PATH
bash miniconda.sh
export PATH=~/miniconda3/bin/:$PATH

# Initialize bash
conda init bash
bash

# Create and activate virtualenv
conda create -c conda-forge -n venv python=3.8
conda activate venv

# Export the library path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib

# Install library for interpreter
pip install nvidia-pytriton

Building binaries from source

The binary package can be built from the source, allowing access to unreleased hotfixes, the ability to modify the PyTriton code, and compatibility with various Triton Inference Server versions, including custom server builds. For further information on building the PyTriton binary, refer to the Building page.