TritonConfig

pytriton.triton.TritonConfig dataclass

Triton Inference Server configuration class for customization of server execution.

The arguments are optional. If value is not provided the defaults for Triton Inference Server are used. Please, refer to https://github.com/triton-inference-server/server/ for more details.

Parameters:

• id (Optional[str], default: None ) –

  Identifier for this server.

• log_verbose (Optional[int], default: None ) –

  Set verbose logging level. Zero (0) disables verbose logging and values >= 1 enable verbose logging.

• log_file (Optional[Path], default: None ) –

  Set the name of the log output file.

• exit_timeout_secs (Optional[int], default: None ) –

  Timeout (in seconds) when exiting to wait for in-flight inferences to finish.

• exit_on_error (Optional[bool], default: None ) –

  Exit the inference server if an error occurs during initialization.

• strict_readiness (Optional[bool], default: None ) –

  If true /v2/health/ready endpoint indicates ready if the server is responsive and all models are available.

• allow_http (Optional[bool], default: None ) –

  Allow the server to listen for HTTP requests.

• http_address (Optional[str], default: None ) –

  The address for the http server to bind to. Default is 0.0.0.0.

• http_port (Optional[int], default: None ) –

  The port for the server to listen on for HTTP requests. Default is 8000.

• http_header_forward_pattern (Optional[str], default: None ) –

  The regular expression pattern that will be used for forwarding HTTP headers as inference request parameters.

• http_thread_count (Optional[int], default: None ) –

  Number of threads handling HTTP requests.

• allow_grpc (Optional[bool], default: None ) –

  Allow the server to listen for GRPC requests.

• grpc_address (Optional[str], default: None ) –

  The address for the grpc server to binds to. Default is 0.0.0.0.

• grpc_port (Optional[int], default: None ) –

  The port for the server to listen on for GRPC requests. Default is 8001.

• grpc_header_forward_pattern (Optional[str], default: None ) –

  The regular expression pattern that will be used for forwarding GRPC headers as inference request parameters.

• grpc_infer_allocation_pool_size (Optional[int], default: None ) –

  The maximum number of inference request/response objects that remain allocated for reuse. As long as the number of in-flight requests doesn't exceed this value there will be no allocation/deallocation of request/response objects.

• grpc_use_ssl (Optional[bool], default: None ) –

  Use SSL authentication for GRPC requests. Default is false.

• grpc_use_ssl_mutual (Optional[bool], default: None ) –

  Use mututal SSL authentication for GRPC requests. This option will preempt grpc_use_ssl if it is also specified. Default is false.

• grpc_server_cert (Optional[Path], default: None ) –

  File holding PEM-encoded server certificate. Ignored unless grpc_use_ssl is true.

• grpc_server_key (Optional[Path], default: None ) –

  Path to file holding PEM-encoded server key. Ignored unless grpc_use_ssl is true.

• grpc_root_cert (Optional[Path], default: None ) –

  Path to file holding PEM-encoded root certificate. Ignored unless grpc_use_ssl is true.

• grpc_infer_response_compression_level (Optional[str], default: None ) –

  The compression level to be used while returning the inference response to the peer. Allowed values are none, low, medium and high. Default is none.

• grpc_keepalive_time (Optional[int], default: None ) –

  The period (in milliseconds) after which a keepalive ping is sent on the transport.

• grpc_keepalive_timeout (Optional[int], default: None ) –

  The period (in milliseconds) the sender of the keepalive ping waits for an acknowledgement.

• grpc_keepalive_permit_without_calls (Optional[bool], default: None ) –

  Allows keepalive pings to be sent even if there are no calls in flight

• grpc_http2_max_pings_without_data (Optional[int], default: None ) –

  The maximum number of pings that can be sent when there is no data/header frame to be sent.

• grpc_http2_min_recv_ping_interval_without_data (Optional[int], default: None ) –

  If there are no data/header frames being sent on the transport, this channel argument on the server side controls the minimum time (in milliseconds) that gRPC Core would expect between receiving successive pings.

• grpc_http2_max_ping_strikes (Optional[int], default: None ) –

  Maximum number of bad pings that the server will tolerate before sending an HTTP2 GOAWAY frame and closing the transport.

• grpc_restricted_protocol –

  Specify restricted GRPC protocol setting. The format of this flag is <protocols>,<key>=<value>. Where <protocol> is a comma-separated list of protocols to be restricted. <key> will be additional header key to be checked when a GRPC request is received, and <value> is the value expected to be matched.

• allow_metrics (Optional[bool], default: None ) –

  Allow the server to provide prometheus metrics.

• allow_gpu_metrics (Optional[bool], default: None ) –

  Allow the server to provide GPU metrics.

• allow_cpu_metrics (Optional[bool], default: None ) –

  Allow the server to provide CPU metrics.

• metrics_interval_ms (Optional[int], default: None ) –

  Metrics will be collected once every <metrics-interval-ms> milliseconds.

• metrics_port (Optional[int], default: None ) –

  The port reporting prometheus metrics.

• metrics_address (Optional[str], default: None ) –

  The address for the metrics server to bind to. Default is the same as http_address.

• allow_sagemaker (Optional[bool], default: None ) –

  Allow the server to listen for Sagemaker requests.

• sagemaker_port (Optional[int], default: None ) –

  The port for the server to listen on for Sagemaker requests.

• sagemaker_safe_port_range (Optional[str], default: None ) –

  Set the allowed port range for endpoints other than the SageMaker endpoints.

• sagemaker_thread_count (Optional[int], default: None ) –

  Number of threads handling Sagemaker requests.

• allow_vertex_ai (Optional[bool], default: None ) –

  Allow the server to listen for Vertex AI requests.

• vertex_ai_port (Optional[int], default: None ) –

  The port for the server to listen on for Vertex AI requests.

• vertex_ai_thread_count (Optional[int], default: None ) –

  Number of threads handling Vertex AI requests.

• vertex_ai_default_model (Optional[str], default: None ) –

  The name of the model to use for single-model inference requests.

• metrics_config (Optional[List[str]], default: None ) –

  Specify a metrics-specific configuration setting. The format of this flag is <setting>=<value>. It can be specified multiple times

• trace_config (Optional[List[str]], default: None ) –

  Specify global or trace mode specific configuration setting. The format of this flag is <mode>,<setting>=<value>. Where <mode> is either 'triton' or 'opentelemetry'. The default is 'triton'. To specify global trace settings (level, rate, count, or mode), the format would be <setting>=<value>. For 'triton' mode, the server will use Triton's Trace APIs. For 'opentelemetry' mode, the server will use OpenTelemetry's APIs to generate, collect and export traces for individual inference requests. More details, including supported settings can be found at Triton trace guide.

• cache_config (Optional[List[str]], default: None ) –

  Specify a cache-specific configuration setting. The format of this flag is <cache_name>,<setting>=<value>. Where <cache_name> is the name of the cache, such as 'local' or 'redis'. Example: local,size=1048576 will configure a 'local' cache implementation with a fixed buffer pool of size 1048576 bytes.

• cache_directory (Optional[str], default: None ) –

  The global directory searched for cache shared libraries. Default is '/opt/tritonserver/caches'. This directory is expected to contain a cache implementation as a shared library with the name 'libtritoncache.so'.

• buffer_manager_thread_count (Optional[int], default: None ) –

  The number of threads used to accelerate copies and other operations required to manage input and output tensor contents.

__post_init__

__post_init__()

Validate configuration for early error handling.

Source code in pytriton/triton.py

def __post_init__(self):
    """Validate configuration for early error handling."""
    if self.allow_http not in [True, None] and self.allow_grpc not in [True, None]:
        raise PyTritonValidationError("The `http` or `grpc` endpoint has to be allowed.")

from_dict classmethod

from_dict(config: Dict[str, Any]) -> TritonConfig

Creates a TritonConfig instance from an input dictionary. Values are converted into correct types.

Parameters:

• config (Dict[str, Any]) –

  a dictionary with all required fields

Returns:

• TritonConfig –

  a TritonConfig instance

Source code in pytriton/triton.py

@classmethod
def from_dict(cls, config: Dict[str, Any]) -> "TritonConfig":
    """Creates a ``TritonConfig`` instance from an input dictionary. Values are converted into correct types.

    Args:
        config: a dictionary with all required fields

    Returns:
        a ``TritonConfig`` instance
    """
    fields: Dict[str, dataclasses.Field] = {field.name: field for field in dataclasses.fields(cls)}
    unknown_config_parameters = {name: value for name, value in config.items() if name not in fields}
    for name, value in unknown_config_parameters.items():
        LOGGER.warning(
            f"Ignoring {name}={value} as could not find matching config field. "
            f"Available fields: {', '.join(map(str, fields))}"
        )

    def _cast_value(_field, _value):
        field_type = _field.type
        is_optional = typing_inspect.is_optional_type(field_type)
        if is_optional:
            field_type = field_type.__args__[0]
        return field_type(_value)

    config_with_casted_values = {
        name: _cast_value(fields[name], value) for name, value in config.items() if name in fields
    }
    return cls(**config_with_casted_values)

from_env classmethod

from_env() -> TritonConfig

Creates TritonConfig from environment variables.

Environment variables should start with PYTRITON_TRITON_CONFIG_ prefix. For example:

PYTRITON_TRITON_CONFIG_GRPC_PORT=45436
PYTRITON_TRITON_CONFIG_LOG_VERBOSE=4

Typical use:

triton_config = TritonConfig.from_env()

Returns:

• TritonConfig –

  TritonConfig class instantiated from environment variables.

Source code in pytriton/triton.py

@classmethod
def from_env(cls) -> "TritonConfig":
    """Creates TritonConfig from environment variables.

    Environment variables should start with `PYTRITON_TRITON_CONFIG_` prefix. For example:

        PYTRITON_TRITON_CONFIG_GRPC_PORT=45436
        PYTRITON_TRITON_CONFIG_LOG_VERBOSE=4

    Typical use:

        triton_config = TritonConfig.from_env()

    Returns:
        TritonConfig class instantiated from environment variables.
    """
    prefix = "PYTRITON_TRITON_CONFIG_"
    config = {name[len(prefix) :].lower(): value for name, value in os.environ.items() if name.startswith(prefix)}
    return cls.from_dict(config)

to_dict

to_dict()

Map config object to dictionary.

Source code in pytriton/triton.py

def to_dict(self):
    """Map config object to dictionary."""
    return dataclasses.asdict(self)