Accelerators

model_navigator.api.triton.AutoMixedPrecisionAccelerator dataclass

Auto-mixed-precision accelerator for TensorFlow. Enable automatic FP16 precision.

Currently empty - no arguments required.

model_navigator.api.triton.GPUIOAccelerator dataclass

GPU IO accelerator for TensorFlow.

Currently empty - no arguments required.

model_navigator.api.triton.OpenVINOAccelerator dataclass

OpenVINO optimization.

Currently empty - no arguments required.

model_navigator.api.triton.OpenVINOAccelerator dataclass

OpenVINO optimization.

Currently empty - no arguments required.

model_navigator.api.triton.TensorRTAccelerator dataclass

TensorRT accelerator configuration.

Read more in Triton Inference server model configuration

Parameters:

• precision (TensorRTOptPrecision, default: FP32 ) –

  The precision used for optimization

• max_workspace_size (Optional[int], default: None ) –

  The maximum GPU memory the model can use temporarily during execution

• max_cached_engines (Optional[int], default: None ) –

  The maximum number of cached TensorRT engines in dynamic TensorRT ops

• minimum_segment_size (Optional[int], default: None ) –

  The smallest model subgraph that will be considered for optimization by TensorRT

model_navigator.api.triton.TensorRTOptPrecision

Bases: Enum

TensorRT optimization allowed precision.

Parameters:

• FP16 –

  fp16 precision

• FP32 –

  fp32 precision