PyTritonAdapter API
model_navigator.api.pytriton.PyTritonAdapter
model_navigator.api.pytriton.PyTritonAdapter(package, strategy=None, runner_return_type=TensorType.NUMPY)
Provides model and configuration for PyTrtion deployment.
Initialize PyTritonAdapter.
Parameters:
-
package
(
Package
) –A package object to be searched for best possible model.
-
strategy
(
Optional[RuntimeSearchStrategy]
, default:None
) –Strategy for finding the best model. Defaults to
MaxThroughputAndMinLatencyStrategy
-
runner_return_type
(
TensorType
, default:NUMPY
) –The type of the output tensor. Defaults to
TensorType.NUMPY
. If the return_type supports CUDA tensors (e.g. TensorType.TORCH) and the input tensors are on CUDA, there will be no additional data transfer between CPU and GPU.
Source code in model_navigator/api/pytriton.py
batching
property
Returns status of batching support by the runner.
Returns:
-
bool
–True if runner supports batching, False otherwise.
config
property
Returns config for pytriton.
Returns:
-
ModelConfig
–ModelConfig with configuration for PyTrtion bind method.
inputs
property
outputs
property
model_navigator.api.pytriton.ModelConfig
dataclass
Additional model configuration for running model through Triton Inference Server.
Parameters:
-
batching
(
bool
, default:True
) –Flag to enable/disable batching for model.
-
max_batch_size
(
int
, default:4
) –The maximal batch size that would be handled by model.
-
batcher
(
DynamicBatcher
, default:dataclasses.field(default_factory=DynamicBatcher)
) –Configuration of Dynamic Batching for the model.
-
response_cache
(
bool
, default:False
) –Flag to enable/disable response cache for the model
model_navigator.api.pytriton.DynamicBatcher
dataclass
Dynamic batcher configuration.
More in Triton Inference Server documentation
Parameters:
-
max_queue_delay_microseconds
(
int
, default:0
) –The maximum time, in microseconds, a request will be delayed in the scheduling queue to wait for additional requests for batching.
-
preferred_batch_size
(
Optional[list]
, default:None
) –Preferred batch sizes for dynamic batching.
-
preserve_ordering
(
bool
, default:False
) –Should the dynamic batcher preserve the ordering of responses to match the order of requests received by the scheduler.
-
priority_levels
(
int
, default:0
) –The number of priority levels to be enabled for the model.
-
default_priority_level
(
int
, default:0
) –The priority level used for requests that don't specify their priority.
-
default_queue_policy
(
Optional[QueuePolicy]
, default:None
) –The default queue policy used for requests.
-
priority_queue_policy
(
Optional[Dict[int, QueuePolicy]]
, default:None
) –Specify the queue policy for the priority level.
model_navigator.api.pytriton.Tensor
dataclass
Model input and output definition for Triton deployment.
Parameters:
-
shape
(
tuple
) –Shape of the input/output tensor.
-
dtype
(
Union[dtype, Type[numpy.dtype], Type[object]]
) –Data type of the input/output tensor.
-
name
(
Optional[str]
, default:None
) –Name of the input/output of model.
-
optional
(
Optional[bool]
, default:False
) –Flag to mark if input is optional.
model_navigator.api.pytriton.QueuePolicy
dataclass
Model queue policy configuration.
More in Triton Inference Server documentation
Parameters:
-
timeout_action
(
TimeoutAction
, default:REJECT
) –The action applied to timed-out request.
-
default_timeout_microseconds
(
int
, default:0
) –The default timeout for every request, in microseconds.
-
allow_timeout_override
(
bool
, default:False
) –Whether individual request can override the default timeout value.
-
max_queue_size
(
int
, default:0
) –The maximum queue size for holding requests.
model_navigator.api.pytriton.TimeoutAction
Bases: Enum
Timeout action definition for timeout_action QueuePolicy field.
Parameters:
-
REJECT
(
str
) –Reject the request and return error message accordingly.
-
DELAY
(
str
) –Delay the request until all other requests at the same (or higher) priority levels that have not reached their timeouts are processed.