Skip to content

Adapter

model_navigator.pytriton.PyTritonAdapter 

PyTritonAdapter(package, strategies=None, runner_return_type=TensorType.NUMPY)

Provides model and configuration for PyTrtion deployment.

Initialize PyTritonAdapter.

Parameters:

  • package (Package) –

    A package object to be searched for best possible model.

  • strategies (Optional[List[RuntimeSearchStrategy]], default: None ) –

    List of strategies for finding the best model. Strategies are selected in provided order. When first fails, next strategy from the list is used. When none provided the strategies defaults to [MaxThroughputAndMinLatencyStrategy, MinLatencyStrategy]

  • runner_return_type (TensorType, default: NUMPY ) –

    The type of the output tensor. Defaults to TensorType.NUMPY. If the return_type supports CUDA tensors (e.g. TensorType.TORCH) and the input tensors are on CUDA, there will be no additional data transfer between CPU and GPU.

Source code in model_navigator/pytriton/__init__.py 
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
def __init__(
    self,
    package: Package,
    strategies: Optional[List[RuntimeSearchStrategy]] = None,
    runner_return_type: TensorType = TensorType.NUMPY,
):
    """Initialize PyTritonAdapter.

    Args:
        package: A package object to be searched for best possible model.
        strategies: List of strategies for finding the best model. Strategies are selected in provided order. When
                    first fails, next strategy from the list is used. When none provided the strategies
                    defaults to [`MaxThroughputAndMinLatencyStrategy`, `MinLatencyStrategy`]
        runner_return_type: The type of the output tensor. Defaults to `TensorType.NUMPY`.
            If the return_type supports CUDA tensors (e.g. TensorType.TORCH) and the input tensors are on CUDA,
            there will be no additional data transfer between CPU and GPU.
    """
    self._package = package
    self._strategies = strategies or [MaxThroughputAndMinLatencyStrategy(), MinLatencyStrategy()]
    self._runner = self._package.get_runner(strategies=strategies, return_type=runner_return_type)
    self._batching = self._package.status.config.get("batch_dim", None) == 0

batching property 

batching

Returns status of batching support by the runner.

Returns:

  • bool –

    True if runner supports batching, False otherwise.

config property 

config

Returns config for pytriton.

Returns:

  • ModelConfig –

    ModelConfig with configuration for PyTrtion bind method.

inputs property 

inputs

Returns inputs configuration.

Returns:

  • List[Tensor] –

    List with Tensor objects describing inputs configuration of runner

outputs property 

outputs

Returns outputs configuration.

Returns:

  • List[Tensor] –

    List with Tensor objects describing outputs configuration of runner

runner property 

runner

Returns runner.

Runner must be activated before use with activate() method.

Returns:

  • NavigatorRunner –

    Model Navigator runner.