Skip to content

Adapter

model_navigator.api.pytriton.PyTritonAdapter 

PyTritonAdapter(package, strategy=None, runner_return_type=TensorType.NUMPY)

Provides model and configuration for PyTrtion deployment.

Initialize PyTritonAdapter.

Parameters:

  • package (Package) –

    A package object to be searched for best possible model.

  • strategy (Optional[RuntimeSearchStrategy], default: None ) –

    Strategy for finding the best model. Defaults to MaxThroughputAndMinLatencyStrategy

  • runner_return_type (TensorType, default: NUMPY ) –

    The type of the output tensor. Defaults to TensorType.NUMPY. If the return_type supports CUDA tensors (e.g. TensorType.TORCH) and the input tensors are on CUDA, there will be no additional data transfer between CPU and GPU.

Source code in model_navigator/api/pytriton.py 
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
def __init__(
    self,
    package: Package,
    strategy: Optional[RuntimeSearchStrategy] = None,
    runner_return_type: TensorType = TensorType.NUMPY,
):
    """Initialize PyTritonAdapter.

    Args:
        package: A package object to be searched for best possible model.
        strategy: Strategy for finding the best model. Defaults to `MaxThroughputAndMinLatencyStrategy`
        runner_return_type: The type of the output tensor. Defaults to `TensorType.NUMPY`.
            If the return_type supports CUDA tensors (e.g. TensorType.TORCH) and the input tensors are on CUDA,
            there will be no additional data transfer between CPU and GPU.
    """
    self._package = package
    self._strategy = MaxThroughputAndMinLatencyStrategy() if strategy is None else strategy
    self._runner = self._package.get_runner(strategy=self._strategy, return_type=runner_return_type)
    self._batching = self._package.status.config.get("batch_dim", None) == 0

batching property 

batching

Returns status of batching support by the runner.

Returns:

  • bool –

    True if runner supports batching, False otherwise.

config property 

config

Returns config for pytriton.

Returns:

  • ModelConfig –

    ModelConfig with configuration for PyTrtion bind method.

inputs property 

inputs

Returns inputs configuration.

Returns:

  • List[Tensor] –

    List with Tensor objects describing inputs configuration of runner

outputs property 

outputs

Returns outputs configuration.

Returns:

  • List[Tensor] –

    List with Tensor objects describing outputs configuration of runner

runner property 

runner

Returns runner.

Runner must be activated before use with activate() method.

Returns:

  • NavigatorRunner –

    Model Navigator runner.