Config
Classes, enums and types used to configure Model Navigator.
model_navigator.api.config
Definition of enums and classes representing configuration for Model Navigator.
CustomConfig
Bases: ABC
Base class used for custom configs. Input for Model Navigator optimize
method.
defaults
from_dict
classmethod
CustomConfigForFormat
dataclass
Bases: DataObject
, CustomConfig
Abstract base class used for custom configs representing particular format.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
name
abstractmethod
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
CustomConfigForTensorRT
dataclass
Bases: CustomConfigForFormat
Abstract base class used for custom configs representing particular TensorRT format.
defaults
Update parameters to defaults.
Source code in model_navigator/api/config.py
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
name
abstractmethod
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
DeviceKind
Format
Bases: Enum
All model formats supported by Model Navigator 'optimize' function.
Parameters:
-
PYTHON
(
str
) –Format indicating any model defined in Python.
-
TORCH
(
str
) –Format indicating PyTorch model.
-
TENSORFLOW
(
str
) –Format indicating TensorFlow model.
-
JAX
(
str
) –Format indicating JAX model.
-
TORCHSCRIPT
(
str
) –Format indicating TorchScript model.
-
TF_SAVEDMODEL
(
str
) –Format indicating TensorFlow SavedModel.
-
TF_TRT
(
str
) –Format indicating TensorFlow TensorRT model.
-
TORCH_TRT
(
str
) –Format indicating PyTorch TensorRT model.
-
ONNX
(
str
) –Format indicating ONNX model.
-
TENSORRT
(
str
) –Format indicating TensorRT model.
JitType
OnnxConfig
dataclass
Bases: CustomConfigForFormat
ONNX custom config used for ONNX export and conversion.
Parameters:
-
opset
(
Optional[int]
, default:DEFAULT_ONNX_OPSET
) –ONNX opset used for conversion.
-
dynamic_axes
(
Optional[Dict[str, Union[Dict[int, str], List[int]]]]
, default:None
) –Dynamic axes for ONNX conversion.
-
onnx_extended_conversion
(
bool
, default:False
) –Enables additional conversions from TorchScript to ONNX.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
OptimizationProfile
dataclass
Bases: DataObject
Optimization profile configuration.
For each batch size profiler will run measurements in windows of fixed number of queries. Batch sizes are profiled in the ascending order.
Profiler will run multiple trials and will stop when the measurements
are stable (within stability_percentage
from the mean) within three consecutive windows.
If the measurements are not stable after max_trials
trials, the profiler will stop with an error.
Profiler will also stop profiling when the throughput does not increase at least by throughput_cutoff_threshold
.
Parameters:
-
max_batch_size
(
Optional[int]
, default:None
) –Maximal batch size used during conversion and profiling. None mean automatic search is enabled.
-
batch_sizes
–
List of batch sizes to profile. None mean automatic search is enabled.
-
window_size
(
int
, default:50
) –Number of requests to measure in each window.
-
stability_percentage
(
float
, default:10.0
) –Allowed percentage of variation from the mean in consecutive windows.
-
stabilization_windows
(
int
, default:3
) –Number consecutive windows selected for stabilization.
-
min_trials
(
int
, default:3
) –Minimal number of window trials.
-
max_trials
(
int
, default:10
) –Maximum number of window trials.
-
throughput_cutoff_threshold
(
float
, default:DEFAULT_PROFILING_THROUGHPUT_CUTOFF_THRESHOLD
) –Minimum throughput increase to continue profiling.
-
dataloader
(
Optional[SizedDataLoader]
, default:None
) –Optional dataloader for profiling. Use only 1 sample.
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
Instantiate OptimizationProfile class from a dictionary.
Parameters:
-
optimization_profile_dict
(
Mapping
) –Data dictionary.
Returns:
-
OptimizationProfile
–OptimizationProfile
Source code in model_navigator/api/config.py
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Append dataloader
field to filtered fields during dump.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/api/config.py
ShapeTuple
dataclass
Bases: DataObject
Represents a set of shapes for a single binding in a profile.
Each element of the tuple represents a shape for a single dimension of the binding.
Parameters:
-
min
(
Tuple[int]
) –The minimum shape that the profile will support.
-
opt
(
Tuple[int]
) –The shape for which TensorRT will optimize the engine.
-
max
(
Tuple[int]
) –The maximum shape that the profile will support.
__iter__
__repr__
__str__
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
SizedIterable
TensorFlowConfig
dataclass
Bases: CustomConfigForFormat
TensorFlow custom config used for SavedModel export.
Parameters:
-
jit_compile
(
Tuple[Optional[bool], ...]
, default:(None)
) –Enable or Disable jit_compile flag for tf.function wrapper for Jax infer function.
-
enable_xla
(
Tuple[Optional[bool], ...]
, default:(None)
) –Enable or Disable enable_xla flag for jax2tf converter.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
TensorFlowTensorRTConfig
dataclass
Bases: CustomConfigForTensorRT
TensorFlow TensorRT custom config used for TensorRT SavedModel export.
Parameters:
-
minimum_segment_size
(
int
, default:DEFAULT_MIN_SEGMENT_SIZE
) –Min size of subgraph.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
Instantiate TensorFlowTensorRTConfig from adictionary.
Source code in model_navigator/api/config.py
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
TensorRTCompatibilityLevel
TensorRTConfig
dataclass
Bases: CustomConfigForTensorRT
TensorRT custom config used for TensorRT conversion.
Parameters:
-
optimization_level
(
Optional[int]
, default:None
) –Optimization level for TensorRT conversion. Allowed values are fom 0 to 5. Where default is 3 based on TensorRT API documentation.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
Instantiate TensorRTConfig from adictionary.
Source code in model_navigator/api/config.py
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
TensorRTPrecision
TensorRTPrecisionMode
TensorRTProfile
Bases: Dict[str, ShapeTuple]
Single optimization profile that can be used to build an engine.
More specifically, it is an Dict[str, ShapeTuple]
which maps binding
names to a set of min/opt/max shapes.
__getitem__
Retrieves the shapes registered for a given input name.
Returns:
-
ShapeTuple –
A named tuple including ``min``, ``opt``, and ``max`` members for the shapes corresponding to the input.
Source code in model_navigator/api/config.py
__repr__
__str__
String representation.
add
A convenience function to add shapes for a single binding.
Parameters:
-
name
(
str
) –The name of the binding.
-
min
(
Tuple[int]
) –The minimum shape that the profile will support.
-
opt
(
Tuple[int]
) –The shape for which TensorRT will optimize the engine.
-
max
(
Tuple[int]
) –The maximum shape that the profile will support.
Returns:
-
Profile –
self, which allows this function to be easily chained to add multiple bindings, e.g., TensorRTProfile().add(...).add(...)
Source code in model_navigator/api/config.py
from_dict
classmethod
Create a TensorRTProfile from a dictionary.
Parameters:
-
profile_dict
(
Dict[str, Dict[str, Tuple[int, ...]]]
) –A dictionary mapping binding names to a dictionary containing
min
,opt
, andmax
keys.
Returns:
-
TensorRTProfile –
A TensorRTProfile object.
Source code in model_navigator/api/config.py
TensorType
TorchConfig
dataclass
Bases: CustomConfigForFormat
Torch custom config used for TorchScript export.
Parameters:
-
jit_type
(
Union[Union[str, JitType], Tuple[Union[str, JitType], ...]]
, default:(SCRIPT, TRACE)
) –Type of TorchScript export.
-
strict
(
bool
, default:True
) –Enable or Disable strict flag for tracer used in TorchScript export, default: True.
defaults
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
TorchTensorRTConfig
dataclass
Bases: CustomConfigForTensorRT
Torch custom config used for TensorRT TorchScript conversion.
defaults
Update parameters to defaults.
Source code in model_navigator/api/config.py
filter_data
staticmethod
Filter fields in dictionary.
Parameters:
Returns:
-
–
Filtered dictionary
Source code in model_navigator/utils/common.py
from_dict
classmethod
Instantiate TorchTensorRTConfig from adictionary.
Source code in model_navigator/api/config.py
name
classmethod
parse_data
staticmethod
Parse values in provided data.
Parameters:
-
data
(
Dict
) –Dictionary with data to parse
Returns:
-
–
Parsed dictionary
Source code in model_navigator/utils/common.py
parse_value
staticmethod
Parse value to jsonable format.
Parameters:
-
value
(
Any
) –Value to be parsed.
Returns:
Source code in model_navigator/utils/common.py
to_dict
Serialize to a dictionary.
Parameters:
-
filter_fields
(
Optional[List[str]]
, default:None
) –List of fields to filter out. Defaults to None.
-
parse
(
bool
, default:False
) –If True recursively parse field values to jsonable representation. Defaults to False.
Returns:
-
Dict(
Dict
) –Data serialized to a dictionary.
Source code in model_navigator/utils/common.py
map_custom_configs
Map custom configs from list to dictionary.
Parameters:
-
custom_configs
(
Optional[Sequence[CustomConfig]]
) –List of custom configs passed to API method
Returns:
-
Dict
–Mapped configs to dictionary
Source code in model_navigator/api/config.py
model_navigator.api.MaxThroughputAndMinLatencyStrategy
Bases: RuntimeSearchStrategy
Get runtime with the highest throughput and the lowest latency.
model_navigator.api.MaxThroughputStrategy
Bases: RuntimeSearchStrategy
Get runtime with the highest throughput.
model_navigator.api.MinLatencyStrategy
Bases: RuntimeSearchStrategy
Get runtime with the lowest latency.