For AidliteSDK, it handles different models from different frameworks, and each model has different input data types and output data types. When setting the input and output data types of the model in the usage workflow mentioned above, this data type enumeration is needed.
Member Variable
Type
Value
Description
TYPE_DEFAULT
uint8_t
0
Invalid DataType
TYPE_UINT8
uint8_t
1
Unsigned byte data
TYPE_INT8
uint8_t
2
Byte data
TYPE_UINT32
uint8_t
3
Uint32 data
TYPE_FLOAT32
uint8_t
4
Float32 data
TYPE_INT32
uint8_t
5
Int32 data
TYPE_INT64
uint8_t
6
Int64 data
TYPE_UINT64
uint8_t
7
Uint64 data
TYPE_INT16
uint8_t
8
Int16 data
TYPE_UINT16
uint8_t
9
Uint16 data
TYPE_FLOAT16
uint8_t
10
Float16 data
TYPE_BOOL
uint8_t
11
Bool data
Inference Implementation Type enum ImplementType
Member Variable
Type
Value
Description
TYPE_DEFAULT
uint8_t
0
Invalid ImplementType
TYPE_MMKV
uint8_t
1
Implementation via MMKV (no current implementation, may be removed later)
As mentioned earlier, AidliteSDK integrates multiple deep learning inference frameworks, so in the usage workflow described above, you need to set which framework's model is currently being used, which requires this framework type enumeration.
Member Variable
Type
Value
Description
TYPE_DEFAULT
uint8_t
0
Invalid FrameworkType
TYPE_SNPE
uint8_t
1
SNPE1.x(dlc) model
TYPE_TFLITE
uint8_t
2
TFLite model
TYPE_RKNN
uint8_t
3
RKNN model
TYPE_QNN
uint8_t
4
QNN model
TYPE_SNPE2
uint8_t
5
SNPE2.x(dlc) model
TYPE_NCNN
uint8_t
6
NCNN model
TYPE_MNN
uint8_t
7
MNN model
TYPE_TNN
uint8_t
8
TNN model
TYPE_PADDLE
uint8_t
9
Paddle model
TYPE_ONNX
uint8_t
11
ONNX model
TYPE_QNN216
uint8_t
101
QNN2.16 model
TYPE_SNPE216
uint8_t
102
SNPE2.16 model
TYPE_QNN223
uint8_t
103
QNN2.23 model
TYPE_SNPE223
uint8_t
104
SNPE2.23 model
TYPE_QNN229
uint8_t
105
QNN2.29 model
TYPE_SNPE229
uint8_t
106
SNPE2.29 model
TYPE_QNN231
uint8_t
107
QNN2.31 model
TYPE_SNPE231
uint8_t
108
SNPE2.31 model
💡Note
Special Notes:
If the user environment has multiple backend implementation packages installed (such as AidLite-QNN216/AidLite-QNN223/AidLite-QNN229) and the FrameworkType in the code is set to TYPE_QNN (i.e., no specific QNN backend version is specified), then the actually running QNN backend should be the one with the highest QNN version (i.e., AidLite-QNN229 is preferred for inference). SNPE2 is handled similarly.
The same program cannot use different QNN version backend packages simultaneously. This means if the program first specifies TYPE_QNN216 for inference, then TYPE_QNN223/TYPE_QNN229 cannot be used for inference. SNPE2 is handled similarly.
For each deep learning inference framework, it may support running on different acceleration devices (such as SNPE/QNN models running on DSP devices, RKNN models running on NPU devices). Therefore, in the usage workflow described above, you need to set which device the current model is expected to run on, which requires this acceleration hardware enumeration.
AidliteSDK provides interfaces for setting logs (which will be introduced later). You need to provide the interface with the current log level to use, so this log level enumeration is needed.
Each model has its own inputs and outputs. For models that developers are not familiar with, they may need to query the input and output tensor information of the model during development to facilitate pre- and post-processing operations for model inference. Aidlite provides interfaces for developers to query detailed information about model inputs and outputs.
As mentioned earlier, before creating an inference interpreter, you need to set detailed parameters related to the specific model. The Model class is mainly used to record model file information, structure information, and model-related content during runtime.
To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as snpe, tflite, etc.) requires only one model file, call this interface.
API
create_instance
Description
Construct a Model type instance object by passing the path name of the model file
Parameters
model_path: Path name of the model file
Return Value
If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object
cpp
// Create model object using inceptionv3_float32.dlc file in current path, error if return value is nullModel* model = Model::create_instance("./inceptionv3_float32.dlc");if(model == nullptr){ printf("Create model failed !\n"); return EXIT_FAILURE;}
To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as ncnn, tnn, etc.) involves two model files, call this interface.
API
create_instance
Description
Construct a Model type instance object by passing the path name of the model file
Parameters
model_path: Path name of the model file
Return Value
If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object
cpp
// Create model object using two ncnn model files in current path, error if return value is nullModel* model = Model::create_instance("./mobilenet_ssd_voc_ncnn.param", "./mobilenet_ssd_voc_ncnn.bin");if(model == nullptr){ printf("Create model failed !\n"); return EXIT_FAILURE;}
After the model instance object is successfully created, you need to set the input and output data types of the model and the shape information of the input and output tensor data.
API
set_model_properties
Description
Set model properties, input and output data shapes and data types
Parameters
input_shapes: Shape array of input tensors, two-dimensional array structure
input_data_type: Data type of input tensors, DataType enumeration type
output_shapes: Shape array of output tensors, two-dimensional array structure
output_data_type: Data type of output tensors, DataType enumeration type
Return Value
void
cpp
// Data types use the aforementioned DataType enum, input/output shapes are 2D arraysstd::vector<std::vector<uint32_t>> input_shapes = {{1,299,299,3}};std::vector<std::vector<uint32_t>> output_shapes = {{1,1001}};model->set_model_properties(input_shapes, DataType::TYPE_FLOAT32, output_shapes, DataType::TYPE_FLOAT32);
If the model framework (such as SNPE, TFLite, etc.) involves only one model file, and the Model object construction is successful, the input model file parameter will be converted to an absolute path form.
API
get_model_absolute_path
Description
Used to get the storage path of the model file
Parameters
void
Return Value
Model file path string object corresponding to the Model object
If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model structure file parameter will be converted to an absolute path form.
API
get_model_struct_absolute_path
Description
Used to get the storage path of the model structure file
Parameters
void
Return Value
Model structure file path string object corresponding to the Model object
If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model weight parameter file parameter will be converted to an absolute path form.
API
get_model_weight_absolute_path
Description
Used to get the absolute path of the model weight parameter file
Parameters
void
Return Value
String object representing the model weight parameter file path corresponding to the Model object
As mentioned in the previous section, before creating an inference interpreter, in addition to setting specific Model information, you also need to set some configuration information for inference. The Config class is used to record configuration options that need to be set in advance, and these configuration items will be used at runtime.
To set runtime configuration information, you first need to have a configuration instance object. This function is used to create a Config instance object.
API
create_instance
Description
Used to construct an instance object of the Config class
Parameters
void
Return Value
If the value is nullptr, it indicates that the function failed to construct the object; otherwise, it is a pointer to the Config object
cpp
// Create configuration instance object, error if return value is nullConfig* config = Config::create_instance();if(config == nullptr){ printf("Create config failed !\n"); return EXIT_FAILURE;}
Used to store runtime context involved in the execution process, including Model objects and Config objects. More runtime data may be extended in the future.
Special note: The memory space pointed to by the model and config pointer parameters must be allocated on the heap. After Context is successfully constructed, their lifecycle will be managed by Context (Context will automatically release model and config objects when released), so developers do not need to manually release the space of both, otherwise it will report an error of duplicate object release.
The Interpreter class object instance is the main body for performing inference operations and is used to execute specific inference processes. In the inference workflow mentioned in the previous sections, after creating the interpreter object, all operations are based on the interpreter object, making it the absolute core content of AidliteSDK.
To perform inference-related operations, an inference interpreter is essential. This function is used to construct an inference interpreter instance object.
API
create_instance
Description
Construct an Interpreter type object using various data managed by the Context object
Parameters
context: Pointer to Context instance object
Return Value
If nullptr, it indicates the function failed to construct the object; otherwise, it is a pointer to the Interpreter object
cpp
// Use Context object pointer to create interpreter object, error if return value is nullInterpreter* current_interpreter = Interpreter::create_instance(context);if(current_interpreter == nullptr){ printf("Create Interpreter failed !\n"); return EXIT_FAILURE;}
After the interpreter object is created, some initialization operations need to be performed (such as environment checks, resource construction, etc.).
API
init
Description
Complete the initialization work required for inference
Parameters
void
Return Value
A value of 0 indicates successful initialization; otherwise, a non-zero value indicates failure
cpp
// Initialize interpreter, error if return value is non-zeroint result = fast_interpreter->init();if(result != EXIT_SUCCESS){ printf("sample : interpreter->init() failed !\n"); return EXIT_FAILURE;}
After the interpreter object completes initialization operations, the required model files can be loaded for the interpreter to complete model loading work. The subsequent inference process uses the loaded model resources.
API
load_model
Description
Complete the model loading related work. Since the model file path has been set in the Model object as mentioned above, the model loading operation can be executed directly
Parameters
void
Return Value
A value of 0 indicates successful model loading operation; otherwise, a non-zero value indicates failure
cpp
// Load model for interpreter, error if return value is non-zeroint result = fast_interpreter->load_model();if(result != EXIT_SUCCESS){ printf("sample : interpreter->load_model() failed !\n"); return EXIT_FAILURE;}
Get Input Tensor Details get_input_tensor_info()
If you want to get detailed information about the model's input tensors, you can use this interface to obtain it.
API
get_input_tensor_info
Description
Get detailed information of input Tensor in the model
Parameters
in_tensor_info: Array storing detailed information of input tensor
Return Value
A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note
This interface is supported starting from version 2.2.5
cpp
// Get model input details data, error if return value is non-zerostd::vector<std::vector<TensorInfo>> in_tensor_info;result = interpreter->get_input_tensor_info(in_tensor_info);if(result != EXIT_SUCCESS){ printf("interpreter->get_input_tensor_info() failed !\n"); return EXIT_FAILURE;}for(int gi = 0; gi < in_tensor_info.size(); ++gi){ for(int ti = 0; ti < in_tensor_info[gi].size(); ++ti){ std::string tensor_shape_str = std::to_string(in_tensor_info[gi][ti].shape[0]); for(int si = 1; si < in_tensor_info[gi][ti].dimensions; ++si){ tensor_shape_str += " "; tensor_shape_str += std::to_string(in_tensor_info[gi][ti].shape[si]); } printf("Input tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n", gi, ti, in_tensor_info[gi][ti].name.c_str(), in_tensor_info[gi][ti].element_count, (int)in_tensor_info[gi][ti].element_type, in_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str()); }}
Get Output Tensor Details get_output_tensor_info()
If you want to get detailed information about the model's output tensors, you can use this interface to obtain it.
API
get_output_tensor_info
Description
Get detailed information of output Tensor in the model
Parameters
out_tensor_info: Array storing detailed information of output tensor
Return Value
A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note
This interface is supported starting from version 2.2.5
cpp
// Get model output details data, error if return value is non-zerostd::vector<std::vector<TensorInfo>> out_tensor_info;result = onnx_interpreter->get_output_tensor_info(out_tensor_info);if(result != EXIT_SUCCESS){ printf("interpreter->get_output_tensor_info() failed !\n"); return EXIT_FAILURE;}for(int gi = 0; gi < out_tensor_info.size(); ++gi){ for(int ti = 0; ti < out_tensor_info[gi].size(); ++ti){ std::string tensor_shape_str = std::to_string(out_tensor_info[gi][ti].shape[0]); for(int si = 1; si < out_tensor_info[gi][ti].dimensions; ++si){ tensor_shape_str += " "; tensor_shape_str += std::to_string(out_tensor_info[gi][ti].shape[si]); } printf("Output tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n", gi, ti, out_tensor_info[gi][ti].name.c_str(), out_tensor_info[gi][ti].element_count, (int)out_tensor_info[gi][ti].element_type, out_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str()); }}
As mentioned in the workflow introduction earlier, before setting input data, different preprocessing operations need to be completed for different models to adapt to specific models.
API
set_input_tensor
Description
Set the data of input tensor required for inference
Parameters
in_tensor_idx/in_tensor_name: Index value/name string of input tensor
input_data: Pointer to the data storage address of input tensor
Return Value
A value of 0 indicates successful setting of input Tensor; otherwise, a non-zero value indicates failure
Special Note
Parameter in_tensor_name is supported starting from version 2.2.5
cpp
// Set input data for inference, error if return value is non-zero// int result = fast_interpreter->set_input_tensor(0, input_tensor_data);int result = fast_interpreter->set_input_tensor("in_tensor_name", input_tensor_data);if(result != EXIT_SUCCESS){ printf("interpreter->set_input_tensor() failed !\n"); return EXIT_FAILURE;}
As mentioned in the previous sections about the workflow, after setting up the input data, the next step is naturally to execute the inference process on the input data.
API
invoke
Description
Execute the inference computation process
Parameters
void
Return Value
A value of 0 indicates successful inference execution, otherwise a non-zero value indicates execution failure
After inference is completed, the result data obtained from inference needs to be extracted. As mentioned in the previous sections about the workflow, after extracting the result data, these result data can be processed to determine whether the results are correct.
API
get_output_tensor
Description
Get inference result data after successful inference
Parameters
out_tensor_idx/out_tensor_name: Index value/name string of the output result tensor
output_data: This value is the address of a pointer variable, the function internally resets the value of this pointer variable
output_length: The data size (in bytes) of this result output tensor
Return Value
A value of 0 indicates successful output tensor retrieval, otherwise a non-zero value indicates execution failure
Special Note
Parameter out_tensor_name is supported starting from version 2.2.5
cpp
// Get inference result data, non-zero return value indicates errorfloat* out_data = nullptr;uint32_t output_tensor_length = 0;// int result = fast_interpreter->get_output_tensor(0, (void**)&out_data, &output_tensor_length);int result = fast_interpreter->get_output_tensor("out_tensor_name", (void**)&out_data, &output_tensor_length);if(result != EXIT_SUCCESS){ printf("interpreter->get_output_tensor() failed !\n"); return EXIT_FAILURE;}
As mentioned earlier, the interpreter object needs to perform init() initialization operations and model loading operations. Correspondingly, the interpreter also needs to perform some cleanup operations to destroy previously created resources.
API
destroy
Description
Perform necessary cleanup operations
Parameters
void
Return Value
A value of 0 indicates successful cleanup operation, otherwise a non-zero value indicates execution failure
Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as SNPE, TFLite, etc.) only requires one model file, call this interface.
API
build_interpreter_from_path
Description
Create corresponding interpreter object directly through the path name of the model file
Parameters
path: Path name of the model file
Return Value
Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object
Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as NCNN, TNN, etc.) involves two model files, call this interface.
API
build_interpreter_from_path
Description
Create corresponding interpreter object directly through the path name of the model file
Parameters
model_struct_path: Path name of the model structure file
model_weight_path: Path name of the model weight parameter file
Return Value
Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object
Build inference interpreter object. In addition to providing model file paths, Model objects can also be provided, which allows setting not only the model file path and name, but also the input/output data types and input/output data shape information of the model.
API
build_interpreter_from_model
Description
Create corresponding interpreter object by passing in Model object. All Config-related parameters use default values
Parameters
model: Model type object containing model-related data
Return Value
Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object
💡Note
Special note: The memory space pointed to by the model parameter must be allocated on the heap. After successful Interpreter construction, its lifecycle will be managed by the Interpreter (when the Interpreter is released, it will automatically release the model object), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.
Build inference interpreter object. In addition to the aforementioned methods, Model objects and Config objects can also be provided simultaneously, which allows providing not only model-related information but also more runtime configuration parameters.
API
build_interpreter_from_model_and_config
Description
Create corresponding interpreter object by passing in Model object and Config
Parameters
model: Model type object containing model-related data
config: Config type object containing configuration parameters
Return Value
Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object
💡Note
Special note: The memory space pointed to by both model and config parameters must be allocated on the heap. After successful Interpreter construction, their lifecycles will be managed by the Interpreter (when the Interpreter is released, it will automatically release both model and config objects), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.
cpp
// Build interpreter by passing in Model and Config objects, null return value indicates errorstd::unique_ptr<Interpreter>&& fast_interpreter = InterpreterBuilder::build_interpreter_from_model_and_config(model, config);if(fast_interpreter == nullptr){ printf("build_interpreter failed !\n"); return EXIT_FAILURE;}