Skip to content

AidLite C++ API Documentation

💡Note: When developing with Aidlite-SDK C++, you need to be aware of the following:

  • Include the header file during compilation, located at: /usr/local/include/aidlux/aidlite/aidlite.hpp
  • Link the library file during linking, located at: /usr/local/lib/libaidlite.so

Model Data Type enum DataType

For AidliteSDK, it handles different models from different frameworks, and each model has different input data types and output data types. When setting the input and output data types of the model in the usage workflow mentioned above, this data type enumeration is needed.

Member VariableTypeValueDescription
TYPE_DEFAULTuint8_t0Invalid DataType
TYPE_UINT8uint8_t1Unsigned byte data
TYPE_INT8uint8_t2Byte data
TYPE_UINT32uint8_t3Uint32 data
TYPE_FLOAT32uint8_t4Float32 data
TYPE_INT32uint8_t5Int32 data
TYPE_INT64uint8_t6Int64 data
TYPE_UINT64uint8_t7Uint64 data
TYPE_INT16uint8_t8Int16 data
TYPE_UINT16uint8_t9Uint16 data
TYPE_FLOAT16uint8_t10Float16 data
TYPE_BOOLuint8_t11Bool data

Inference Implementation Type enum ImplementType

Member VariableTypeValueDescription
TYPE_DEFAULTuint8_t0Invalid ImplementType
TYPE_MMKVuint8_t1Implementation via MMKV (no current implementation, may be removed later)
TYPE_REMOTEuint8_t2Backend implementation via IPC
TYPE_LOCALuint8_t3Backend implementation via local calls

Model Framework Type enum FrameworkType

As mentioned earlier, AidliteSDK integrates multiple deep learning inference frameworks, so in the usage workflow described above, you need to set which framework's model is currently being used, which requires this framework type enumeration.

Member VariableTypeValueDescription
TYPE_DEFAULTuint8_t0Invalid FrameworkType
TYPE_SNPEuint8_t1SNPE1.x(dlc) model
TYPE_TFLITEuint8_t2TFLite model
TYPE_RKNNuint8_t3RKNN model
TYPE_QNNuint8_t4QNN model
TYPE_SNPE2uint8_t5SNPE2.x(dlc) model
TYPE_NCNNuint8_t6NCNN model
TYPE_MNNuint8_t7MNN model
TYPE_TNNuint8_t8TNN model
TYPE_PADDLEuint8_t9Paddle model
TYPE_ONNXuint8_t11ONNX model
TYPE_QNN216uint8_t101QNN2.16 model
TYPE_SNPE216uint8_t102SNPE2.16 model
TYPE_QNN223uint8_t103QNN2.23 model
TYPE_SNPE223uint8_t104SNPE2.23 model
TYPE_QNN229uint8_t105QNN2.29 model
TYPE_SNPE229uint8_t106SNPE2.29 model
TYPE_QNN231uint8_t107QNN2.31 model
TYPE_SNPE231uint8_t108SNPE2.31 model

💡Note

Special Notes:

  • If the user environment has multiple backend implementation packages installed (such as AidLite-QNN216/AidLite-QNN223/AidLite-QNN229) and the FrameworkType in the code is set to TYPE_QNN (i.e., no specific QNN backend version is specified), then the actually running QNN backend should be the one with the highest QNN version (i.e., AidLite-QNN229 is preferred for inference). SNPE2 is handled similarly.
  • The same program cannot use different QNN version backend packages simultaneously. This means if the program first specifies TYPE_QNN216 for inference, then TYPE_QNN223/TYPE_QNN229 cannot be used for inference. SNPE2 is handled similarly.

Acceleration Hardware Type enum AccelerateType

For each deep learning inference framework, it may support running on different acceleration devices (such as SNPE/QNN models running on DSP devices, RKNN models running on NPU devices). Therefore, in the usage workflow described above, you need to set which device the current model is expected to run on, which requires this acceleration hardware enumeration.

Member VariableTypeValueDescription
TYPE_DEFAULTuint8_t0Invalid AccelerateType
TYPE_CPUuint8_t1CPU acceleration
TYPE_GPUuint8_t2GPU acceleration
TYPE_DSPuint8_t3DSP acceleration
TYPE_NPUuint8_t4NPU acceleration

Log Level enum LogLevel

AidliteSDK provides interfaces for setting logs (which will be introduced later). You need to provide the interface with the current log level to use, so this log level enumeration is needed.

Member VariableTypeValueDescription
INFOuint8_t0Information
WARNINGuint8_t1Warning
ERRORuint8_t2Error
FATALuint8_t3Fatal error

Input/Output Information Class class TensorInfo

Each model has its own inputs and outputs. For models that developers are not familiar with, they may need to query the input and output tensor information of the model during development to facilitate pre- and post-processing operations for model inference. Aidlite provides interfaces for developers to query detailed information about model inputs and outputs.

Member Variable List

The TensorInfo object is used to store detailed information about model input and output tensors, including the following parameters:

Member Variable version
Type uint32_t
Default Value No default value
Description This member is used to identify the TensorInfo version. If more information needs to be extended in the future, the version number will be updated
Member Variable index
Type uint32_t
Default Value No default value
Description Index value of the Tensor
Member Variable name
Type std::string
Default Value No default value
Description Name string of the Tensor
Member Variable element_count
Type uint64_t
Default Value 0
Description Number of data elements in the Tensor
Member Variable element_type
Type enum DataType
Default Value DataType::TYPE_DEFAULT
Description Type of data elements in the Tensor
Member Variable dimensions
Type uint32_t
Default Value 0
Description Dimension value of data elements in the Tensor
Member Variable shape
Type std::vector< uint64_t >
Default Value No default value
Description Shape description of data elements in the Tensor

Model Class .class Model

As mentioned earlier, before creating an inference interpreter, you need to set detailed parameters related to the specific model. The Model class is mainly used to record model file information, structure information, and model-related content during runtime.

Create Instance Object .create_instance()

To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as snpe, tflite, etc.) requires only one model file, call this interface.

API create_instance
Description Construct a Model type instance object by passing the path name of the model file
Parameters model_path: Path name of the model file
Return Value If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object
cpp
// Create model object using inceptionv3_float32.dlc file in current path, error if return value is null
Model* model = Model::create_instance("./inceptionv3_float32.dlc");
if(model == nullptr){
    printf("Create model failed !\n");
    return EXIT_FAILURE;
}

Create Instance Object create_instance()

To set detailed information related to the model, you first need to have a model instance object. This function is used to create a Model instance object. If the model framework (such as ncnn, tnn, etc.) involves two model files, call this interface.

API create_instance
Description Construct a Model type instance object by passing the path name of the model file
Parameters model_path: Path name of the model file
Return Value If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Model object
cpp
// Create model object using two ncnn model files in current path, error if return value is null
Model* model = Model::create_instance("./mobilenet_ssd_voc_ncnn.param", 
     "./mobilenet_ssd_voc_ncnn.bin");
if(model == nullptr){
    printf("Create model failed !\n");
    return EXIT_FAILURE;
}

Set Model Properties set_model_properties()

After the model instance object is successfully created, you need to set the input and output data types of the model and the shape information of the input and output tensor data.

API set_model_properties
Description Set model properties, input and output data shapes and data types
Parameters input_shapes: Shape array of input tensors, two-dimensional array structure
input_data_type: Data type of input tensors, DataType enumeration type
output_shapes: Shape array of output tensors, two-dimensional array structure
output_data_type: Data type of output tensors, DataType enumeration type
Return Value void
cpp
// Data types use the aforementioned DataType enum, input/output shapes are 2D arrays
std::vector<std::vector<uint32_t>> input_shapes = {{1,299,299,3}};
std::vector<std::vector<uint32_t>> output_shapes = {{1,1001}};
model->set_model_properties(input_shapes, DataType::TYPE_FLOAT32, 
    output_shapes, DataType::TYPE_FLOAT32);

Get Model Path get_model_absolute_path()

If the model framework (such as SNPE, TFLite, etc.) involves only one model file, and the Model object construction is successful, the input model file parameter will be converted to an absolute path form.

API get_model_absolute_path
Description Used to get the storage path of the model file
Parameters void
Return Value Model file path string object corresponding to the Model object

Get Model Path get_model_struct_absolute_path()

If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model structure file parameter will be converted to an absolute path form.

API get_model_struct_absolute_path
Description Used to get the storage path of the model structure file
Parameters void
Return Value Model structure file path string object corresponding to the Model object

Get Model Path get_model_weight_absolute_path()

If the model framework (such as NCNN, TNN, etc.) involves two model files, and the Model object construction is successful, the input model weight parameter file parameter will be converted to an absolute path form.

API get_model_weight_absolute_path
Description Used to get the absolute path of the model weight parameter file
Parameters void
Return Value String object representing the model weight parameter file path corresponding to the Model object

Configuration Class class Config

As mentioned in the previous section, before creating an inference interpreter, in addition to setting specific Model information, you also need to set some configuration information for inference. The Config class is used to record configuration options that need to be set in advance, and these configuration items will be used at runtime.

Create Instance Object create_instance()

To set runtime configuration information, you first need to have a configuration instance object. This function is used to create a Config instance object.

API create_instance
Description Used to construct an instance object of the Config class
Parameters void
Return Value If the value is nullptr, it indicates that the function failed to construct the object; otherwise, it is a pointer to the Config object
cpp
// Create configuration instance object, error if return value is null
Config* config = Config::create_instance();
if(config == nullptr){
    printf("Create config failed !\n");
    return EXIT_FAILURE;
}

Member Variable List

The Config object is used to set runtime configuration information, including the following parameters:

Member Variable accelerate_type
Type enum AccelerateType
Default Value AccelerateType.TYPE_CPU
Description Type of acceleration hardware
Member Variable implement_type
Type enum ImplementType
Default Value ImplementType.TYPE_DEFAULT
Description Differentiation of underlying function implementation methods
Member Variable framework_type
Type enum FrameworkType
Default Value FrameworkType.TYPE_DEFAULT
Description Type of underlying inference framework
Member Variable number_of_threads
Type int8_t
Default Value -1
Description Number of threads, valid when greater than 0
Member Variable snpe_out_names
Type String array
Default Value Empty
Description List of model output node names
Member Variable is_quantify_model
Type Int32_t
Default Value 0
Description Whether it is a quantized model, 1 indicates quantized model (only needs to be set for Remote)
Member Variable remote_timeout
Type Int32_t
Default Value -1
Description Interface timeout duration (milliseconds, valid when greater than 0) (only needs to be set for Remote)
cpp
config->framework_type = FrameworkType::TYPE_SNPE;
config->accelerate_type = AccelerateType::TYPE_CPU;
config->is_quantify_model = 0;
// ...
config->snpe_out_names.push_back("InceptionV3/Softmax");

Context Class class Context

Used to store runtime context involved in the execution process, including Model objects and Config objects. More runtime data may be extended in the future.

Constructor Context()

API Context
Description Construct Context instance object
Parameters model: Model instance object pointer
config: Config instance object pointer
Return Value Normal: Context instance object

💡Note

Special note: The memory space pointed to by the model and config pointer parameters must be allocated on the heap. After Context is successfully constructed, their lifecycle will be managed by Context (Context will automatically release model and config objects when released), so developers do not need to manually release the space of both, otherwise it will report an error of duplicate object release.

Get Model Member Variable get_model()

API get_model
Description Get the Model object pointer managed by context
Parameters void
Return Value Pointer to Model object

Get Config Member Variable get_config()

API get_config
Description Get the Config object pointer managed by context
Parameters void
Return Value Pointer to Config object

Interpreter Class class Interpreter

The Interpreter class object instance is the main body for performing inference operations and is used to execute specific inference processes. In the inference workflow mentioned in the previous sections, after creating the interpreter object, all operations are based on the interpreter object, making it the absolute core content of AidliteSDK.

Create Instance Object create_instance()

To perform inference-related operations, an inference interpreter is essential. This function is used to construct an inference interpreter instance object.

API create_instance
Description Construct an Interpreter type object using various data managed by the Context object
Parameters context: Pointer to Context instance object
Return Value If nullptr, it indicates the function failed to construct the object; otherwise, it is a pointer to the Interpreter object
cpp
// Use Context object pointer to create interpreter object, error if return value is null
Interpreter* current_interpreter = Interpreter::create_instance(context);
if(current_interpreter == nullptr){
    printf("Create Interpreter failed !\n");
    return EXIT_FAILURE;
}

Initialize init()

After the interpreter object is created, some initialization operations need to be performed (such as environment checks, resource construction, etc.).

API init
Description Complete the initialization work required for inference
Parameters void
Return Value A value of 0 indicates successful initialization; otherwise, a non-zero value indicates failure
cpp
// Initialize interpreter, error if return value is non-zero
int result = fast_interpreter->init();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->init() failed !\n");
    return EXIT_FAILURE;
}

Load Model load_model()

After the interpreter object completes initialization operations, the required model files can be loaded for the interpreter to complete model loading work. The subsequent inference process uses the loaded model resources.

API load_model
Description Complete the model loading related work. Since the model file path has been set in the Model object as mentioned above, the model loading operation can be executed directly
Parameters void
Return Value A value of 0 indicates successful model loading operation; otherwise, a non-zero value indicates failure
cpp
// Load model for interpreter, error if return value is non-zero
int result = fast_interpreter->load_model();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->load_model() failed !\n");
    return EXIT_FAILURE;
}

Get Input Tensor Details get_input_tensor_info()

If you want to get detailed information about the model's input tensors, you can use this interface to obtain it.

API get_input_tensor_info
Description Get detailed information of input Tensor in the model
Parameters in_tensor_info: Array storing detailed information of input tensor
Return Value A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note This interface is supported starting from version 2.2.5
cpp
// Get model input details data, error if return value is non-zero
std::vector<std::vector<TensorInfo>> in_tensor_info;
result = interpreter->get_input_tensor_info(in_tensor_info);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_input_tensor_info() failed !\n");
    return EXIT_FAILURE;
}
for(int gi = 0; gi < in_tensor_info.size(); ++gi){
    for(int ti = 0; ti < in_tensor_info[gi].size(); ++ti){

        std::string tensor_shape_str = std::to_string(in_tensor_info[gi][ti].shape[0]);
        for(int si = 1; si < in_tensor_info[gi][ti].dimensions; ++si){
            tensor_shape_str += " ";
            tensor_shape_str += std::to_string(in_tensor_info[gi][ti].shape[si]);
        }

        printf("Input  tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n",
                gi, ti, in_tensor_info[gi][ti].name.c_str(), in_tensor_info[gi][ti].element_count, 
                (int)in_tensor_info[gi][ti].element_type, in_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str());
    }
}

Get Output Tensor Details get_output_tensor_info()

If you want to get detailed information about the model's output tensors, you can use this interface to obtain it.

API get_output_tensor_info
Description Get detailed information of output Tensor in the model
Parameters out_tensor_info: Array storing detailed information of output tensor
Return Value A value of 0 indicates successful retrieval of Tensor details; otherwise, a non-zero value indicates failure
Special Note This interface is supported starting from version 2.2.5
cpp
// Get model output details data, error if return value is non-zero
std::vector<std::vector<TensorInfo>> out_tensor_info;
result = onnx_interpreter->get_output_tensor_info(out_tensor_info);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_output_tensor_info() failed !\n");
    return EXIT_FAILURE;
}
for(int gi = 0; gi < out_tensor_info.size(); ++gi){
    for(int ti = 0; ti < out_tensor_info[gi].size(); ++ti){

        std::string tensor_shape_str = std::to_string(out_tensor_info[gi][ti].shape[0]);
        for(int si = 1; si < out_tensor_info[gi][ti].dimensions; ++si){
            tensor_shape_str += " ";
            tensor_shape_str += std::to_string(out_tensor_info[gi][ti].shape[si]);
        }

        printf("Output tensor : Graph[%d]-Tensor[%d]-name[%s]-element_count[%ld]-element_type[%d]-dimensions[%d]-shape[%s]\n",
                gi, ti, out_tensor_info[gi][ti].name.c_str(), out_tensor_info[gi][ti].element_count, 
                (int)out_tensor_info[gi][ti].element_type, out_tensor_info[gi][ti].dimensions, tensor_shape_str.c_str());
    }
}

Set Input Data set_input_tensor()

As mentioned in the workflow introduction earlier, before setting input data, different preprocessing operations need to be completed for different models to adapt to specific models.

API set_input_tensor
Description Set the data of input tensor required for inference
Parameters in_tensor_idx/in_tensor_name: Index value/name string of input tensor
input_data: Pointer to the data storage address of input tensor
Return Value A value of 0 indicates successful setting of input Tensor; otherwise, a non-zero value indicates failure
Special Note Parameter in_tensor_name is supported starting from version 2.2.5
cpp
// Set input data for inference, error if return value is non-zero
// int result = fast_interpreter->set_input_tensor(0, input_tensor_data);
int result = fast_interpreter->set_input_tensor("in_tensor_name", input_tensor_data);
if(result != EXIT_SUCCESS){
    printf("interpreter->set_input_tensor() failed !\n");
    return EXIT_FAILURE;
}

Execute Inference .invoke()

As mentioned in the previous sections about the workflow, after setting up the input data, the next step is naturally to execute the inference process on the input data.

API invoke
Description Execute the inference computation process
Parameters void
Return Value A value of 0 indicates successful inference execution, otherwise a non-zero value indicates execution failure
cpp
// Execute inference operation, non-zero return value indicates error
int result = fast_interpreter->invoke();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter->invoke() failed !\n");
    return EXIT_FAILURE;
}

Get Output Data .get_output_tensor()

After inference is completed, the result data obtained from inference needs to be extracted. As mentioned in the previous sections about the workflow, after extracting the result data, these result data can be processed to determine whether the results are correct.

API get_output_tensor
Description Get inference result data after successful inference
Parameters out_tensor_idx/out_tensor_name: Index value/name string of the output result tensor
output_data: This value is the address of a pointer variable, the function internally resets the value of this pointer variable
output_length: The data size (in bytes) of this result output tensor
Return Value A value of 0 indicates successful output tensor retrieval, otherwise a non-zero value indicates execution failure
Special Note Parameter out_tensor_name is supported starting from version 2.2.5
cpp
// Get inference result data, non-zero return value indicates error
float* out_data = nullptr;
uint32_t output_tensor_length = 0;
// int result = fast_interpreter->get_output_tensor(0, (void**)&out_data, &output_tensor_length);
int result = fast_interpreter->get_output_tensor("out_tensor_name", (void**)&out_data, &output_tensor_length);
if(result != EXIT_SUCCESS){
    printf("interpreter->get_output_tensor() failed !\n");
    return EXIT_FAILURE;
}

Resource Cleanup .destroy()

As mentioned earlier, the interpreter object needs to perform init() initialization operations and model loading operations. Correspondingly, the interpreter also needs to perform some cleanup operations to destroy previously created resources.

API destroy
Description Perform necessary cleanup operations
Parameters void
Return Value A value of 0 indicates successful cleanup operation, otherwise a non-zero value indicates execution failure
cpp
// Execute interpreter cleanup process, non-zero return value indicates error
int result = fast_interpreter->destroy();
if(result != EXIT_SUCCESS){
    printf("sample : interpreter-> destroy() failed !\n");
    return EXIT_FAILURE;
}

Interpreter Creation Class .class InterpreterBuilder

Unified interpreter creation function for Interpreter objects. Use this class to create the required interpreter objects.

Build Interpreter .build_interpreter_from_path()

Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as SNPE, TFLite, etc.) only requires one model file, call this interface.

API build_interpreter_from_path
Description Create corresponding interpreter object directly through the path name of the model file
Parameters path: Path name of the model file
Return Value Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

Build Interpreter .build_interpreter_from_path()

Build inference interpreter object. Different parameters can be provided, the simplest being just providing the path and name of the model file. If the model framework (such as NCNN, TNN, etc.) involves two model files, call this interface.

API build_interpreter_from_path
Description Create corresponding interpreter object directly through the path name of the model file
Parameters model_struct_path: Path name of the model structure file
model_weight_path: Path name of the model weight parameter file
Return Value Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

Build Interpreter .build_interpreter_from_model()

Build inference interpreter object. In addition to providing model file paths, Model objects can also be provided, which allows setting not only the model file path and name, but also the input/output data types and input/output data shape information of the model.

API build_interpreter_from_model
Description Create corresponding interpreter object by passing in Model object. All Config-related parameters use default values
Parameters model: Model type object containing model-related data
Return Value Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

💡Note

Special note: The memory space pointed to by the model parameter must be allocated on the heap. After successful Interpreter construction, its lifecycle will be managed by the Interpreter (when the Interpreter is released, it will automatically release the model object), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.

Build Interpreter .build_interpreter_from_model_and_config()

Build inference interpreter object. In addition to the aforementioned methods, Model objects and Config objects can also be provided simultaneously, which allows providing not only model-related information but also more runtime configuration parameters.

API build_interpreter_from_model_and_config
Description Create corresponding interpreter object by passing in Model object and Config
Parameters model: Model type object containing model-related data
config: Config type object containing configuration parameters
Return Value Returns a unique_ptr pointer of Interpreter type. If it's a nullptr value, it indicates function object construction failure; otherwise it's a pointer to the Interpreter object

💡Note

Special note: The memory space pointed to by both model and config parameters must be allocated on the heap. After successful Interpreter construction, their lifecycles will be managed by the Interpreter (when the Interpreter is released, it will automatically release both model and config objects), so developers do not need to manually release the space for both objects, otherwise duplicate object release errors will occur.

cpp
// Build interpreter by passing in Model and Config objects, null return value indicates error
std::unique_ptr<Interpreter>&& fast_interpreter = 
 InterpreterBuilder::build_interpreter_from_model_and_config(model, config);
if(fast_interpreter == nullptr){
    printf("build_interpreter failed !\n");
    return EXIT_FAILURE;
}

Other Methods

In addition to the aforementioned inference-related interfaces, Aidlite-SDK also provides the following auxiliary interfaces.

Get SDK Version Information .get_library_version()

API get_library_version
Description Used to get version-related information of the current aidlite-sdk.
Parameters void
Return Value Returns the version information string of the current aidlite-sdk

Set Log Level .set_log_level()

API set_log_level
Description Set the current minimum log level, output log data greater than or equal to this log level. By default, prints WARNING level and above logs
Parameters log_level: Enumeration type LogLevel value
Return Value Returns 0 by default

Log Output to Standard Terminal .log_to_stderr()

API log_to_stderr
Description Set log information output to standard error terminal
Parameters void
Return Value Returns 0 by default

Log Output to Text File .log_to_file()

API log_to_file
Description Set log information output to specified text file
Parameters path_and_prefix: Storage path and name prefix of the log file
also_to_stderr: Indicates whether to simultaneously output logs to stderr terminal, default value is false
Return Value A value of 0 indicates successful execution, otherwise a non-zero value indicates execution failure

Get Recent Log Information .last_log_msg()

API last_log_msg
Description Get the latest log information for a specific log level, usually to get the latest error information
Parameters log_level: Enumeration type LogLevel value
Return Value Storage address pointer to the latest log information