AidLLM C++ API Documentation
💡Note
When developing with AidGen-SDK C++, you need to know the following basic information:
- Include the header file during compilation, located at
/usr/local/include/aidlux/aidgen/aidllm.hpp - Specify the library file during linking, located at
/usr/local/lib/libaidgen.so
Inference Backend Type.enum LLmBackendType
For AidllmSDK, it supports using different inference backend frameworks to implement large model inference tasks. The inference backends involved are shown below.
| Member Variable Name | Type | Value | Description |
|---|---|---|---|
| TYPE_DEFAULT | uint8_t | 0 | Unknown backend type |
| TYPE_GENIE | uint8_t | 1 | Genie inference backend |
Inference Task Status.enum LLMSentenceState
During the completion of inference tasks, Aidllm may have various states for a single conversation. Developers can understand the current inference task's running status through these status codes.
| Member Variable Name | Type | Value | Description |
|---|---|---|---|
| BEGIN | uint8_t | 0 | Beginning of conversation |
| CONTINUE | uint8_t | 1 | Continuous inference content in conversation |
| END | uint8_t | 2 | End of conversation |
| COMPLETE | uint8_t | 3 | Current conversation successfully completed |
| ABORT | uint8_t | 4 | Current conversation passively terminated |
| ERROR | uint8_t | 5 | Current conversation inference error |
Log Level.enum LogLevel
AidllmSDK provides interfaces for setting logs (to be introduced later), which require specifying the log level to use, so this log level enumeration is needed.
| Member Variable Name | Type | Value | Description |
|---|---|---|---|
| INFO | uint8_t | 0 | Information |
| WARNING | uint8_t | 1 | Warning |
| ERROR | uint8_t | 2 | Error |
| FATAL | uint8_t | 3 | Fatal error |
Inference Callback Function Data Type.struct LLMCallbackData
Aidllm uses callback functions provided by developers during inference tasks. This data type is the parameter passed to the callback function, which developers can use to handle inference result data in their custom callback functions.
Member Variable List
The Config object is used to set runtime configuration information, including the following parameters:
| Member Variable | state |
| Type | enum LLMSentenceState |
| Default Value | |
| Description | Status code of the current inference conversation |
| Member Variable | text |
| Type | std::string |
| Default Value | |
| Description | Result characters/prompt messages corresponding to special status codes of the inference task |
Runtime Context Class.class LLMContext
During Aidllm runtime, some configuration information may need to be set, and runtime-related data also needs to be passed. This runtime context class object is used to complete data flow.
Create Instance Object.create_instance()
To set runtime context information, you need to have a configuration instance object first. This function is used to create an instance object of LLMContext type.
| API | create_instance |
| Description | Used to construct an instance object of LLMContext class |
| Parameters | config_file: Initial configuration file, which can configure key information such as backend type, model file name, etc. |
| Return Value | If nullptr, the function failed to build the object; otherwise, it's a pointer to the LLMContext object |
// Create configuration instance object, return null value indicates error
std::unique_ptr<LLMContext> llm_context_ptr = LLMContext::create_instance("qwen2-7b/qwen2-7b.json");
if(llm_context_ptr == nullptr){
printf("Test sample: LLMContext create_instance failed.\n");
return EXIT_FAILURE;
}Member Variable List
The Config object is used to set runtime configuration information, including the following parameters:
| Member Variable | config_file |
| Type | std::string |
| Default Value | |
| Description | Initial configuration file, configuration file parameter passed when creating object |
| Member Variable | backend_type |
| Type | LLmBackendType |
| Default Value | LLmBackendType.TYPE_DEFAULT |
| Description | The configuration file requires developers to specify the inference backend. After the initialization operation parses the configuration file, this field will be overwritten to identify the backend type specified by the developer |
| Member Variable | model_file_vec |
| Type | std::vector<std::string> |
| Default Value | |
| Description | The configuration file requires developers to specify model files. After the initialization operation parses the configuration file, this field will be overwritten to identify the model files specified by the developer |
| Member Variable | config_overwrite_options |
| Type | std::string |
| Default Value | |
| Description | By setting this field, certain key parameters in the inference process can be specified, thereby affecting inference speed, inference results, etc. |
| Member Variable | android_tmp_directory |
| Type | std::string |
| Default Value | |
| Description | This field is only valid on Android platform. By setting this field, directories with legal permissions for system users can be specified for inference programs to borrow |
Interpreter Class.class LLMInterpreter
LLMInterpreter class object instances are the main body for executing inference operations, used to perform specific inference processes.
Create Instance Object.create_instance()
To execute inference-related operations, an inference interpreter is essential. This function is used to build an instance object of the inference interpreter.
| API | create_instance |
| Description | Construct an LLMInterpreter type object using various data managed by the LLMContext object |
| Parameters | llm_context: Pointer to LLMContext instance object |
| Return Value | If nullptr, the function failed to build the object; otherwise, it's a pointer to the LLMInterpreter object |
// Create interpreter object using LLMContext object pointer, return null value indicates error
std::unique_ptr<LLMInterpreter> llm_interpreter_ptr = LLMInterpreter::create_instance(llm_context_ptr);
if(llm_interpreter_ptr == nullptr){
printf("Test sample: LLMInterpreter create_instance failed.\n");
return EXIT_FAILURE;
}Initialize Operation.initialize()
After the interpreter object is created, some initialization operations need to be performed, such as environment checking, resource building, etc.
| API | initialize |
| Description | Complete the relevant initialization work required for inference |
| Parameters | void |
| Return Value | 0 value indicates successful initialization operation, otherwise non-0 value indicates execution failure |
// Initialize interpreter, return non-zero value indicates error
int init_result = llm_interpreter_ptr->initialize();
if(init_result != EXIT_SUCCESS){
printf("Test sample: aidllm initialize failed.\n");
return EXIT_FAILURE;
}Conversation Inference Operation.run()
After successfully completing the initialization operation, you can execute dialogue operations with the large model. Developers provide custom callback functions to handle continuous inference results during the conversation process.
| API | run |
| Description | Execute one conversation inference |
| Parameters | prompt: Prompt string |
| cb: Custom callback function provided by developers to handle continuous inference results during the conversation process | |
| user_data: Pointer to user data, convenient for using this data in custom callback functions | |
| Return Value | 0 value indicates successful execution of release operation, otherwise non-0 value indicates execution failure |
Conversation Termination Operation.abort()
Users may want to interrupt the currently executing inference conversation in certain situations. This function is used to implement inference termination.
| API | abort |
| Description | Terminate the currently running inference conversation |
| Parameters | void |
| Return Value | 0 value indicates successful initialization operation, otherwise non-0 value indicates execution failure |
// Initialize interpreter, return non-zero value indicates error
int init_result = llm_interpreter_ptr->initialize();
if(init_result != EXIT_SUCCESS){
printf("Test sample: aidllm initialize failed.\n");
return EXIT_FAILURE;
}Finalize Release Operation.finalize()
As mentioned earlier, the interpreter object needs to execute initialize() initialization-related operations. Correspondingly, the interpreter also needs to execute some release operations to destroy previously created resources, etc.
| API | finalize |
| Description | Complete necessary de-initialization release operations |
| Parameters | void |
| Return Value | 0 value indicates successful execution of release operation, otherwise non-0 value indicates execution failure |
// Execute interpreter de-initialization process, return non-zero value indicates error
int fin_result = llm_interpreter_ptr->finalize();
if(fin_result != EXIT_SUCCESS){
printf("Test : aidllm finalize failed.\n");
return EXIT_FAILURE;
}