IoT Platform provides the Link Visual SDK for Android devices. You can use this SDK to develop features for LinkVisual video devices, such as live streaming, video-on-demand (VOD), voice intercom, and image capture.
The Link Visual SDK for Android has the following dependencies.
Dependent SDK | Overview |
Provides bidirectional tunnel capabilities between devices and the cloud. |
Get the SDK
You can obtain the Link Visual SDK for Android by importing dependencies. Then, in the proguard-rules.pro file, you must exclude the classes and methods that do not need to be obfuscated.
Import dependencies
// 1. In the root build.gradle file, add a reference to the Alibaba Cloud Maven repository. allprojects { repositories { maven { url "http://maven.aliyun.com/nexus/content/repositories/releases" } } } // 2. In the app's build.gradle file, add the dependency. implementation 'com.aliyun.iotx:linkvisual-ipc:1.4.6'Obfuscation configuration
# Keep LinkVisual -keep class com.aliyun.iotx.linkvisualipc.** { *; }
Initialize the SDK
You must initialize the Link Visual SDK after the Link SDK is initialized. During initialization, you must pass the device certificate information.
LinkKit.getInstance().init(this, params, new ILinkKitConnectListener() {
@Override
public void onError(AError error) {
Log.d(TAG,
"onError() called with: error = [" + (error == null ? "null" : (error.getCode() + error.getMsg()))
+ "]");
}
@Override
public void onInitDone(Object data) {
Log.d(TAG, "onInitDone() called with: data = [" + JSON.toJSONString(data) + "]");
// Initialize the SDK.
IPCDev.getInstance().init(context, your_productKey, your_deviceName, your_deviceSecret);
}Replace your_productname, your_devicename, and your_devicesecret with your device certificate information.
The Link Visual SDK uses the capabilities of the Link Kit to listen for and process messages. You can register a listener by following these steps.
In Device services, you can register a listener for asynchronous service invocations.
// Register a listener for asynchronous service invocations. LinkKit.getInstance().getDeviceThing().setServiceHandler(service.getIdentifier(), itResRequestHandler); // Listener for asynchronous service invocations. private ITResRequestHandler itResRequestHandler = new ITResRequestHandler() { @Override public void onProcess(String identify, Object result, ITResResponseCallback itResResponseCallback) { Log.d(TAG, "ITResRequestHandler onProcess() called with: identify = [" + identify + "], result = [" + JSON.toJSONString(result) + "], itResResponseCallback = [" + itResResponseCallback + "]"); /** * Add a listener for asynchronous service invocations to the SDK. */ IPCDev.getInstance().notifyAsyncTopicReceived(identify, result, itResResponseCallback); } @Override public void onSuccess(Object o, OutputParams outputParams) { Log.d(TAG, "onSuccess() called with: o = [" + JSON.toJSONString(o) + "], outputParams = [" + JSON .toJSONString(outputParams) + "]"); } @Override public void onFail(Object o, ErrorInfo errorInfo) { Log.d(TAG, "onFail() called with: o = [" + JSON.toJSONString(o) + "], errorInfo = [" + JSON .toJSONString(errorInfo) + "]"); } };You can register a listener for synchronous service invocations.
/** * Register a listener for synchronous service invocations. */ LinkKit.getInstance().registerOnPushListener(connectNotifyListener); // Listener for synchronous service invocations. private IConnectNotifyListener connectNotifyListener = new IConnectNotifyListener() { @Override public void onNotify(String connectId, String topic, AMessage aMessage) { /** * Add a listener for synchronous service invocations to the SDK. */ IPCDev.getInstance().notifySyncTopicReceived(connectId, topic, aMessage); if (CONNECT_ID.equals(connectId) && !TextUtils.isEmpty(topic) && topic.startsWith("/sys/" + productKey + "/" + deviceName + "/rrpc/request")) { Log.d(TAG, "IConnectNotifyListener onNotify() called with: connectId = [" + connectId + "], topic = [" + topic + "], aMessage = [" + new String((byte[])aMessage.data) + "]"); } } @Override public boolean shouldHandle(String connectId, String topic) { return true; } @Override public void onConnectStateChange(String connectId, ConnectState connectState) { } };
Develop the live streaming feature
Live streaming uses RTMP for stream ingest. The supported video formats are H.264 and H.265. The supported audio formats are G.711a and AAC-LC. To develop the stream ingest feature for live streaming, follow these steps.
You can register a live stream event listener and a stream error listener.
After the Link Visual SDK receives a start stream ingest instruction from the server-side, it uses the registered live stream event listener (OnLiveStreamListener) to notify the device when to start or stop stream ingest, or to force an I-frame. You can develop this flow by following these steps.
You can set the live stream event listener.
// Set the live stream event listener. IPCDev.getInstance().getIpcStreamManager().setOnLiveStreamListener(MainActivity.this); // Set the stream error listener. IPCDev.getInstance().getIpcStreamManager().setOnStreamErrorListener(MainActivity.this);You can receive the request to start live stream ingest from the server-side.
public interface OnLiveStreamListener { /** * Receives a request to start live stream ingest. * * @param streamId Stream ID. * @param streamType Stream type. 0 indicates the major stream and 1 indicates the minor stream. * @param preTimeInS Pre-recording time in seconds. */ void onStartPushLiveStreaming(final int streamId, final int streamType, final int preTimeInS); /** * Receives a request to stop stream ingest. * * @param streamId Stream ID. */ void onStopPushStreaming(final int streamId); /** * Receives a request to force an I-frame. * You must immediately create an I-frame and send it. * @param streamId Stream ID. */ void onForceIFrame(int streamId); }
Errors that occur during stream ingest are also reported through the stream error listener. Do not run blocking tasks in the callback interface.
public interface OnStreamErrorListener { /** * Callback for stream exceptions. * @param streamId * @param error For more information, see the StreamError definition. */ void onError(int streamId, StreamError error); }The stream error codes are as follows.
Error code
Identifier
Error description
1
StreamError.ERROR_STREAM_CREATE_FAILED
Failed to create a stream instance.
2
StreamError.ERROR_STREAM_START_FAILED
Failed to start the stream.
3
StreamError.ERROR_STREAM_STOP_FAILED
Failed to stop the stream.
4
StreamError.ERROR_STREAM_SEND_VIDEO_FAILED
Failed to send video data.
5
StreamError.ERROR_STREAM_SEND_AUDIO_FAILED
Failed to send audio data.
6
StreamError.ERROR_STREAM_INVALID_PARAMS
Invalid stream parameters.
The RTMP error codes are as follows.
Error code
Identifier
Description
-1
RTMP_ILLEGAL_INPUT
Invalid input. Check the input parameters.
-2
RTMP_MALLOC_FAILED
Failed to allocate memory.
-3
RTMP_CONNECT_FAILED
Failed to establish an RTMP connection.
-4
RTMP_IS_DISCONNECTED
The RTMP connection is not established.
-5
RTMP_UNSUPPORT_FORMAT
Unsupported audio or video format.
-6
RTMP_SEND_FAILED
Failed to send an RTMP data packet.
-7
RTMP_READ_MESSAGE_FAILED
Failed to read an RTMP message.
-8
RTMP_READ_TIMESTAMP_ERROR
Incorrect input timestamp.
You can handle the request to start live stream ingest.
When the server-side sends a stream ingest request, the
OnLiveStreamListener.onStartPushLiveStreaming(int streamId, int streamType, int preTimeInS)method is called back to notify the device to start stream sampling and ingest.Typically, you need to turn on the camera and microphone for stream sampling. You can call MediaCodec to perform H.264 encoding on the data collected by the camera and perform G.711a encoding on the data collected by the microphone. You must set the audio and video parameters for the corresponding formats in advance. Then, you can call the interfaces for sending audio and video to continuously send the sampled and encoded data.
@Override public void onStartPushLiveStreaming(int streamId, int streamType, int preTimeInS) { this.streamId = streamId; try { // Create video parameters. VideoStreamParams videoStreamParams = new VideoStreamParams(); // For live streams, this parameter is always 0. videoStreamParams.setDurationInS(0); videoStreamParams.setVideoFormat(VideoStreamParams.VIDEO_FORMAT_H264); // Create audio parameters. AudioStreamParams audioStreamParams = new AudioStreamParams(); audioStreamParams.setAudioChannel(AudioStreamParams.AUDIO_CHANNEL_MONO); audioStreamParams.setAudioFormat(AudioStreamParams.AUDIO_FORMAT_G711A); audioStreamParams.setAudioEncoding(AudioStreamParams.AUDIO_ENCODING_16BIT); audioStreamParams.setAudioSampleRate(AudioStreamParams.AUDIO_SAMPLE_RATE_8000); // Set stream ingest parameters. IPCDev.getInstance().getIpcStreamManager().setStreamParams(streamId, videoStreamParams, audioStreamParams); // TODO: Start stream sampling, encoding, and sending audio and video data. } catch (NoSuchStreamException e) { e.printStackTrace(); } }Interface for sending audio and video data (IPCStreamManager).
/** * Sends an audio frame. * * @param streamId Stream ID. * @param directByteBuffer Source data. * @param length Data length. * @param timeStampInMs Timestamp of the audio frame in milliseconds. */ void sendAudioData(int streamId, ByteBuffer directByteBuffer, int length, long timeStampInMs) throws NoSuchStreamException /** * Sends a video frame. * * @param streamId Stream ID. * @param directByteBuffer Source data. * @param length Data length. * @param isIFrame Specifies whether the frame is an I-frame. * @param timeStampInMs Timestamp of the video frame in milliseconds. */ void sendVideoData(int streamId, ByteBuffer directByteBuffer, int length, boolean isIFrame, long timeStampInMs) throws NoSuchStreamException /** * Sends an audio frame. * * @param streamId Stream ID. * @param data Source data. * @param offset Offset. * @param length Data length. * @param timeStampInMs Timestamp of the audio frame in milliseconds. * @deprecated Use {@link #sendAudioData(int, ByteBuffer, int, long)} to replace this method. */ @Deprecated void sendAudioData(int streamId, byte[] data, int offset, int length, long timeStampInMs) throws NoSuchStreamException /** * Sends a video frame. * * @param streamId Stream ID. * @param data Source data. * @param offset Offset. * @param length Data length. * @param isIFrame Specifies whether the frame is an I-frame. * @param timeStampInMs Timestamp of the video frame in milliseconds. * @deprecated Use {@link #sendVideoData(int, ByteBuffer, int, boolean, long)} to replace this method. */ @Deprecated public void sendVideoData(int streamId, byte[] data, int offset, int length, boolean isIFrame, long timeStampInMs) throws NoSuchStreamExceptionYou can print the first 256 bytes of the I-frame and view the result.
The frame structures of H.264 and H.265 have specific requirements. You can print the first 256 bytes of an I-frame to check it. The code for printing is as follows.
for (int i = 0; i < ((buffer_size > 256)?256:buffer_size); i++) { printf("%02x ", buffer[i]); if ((i + 1) % 30 == 0) { printf("\n"); } } printf("\n");The frame structures of H.264 and H.265 are described as follows.
H.264
For H.264, an I-frame must have the following structure:
Frame Separator + SPS + Frame Separator + PPS + Frame Separator + IDR. The frame separator is `0x000001` or `0x00000001`. `0x67` indicates the start of a Sequence Parameter Set (SPS). `0x68` indicates the start of a Picture Parameter Set (PPS). `0x65` indicates the start of an Instantaneous Decoder Refresh (IDR) frame. The following figure shows an example.
H.265
For H.265, an I-frame must have the following structure:
Frame Separator + VPS + Frame Separator + SPS + Frame Separator + PPS + Frame Separator + IDR. The frame separator is `0x000001` or `0x00000001`. `0x40` indicates the start of a Video Parameter Set (VPS). `0x42` indicates the start of an SPS. `0x44` indicates the start of a PPS. `0x26` indicates the start of an IDR frame. The following figure shows an example.
Processes the request to stop stream ingest.
When the server-side sends a request to stop stream ingest, the
OnLiveStreamListener.onStopPushLiveStreaming()method is called back to notify the device to stop stream ingest.Typically, you need to stop collecting data from the camera and microphone, and call the
stopStreaming(int streamId)method of `IPCStreamManager`./** * Receives a request to stop stream ingest. * * @param streamId Stream ID. */ @Override public void onStopPushStreaming(int streamId) { // TODO: Stop sending audio and video data. try { // Call the interface to stop stream ingest. IPCDev.getInstance().getIpcStreamManager().stopStreaming(streamId); } catch (NoSuchStreamException e) { e.printStackTrace(); } }Handle stream faults.
During stream ingest, you can handle stream errors that are received through
OnStreamErrorListener.onError(int streamId, StreamError error).
Develop the video-on-demand feature
Video-on-demand (VOD) uses RTMP for stream ingest. The supported video formats are H.264 and H.265. The supported audio formats are G.711a and AAC-LC. To develop the stream ingest feature for VOD, follow these steps.
You can register a VOD stream event listener and a stream error listener.
After the SDK receives a start stream ingest instruction from the server-side, it uses the previously registered `OnVodStreamListener` video-on-demand stream event listener to notify you of stream ingest events, such as start, end, pause, resume, or seek.
The stream error listener also reports errors that occur during stream ingest. Do not execute blocking tasks in the callback function.
You can handle the request to query the list of recordings on the device.
When the app sends a request to query the list of recordings on the device, the device receives a synchronous service invocation (rrpc/request). In response to the request, the device returns the list of files within the queried time range to the app.
@Override public void onNotify(String connectId, String topic, AMessage aMessage) { Log.d(TAG, "onNotify() called with: connectId = [" + connectId + "], topic = [" + topic + "], aMessage = [" + new String((byte[]) aMessage.data) + "]"); /** * Add a listener to the SDK. */ IPCDev.getInstance().notifySyncTopicReceived(connectId, topic, aMessage); // Handle the synchronous service invocation. if (CONNECT_ID.equals(connectId) && !TextUtils.isEmpty(topic) && topic.contains("rrpc")) { Log.d(TAG, "IConnectNotifyListener onNotify() called with: connectId = [" + connectId + "], topic = [" + topic + "], aMessage = [" + new String((byte[]) aMessage.data) + "]"); int code = 200; String data = "{}"; JSONObject json = JSON.parseObject(new String((byte[]) aMessage.data)); if (json != null) { String method = json.getString("method"); JSONObject params = json.getJSONObject("params"); switch (method) { // Request to query the list of device recordings. case "thing.service.QueryRecordList": int beginTime = params.getIntValue("BeginTime"); int endTime = params.getIntValue("EndTime"); int querySize = params.getIntValue("QuerySize"); int type = params.getIntValue("Type"); appendLog("Received a request to query the list of device recordings: beginTime=" + beginTime + "\tendTime=" + endTime + "\tquerySize=" + querySize + "\ttype=" + type); JSONArray resultArray = new JSONArray(); JSONObject item1 = new JSONObject(); item1.put("FileName", Base64.encode("file1".getBytes(), Base64.DEFAULT)); item1.put("BeginTime", System.currentTimeMillis() / 1000 - 200); item1.put("EndTime", System.currentTimeMillis() / 1000 - 100); item1.put("Size", 1024000); item1.put("Type", 0); resultArray.add(item1); JSONObject item2 = new JSONObject(); item2.put("FileName", Base64.encode("file2".getBytes(), Base64.DEFAULT)); item2.put("BeginTime", System.currentTimeMillis() / 1000 - 100); item2.put("EndTime", System.currentTimeMillis() / 1000); item2.put("Size", 1024000); item2.put("Type", 0); resultArray.add(item2); JSONObject result = new JSONObject(); result.put("RecordList", resultArray); code = 200; data = result.toJSONString(); break; default: break; } } MqttPublishRequest request = new MqttPublishRequest(); request.isRPC = false; request.topic = topic.replace("request", "response"); String resId = topic.substring(topic.indexOf("rrpc/request/") + 13); request.msgId = resId; request.payloadObj = "{\"id\":\"" + resId + "\", \"code\":" + code + ",\"data\":" + data + "}"; LinkKit.getInstance().publish(request, new IConnectSendListener() { @Override public void onResponse(ARequest aRequest, AResponse aResponse) { appendLog("Reported successfully"); } @Override public void onFailure(ARequest aRequest, AError aError) { appendLog("Failed to report:" + aError.toString()); } }); } }ImportantFile names must be Base64-encoded.
You can handle the start stream ingest instruction.
After the app requests a device recording file from the list that is returned in the previous step, the server-side sends a stream ingest instruction. The
OnVodStreamListener.onStartPushVodStreaming(int streamId, String fileName)orOnVodStreamListener.onStartPushVodStreaming(int streamId, int beginTimeUtc, int endTimeUtc)method is called back to notify the device to ingest the audio and video data.Response for playing back device recordings by file
@Override public void onStartPushVodStreaming(int streamId, String fileName) { appendLog("Start VOD stream ingest " + streamId + " File name:" + new String(Base64.decode(fileName, Base64.NO_WRAP))); try { // Create video parameters. VideoStreamParams videoStreamParams = new VideoStreamParams(); // Duration of the video file in seconds. videoStreamParams.setDurationInS(H264_DURATION_IN_S); videoStreamParams.setVideoFormat(VideoStreamParams.VIDEO_FORMAT_H264); // Create audio parameters. AudioStreamParams audioStreamParams = new AudioStreamParams(); audioStreamParams.setAudioChannel(AudioStreamParams.AUDIO_CHANNEL_MONO); audioStreamParams.setAudioFormat(AudioStreamParams.AUDIO_FORMAT_G711A); audioStreamParams.setAudioEncoding(AudioStreamParams.AUDIO_ENCODING_16BIT); audioStreamParams.setAudioSampleRate(AudioStreamParams.AUDIO_SAMPLE_RATE_8000); // Set stream ingest parameters. IPCDev.getInstance().getIpcStreamManager().setStreamParams(streamId, videoStreamParams, audioStreamParams); // TODO: Read the fileName file and call the interface to send audio and video data for stream ingest. // After the file stream ingest is complete, call IPCDev.getInstance().getIpcStreamManager().notifyVodComplete(streamId) to report that the ingest is complete. } catch (NoSuchStreamException e) { e.printStackTrace(); } }Response for playing back device recordings by time
@Override public void onStartPushVodStreaming(int streamId, int beginTimeUtc, int endTimeUtc) { appendLog("Start VOD stream ingest " + streamId + " beginTimeUtc: "+beginTimeUtc + " endTimeUtc:"+endTimeUtc); //TODO: Add the stream ingest logic: // 1. beginTimeUtc and endTimeUtc are usually the start and end times of a day. // 2. After the onStartPushVodStreaming callback is received, start stream ingest from the nearest I-frame after beginTimeUtc. The timestamp must be the UTC time of the corresponding frame. // 3. If there are no recordings in the range from beginTimeUtc to endTimeUtc, or if stream ingest within this range is complete, call IPCDev.getInstance().getIpcStreamManager().notifyVodComplete(streamId) to report that the ingest is complete. // 4. As long as there is data in the range from beginTimeUtc to endTimeUtc, stream ingest must be continuous, even across files. }
You can handle pause or resume instructions.
You can respond to pause or resume stream ingest instructions (OnVodStreamListener) by pausing or resuming the sending of audio and video data.
/** * Receives a request to pause stream ingest. * * @param streamId Stream ID. */ void onPausePushVodStreaming(int streamId); /** * Receives a request to resume stream ingest. * * @param streamId Stream ID. */ void onResumePushVodStreaming(int streamId);You can handle the seek instruction.
When you respond to a seek instruction (OnVodStreamListener), for example, when the progress bar of the app player is moved to 80 seconds, the onSeekTo method is called back. You must continue stream ingest from the nearest I-frame at that `timeStampInS`.
/** * Receives a retargeting request. * * @param streamId Stream ID. * @param timeStampInS Time offset relative to the video start time, in seconds. */ void onSeekTo(int streamId, long timeStampInS);You can handle the stop stream ingest instruction.
When the server-side sends a request to stop the stream ingest, the
OnVodStreamListener.onStopPushLiveStreaming()callback method is invoked to instruct the device to stop the stream ingest.Typically, you need to stop calling the interface to send audio and video data, close the video file, and call the
stopStreaming(int streamId)method of `IPCStreamManager`./** * Receives a request to stop stream ingest. * * @param streamId Stream ID. */ @Override public void onStopPushStreaming(int streamId) { // TODO: Stop sending audio and video data. try { // Call the interface to stop stream ingest. IPCDev.getInstance().getIpcStreamManager().stopStreaming(streamId); catch (NoSuchStreamException e) { e.printStackTrace(); } }You can handle stream errors.
During stream ingest, you can handle stream errors that are received through
OnStreamErrorListener.onError(int streamId, StreamError error).
Develop the voice intercom feature
The voice intercom feature supports half-duplex and full-duplex modes.
One-way talk: The app captures audio and streams it to the device for playback.
Full-duplex: The app and the device must sample and play audio at the same time. The device must support Acoustic Echo Cancellation (AEC). If the device does not support AEC, this solution is not recommended.
The audio formats supported by the voice intercom feature are as follows.
Format | Sample rate | Encoding | Decoding |
G711A | 8 KHz/16 KHz | ✓ | ✓ |
G711U | 8 KHz/16 KHz | ✓ | ✓ |
The intercom feature provides V1 and V2 interfaces. Unless otherwise specified, you must use the V2 interfaces.
V1: You must implement audio sampling, playback, and echo cancellation yourself.
V2: This version supports half-duplex and full-duplex modes. It has built-in audio sampling, playback, and echo cancellation. It also lets you set the gain for sampled audio.
You can register a voice intercom event listener and an error listener.
After the SDK receives a start stream ingest instruction from the server-side, it uses the registered voice intercom event listener (OnLiveIntercomListener) to notify the device when to start or stop the voice intercom and to receive the peer's audio parameters and voice data.
Errors that occur during stream ingest are also reported to the app through the stream error listener. Do not run blocking tasks in the callback interface.
You can select the voice intercom version to use.
// Set the intercom version to V2. This is the default version. IPCDev.getInstance().setLiveIntercomVersionBeforeInit(IPCDev.LiveIntercomVersion.VERSION_2);You can select the intercom mode.
// Set the mode to full-duplex. This is valid only for V2 interfaces. IPCDev.getInstance().setLiveIntercomModeBeforeInit(IPCLiveIntercomV2.LiveIntercomMode.DoubleTalk);You can set the listener.
// Set the voice intercom event listener. IPCDev.getInstance().getIpcLiveIntercom().setOnLiveIntercomListener(MainActivity.this); // Set the voice intercom error callback. IPCDev.getInstance().getIpcLiveIntercom().setOnLiveIntercomErrorListener(MainActivity.this);
The complete code for registering the voice intercom event listener and error listener is as follows.
public interface OnLiveIntercomListener { /** * Receives a request from the app to start the voice intercom. * * @return Returns the format of the upstream audio parameters on the current device, such as sample rate, number of channels, sample bit depth, and audio format. Make sure that the app supports this audio parameter configuration. */ AudioParams onStartVoiceIntercom(); /** * Receives a request to stop the voice intercom. */ void onStopVoiceIntercom(); /** * Receives audio parameters from the app. This indicates that the channel with the app is established and the intercom can start. * @param audioParams Audio parameters of the app. */ void onAudioParamsChange(AudioParams audioParams); /** * Receives PCM data sent from the app. This is typically used for UI display, such as drawing a volume level indicator. * @param buffer * @param size */ void onAudioBufferReceive(byte[] buffer, int size); }public interface OnLiveIntercomErrorListener { /** * An error occurred in the voice intercom. * @param error See {@link LiveIntercomError}. */ void onError(LiveIntercomError error); }You can respond to the request to start the voice intercom (only for V1 interfaces).
You can start the audio recorder to begin audio sampling and send the audio data to the peer.
@Override public AudioParams onStartVoiceIntercom() { appendLog("Received instruction to start voice intercom"); // Received a request to start voice intercom. Start the audio recorder. simpleAudioRecord.setAudioRecordListener(new AudioRecordListener() { @Override public void onRecordStart() { // Recording started. appendLog("Recording started"); } @Override public void onRecordEnd() { // Recording ended. appendLog("Recording ended"); } @Override public void onBufferReceived(byte[] buffer, int offset, int size) { // Received PCM data from the audio recorder. Call the send interface to send it to the peer. IPCDev.getInstance().getIpcLiveIntercom().sendAudioBuffer(buffer, offset, size); } @Override public void onError(int error, String message) { appendLog("Audio recorder error:" + error + " " + message); } }); simpleAudioRecord.start(); // Notify that G.711a is used as the audio sending format. The PCM data will be re-encoded internally. return AudioParams.AUDIOPARAM_MONO_8K_G711A; }Interface for sending audio data (IPCLiveIntercom, only for V1 interfaces).
/** * Sends audio data.<br> * Equivalent to {@link #sendAudioBuffer(byte[] data, int offset, int length, boolean enableEncode)} with enableEncode=true. */ void sendAudioBuffer(byte[] data, int offset, int length); /** * Sends audio data.<br> * * @param data Data buffer. * @param offset Offset. * @param length Length. * @param enableEncode true: The data sent (must be PCM) is re-encoded and sent based on the audio format returned by {@link OnLiveIntercomListener#onStartVoiceIntercom()}. * false: The audio data is sent directly. */ void sendAudioBuffer(byte[] data, int offset, int length, boolean enableEncode);
You can handle audio playback (only for V1 interfaces).
After the voice intercom channel is established between the app and the device, the device receives the audio parameters from the app through
onAudioParamsChange(AudioParams audioParams). You can create a new audio player based on these audio parameters to play back the subsequently received audio data.@Override public void onAudioParamsChange(AudioParams audioParams) { // Received audio parameters from the peer. appendLog("Received audio parameters from the client: " + audioParams.toString()); // Initialize the player. if (simpleStreamAudioTrack != null) { simpleStreamAudioTrack.release(); audioTrackQueue.clear(); } if (acousticEchoCanceler != null) { acousticEchoCanceler.release(); } if (noiseSuppressor != null) { noiseSuppressor.release(); } noiseSuppressor = NoiseSuppressor.create(simpleAudioRecord.getAudioSessionId()); if (noiseSuppressor != null) { noiseSuppressor.setEnabled(true); } simpleStreamAudioTrack = new SimpleStreamAudioTrack(audioParams, AudioManager.STREAM_MUSIC, audioTrackQueue, simpleAudioRecord.getAudioSessionId()); if (AcousticEchoCanceler.isAvailable()) { acousticEchoCanceler = AcousticEchoCanceler.create(simpleAudioRecord.getAudioSessionId()); if (acousticEchoCanceler != null) { appendLog("Echo cancellation enabled"); acousticEchoCanceler.setEnabled(true); } } simpleStreamAudioTrack.start(); } @Override public void onAudioBufferReceive(byte[] buffer, int size) { // Received PCM data from the peer. audioTrackQueue.add(buffer); }You can handle the instruction to stop the voice intercom (only for V1 interfaces).
@Override public void onStopVoiceIntercom() { appendLog("Received instruction to stop voice intercom"); // Received a request to stop voice intercom. Stop recording. simpleAudioRecord.stop(); }You can handle voice intercom errors.
@Override public void onError(LiveIntercomError error) { // A voice intercom error occurred. appendLog("Voice intercom error:" + error.getCode() + " msg:" + error.getMessage()); // Stop recording (only for V1 interfaces). simpleAudioRecord.stop(); // Stop playback (only for V1 interfaces). simpleStreamAudioTrack.stop(); }
The voice intercom error codes are as follows.
Error code | Error description |
LiveIntercomError.INVALID_AUDIO_PARAMS | Invalid device-side audio parameters. |
LiveIntercomError.START_LIVE_INTERCOM_REQUEST_FAILED | Invalid voice intercom request. |
LiveIntercomError.CONNECTION_STREAM_FAILED | Failed to establish a voice intercom stream channel. |
LiveIntercomError.SEND_STREAM_DATA_FAILED | Failed to send audio data. |
LiveIntercomError.RECEIVE_STREAM_DATA_FAILED | Failed to receive audio data. |
LiveIntercomError.INIT_RECORD_FAILED | Audio recorder initialization error. |
LiveIntercomError.START_RECORD_FAILED | Audio recorder startup error. |
LiveIntercomError.READ_RECORD_BUFFER_FAILED | Audio recorder data read error. |
LiveIntercomError.INIT_AUDIO_PLAYER_FAILED | Failed to create the audio player. |
Develop the image capture feature
The image capture feature uses the IUploadPicListener callback to upload images. The callback information includes the upload URL, the ID of the image to be uploaded, and the upload type. After the image is uploaded, the device uses the IUploadPicProcessCallback callback to notify the SDK that the upload task is complete. The image capture feature is primarily used in the following two scenarios.
The app sends an image capture request. The device captures an image and uploads it to the cloud.
A detection event is triggered. The device automatically captures an image and uploads it to the cloud.
You must also check the string length of UploadUrl in the TriggerPicCapture Thing Specification Language (TSL) model for image capture. If the length is less than 512 characters, you must increase it to 512 or more. If you do not increase the length, image uploads are affected.
You can capture and upload an image.
The typical process for an image capture and upload task is as follows: the SDK notifies the device to capture an image, the device captures and uploads the image, and then the device notifies the SDK of the upload result. The detailed process is as follows.
The device registers a listener for image uploads.
/** * Listener for image uploads. */ public interface IUploadPicListener { /** * Triggers an upload. * @param uploadInfo Upload information. * @param callback Callback for the upload result. */ void onUpload(UploadInfo uploadInfo, IUploadPicProcessCallback callback); }When the app triggers an image capture, the device receives a notification to capture and upload an image through the
onUploadmethod ofIUploadPicListener.You can obtain the image upload URL from the `uploadInfo` parameter of the `onUpload` callback.
/** * Uploads an image. */ public class UploadInfo { /** * Upload URL. */ private String uploadUrl; /** * Image ID for the upload (user-generated, available only for event reporting). */ private String picId; /** * Upload type. 0 indicates an image capture. 1 indicates an event image upload. */ private int type; }The device uses the
IUploadPicProcessCallbackcallback to notify the SDK of the upload result./** * Callback for the image upload result. */ public interface IUploadPicProcessCallback { /** * The upload task is successful. */ void onSucceed(); /** * The upload task failed. * * @param errorMsg Error message. */ void onFailed(String errorMsg); }You can listen for the registration result.
After the LinkKit SDK is successfully initialized, the device can register the listener using the following code.
// Register a listener for image uploads. When the app triggers an image capture or the device reports an event, the onUpload callback is received. IPCDev.getInstance().registerUploadPicListener(uploadPicListener); /** * Image upload callback. In this callback, picId is not empty only for event image uploads, and it is the ID uploaded by the user when reporting the event. * type is the upload type. 0 indicates an image capture triggered by the app. 1 indicates an event image upload by the device. */ private IUploadPicListener uploadPicListener = new IUploadPicListener() { @Override public void onUpload(final UploadInfo uploadInfo, final IUploadPicProcessCallback callback) { final String url = uploadInfo.getUploadUrl(); final String picId = uploadInfo.getPicId(); final int type = uploadInfo.getType(); switch (type) { // Image capture report. case 0: // Capture and upload the image. upLoadFile(url, pic, new CallBack(){ @Override public void onSucceed() { Log.d(TAG, "uploadImage onSucceed"); if (callback != null) { callback.onSucceed(); } } @Override public void onFailed(String error) { Log.e(TAG, "uploadImage onFailed:" + error); if (callback != null) { callback.onFailed(error); } } }); break; // Event image report. case 1: // Upload the image. upLoadFile(url, pic, new CallBack(){ @Override public void onSucceed() { Log.d(TAG, "uploadImage onSucceed"); if (callback != null) { callback.onSucceed(); } } @Override public void onFailed(String error) { Log.e(TAG, "uploadImage onFailed:" + error); if (callback != null) { callback.onFailed(error); } } }); break; default: break; } }};
Event reporting triggers an image upload.
The event reporting process is as follows: the device triggers and reports an event, the SDK notifies the device to upload an image, the device uploads the image, and then the device notifies the SDK of the upload result. The detailed process is as follows.
The device registers a listener for image uploads.
When an event is triggered on the device, the device reports the event through the interface provided by the SDK (the image ID is required).
The SDK notifies the device to upload the image through the
onUploadmethod ofIUploadPicListener.The device can obtain the upload URL and image ID from the UploadInfo parameter in the
onUploadmethod.The device uploads the image using the URL.
The device uses the
IUploadPicProcessCallbackcallback to notify the SDK of the upload result.
The following is an example of the event reporting interface.
// Reporting an alarm event consists of two steps: 1. Report the alarm. 2. Upload the image. // To report an alarm event, the device must generate an alarm image ID. This ID must be unique on the device. We recommend using a Unix timestamp. // However, you need to check the string length of 'UploadUrl' in the 'TriggerPicCapture' TSL model. If it is less than 512, increase it to 512 or more. // After an event is reported, the onUpload callback within IUploadPicListener (requires listener registration) is received. Then, upload the corresponding image to the URL specified in the callback. final String alarmPicId = String.valueOf(System.currentTimeMillis() / 1000); IPCDev.getInstance().reportAlarmEvent(alarmPicId, 1, new ReportAlarmEventListener() { @Override public void onSucceed() { Log.d(TAG, " report onSucceed alarmPicId:" + alarmPicId); appendLog("Event reported successfully"); } @Override public void onFailed(String msg) { Log.e(TAG, " report onFailed e:" + msg); appendLog("Failed to report event e:" + msg); } });