Links

Client Integration

Client integration is based on web socket provided by Core runner service.

Java/Android Client

If you want to create your own client in Java or Android platforms, you can use Promethist Common Client library for that. The whole source code of it is a part of Promethist Core project and can be found on GitHub.
We recommend using Java version 11 (OpenJDK).
First, add PromethistAI artifact repository to your project
Maven
Gradle
<repositories>
<repository>
<id>promethistai</id>
<name>PromethistAI repository</name>
<url>https://gitlab.com/api/v4/groups/5074098/-/packages/maven</url>
</repository>
</repositories>
allprojects {
repositories {
maven { url 'https://gitlab.com/api/v4/groups/5074098/-/packages/maven' }
}
}
and set dependency to promethist-client-common to your project or module
Maven
Gradle
<dependency>
<groupId>ai.flowstorm.client</groupId>
<artifactId>flowstorm-client-lib</artifactId>
<version>2.68.0</version>
</dependency>
dependencies {
implementation 'ai.flowstorm.client:flowstorm-client-lib:2.68.0'
}
Latest stable version number can be checked here

Callback Object

First of all, every client has to implement BotClientCallback interface - following example demonstrates it (you can replace method implementations just printing what's has occurred with audio/visual processing logic of your client)
class Callback : BotClientCallback {
override fun onOpen(client: BotClient) = println("{Open}")
override fun onReady(client: BotClient) = println("{Ready}")
override fun onClose(client: BotClient) = println("{Closing}")
override fun onError(client: BotClient, text: String) = println("{Error $text}")
override fun onFailure(client: BotClient, t: Throwable) = t.printStackTrace()
override fun audioCancel() = println("{Audio Cancel}")
override fun command(client: BotClient, command: String, code: String?) = println("{Command $command $code}")
override fun httpRequest(client: BotClient, url: String, request: HttpRequest?) = HttpUtil.httpRequest(url, request)
override fun onSessionId(client: BotClient, sessionId: String?) = println("{Session ID $sessionId}")
override fun onBotStateChange(client: BotClient, newState: BotClient.State) = println("{State change to $newState}")
override fun onWakeWord(client: BotClient) = println("{Wake word}")
override fun audio(client: BotClient, data: ByteArray) = println("{Audio ${data.size} bytes}")
override fun video(client: BotClient, url: String) = println("{Video $url}")
override fun image(client: BotClient, url: String) = println("{image $url}")
override fun text(client: BotClient, text: String) = println("< $text")
override fun onRecognized(client: BotClient, text: String) = println("> $text")
override fun onLog(client: BotClient, logs: MutableList<LogEntry>) = logs.forEach { println(it) }
}
Then, you need to create BotContext object defining relation between client a conversational application
val context = BotContext(
"https://core.promethist.com", // Core service URL
"5ea17702d28fd40eec1e9076", // application ID
"device_ID", // CHANGE TO unique sender device identification
Dynamic("clientType" to "client_type:1.0.0-SNAPSHOT"), // CHANGE TO your-client-name:version
autoStart = false // set to true if you want to start application immediately as soon as client is open
)

Bot Client

Client is represented by BotClient class which implements network communication with the Core service using web socket, processing input audio and playing output audio. To understand better networking part of implementation please take a look on web socket page too.

State

It is always in one of following states accessible via public property state
State
Description
Sleeping
No conversation is going on
Listening
Waiting for user input (if input audio device is available as described below, sending input audio to the server; otherwise waiting for text input by calling doText method)
Processing
User text input sent or text recognized in input audio on server side, waiting for response
Responding
Processing server response = display response text(s)/images and playing audio(s). As soon as the last audio play is finished it will go back to Listening state IF the response have not finished session (in that case client goes immediately to Sleeping state)
Open
Short time state between opening web socket and obtaining response to Init event.
Closed
Network connection is closed (after calling close method)
Failed
Network connection has failed. Client will keep trying to restore it every 20 seconds

Methods

Method
Description
open()
Open communication to the server. Call this method first and once before any other
doText(text: String, attributes: PropertyMap = emptyMap())
Send text input to the server e.g. #signal, what's the weather. Optionally you can also use this method to update some client attributes
doIntro()
Initiate conversation - equal to call toText("#intro")
touch()
Touch client (usually - client calls it after user clicks or touches "play" button). Depending on client state
  • Sleeping - cancel audio playing, calls doIntro()
  • Responding - if client has pausing enabled, it will pause; otherwise output audio will be canceled and client will go to Listening
  • Paused - resume output audio
  • Closed or Failed - error audio response
close()
Close client. Output audio play is cancelled, input audio device and network socked are close.

Text Bot

If you want to create client without audio support, you can instantiate BotClient without AudioDevice object
val audioDevice: AudioDevice? = null
val client = BotClient(context, OkHttp3BotClientSocket(context.url), audioDevice, Callback())
client.open()
You can try to create simple client using standard input/output handling TextConsole object
val console = TextConsole() {
override fun afterInput(text: String) = client.doText(text)
}
console.run()

Voice Bot

If you want to create voice (input audio) using client, you have to create object of AudioDevice class and pass it to BotClient constructor. When using Java JRE, you can take Microphone class from Standalone application using Java Sound API.
val audioDevice = Microphone()
// if your microphone has multiple channels, you have to specify
// count and index of microphone channel to be used
val audioDevice = Microphone(channels = 6, channel = 1)
Microphone sample rate is set to 16000 Hz, size 16 bits, signed, little endian by default.

Custom Input Audio Device

If case that you have specific device providing audio input you can implement your own InputAudioDevice
class MyAudioDevice : InputAudioDevice() {
override fun read(buffer: ByteArray): Int = TODO("Add audio data reading code here")
}
Example of custom InputAudioDevice implementation for Android
class AudioRecorder(var autoClose: Boolean = false) : InputAudioDevice() {
companion object {
private val SAMPLE_RATE_CANDIDATES = intArrayOf(16000, 11025, 22050, 44100)
private val CHANNEL = AudioFormat.CHANNEL_IN_MONO
private val ENCODING = AudioFormat.ENCODING_PCM_16BIT
}
var bufferSize: Int = 0
var audioRecord: AudioRecord? = null
override fun start() {
audioRecord = createAudioRecord()
audioRecord!!.startRecording()
if (audioRecord?.recordingState == AudioRecord.RECORDSTATE_RECORDING)
super.start()
}
override fun stop() {
super.stop()
if (audioRecord?.state == AudioRecord.RECORDSTATE_RECORDING)
audioRecord?.stop()
audioRecord?.release()
audioRecord = null
}
override fun read(buffer: ByteArray): Int = audioRecord!!.read(buffer, 0, buffer.size)
private fun createAudioRecord(): AudioRecord {
for (sampleRate in SAMPLE_RATE_CANDIDATES) {
var sizeInBytes = AudioRecord.getMinBufferSize(sampleRate, CHANNEL, ENCODING)
if (sizeInBytes == AudioRecord.ERROR_BAD_VALUE) {
continue
}
sizeInBytes = sizeInBytes * 4
val audioRecord = AudioRecord(
MediaRecorder.AudioSource.MIC,
sampleRate,
CHANNEL,
ENCODING, sizeInBytes
)
if (audioRecord.state == AudioRecord.STATE_INITIALIZED) {
bufferSize = sizeInBytes
return audioRecord
} else {
audioRecord.release()
}
}
throw RuntimeException("Cannot instantiate AudioRecorder")
}
}