Client Integration

Client integration is based on web socket provided by Core runner service.
Java/Android Client
If you want to create your own client in Java or Android platforms, you can use Promethist Common Client library for that. The whole source code of it is a part of Promethist Core project and can be found on GitHub. 
We recommend using Java version 11 (OpenJDK).
First, add PromethistAI artifact repository to your project
Maven
Gradle
<repositories>
    <repository>
        <id>promethistai</id>
        <name>PromethistAI repository</name>
        <url>https://gitlab.com/api/v4/groups/5074098/-/packages/maven</url>
    </repository>
</repositories>
allprojects {
    repositories {
        maven { url 'https://gitlab.com/api/v4/groups/5074098/-/packages/maven' }
    }
}
and set dependency to promethist-client-common  to your project or module
Maven
Gradle
<dependency>
    <groupId>ai.flowstorm.client</groupId>
    <artifactId>flowstorm-client-lib</artifactId>
    <version>2.68.0</version>
</dependency>
dependencies {
    implementation 'ai.flowstorm.client:flowstorm-client-lib:2.68.0'
}
Latest stable version number can be checked here​
Callback Object
First of all, every client has to implement BotClientCallback interface - following example demonstrates it (you can replace method implementations just printing what's has occurred with audio/visual processing logic of your client)
class Callback : BotClientCallback {
​
    override fun onOpen(client: BotClient) = println("{Open}")
    override fun onReady(client: BotClient) = println("{Ready}")
    override fun onClose(client: BotClient) = println("{Closing}")
    override fun onError(client: BotClient, text: String) = println("{Error $text}")
    override fun onFailure(client: BotClient, t: Throwable) = t.printStackTrace()
    override fun audioCancel() = println("{Audio Cancel}")
    override fun command(client: BotClient, command: String, code: String?) = println("{Command $command $code}")
    override fun httpRequest(client: BotClient, url: String, request: HttpRequest?) = HttpUtil.httpRequest(url, request)
​
    override fun onSessionId(client: BotClient, sessionId: String?) = println("{Session ID $sessionId}")
    override fun onBotStateChange(client: BotClient, newState: BotClient.State) = println("{State change to $newState}")
    override fun onWakeWord(client: BotClient) = println("{Wake word}")
​
    override fun audio(client: BotClient, data: ByteArray) = println("{Audio ${data.size} bytes}")
    override fun video(client: BotClient, url: String) = println("{Video $url}")
    override fun image(client: BotClient, url: String) = println("{image $url}")
    override fun text(client: BotClient, text: String) = println("< $text")
​
    override fun onRecognized(client: BotClient, text: String) = println("> $text")
    override fun onLog(client: BotClient, logs: MutableList<LogEntry>) = logs.forEach { println(it) }
}
Then, you need to create BotContext object defining relation between client a conversational application
val context = BotContext(
    "https://core.promethist.com", // Core service URL
    "5ea17702d28fd40eec1e9076", // application ID
    "device_ID", // CHANGE TO unique sender device identification
    Dynamic("clientType" to "client_type:1.0.0-SNAPSHOT"), // CHANGE TO your-client-name:version
    autoStart = false // set to true if you want to start application immediately as soon as client is open
)
Bot Client 
Client is represented by BotClient class which implements network communication with the Core service using web socket, processing input audio and playing output audio. To understand better networking part of implementation please take a look on web socket page too.
State
It is always in one of following states accessible via public property state
State
Description
Sleeping
No conversation is going on
Listening
Waiting for user input (if input audio device is available as described below, sending input audio to the server; otherwise waiting for text input by calling doText method)
Processing
User text input sent or text recognized in input audio on server side, waiting for response
Responding
Processing server response = display response text(s)/images and playing audio(s). As soon as the last audio play is finished it will go back to Listening state IF the response have not finished session (in that case client goes immediately to Sleeping state)
Open
Short time state between opening web socket and obtaining response to Init event.
Closed
Network connection is closed (after calling close method) 
Failed
Network connection has failed. Client will keep trying to restore it every 20 seconds
Methods
Method
Description
open()
Open communication to the server. Call this method first and once before any other 
doText(text: String, attributes: PropertyMap = emptyMap())
Send text input to the server e.g. #signal, what's the weather. Optionally you can also use this method to update some client attributes​
doIntro()
Initiate conversation - equal to call toText("#intro")
touch()
Touch client (usually - client calls it after user clicks or touches "play" button). Depending on client state
Sleeping - cancel audio playing, calls doIntro() 
Responding - if client has pausing enabled, it will pause; otherwise output audio will be canceled and client will go to Listening
Paused - resume output audio
Closed or Failed - error audio response 
close()
Close client. Output audio play is cancelled, input audio device and network socked are close. 
Text Bot
If you want to create client without audio support, you can instantiate BotClient without AudioDevice object
val audioDevice: AudioDevice? = null
val client = BotClient(context, OkHttp3BotClientSocket(context.url), audioDevice, Callback())
client.open()
You can try to create simple client using standard input/output handling TextConsole object
val console = TextConsole() {
    override fun afterInput(text: String) = client.doText(text)
}
console.run() 
 Voice Bot
If you want to create voice (input audio) using client, you have to create object of  AudioDevice class and pass it to BotClient  constructor. When using Java JRE, you can take Microphone class from Standalone application using Java Sound API.
val audioDevice = Microphone()
​
// if your microphone has multiple channels, you have to specify 
// count and index of microphone channel to be used
val audioDevice = Microphone(channels = 6, channel = 1)
Microphone sample rate is set to 16000 Hz, size 16 bits, signed, little endian by default.  
Custom Input Audio Device
If case that you have specific device providing audio input you can implement your own InputAudioDevice​
class MyAudioDevice : InputAudioDevice() {
    override fun read(buffer: ByteArray): Int = TODO("Add audio data reading code here")
}
Example of custom InputAudioDevice implementation for Android
class AudioRecorder(var autoClose: Boolean = false) : InputAudioDevice() {
​
    companion object {
​
        private val SAMPLE_RATE_CANDIDATES = intArrayOf(16000, 11025, 22050, 44100)
        private val CHANNEL = AudioFormat.CHANNEL_IN_MONO
        private val ENCODING = AudioFormat.ENCODING_PCM_16BIT
    }
​
    var bufferSize: Int = 0
    var audioRecord: AudioRecord? = null
​
    override fun start() {
        audioRecord = createAudioRecord()
        audioRecord!!.startRecording()
        if (audioRecord?.recordingState == AudioRecord.RECORDSTATE_RECORDING)
            super.start()
    }
​
    override fun stop() {
        super.stop()
        if (audioRecord?.state == AudioRecord.RECORDSTATE_RECORDING)
            audioRecord?.stop()
        audioRecord?.release()
        audioRecord = null
    }
​
    override fun read(buffer: ByteArray): Int = audioRecord!!.read(buffer, 0, buffer.size)
​
    private fun createAudioRecord(): AudioRecord {
        for (sampleRate in SAMPLE_RATE_CANDIDATES) {
            var sizeInBytes = AudioRecord.getMinBufferSize(sampleRate, CHANNEL, ENCODING)
            if (sizeInBytes == AudioRecord.ERROR_BAD_VALUE) {
                continue
            }
            sizeInBytes = sizeInBytes * 4
            val audioRecord = AudioRecord(
                MediaRecorder.AudioSource.MIC,
                sampleRate,
                CHANNEL,
                ENCODING, sizeInBytes
            )
            if (audioRecord.state == AudioRecord.STATE_INITIALIZED) {
                bufferSize = sizeInBytes
                return audioRecord
            } else {
                audioRecord.release()
            }
        }
        throw RuntimeException("Cannot instantiate AudioRecorder")
    }
}

Last modified 3mo ago

State	Description
`Sleeping`	No conversation is going on
`Listening`	Waiting for user input (if input audio device is available as described below, sending input audio to the server; otherwise waiting for text input by calling `doText` method)
`Processing`	User text input sent or text recognized in input audio on server side, waiting for response
`Responding`	Processing server response = display response text(s)/images and playing audio(s). As soon as the last audio play is finished it will go back to `Listening` state IF the response have not finished session (in that case client goes immediately to `Sleeping` state)
`Open`	Short time state between opening web socket and obtaining response to Init event.
`Closed`	Network connection is closed (after calling `close` method)
`Failed`	Network connection has failed. Client will keep trying to restore it every 20 seconds

Method	Description
`open()`	Open communication to the server. Call this method first and once before any other
`doText(text: String, attributes: PropertyMap = emptyMap())`	Send text input to the server e.g. `#signal`, `what's the weather`. Optionally you can also use this method to update some client attributes
`doIntro()`	Initiate conversation - equal to call `toText("#intro")`
`touch()`	Touch client (usually - client calls it after user clicks or touches "play" button). Depending on client state `Sleeping` - cancel audio playing, calls `doIntro()` `Responding` - if client has pausing enabled, it will pause; otherwise output audio will be canceled and client will go to `Listening` `Paused` - resume output audio `Closed` or `Failed` - error audio response
`close()`	Close client. Output audio play is cancelled, input audio device and network socked are close.