Creating Jami plugins

NOTE: this page introduces the Jami Plugins SDK.

Jami Plugins

As from September of 2020, Jami team has added plugins as a call feature for GNU/Linux, Windows, and Android users. This meaning that now you can personalize your call and chat experience by using one of our available plugins. But that is not all, you can also transform your awesome ideas into a brand new plugin!

Here you will be guided throught the SDK that will help you start your plugin developpment. The text is organized as:

• A description of our #SDK;

• An example of how to create your own base plugin with our SDK - #Create my first plugin.

SDK

We developped a Plugin System for Jami and we have a few plugins available to be used. However as an open source project, we now desire users to be able to create, use, and distribute their own plugins. To achieve that goal, we also developped a Jami Plugins SDK. This kit is fully writen in python, and can be invoked running pluginMainSDK.py from <plugins> folder. To get started you must:

mkdir jami-plugins && cd jami-plugins
git clone https://review.jami.net/jami-daemon daemon
git clone https://review.jami.net/jami-plugins plugins
cd plugins
pip3 install -r SDK/requirements.txt
python3 SDK/pluginMainSDK.py

You will notice that this script will generate a Jami Plugins SDK shell that allows users to:

• Create full plugin skeleton;

• Create, modify or delete a manifest.json;

• Create a package.json;

• Create or delete a preference;

• Create functionality;

• Create main;

• Assemble files;

• Build;

• Merge jpls.

Each one of these functionalities will be detailled next. We also will explain the importance of the files it generates and any related SDK limitations.

Create full plugin skeleton

This option performs a sequence of actions to properly create all base files needed to start a plugin development. The steps are:

1. gather authorship information;

2. define a plugin name;

3. create manifest;

4. create functionalities and preferences;

5. create main file;

6. define basic package.json;

7. define basic build files (build.sh and CMakeLists.txt).

If all is completed successfully, the plugin may be build, installed, and loaded. Also the functionallities may be toggled, however, since their data processes are not implemented, they will perform no action. In our HelloWorld plugin, we implement a simple process using OpenCV. For more complex options, you can refer to our available plugins at gitlab:jami-plugins. Feel free to implement any ideas you may have, but you should respect those constrains:

• use ffmpeg, opencv, and onnx from jami-daemon project;

• if using tensorflow, choose version 2.1.0. Why?

  • We have all needed header files of tensorflow 2.1.0 in /contrib/libs.tar.gz;

  • We provide docker images with libraries for Tensorflow C++ API and Tensorflow Lite, for both Android and Linux development.

• if you need other libraries, check if we support it’s build with jami-daemon project, otherwise, you will have to build it and ensure correct link to the plugin libraries.

To fully create a basic plugin with pre-implementation of desired functionalities APIs, preferences, package, manifest, main file, and basic build related files, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) plugin

The SDK will ask other informations needed to correctly accomplish the task.

Create, modify or delete manifest.json

Every plugin must have a manifest. This file contains the official name, a description, and carries the plugin build version as in the example bellow. Without it, the plugin system will not be able to find the plugin library it should load. Due to it’s importance, every time Jami Plugin SDK is told to create files to a non existing plugin or to a plugin with no manifest, it will first create a new manifest.

{
    "name": "HelloWorld",
    "description" : "HelloWorld plugin will guide you throught Jami Plugins SDK use!",
    "version" : "1.0.0"
}

To create/modify (or to delete) a manifest.json, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) manifest (-del)

The SDK will ask other informations needed to correctly accomplish the task.

Create a package.json

Jami currently supports plugins for GNU/Linux, Android and Windows. For the latter, the build system used is the same as for Jami, it is, we call the plugin build using a python script. This script will look for a package.json file to aquire build informations and commands. Without this file our build pipeline for Windows will not work.

An example package.json file is shown bellow.

• name and version in the package.json and manifest.json files should match.

• extractLibs indicates to the build system if the files under <jami-plugins>/contrib/libs.tar.gz will be used. This archive contains header files for Tensorflow. Thus, you only need to set extractLibs to true if you plan to use this library.

• To play with audio or video, the plugin is dependent of ffmpeg. By adding it to deps, the build system will automatically compile this library from <jami-daemon>/contrib if needed. We also provide OpenCV build from inside <jami-daemon>/contrib ! If you want to use Tensorflow, we provide built libraries for GNU/Linux and Android with our docker images here and here. For more information about OpenCV and Tensorflow build, please refer to jami-plugins technical documentation. There we have a step-by-step!

• If you’re using cmake, your can set configuration definition in defines property. Exemple: if your configuration line is of the form cmake -DCPU=TRUE .. you may set "defines": ["CPU=TRUE"].

• Any command directly related to the plugin build can be defined inside custom_scripts. Bellow we create the build folder to with the plugins project will be configured with mkdir msvc and we also set the build command as cmake --build ./msvc --config Release. Our example CMakeLists.txt may have pre and post build instruction that are not listed here.

{
    "name": "Hello World",
    "version": "1.0.0",
    "extractLibs": false,
    "deps": [
        "ffmpeg"
    ],
    "defines": [],
    "custom_scripts": {
        "pre_build": [
            "mkdir msvc"
        ],
        "build": [
            "cmake --build ./msvc --config Release"
            ],
        "post_build": []
    }
}

To create a package.json, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) package

The SDK will ask other informations needed to correctly accomplish the task. After the base package.json creation, the user must add or modify any information not previewed by the SDK process.

Create or delete a preference

A preference is a internal variable that will be used upon loading or while running the plugin. There is limited types of preferences supported by the plugin system and each of them must contain generic and specific informations. Those informations must be placed under a certain structure that will form one preference and each preference must be listed inside a preference.json file.

The generic properties of a preference are those that must be set by any type of preference: category, type, key, title, summary, defaultValue, and scope. The specific ones are linked to the type of the preference constructed. For EDITTEXT preferences there is no other property to be set. For PATH preferences we have: mimeType. For LIST preferences we have: entries and entryValues. A LIST preference must have a list of possible values to be used and a list of ‘names’ for these values. For example: If you have two entriesValues: ‘0’ and ‘1’, these values may not be understandable by the user. Jami’s UI will take the values from entries to be shown as ‘names’ for each one of these entryValues. Then you can call ‘0’ and ‘1’ as ‘sent’ and ‘received’. The UI will show these names that are more user friendly! It is important to note that entries and entryValues must have the same number of items.

Another important point to be noted for the preferences is that their values could be modified during runtime if, and only if, two conditions are satisfied:

1. the code that applies the new value is within your functionality implementation and;

2. this functionality is listed in the preference’s scope property.

To better explain, we have to detail what is a scope and how a preference changement is implemented inside the plugin.

• Scope: A scope is a list of functionalities that can modify a preference value even if the functionality is under use. It is, imagine you have a functionality called “Circle” that prints a colored circle to your video. Consider also that the color of that circle is set by a preference and that this preference lists “Circle” as one of it’s scopes. In that scenario “Circle” will be able to modify the default circle color to another one.

• Code implementation: Continuing our example above, “Circle” also is the implementation of an abstract API class from Daemon (for more details check #Create functionality. When a user changes a preference value, the plugin system will call the setPreferenceAttribute implementation for each of the functionalities listed by the preference’s scope. By it’s turn, this function will match the preference unique key and call an internal function to apply the new value.

For a pratical example, you can check the ‘Foreground Segmentation’ functionality of the GreenScreen plugin - pluginMediaHandler.cpp and preferences-onnx.json. This plugin has both LIST and PATH preferences and also has one preference that can be modified during runtime. Can you tell wich one?

To create (or delete) a preference, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) preference (-del)

The SDK will ask other informations needed to correctly accomplish the task. If a preference is created from outside a functionality creation pipeline, any API implementation will not be changed in order to avoid overwrittings. Thus, if you want to allow values changement during runtime, you may need to manually modify your functionality implementation to fit running time changements conditions.

Create functionality

A Jami plugin may wrap one or multiple functionalities. It is, the same plugins may have a functionality to change background and to draw a circle to a video for example. Each functionality must implement an abstract API class defined by Jami plugins System. That way, we can say that one functionality is one implementation of an API. Currently we have the MediaHandler which allows access to audio or video frames and we have ChatHandler which allows acces to messages exchanged in a conversation.

When defining a new a functionality, the SDK will create basic header and cpp files for you work on. However, your new functionality can not be added to the scopes of a previously existing preference. For more information, please refer to Create or delete a preference.

To create a functionality, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) functionality

The SDK will ask other informations needed to correctly accomplish the task.

Create main

This option create plugin’s main.cpp. A file that implements the plugin external loading function that Jami Plugin System will call to initialize and register all functionalities for latter use.

The SDK is set to rewrite the main file every time you create a new functionality. Thus, if you want to manually create or delete a functionality, we recomend calling this option instead.

To create a main.cpp, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) main

The SDK will ask other informations needed to correctly accomplish the task.

Assemble files

The final plugin file is an archive compressed to a JPL format. This archive contains libraries, manifest.json, preferences.json, icons and other custom important files for your plugin. OBS.: files added by the developper must be organized under the data/ folder.

The SDK assemble option has two different behaviors:

• it creates a build folder and copies there all files that will be compressed to the final plugin archive. For linux host: <plugins>/<HelloWorld>/build-local/jpl/ and for a windows host: <plugins>/<HelloWorld>/msvc/jpl/;

• it compresses all files inside the build folder to the jpl archive wich is output under <plugins>/build.

Both process should be called from inside the CMakeLists.txt as POST_BUILD and PRE_BUILD commands, respectively. Also, the build.sh script should call them. For more information about CMakeLists.txt and build.sh files, please refere to build and to our available plugins at gitlab:jami-plugins.

To create a build folder and copy all important files there, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) assemble -pre

To compress all assembled to a jpl archive, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) assemble

The SDK will ask other informations needed to correctly accomplish the task.

Build

The SDK build option has two different behaviors:

• it creates basic CMakeLists.txt and buils.sh files;

• it build the plugin library with default options defined at the files mentioned above.

A description of thes CMakeLists.txt and buils.sh files are found further in this section.

To create basic CMakeLists.txt and buils.sh files, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) build -def

To build plugin library with default configuration, from inside Jami Plugins SDK shell, the user must call:

(Jami Plugins SDK) build

The SDK will ask other informations needed to correctly accomplish the task. For the moment, the build option does not support cross-compilation neither non default builds. If you have build variables to be set differently than the default option, please directly use the <plugins>/build-plugin.py script.

CMakeLists.txt

This file is used only by Jami’s Windows build pipeline.

If you need to pass any defines to cmake generation, your definitions can be in package.json as explained in the package section. The package.json created specifies the default configuration that calling the build from Jami Plugins SDK will consider. If you want to build with no default configuration, you can: directly use the script mentioned above and pass non-default definitions as an argument or; you also can manually change your package.json file. For examples, you can refer to our available plugins at gitlab:jami-plugins.

Another important information about CMakeLists.txt is that it has to add custom commands. For PRE_BUILD, it must call the pre-assemble functionality process. For POST_BUILD, it must copy the built library to the build folder and call the assemble functionality process. In the end, your jpl archive may be found under <plugins>/build. The CMakeLists.txt file automatically created by our SDK, already respects these constraints.

build.sh

This file is used by Jami to build plugins for GNU/Linux and Android platforms.

The basic script consider the environment variable DAEMON that must point to the jami-daemon folder. Besides, you can pass an argument for the platform used like -t android if you want to cross-compile for Android. Further custom definitions and environment variables should be handled by the plugin developper. If you want to build with no default configuration, you can modify the environment variables values and then call the build. Ie: for android, you can set which ABI you want to build with export ANDROID_ABI="arm64-v8a armeabi-v7a. For other examples, you can refer to our technical documentation and to our available plugins.

Another important information about build.sh is that it has to call pre and post assemble. Before the build, it must call the pre-assemble functionality process. After it, it must copy the built library to the build folder and call the assemble functionality process. In the end, your jpl archive may be found under <plugins>/build. The build.sh file automatically created by our SDK, already respects these constraints.

Merge jpls

If you have more than one jpl archives, like one build for Android and anothe for GNU/Linux platforms, you can merge them into one to easy it’s distribution. However, you should know that merging two or more jpls may inccur orverwritting some of the files inside them if they are not equal for all archives. The only files that may not present conflicting contents are the ones that do not repeate themselves. If conflicts occur, files from the first jpl in the arguments will prevail over the others.

To merge two or more jpls, from inside Jami Plugins SDK shell, the user must simple call:

(Jami Plugins SDK) merge

The SDK will ask other informations needed to correctly accomplish the task.

Create my first plugin

Through this section we will present a step-by-step construction of a HelloWorld plugin using our SDK. Our goal is to print a coloured circle in the midle of the video frames using OpenCV. The color of that circle will be defined by a preference which will be changeable during runtime. Also we can set a stream preferece to define if the plugin will modify the video sent or the one received, this time we don’t want to allow a changement during runtime. We can define a second functionality that will aways draw a circle in the right top corner, with the color defined by the same preference as the previous functionality but that cannot be changed during runtime. At the end we will exemplify how to build your plugin with and without the SDK.

Step 1 - prepare developpment environment

The first step towards plugin development is to properly prepare the environment.

mkdir jami-plugins && cd jami-plugins
git clone https://review.jami.net/jami-daemon daemon
git clone https://review.jami.net/jami-plugins plugins
cd plugins
pip3 install -r SDK/requirements.txt

Step 2 - create HelloWorld with one functionality

• Use Jami Plugins SDK to create the plugin skeleton

python3 SDK/pluginMainSDK.py

(Jami Plugins SDK) plugin

Tell us who you are?

What’s your name? Aline Gondim Santos

What’s your e-mail? aline.gondimsantos@savoirfairelinux.com

Leave Empty or Press ‘q’ to quit this option.

Now, you need to tell how you want yout plugin to be called.

Choose a cool name for your plugin: Hello World

Nice! Your HelloWorld will be awesome!

Defining a manifest for “HelloWorld” plugin, we gonna need more information..

Press ‘q’ to quit this option.

Tell us a description: HelloWorld draws a circle in the center of a call’s video

Version must be of the form X.Y.Z Set plugin version (default 0.0.0): 1.0.0

Chose a functionality name: CenterCircle

Choose a API for functionality “CenterCircle”.

Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences.

Enter a data type number: 1

Add another functionaliy? [y/N]

Would you like to add a preference? [y/n] y

Your preferences options available are: (0) List; (1) Path; (2) EditText;

Which preference type do you choose: 0 Type a value for category: stream Type a value for key: videostream Type a value for title: Video stream Type a value for summary: select a video direction Type a value for defaultValue: 0

Would you like to add a scope? [y/n] n

Do you want to add a new entry Value? [y/n] y Type one new entry: 0

Do you want to add a new entry Value? [y/n] y Type one new entry: 1

Do you want to add a new entry Value? [y/n] n Type an entry name for ‘0’: sent Type an entry name for ‘1’: received

Would you like to add a preference? [y/n] y

Your preferences options available are: (0) List; (1) Path; (3) EditText;

Which preference type do you choose: 0 Type a value for category: color Type a value for key: color Type a value for title: Circle color Type a value for summary: select a color Type a value for defaultValue: ##00FF00

Would you like to add a scope? [y/n] y

Possible values for scope: (0) centerCircle;

Which scope do you choose: 0

Do you want to add a new entry Value? [y/n] y Type one new entry: #0000FF

Do you want to add a new entry Value? [y/n] y Type one new entry: #00FF00

Do you want to add a new entry Value? [y/n] y Type one new entry: #FF0000

Do you want to add a new entry Value? [y/n] n Type an entry name for ‘#0000FF’: blue Type an entry name for ‘#00FF00’: green Type an entry name for ‘#FF0000’: red

Would you like to add a preference? [y/n] n

The preference Circle color will be changeable during running time for centerCircle functionality? [y/n] y

Package ok.

CMakeLists.txt and build.sh ok.

• modify CenterCircleMediaHandler.cpp, CenterCircleVideoSubscriver.h, CenterCircleVideoSubscriber.cpp;

• You will need to modify package.json, CMakeLists.txt and build.sh to add OpenCV dependencies.

All final files can be found here.

Step 3 - build

Before building the HelloWorld, you should compile ffmpeg and OpenCV dependencies. This can be achieved by following the build instructions for OpenCV here. Ffmpeg will be automatically build if you follow those instructions. To build you plugin, you can either:

• Call the build from Jami Plugins SDK (works for GNU/Linux and Windows):

python3 SDK/pluginMainSDK.py

(Jami Plugins SDK) build

Leave Empty or Press ‘q’ to quit this option.

Plugin to pass build related step: HelloWorld

DAEMON not provided, building with ./../../daemon

• Call plugin-build.py script (works for GNU/Linux, Windows and Android):

• GNU/Linux or Windows:

python3 build-plugin.py –projects=HelloWorld

• Android:

python3 build-plugin.py –projects=HelloWorld –distribution=android

OBS: For Android, you can set ANDROID_ABI environment variable to the ABI you want to build. Currently Jami supports x86_64, armeabi-v7a, and arm64-v8a. Plugins will be build for the three options by default.

Step 4 - create another functionality for HelloWorld and rebuild

Now that you tested and your HelloWorld is working, you can try to do add another functionality to it.

python3 SDK/pluginMainSDK.py

(Jami Plugins SDK) functionality

Tell us who you are?

What’s your name? Aline Gondim Santos

What’s your e-mail? aline.gondimsantos@savoirfairelinux.com

Leave Empty or Press ‘q’ to quit this option.

Plugin to be modified or created: HelloWorld

Chose a functionality name: CoinCircle

Choose a API for functionality “CoinCircle”.

Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences.

Enter a data type number: 1

Add another functionaliy? [y/N] n

Would you like to add a preference? [y/n] n

CMakeLists.txt and build.sh ok.

Again, all code modifications are available here.

Optionally, you can change your plugin version and description:

python3 SDK/pluginMainSDK.py

(Jami Plugins SDK) manifest Leave Empty or Press ‘q’ to quit this option.

Plugin to be modified or created: HelloWorld New description for your plugin (ignore to keep previous description): HelloWorld can draw a circle at the center or at the top-left of a call’s video New plugin version (ignore to keep previous version): 2.0

Finally, you can rebuild the plugin using the same steps as before!

Now you can start your own creations! Do not forget to tag Jami or Savoir-Faire Linux in your repository, this way we can get to know how the community is developping their own plugins!