Creating Jami extensions

NOTE: This page introduces the Jami Plugin SDK.

Jami extensions

In September 2020, the Jami team added extensions as a call feature for GNU/Linux, Windows, macOS, and Android users. This enables personalized call and chat experiences by using available extensions. Additionally, anyone is also able to transform awesome ideas into brand new Jami extensions!

This is a guide for the SDK to help start extension development. The text is organized as:

A description of the SDK;
An example of how to create a base extension with the Jami Plugin SDK - #Create my first extension.

SDK

An Extension System for Jami was developed and there are extensions available to be used. However as an open-source project, it allows users to create, use, and distribute their own extensions. To achieve that goal, a Jami Plugin SDK was also developed. This kit is written in Python, and can be invoked running pluginMainSDK.py from <plugins> folder. Requirement to get started:

git clone https://review.jami.net/jami-plugins plugins
cd plugins
git clone https://review.jami.net/jami-daemon daemon
pip3 install -r SDK/requirements.txt
python3 SDK/pluginMainSDK.py

Notice that this script will generate a Jami Plugin SDK shell that allows users to:

Create full extension skeleton;
Create, modify or delete a manifest.json;
Create a package.json;
Create or delete a preference;
Create functionality;
Create main;
Assemble files;
Build;
Merge jpl files.

Each one of these functionalities will be detailed next. The importance of the files it generates is explained and any related SDK limitations.

Create full extension skeleton

This option performs a sequence of actions to properly create all base files needed to start an extension development. The steps are:

gather authorship information;
define an extension name;
create manifest;
create functionalities and preferences;
create main file;
define basic package.json;
define basic build file (CMakeLists.txt).

If all is completed successfully, the extension may be build, installed, and loaded. Also the functionalities may be toggled, however, since their data processes are not implemented, they will perform no action. In the HelloWorld extension, a simple process using OpenCV is implemented. For more complex options, please refer to the available extensions at gitlab:jami-plugins. Feel free to implement any ideas while respecting those constrains:

use FFmpeg, OpenCV, and ONNX from jami-daemon project;
if using TensorFlow, choose version 2.1.0. Why?
- All required header files of TensorFlow 2.1.0 are in /contrib/libs.tar.gz;
- Docker images with libraries for TensorFlow C++ API and LiteRT (formerly TensorFlow Lite), for both Android and Linux development, are provided.
if other libraries are required, check if the build is supported within the jami-daemon project; otherwise, it will require building with the correct links to extension libraries.

To fully create a basic extension with pre-implementation of desired functionalities APIs, preferences, package, manifest, main file, and basic build related files, from inside Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) extension

The SDK will ask other information needed to correctly accomplish the task.

Create, modify or delete manifest.json

Every extension must have a manifest. This file contains the official name, a description, and carries the extension build version as in the example below. Without it, the Extension System is unable to find the extension library it should load. Due to its importance, every time Jami Plugin SDK is told to create files to a non-existent extension or to an extension with no manifest, it will first create a new manifest.

{
    "name": "HelloWorld",
    "description" : "HelloWorld extension to help show Jami Plugin SDK use!",
    "version" : "1.0.0"
}

To create/modify (or to delete) a manifest.json, from inside Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) manifest (-del)

The SDK will ask other information needed to correctly accomplish the task.

Create a package.json

Jami currently supports extensions for GNU/Linux, macOS, Windows, and Android. For Windows, the build system used is the same as for Jami, it is, the extension build is called using a Python script. This script will look for a package.json file to acquire build information and commands. Without this file the build pipeline for Windows are unable to run.

An example package.json file is shown below.

name and version in the package.json and manifest.json files should match.
extractLibs indicates to the build system if the files under <jami-plugins>/contrib/libs.tar.gz will be used. This archive contains header files for TensorFlow. Thus, setting extractLibs to true is only required if this library is planned to be used.
To play with audio or video, the extension is dependent on FFmpeg. By adding it to deps, the build system will automatically compile this library from <jami-daemon>/contrib if required. OpenCV build is provided from inside <jami-daemon>/contrib! Docker images have been created with CUDA and TensorFlow libraries available for GNU/Linux builds here and for Android builds here. For more information about OpenCV and TensorFlow build, please refer to jami-extensions technical documentation.
If CMake is used, the configuration definition is able to be set in the defines property. Example: if the configuration line is of the form cmake -DCPU=TRUE … set "defines": ["CPU=TRUE"].
Any command directly related to the extension build can be defined inside custom_scripts. Below the build folder is msvc, created with mkdir msvc and the build command is cmake --build ./msvc --config Release. The example CMakeLists.txt may have pre and post build instruction that are not listed here.

{
    "name": "Hello World",
    "version": "1.0.0",
    "extractLibs": false,
    "deps": [
        "ffmpeg"
    ],
    "defines": [],
    "custom_scripts": {
        "pre_build": [
            "mkdir msvc"
        ],
        "build": [
            "cmake --build ./msvc --config Release"
            ],
        "post_build": []
    }
}

To create a package.json, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) package

The SDK will ask other information needed to correctly accomplish the task. After the base package.json creation, the user must add or modify any information not previewed by the SDK process.

Create or delete a preference

A preference is a internal variable that will be used upon loading or while running the extension. There is limited types of preferences supported by the Extension System and each of them must contain generic and specific information. Those information must be placed under a certain structure that will form one preference and each preference must be listed inside a preference.json file.

The generic properties of a preference are those that must be set by any type of preference: category, type, key, title, summary, defaultValue, and scope. The specific ones are linked to the type of the preference constructed. For EDITTEXT preferences there is no other property to be set. For PATH preferences we have: mimeType. For LIST preferences we have: entries and entryValues. A LIST preference must have a list of possible values to be used and a list of 'names' for these values. For example: If there are two entriesValues: '0' and '1', these values may not be understandable by the user. Jami's UI will take the values from entries to be shown as 'names' for each one of these entryValues. Then '0' and '1' can be called as 'sent' and 'received'. The UI will show these names that are more user friendly! It is important to note that entries and entryValues must have the same number of items.

Another important point to be noted for the preferences is that their values could be modified at runtime if, and only if, two conditions are satisfied:

the code that applies the new value is within the functionality implementation; and,
this functionality is listed in the preference's scope property.

To better explain, we have to detail what is a scope and how a preference change is implemented inside the extension.

Scope: A scope is a list of functionalities that can modify a preference value even if the functionality is under use. It is, imagine a functionality called "Circle" that prints a colored circle to a video. Consider also that the color of that circle is set by a preference and that this preference lists "Circle" as one of its scopes. In that scenario "Circle" will be able to modify the default circle color to another one.
Code implementation: Continuing the example above, "Circle" also is the implementation of an abstract API class from Daemon (for more details check #Create functionality. When a user changes a preference value, the Extension System will call the setPreferenceAttribute implementation for each of the functionalities listed by the preference's scope. By its turn, this function will match the preference unique key and call an internal function to apply the new value.

For a practical example, the 'Foreground Segmentation' functionality is able to be checked in the GreenScreen extension - pluginMediaHandler.cpp and preferences-onnx.json. This extension has both LIST and PATH preferences and also has one preference that can be modified at runtime. Can you tell which one?

To create (or delete) a preference, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) preference (-del)

The SDK will ask other information needed to correctly accomplish the task. If a preference is created from outside a functionality creation pipeline, any API implementation will not be changed in order to avoid being overwritten. Thus, if value changes are allowed at runtime, functionality implementation may require manual modification to satisfy runtime change conditions.

Create functionality

A Jami extension may wrap one or multiple functionalities. It is, the same extension may have a functionality to change background and to draw a circle to a video for example. Each functionality must implement an abstract API class defined by the Jami Extension System. That way, we can say that one functionality is one implementation of an API. Currently we have the MediaHandler which allows access to audio or video frames and we have ChatHandler which allows access to messages exchanged in a conversation.

When defining a new functionality, the SDK will create basic header and cpp files. However, new functionality is unable to be added to the scope of a previously existing preference. For more information, please refer to Create or delete a preference.

To create a functionality, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) functionality

The SDK will ask other information needed to correctly accomplish the task.

Create main

This option creates main.cpp for the extension. A file that implements the extension external loading function that the Jami Extension System will call to initialize and register all functionalities for latter use.

The SDK is set to rewrite the main file every time a new functionality is created. Thus, if a functionality is to be manually created or deleted, it is recommended to call this option instead.

To create a main.cpp, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) main

The SDK will ask other information needed to correctly accomplish the task.

Assemble files

The final extension file is an archive compressed to a JPL (Jami PLugin) format. This archive contains libraries, manifest.json, preferences.json, icons and other custom important files for the extension. OBS.: files added by the developer must be organized under the data/ folder.

The SDK assemble option has two different behaviors:

it creates a build folder and copies there all files that will be compressed to the final extension archive. For a Linux host: <plugins>/<HelloWorld>/build-local/jpl/ and for a Windows host: <plugins>/<HelloWorld>/msvc/jpl/;
it compresses all files inside the build folder to the jpl archive which is output under <plugins>/build.

Both process should be called from inside the CMakeLists.txt as POST_BUILD and PRE_BUILD commands, respectively. For more information about CMakeLists.txt file, please refer to build and to the available extensions at gitlab:jami-plugins.

To create a build folder and copy all important files there, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) assemble -pre

To compress all assembled to a jpl archive, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) assemble

The SDK will ask other information needed to correctly accomplish the task.

Build

The SDK build option has two different behaviors:

it creates basic CMakeLists.txt file;
it build the extension library with default options defined at the files mentioned above.

A description of the CMakeLists.txt file are found further in this section.

To create basic CMakeLists.txt file, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) build -def

To build extension library with default configuration, from inside the Jami Plugin SDK shell, the user must call:

(Jami Plugin SDK) build

The SDK will ask other information needed to correctly accomplish the task. For the moment, the build option does not support cross-compilation neither non default builds. If build variables are set differently to the default option, please directly use the <plugins>/build-plugin.py script.

CMakeLists.txt

This file is used in every Jami build pipeline, i.e. GNU/Linux, macOS, Windows, and Android. Extensions are unavailable on iOS due to the Apple App Store security policy.

If it is required to pass any defines to CMake generation, the definitions can be in package.json as explained in the package section. The package.json created specifies the default configuration that calling the build from the Jami Plugin SDK will consider. If it is required to build with no default configuration, it is possible to: directly use the script mentioned above and pass non-default definitions as an argument; or, manually change the package.json file. For examples, please refer to the available extensions at gitlab:jami-plugins.

Another important information about CMakeLists.txt is that it has to add custom commands. For PRE_BUILD, it must call the pre-assemble functionality process. For POST_BUILD, it must copy the built library to the build folder and call the assemble functionality process. In the end, the jpl archive will be located in <plugins>/build. The CMakeLists.txt file automatically created by the Jami Plugin SDK, already respects these constraints.

Merge jpl files

If there is more than one jpl archive, like one build for Android and another for GNU/Linux platforms, they can be merged into one to ease its distribution. However, it is important to note that merging two or more jpl files may overwrite some of the files inside them if they are not equal for all archives. The only files that may not present conflicting contents are the ones that do not repeat themselves. If conflicts occur, files from the first jpl in the arguments will prevail over the others.

To merge two or more jpl files, from inside the Jami Plugin SDK shell, the user must simple call:

(Jami Plugin SDK) merge

The SDK will ask other information needed to correctly accomplish the task.

Create my first extension

Through this section we will present a step-by-step construction of a HelloWorld extension using the Jami Plugin SDK. The goal is to print a colored circle in the middle of the video frames using OpenCV. The color of that circle will be defined by a preference which will be changeable at runtime. Also we can set a stream preference to define if the extension will modify the video sent or the one received, this time we don't want to allow a change at runtime. We can define a second functionality that will always draw a circle in the right top corner, with the color defined by the same preference as the previous functionality but that cannot be changed at runtime. At the end we will exemplify how to build extensions with and without the SDK.

Step 1 - prepare development environment

The first step towards extension development is to properly prepare the environment.

mkdir jami-plugins && cd jami-plugins
git clone https://review.jami.net/jami-daemon daemon
git clone https://review.jami.net/jami-plugins plugins
cd plugins
pip3 install -r SDK/requirements.txt

Step 2 - create HelloWorld with one functionality

Use the Jami Plugin SDK to create the extension skeleton

python3 SDK/pluginMainSDK.py

(Jami Plugin SDK) extension

Tell us who you are?

Enter author’s name: Aline Gondim Santos

Enter author’s e-mail address: aline.gondimsantos@savoirfairelinux.com

Leave blank or press 'q' to quit this option.

Enter the extension unique identifier to be created.

Enter a unique identifier for the extension: Hello World

The extension identifier is HelloWorld

Defining the manifest for "HelloWorld" extension.

Press 'q' to quit this option.

Enter Extension description: HelloWorld draws a circle in the center of a call's video

Version must be of the form X.Y.Z Enter Extension version (leave blank for '0.0.0'): 1.0.0

Enter functionality name: CenterCircle

Select an API for the "CenterCircle" functionality.

Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences.

Enter datatype number: 1

Add another functionality? [y/N]

Add a preference? [y/n] y

Available preferences: (0) List; (1) Path; (2) EditText;

Select preference type: 0 Type a value for category: stream Type a value for key: videostream Type a value for title: Video stream Type a value for summary: select a video direction Type a value for defaultValue: 0

Add a scope? [y/n] n

Add a new entry value? [y/n] y Type one new entry: 0

Add a new entry value? [y/n] y Type one new entry: 1

Add a new entry value? [y/n] n Enter an entry name for '0': sent Enter an entry name for '1': received

Add a preference? [y/n] y

Available preferences: (0) List; (1) Path; (3) EditText;

Select preference type: 0 Type a value for category: color Type a value for key: color Type a value for title: Circle color Type a value for summary: select a color Type a value for defaultValue: ##00FF00

Add a scope? [y/n] y

Available values for scope: (0) centerCircle;

Select scope: 0

Add a new entry value? [y/n] y Enter one new entry: #0000FF

Add a new entry value? [y/n] y Enter one new entry: #00FF00

Add a new entry value? [y/n] y Enter one new entry: #FF0000

Add a new entry value? [y/n] n Enter an entry name for '#0000FF': blue Enter an entry name for '#00FF00': green Enter an entry name for '#FF0000': red

Add a preference? [y/n] n

Allow the Circle preference to change at runtime for the centerCircle functionality? [y/n] y

Package OK.

CMakeLists.txt OK.

modify CenterCircleMediaHandler.cpp, CenterCircleVideoSubscriver.h, CenterCircleVideoSubscriber.cpp;
Modifying package.json and CMakeLists.txt is required to add OpenCV dependencies.

All final files can be found here.

Step 3 - build

Before building the HelloWorld, FFmpeg and OpenCV dependencies should be compiled. This can be achieved by following the build instructions for OpenCV here. FFmpeg will build automatically if you follow those instructions. To build the extension, either:

Call the build from the Jami Plugin SDK (works for GNU/Linux and Windows):

python3 SDK/pluginMainSDK.py

(Jami Plugin SDK) build

Leave blank or press 'q' to quit this option.

Extension to pass build related step: HelloWorld

DAEMON not provided, building with ./../../daemon

Call plugin-build.py script (works for GNU/Linux, Windows and Android):
GNU/Linux or Windows:

python3 build-plugin.py --projects=HelloWorld

Android:

python3 build-plugin.py --projects=HelloWorld --distribution=android

OBS: For Android, set ANDROID_ABI environment variable to the ABI required to build. Currently Jami supports x86_64, armeabi-v7a, and arm64-v8a. Extensions will be build for the three options by default.

Step 4 - create another functionality for HelloWorld and rebuild

Once HelloWorld is working and passed testing, attempt adding another functionality.

python3 SDK/pluginMainSDK.py

(Jami Plugin SDK) functionality

Tell us who you are?

Enter author’s name: Aline Gondim Santos

Enter author’s e-mail address: aline.gondimsantos@savoirfairelinux.com

Leave blank or press 'q' to quit this option.

Extension to be modified or created: HelloWorld

Enter functionality name: CoinCircle

Select an API for the "CoinCircle" functionality.

Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences.

Enter datatype number: 1

Add another functionality? [y/N] n

Add a preference? [y/n] n

CMakeLists.txt OK.

Again, all code modifications are available here.

Optionally, the extension version and description can be changed:

python3 SDK/pluginMainSDK.py

(Jami Plugin SDK) manifest Leave blank or press 'q' to quit this option.

Extension to be modified or created: HelloWorld Enter Extension description (leave blank to keep previous description): HelloWorld can draw a circle at the center or at the top-left of a call's video Enter Extension version (leave blank to keep previous version): 2.0

Finally, rebuild the extension using the same steps as before!

Now you can start your own creations! Remember to tag Jami and Savoir-Faire Linux Inc. in your repository, this way we can get to know how the community is developing their own extensions!