# Creating Jami extensions ```{note} This page introduces the Jami Plugin SDK. ``` ## Jami extensions In September 2020, the Jami team added extensions as a call feature for GNU/Linux, Windows, macOS, and Android users. This enables personalized call and chat experiences by using available extensions. Additionally, anyone is also able to transform awesome ideas into brand new Jami extensions! This is a guide for the SDK to help start extension development. The text is organized as: * A description of the [SDK](#sdk); * An example of how to create a base extension with the Jami Plugin SDK - [#Create my first extension](#create-my-first-extension). ### SDK An Extension System for Jami was developed and there are extensions available to be used. However as an open-source project, it allows users to create, use, and distribute their own extensions. To achieve that goal, a Jami Plugin SDK was also developed. This kit is written in Python, and can be invoked running [pluginMainSDK.py]('') from `` folder. Requirement to get started: ```bash git clone https://review.jami.net/jami-plugins plugins cd plugins git clone https://review.jami.net/jami-daemon daemon pip3 install -r SDK/requirements.txt python3 SDK/pluginMainSDK.py ``` Notice that this script will generate a Jami Plugin SDK shell that allows users to: * [Create full extension skeleton](#create-full-extension-skeleton); * [Create, modify or delete a manifest.json](#create-modify-or-delete-manifest-json); * [Create a package.json](#create-a-package-json); * [Create or delete a preference](#create-or-delete-a-preference); * [Create functionality](#create-functionality); * [Create main](#create-main); * [Assemble files](#assemble-files); * [Build](#build); * [Merge jpl files](#merge-jpl-files). Each one of these functionalities will be detailed next. The importance of the files it generates is explained and any related SDK limitations. #### Create full extension skeleton This option performs a sequence of actions to properly create all base files needed to start an extension development. The steps are: 1. gather authorship information; 2. define an extension name; 3. create manifest; 4. create functionalities and preferences; 5. create main file; 6. define basic package.json; 7. define basic build file (CMakeLists.txt). If all is completed successfully, the extension may be build, installed, and loaded. Also the functionalities may be toggled, however, since their data processes are not implemented, they will perform no action. In the `HelloWorld` extension, a simple process using **OpenCV** is implemented. For more complex options, please refer to the available extensions at {gitlab-project}`jami-plugins`. Feel free to implement any ideas while respecting those constrains: * use FFmpeg, OpenCV, and ONNX from jami-daemon project; * if using TensorFlow, choose version 2.1.0. Why? * All required header files of TensorFlow 2.1.0 are in /contrib/libs.tar.gz; * Docker images with libraries for TensorFlow C++ API and LiteRT (formerly TensorFlow Lite), for both Android and Linux development, are provided. * if other libraries are required, check if the build is supported within the jami-daemon project; otherwise, it will require building with the correct links to extension libraries. To fully create a basic extension with pre-implementation of desired functionalities APIs, preferences, package, manifest, main file, and basic build related files, from inside Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) plugin ``` The SDK will ask other information needed to correctly accomplish the task. #### Create, modify or delete manifest.json Every extension must have a manifest. This file contains the official name, a description, and carries the extension build version as in the example below. Without it, the Extension System is unable to find the extension library it should load. Due to its importance, every time Jami Plugin SDK is told to create files to a non-existent extension or to an extension with no manifest, it will first create a new manifest. ```bash { "name": "HelloWorld", "description" : "HelloWorld extension to help show Jami Plugin SDK use!", "version" : "1.0.0" } ``` To create/modify (or to delete) a manifest.json, from inside Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) manifest (-del) ``` The SDK will ask other information needed to correctly accomplish the task. #### Create a package.json Jami currently supports extensions for GNU/Linux, macOS, Windows, and Android. For Windows, the build system used is the same as for Jami, it is, the extension build is called using a Python script. This script will look for a `package.json` file to acquire build information and commands. Without this file the build pipeline for Windows are unable to run. An example package.json file is shown below. * `name` and `version` in the package.json and manifest.json files should match. * `extractLibs` indicates to the build system if the files under `/contrib/libs.tar.gz` will be used. This archive contains header files for **TensorFlow**. Thus, setting `extractLibs` to true is only required if this library is planned to be used. * To play with audio or video, the extension is dependent on **FFmpeg**. By adding it to `deps`, the build system will automatically compile this library from `/contrib` if required. **OpenCV** build is provided from inside `/contrib`! Docker images have been created with CUDA and TensorFlow libraries available for GNU/Linux builds [here](https://hub.docker.com/repository/docker/sflagsantos/tensorflow-cuda) and for Android builds [here](https://hub.docker.com/repository/docker/sflagsantos/tensorflowlite). For more information about OpenCV and TensorFlow build, please refer to relevant extension technical documentation. * If CMake is used, the configuration definition is able to be set in the `defines` property. Example: if the configuration line is of the form `cmake -DCPU=TRUE …` set `"defines": ["CPU=TRUE"]`. * Any command directly related to the extension build can be defined inside `custom_scripts`. Below the build folder is `msvc`, created with `mkdir msvc` and the build command is `cmake --build ./msvc --config Release`. The example CMakeLists.txt may have pre and post build instruction that are not listed here. ```bash { "name": "Hello World", "version": "1.0.0", "extractLibs": false, "deps": [ "ffmpeg" ], "defines": [], "custom_scripts": { "pre_build": [ "mkdir msvc" ], "build": [ "cmake --build ./msvc --config Release" ], "post_build": [] } } ``` To create a package.json, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) package ``` The SDK will ask other information needed to correctly accomplish the task. After the base package.json creation, the user must add or modify any information not previewed by the SDK process. #### Create or delete a preference A preference is a internal variable that will be used upon loading or while running the extension. There is limited types of preferences supported by the Extension System and each of them must contain generic and specific information. Those information must be placed under a certain structure that will form one preference and each preference must be listed inside a `preference.json` file. The generic properties of a preference are those that must be set by any type of preference: `category`, `type`, `key`, `title`, `summary`, `defaultValue`, and `scope`. The specific ones are linked to the type of the preference constructed. For `EDITTEXT` preferences there is no other property to be set. For `PATH` preferences we have: `mimeType`. For `LIST` preferences we have: `entries` and `entryValues`. A `LIST` preference must have a list of possible values to be used and a list of 'names' for these values. For example: If there are two `entriesValues`: '0' and '1', these values may not be understandable by the user. Jami's UI will take the values from `entries` to be shown as 'names' for each one of these `entryValues`. Then '0' and '1' can be called as 'sent' and 'received'. The UI will show these names that are more user friendly! It is important to note that `entries` and `entryValues` must have the same number of items. Another important point to be noted for the preferences is that their values could be modified at runtime if, and only if, two conditions are satisfied: 1. the code that applies the new value is within the functionality implementation; and, 2. this functionality is listed in the preference's `scope` property. To better explain, we have to detail what is a scope and how a preference change is implemented inside the extension. * **Scope**: A scope is a list of functionalities that can modify a preference value even if the functionality is under use. It is, imagine a functionality called "Circle" that prints a colored circle to a video. Consider also that the color of that circle is set by a preference and that this preference lists "Circle" as one of its scopes. In that scenario "Circle" will be able to modify the default circle color to another one. * **Code implementation**: Continuing the example above, "Circle" also is the implementation of an abstract API class from Daemon (for more details check [#Create functionality](#create-functionality). When a user changes a preference value, the Extension System will call the `setPreferenceAttribute` implementation for each of the functionalities listed by the preference's scope. By its turn, this function will match the preference unique `key` and call an internal function to apply the new value. For a practical example, the 'Foreground Segmentation' functionality is able to be checked in the GreenScreen extension - [pluginMediaHandler.cpp](https://git.jami.net/savoirfairelinux/jami-plugins/blob/master/GreenScreen/pluginMediaHandler.cpp) and [preferences-onnx.json](https://git.jami.net/savoirfairelinux/jami-plugins/blob/master/GreenScreen/preferences-onnx.json). This extension has both `LIST` and `PATH` preferences and also has one preference that can be modified at runtime. Can you tell which one? To create (or delete) a preference, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) preference (-del) ``` The SDK will ask other information needed to correctly accomplish the task. If a preference is created from outside a functionality creation pipeline, any API implementation will not be changed in order to avoid being overwritten. Thus, if value changes are allowed at runtime, functionality implementation may require manual modification to satisfy runtime change conditions. #### Create functionality A Jami extension may wrap one or multiple functionalities. It is, the same extension may have a functionality to change background and to draw a circle to a video for example. Each functionality must implement an abstract API class defined by the Jami Extension System. That way, we can say that one functionality is one implementation of an API. Currently we have the `MediaHandler` which allows access to audio or video frames and we have `ChatHandler` which allows access to messages exchanged in a conversation. When defining a new functionality, the SDK will create basic header and cpp files. However, new functionality is unable to be added to the scope of a previously existing preference. For more information, please refer to [Create or delete a preference](#create-or-delete-a-preference). To create a functionality, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) functionality ``` The SDK will ask other information needed to correctly accomplish the task. #### Create main This option creates `main.cpp` for the extension. A file that implements the extension external loading function that the Jami Extension System will call to initialize and register all functionalities for latter use. The SDK is set to rewrite the main file every time a new functionality is created. Thus, if a functionality is to be manually created or deleted, it is recommended to call this option instead. To create a `main.cpp`, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) main ``` The SDK will ask other information needed to correctly accomplish the task. #### Assemble files The final extension file is an archive compressed to a `JPL` (Jami PLugin) format. This archive contains libraries, manifest.json, preferences.json, icons and other custom important files for the extension. OBS.: files added by the developer must be organized under the `data/` folder. The SDK assemble option has two different behaviors: * it creates a build folder and copies there all files that will be compressed to the final extension archive. For a Linux host: `//build-local/jpl/` and for a Windows host: `//msvc/jpl/`; * it compresses all files inside the build folder to the jpl archive which is output under `/build`. Both process should be called from inside the CMakeLists.txt as POST_BUILD and PRE_BUILD commands, respectively. For more information about CMakeLists.txt file, please refer to [build](#build) and to the available extensions at {gitlab-project}`jami-plugins`. To create a build folder and copy all important files there, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) assemble -pre ``` To compress all assembled to a jpl archive, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) assemble ``` The SDK will ask other information needed to correctly accomplish the task. #### Build The SDK build option has two different behaviors: * it creates basic CMakeLists.txt file; * it build the extension library with default options defined at the files mentioned above. A description of the CMakeLists.txt file are found further in this section. To create basic CMakeLists.txt file, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) build -def ``` To build extension library with default configuration, from inside the Jami Plugin SDK shell, the user must call: ``` (Jami Plugin SDK) build ``` The SDK will ask other information needed to correctly accomplish the task. For the moment, the build option does not support cross-compilation neither non default builds. If build variables are set differently to the default option, please directly use the `/build-plugin.py` script. ##### CMakeLists.txt This file is used in every Jami build pipeline, i.e. GNU/Linux, macOS, Windows, and Android. Extensions are unavailable on iOS due to the Apple App Store security policy. If it is required to pass any defines to CMake generation, the definitions can be in `package.json` as explained in the package section. The package.json created specifies the default configuration that calling the build from the Jami Plugin SDK will consider. If it is required to build with no default configuration, it is possible to: directly use the script mentioned above and pass non-default definitions as an argument; or, manually change the package.json file. For examples, please refer to the available extensions at {gitlab-project}`jami-plugins`. Another important information about CMakeLists.txt is that it has to add custom commands. For PRE_BUILD, it must call the pre-assemble functionality process. For POST_BUILD, it must copy the built library to the build folder and call the assemble functionality process. In the end, the jpl archive will be located in `/build`. The CMakeLists.txt file automatically created by the Jami Plugin SDK, already respects these constraints. #### Merge jpl files If there is more than one jpl archive, like one build for Android and another for GNU/Linux platforms, they can be merged into one to ease its distribution. However, it is important to note that merging two or more jpl files may overwrite some of the files inside them if they are not equal for all archives. The only files that may not present conflicting contents are the ones that do not repeat themselves. If conflicts occur, files from the first jpl in the arguments will prevail over the others. To merge two or more jpl files, from inside the Jami Plugin SDK shell, the user must simple call: ``` (Jami Plugin SDK) merge ``` The SDK will ask other information needed to correctly accomplish the task. ### Create my first extension Through this section we will present a step-by-step construction of a `HelloWorld` extension using the Jami Plugin SDK. The goal is to print a colored circle in the middle of the video frames using **OpenCV**. The color of that circle will be defined by a preference which will be changeable at runtime. Also we can set a stream preference to define if the extension will modify the video sent or the one received, this time we don't want to allow a change at runtime. We can define a second functionality that will always draw a circle in the right top corner, with the color defined by the same preference as the previous functionality but that cannot be changed at runtime. At the end we will exemplify how to build extensions with and without the SDK. #### Step 1 - prepare development environment The first step towards extension development is to properly prepare the environment. ```bash mkdir jami-plugins && cd jami-plugins git clone https://review.jami.net/jami-daemon daemon git clone https://review.jami.net/jami-plugins plugins cd plugins pip3 install -r SDK/requirements.txt ``` #### Step 2 - create HelloWorld with one functionality * Use the Jami Plugin SDK to create the extension skeleton ```python python3 SDK/pluginMainSDK.py ``` > (Jami Plugin SDK) extension > Tell us who you are? > Enter author’s name: Aline Gondim Santos > Enter author’s e-mail address: aline.gondimsantos@savoirfairelinux.com > Leave blank or press 'q' to quit this option. > Enter the extension unique identifier to be created. > Enter a unique identifier for the extension: Hello World > The extension identifier is HelloWorld > Defining the manifest for "HelloWorld" extension. > Press 'q' to quit this option. > Enter Extension description: HelloWorld draws a circle in the center of a call's video > Version must be of the form X.Y.Z Enter Extension version (leave blank for '0.0.0'): 1.0.0 > Enter functionality name: CenterCircle > Select an API for the "CenterCircle" functionality. > Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences. > Enter datatype number: 1 > Add another functionality? [y/N] > Add a preference? [y/n] y > Available preferences: (0) List; (1) Path; (2) EditText; > Select preference type: 0 Type a value for category: stream Type a value for key: videostream Type a value for title: Video stream Type a value for summary: select a video direction Type a value for defaultValue: 0 > Add a scope? [y/n] n > Add a new entry value? [y/n] y Type one new entry: 0 > Add a new entry value? [y/n] y Type one new entry: 1 > Add a new entry value? [y/n] n Enter an entry name for '0': sent Enter an entry name for '1': received > Add a preference? [y/n] y > Available preferences: (0) List; (1) Path; (3) EditText; > Select preference type: 0 Type a value for category: color Type a value for key: color Type a value for title: Circle color Type a value for summary: select a color Type a value for defaultValue: ##00FF00 > Add a scope? [y/n] y > Available values for scope: (0) centerCircle; > Select scope: 0 > Add a new entry value? [y/n] y Enter one new entry: #0000FF > Add a new entry value? [y/n] y Enter one new entry: #00FF00 > Add a new entry value? [y/n] y Enter one new entry: #FF0000 > Add a new entry value? [y/n] n Enter an entry name for '#0000FF': blue Enter an entry name for '#00FF00': green Enter an entry name for '#FF0000': red > Add a preference? [y/n] n > Allow the Circle preference to change at runtime for the centerCircle functionality? [y/n] y > Package OK. > CMakeLists.txt OK. * modify CenterCircleMediaHandler.cpp, CenterCircleVideoSubscriver.h, CenterCircleVideoSubscriber.cpp; * Modifying package.json and CMakeLists.txt is required to add OpenCV dependencies. All final files can be found {gitlab-project}`here `. #### Step 3 - build Before building the HelloWorld, FFmpeg and OpenCV dependencies should be compiled. This can be achieved by following the build instructions for OpenCV {doc}`here `. FFmpeg will build automatically if you follow those instructions. To build the extension, either: * Call the build from the Jami Plugin SDK (works for GNU/Linux and Windows): ``` python python3 SDK/pluginMainSDK.py ``` > (Jami Plugin SDK) build > Leave blank or press 'q' to quit this option. > Extension to pass build related step: HelloWorld > DAEMON not provided, building with ./../../daemon * Call plugin-build.py script (works for GNU/Linux, Windows and Android): * GNU/Linux or Windows: > python3 build-plugin.py --projects=HelloWorld * Android: > python3 build-plugin.py --projects=HelloWorld --distribution=android OBS: For Android, set `ANDROID_ABI` environment variable to the ABI required to build. Currently Jami supports `x86_64`, `armeabi-v7a`, and `arm64-v8a`. Extensions will be build for the three options by default. #### Step 4 - create another functionality for HelloWorld and rebuild Once HelloWorld is working and passed testing, attempt adding another functionality. ``` python python3 SDK/pluginMainSDK.py ``` > (Jami Plugin SDK) functionality > Tell us who you are? > Enter author’s name: Aline Gondim Santos > Enter author’s e-mail address: aline.gondimsantos@savoirfairelinux.com > Leave blank or press 'q' to quit this option. > Extension to be modified or created: HelloWorld > Enter functionality name: CoinCircle > Select an API for the "CoinCircle" functionality. > Available APIs: (1) video during a call (Media Handler API) (2) audio during a call (Media Handler API) (3) chat messages (Chat Handler API) For more information about the API, call help preferences. > Enter datatype number: 1 > Add another functionality? [y/N] n > Add a preference? [y/n] n > CMakeLists.txt OK. Again, all code modifications are available {gitlab-project}`here `. Optionally, the extension version and description can be changed: ``` python python3 SDK/pluginMainSDK.py ``` > (Jami Plugin SDK) manifest Leave blank or press 'q' to quit this option. > Extension to be modified or created: HelloWorld Enter Extension description (leave blank to keep previous description): HelloWorld can draw a circle at the center or at the top-left of a call's video Enter Extension version (leave blank to keep previous version): 2.0 Finally, rebuild the extension using the same steps as before! Now you can start your own creations! Remember to tag Jami and Savoir-Faire Linux Inc. in your repository, this way we can get to know how the community is developing their own extensions!