# Conference protocol This document aims to describe the evolutions we will do for managing conferences (audio/video). The goal is to improve the current implementation which simply merges SIP calls and provide a grid view, to a view where participants are listed, can be muted independently, or the video layout changed (to show only one participant) ## Definitions + Host: Is the user who mix the audio/video streams for the others + Participant: Every user in the conference, even the host ## Disclaimer This document only describes the first steps for now. This means the identification of participants and position in the video mixer sent to all participants. ### Possible layouts + GRID: Every member is shown with the same height/width + ONE_BIG_WITH_SMALL: One member is zoomed and the other preview is shown + ONE_BIG: One member take the full screen rendered Two new methods are available to manage the conference Layout in CallManager: ```cpp /** * Change the conference layout * @param confId * @param layout 0 = matrix, 1 = one big, others in small, 2 = one in big */ void setConferenceLayout(const std::string& confId, int layout); /** * Change the active participant (used in layout != matrix) * @param confId * @param participantId If participantId not found, the local video will be shown */ void setActiveParticipant(const std::string& confId, const std::string& participantId); ``` ### Implementation The implementation is pretty straightforward. Everything is managed by `conference.cpp` (to link participant to sources) and `video_mixer.cpp` (to render the wanted layout). ## Syncing Conferences Informations Note: Actually, the word participant is used for callId mixed in a conference. This can lead at first to some problems for the API and must be fixed in the future The goal is to notify all participants of the metadata of the rendered video. This means what participant is in the conference and where the video is located. If a participant is itself a conference, its incoming layout info should be merged when sent to other participants. Layout info must not be merged when sent back to a conference. ### Layout Info The Layout is stored as a VectorMapStringString for clients and internally with a vector with the following format: ``` Layout = { { "uri": "participant", "x":"0", "y":"0", "w": "0", "h": "0", "isModerator": "true" }, { "uri": "participant1", "x":"0", "y":"0", "w": "0", "h": "0", "isModerator": "false" } (...) } ``` Possible keys are: + uri = account's uri + device = device's id + media = media's id + active = if the participant is active + x = position (x) in the video + y = position (y) in the video + w = size (width) in the video + h = size (height) in the video + videoMuted = if the video is muted + audioLocalMuted = if the audio is locally muted + audioModeratorMuted = if the audio is muted by moderators + isModerator = if it's a moderator + handRaised = if the hand is raised + voiceActivity = if the stream has voice activity + recording = if the peer is recording the conference ### New API A new method (in CallManager) and a new signal to respectively get current conference infos and updates are available: ```cpp VectorMapStringString getConferenceInfos(const std::string& confId); void onConferenceInfosUpdated(const std::string& confId, const VectorMapStringString& infos); ``` ### Implementation The `Conference` Object (which only exists if we mix calls, this means that we are the master) manages the information for the whole conference, based on the LayoutInfos of each `Call` object. The getConferenceInfos will retrieve info directly from this object. So, every `Call` object now has a LayoutInfo and if updated, ask the `Conference` object to updates its info. The master of a conference sends its info via the SIP channel as a message with the following MIME type: `application/confInfo+json` So, if a call receives some confInfo, we know that this call is a member of a conference. To summarize, `Call` manages received layouts, `Conference`-managed sent layouts. ## Changing the state of the conference To change the state of the conference, participants needs to send orders that the host will handle. The protocol have the following needs: It should handle orders at multiple levels. In fact for a conference the is 3 levels to define a participant: + The account which is the identity of the participant + Devices, because each account can join via multiple devices + Medias, because there can be multiple videos by devices (eg 1 camera and 1 screen sharing) To save bandwidth, clients should be able to send multiple orders at once. ### General actions To change a layout, the moderator can send a payload with "application/confOrder+json" as type: where **0** is a grid, **1** is one user in big, others in small, **2** is one in big ### Account's actions For now, there is no action supported, however, in the future `moderator: true/false` should be handled to change a moderator. ### Device's actions + `hangup: true` to hangup a device from the conference (only moderators) + `raisehand: true/false` to change the raise hand's status. Only doable by the device itself, else dropped. ### Media's actions + `muteAudio` only doable by moderators to mute the audio of a participant + `muteVideo` not supported yet. + `active` to mark the media as active. + `voiceActivity` to indiciate a media stream's voice activity status (only relevant for audio) ### Example So, the `application/confOrder+json` will contains: ```json { "989587609427420" : { "moderator": true/false, "devices": { "40940943604396R64363": { "hangup": true, "raisehand": true/false, "media":{ "3532532662432" : { "muteAudio": true/false, "muteVideo": true/false, "active": true/false, "voiceActivity": true/false } } } } }, "layout": 0/1/2, } ``` Note: the type of the media should be included in conferences informations and can be used for the client to improve display (e.g. do not crop screen sharing) ### Controlling moderators There is actually 3 possibilities: + Changing account's config to add a list of moderators (In the config.yml (`defaultModerators` can contains a list of default moderators) + If `localModeratorsEnabled` is true, all accounts of the device will be moderators + If `allModeratorsEnabled` is true, anybody in the conference will be a moderator ## Future + Separate streams to allow more controls? ## Notes/Comments It's likely that the protocol will evolve for future needs. I believe it's best if we have a "version" field. The older version will be recognized if this field is missing.