DoreSegment Agora Extension Guidelines

 

 

Product overview

Introduction

Use DoreSegment to recognize objects and identify exactly which pixels belong to each object. Great for creating pixel-level masks, performing photo compositing and more. 

DoreSegment, at its core, separates a digital image into two parts: background and foreground segment (human segment). DoreSegment is providing a Background Removal extension for agora framework. Digital image is made up of millions of individual pixels, and the goal of DoreSegment is to assign each of those pixels to the object to which it belongs. Segmenting an image allows us to separate the foreground from background, identify the human and clearly mark the boundaries that separate a human body from the entire image.

Extension workflow 

 

DoreSegment Extension works with Agora SDK as a low level component to update each video frame.

Features

  • Extract human segment and remove background from video frame
  • User can change background at run time 

 Supported platforms

  • IOS (v12 or above)

Pricing

  • Pricing Model: Yearly subscription 

  • Pricing Details : $200 per year  (unlimited usage)

  • Pricing Calculation : subscription license will be generated while creating a new project, and each subscription will be valid 365 days (one year) from purchase date.

Integration guide 

Prerequisites

  • Xocde Version 11.3 or above

  • IOS Version 12 or above

  • Agora IOS SDK 

  • iPhone 6s or Above

Project Setup

Before importing the DoreSegment framework make sure the Agora IOS SDK is imported into your project. Please refer Agora SDK documentation to install Agora SDK in your project  

https://www.agora.io/en/blog/building-a-one-to-many-ios-video-app-with-agora/

How to implement

There are two ways you can import the DoreSegment Extension framework, 1. Import through cocoapods repository 2. Manual import

Import through Cocoapod  

1. Add the below line in pod file 

         pod ‘DoreSegmentExt’ 

2. Run ‘pod install’ command

Manual Import

1. Download the below Framework files from the download link (download latest release version) 

FrameworkFiles:

  • DoreCoreAI.framework 
  • DoreSegment.framework

Extension download link : https://docs.agora.io/en/extension_customer/VisionLab_downloads

All Releases : https://github.com/doreai/AgoraIO/releases/ 

2. Open your project in xcode, goto target - > General - > Frameworks, Libraries and Embedded content.  Click + button to add a new framework, add “DoreCoreAI.framework”, and “DoreSegment.framework” files.

3. Choose “Embed & Sign” in embed option

4. Go to build settings and change “Enable Bitcode” to “No”

How to enable/start DoreSegment extension    

 1. Import DoreSegment & AgoraRtcKit

 2Setup Agora video configurations  

Add the below line to enable DoreSegment Extension 

agoraKit!.enableExtension(withVendor: "DoreAI", extension: "DoreSegment", enabled: true)

Use the below Agora Video Encoder Configuration:

Size : 480x360  (for better performance)

frameRate : .fps24

Bitrate : AgoraVideoBitrateStandard

orientationMode : .fixedPortrait 

Mirrormode : .auto  

3. Pass the below json properties to start segmentation (start background remover)

Use “setExtensionPropertyWithVendor” method to start extension

 let jsonObject: NSMutableDictionary = NSMutableDictionary()

        jsonObject.setValue("xxxx", forKey: "apiKey")

        jsonObject.setValue("xxxx", forKey: "license")

        let jsonData: NSData

        do {

            jsonData = try JSONSerialization.data(withJSONObject: jsonObject, options: JSONSerialization.WritingOptions()) as NSData

            let jsonString = NSString(data: jsonData as Data, encoding: String.Encoding.utf8.rawValue)! as String

            agoraKit!.setExtensionPropertyWithVendor("DoreAI", extension: "DoreSegment", key: "start", value: jsonString)            

        } catch _ {

            print ("JSON Failure")

        }

Json property

Key

Value

apiKey

[Project api key value]

license

[Project license key value]

 Extension property

Key

Value

start

[Json property string]

How to change background

Use the below code to change background at run time. Background image (UIImage) should be passed by base64 format as below

        let stringPath = Bundle.main.path(forResource: "bg1", ofType: "png")

        let bgImage:UIImage = UIImage(contentsOfFile: stringPath!)!

        let base64Txt:String = (bgImage.pngData()?.base64EncodedString())!

        agoraKit!.setExtensionPropertyWithVendor("DoreAI", extension: "DoreSegment", key: "bgImage", value: base64Txt)

 

Sample Project Code

Download Sample project code with the below link 

https://github.com/doreai/AgoraIO/tree/master/DoreSegmentExtExample

  

 

Tutorial Video:

 

Demo Video :