1 How to start
2 What is a Spark?
3 How does a Spark work?
4 Interfaces
5 Configuration file
- 5.1 Configuration API
6 Creating a Spark
- 6.1 Example of a Spark
- 6.2 Motion Control Sparks: publisher-subscriber architecture
  - 6.2.1 First step: subscribing to the publisher
  - 6.2.2 Second step: telling the publisher what to do
7 How to upload a Spark
8 Eclipse CDT templates
- 8.1 C++ Spark header template
- 8.2 C++ Spark source template

How to start

In the near future, it will be possible to develop Sparks using several programming languages. Currently however, the following documentation applies specifically to C++.

The fastest way to start Spark development is to download a VMWare appliance with all the required software already installed. Learn more on the virtual appliance here.

What is a Spark?

Avatars developed in Fiona are made of Sparks. They represent certain features or processes within the operation of the avatar. A Spark is a wrapper for one of those functionalities and it can talk to other sparks about what it is doing. They offer and share information with each other, and they use that information to do a specific thing. For instance, there is a Spark that is able to render the face of a virtual agent in relation to data given to it by other Sparks as the position of the eyes, which in turn is extracted from another Spark that tracks human faces. This is only an example of Sparks working together to share data and make things happen.

More specifically, a Spark is a plugin or shared object which talks to other Sparks by using interfaces. These interfaces set the methods that will be used to send and receive data. There are two types of interfaces, provided and required (they will be explained in greater detail). Provided interfaces are used by Sparks to offer methods and required interfaces are used to call those methods. We use a component-based programming paradigm where the Spark is the component to develop and it must inherit from a base class called 'Component'.

 class ExampleSpark :
	public Component
{
...

A configuration file (which will be better explained) is used to set some necessary parameters and/or indicate the path of extra needed files.

How does a Spark work?

Interfaces

Interfaces are abstract base classes that describe the behavior or capabilities of a class without committing to a particular implementation. They have prototypes of pure virtual (or abstract) functions that simply act as placeholders and have to be defined by derived classes. Depending on whether you want the Spark to implement a function or call it, you might use provided or required interfaces. The interface class declaration is the same in both cases; the only difference is how they are used.

Interfaces are named starting with the capital letter I followed by some descriptive word, such as ICamera.

The following image shows two Sparks connected via interfaces.

FFMpegAVOutSpark provides IAudioConsumer and IVideoConsumer interfaces. RemoteRendererSpark uses those interfaces to pass audio buffers and video frames to FFMpegAVOutSpark which needs that data to publish a stream. So FFMpegAVOutSpark is responsible for the implementation of the interfaces (as provided interfaces) and RemoteRendererSpark uses the interfaces to call the methods (as required interfaces).

Provided interfaces

These kinds of interfaces are used to implement the methods declared on the abstract class. To use them, a Spark class has to be a derived class from the interface. That way, it inherits all the virtual methods. It is mandatory to implement all the virtual functions declared in a interface.

 class ExampleSpark :
	public Component,
        public IProvidedInterface1,
        public IProvidedInterface2, ...
{
...

Required interfaces

Required interfaces are used when a Spark needs to notify another Spark, or require some data from it. So connections between Sparks are made with interfaces. Some Sparks implement an interface's procedure (using a provided interface) that another Spark calls through a required interface. The two Sparks are then connected. To use a required interface, it is necessary to declare a pointer to the interface class type as a Spark class member and initialize it with the function initializeRequiredInterfaces().

private:
   IExampleInterface *myExampleInterface;
   void initializeRequiredInterfaces() {	
	requestRequiredInterface<IExampleInterface>(&myExampleInterface);		
   }

List of interfaces

IAnimation

Used to launch an animation (avatar moves).
Methods

void playAnimation(char *fileName); //Launches animation defined in the passed file. It must has the .anim extension.
void update(); //Necessary to refresh animation movement.

IApplication

Used to launch the avatar process.
Method

void run();

IASyncFatalErrorHandler

Used to handle asynchronous errors produced in non-main threads.
Method

void handleError(); //Logs and sends an error message. Threads must have this handler to give up with a goblal fatal error.

IAudioQueue

Used to manage an audio buffer queue.
Methods

int getStoredAudioSize(); //Returns audio queue size (in bytes).
void queueAudioBuffer(char *buffer, int size); //Pushes an audio buffer
void dequeueAudioBuffer(char *buffer, int size); //Pops an audio buffer

ICamera

Used to set position, angles and different parameters of the avatar's camera.
Methods

int void setCameraPosition(float X, float Y, float Z);// Chooses the position of the camera which shows the avatar
int void setCameraRotation(float X, float Y, float Z);// Chooses angles of the camera which shows the avatar
int void setCameraParameters(bool isOrtho, float VisionAngle, float nearClippingPlane, float farClippingPlane);// Chooses different parameters of the camera

IConcurrent

Used to start/stop threads.
Methods

void start(); //Starts the thread.
void stop(); //Stops the thread.

IControlVoice

Used to control voice and lip movement
Methods

void startSpeaking(); //Starts a speech and the lip movement.
void stopSpeaking(); //Notifies when to stop moving the lips.
void startVoice(); //Notifies when to start moving the lips.

IDetectedFacePositionConsumer

Used to send and receive the position of the detected human face through the camera.
Method

void consumeDetectedFacePosition(bool isFaceDetected, double x, double y);

IEyes

Sets eye position.
Method

void rotateEye(float pan,float tilt); //Sets the position of the eyes in terms of calculated pan and tilt.

IFaceExpression

Used to change the value a MorphTarget has.
Method

void setFaceExpression(char *expressionName,float intensity);

IFlow<class T>

A templated interface used to exchange data between Sparks.
Method

void processData(T myItem);

There are several accepted types: char*, AudioWrap* and Image*. Thus, the following interfaces could be used.

IFlow<char*>

Normally used to send and receive text.

IFlow<AudioWrap*>

AudioWrap is used to send and receive audio packets:

#ifndef AUDIOWRAP_H_
#define AUDIOWRAP_H_
#include <stdint.h>
class AudioWrap {
public:
	AudioWrap(int16_t *audioBuffer, int bufferSizeInBytes);
	int16_t *audioBuffer;
	int bufferSizeInBytes;
};
#endif /* AUDIOWRAP_H_ */

IFlow<Image*>

Image is a common interface for FFMpeg and CVCapture frames consumed by OpenCV:

#ifndef __IMAGE_H
#define __IMAGE_H
#include <opencv2/opencv.hpp>
class Image {
public:
	Image(int w, int h) : width(w), height(h) {}
	int getWidth() { return width; }
	int getHeigth() { return height; }
	virtual IplImage *getIplImage() { return iplImage; }
	virtual ~Image() {}
protected:
	int width;
	int height;
	IplImage *iplImage;
};
#endif

IFrameEventPublisher

Used to update avatar's movements produced by sparks that control these motions.
Method

void addFrameEventSubscriber(FrameEventSubscriber *frameEventSubscriber);//Each motion control spark has to subscribe to CharacterEmbodiment (publisher)

INeck

Sets head position.
Method

void rotateHead(float pan, float tilt); //Sets the position of the head in terms of calculated pan and tilt.

IThreadProc

Used to work with our threads.
Sparks which want to use ThreadSpark must implement this interface. ThreadSpark is a wrapper, of an Unix Thread, which provides some control over the created thread. Therefore, developers should place within this process method the code they want to be executed inside the thread.
Method

void process(); //ThreadSpark calls this function. It must be implemented in developed sparks that work with threads.

Configuration file

This file must has the same name as the developed Spark, ending with the extension '.ini', e.g. ExampleSpark.ini. This is the place to add your Spark settings and parameters. Later on, somebody using your Spark will be able to change and edit some suitable parameters you may want to be configurable. We encourage developers not to hard-code Spark settings and use this configuration file and the available API to get its values.

Parameters must follow the skeleton Parameter_With_More_Than_One_Word, this is, words must be separated by low dashes and begin with capital letter. Right now, there are a few parameter types to choose:
- String
- Integer
- Float
- Boolean
Additionally, you could create your own "list type", which consists of a list of values and will be treated as a 'String' parameter. See more info on Upload Spark process.

Example of a configuration file

...
Character="3D_elvira";   /*String */
Camera_Control_Connected=true; /* Boolean */
Audio_Video_Config_Width = 768;  /* Integer */
Audio_Video_Config_Height = 550;  /* Integer */
Alpha_Value = 10.0;   /* Float */
...

Configuration API

int getComponentConfiguration()->getInt(char *settingPath);
float getComponentConfiguration()->getFloat(char *settingPath);
bool getComponentConfiguration()->getBool(char *settingPath);
std::string getComponentConfiguration()->getString(char *settingPath);
void getComponentConfiguration()->getString(char *settingPath,char *buff,int len);

Creating a Spark

This guide covers the development of a Spark written in the C++ programming language. There are source and header templates for Eclipse CDT available at the end of this guide.

First thing to do is to create a C++ project (shared library project) with the name of the Spark. We recommend that you give your Spark a descriptive name. The name should end with the word Spark, for example, ExampleSpark. Then create a C++ class and include the necessary headers such as Component.h and the interfaces you want to implement and use. The created Spark class must derive from Component and the provided interfaces.

We suggest the following Spark project structure since it will be the model to match in the Upload Spark process:
-/src folder with your sources.
-/include folder with your headers.
-/bin folder with your binary (shared object .so).
-/dependencies folder with your external libraries used.

ExampleSpark.h

/// @file ExampleSpark.h
/// @brief Component ExampleSpark main class.
#ifndef __ExampleSpark_H
#define __ExampleSpark_H
#include "Component.h"
//Example of provided interface
#include "IProvidedInterface1.h"
//Example of required interface
#include "IRequiredInterface1.h"
/// @brief This is the main class for component ExampleSpark.
///
/// 
class ExampleSpark :
	public Component,
	public IProvidedInterface1
{

Define the class constructor. It will always have the same structure.

public:
		ExampleSpark(
				char *instanceName,
				ComponentSystem *cs
		) : Component(instanceName, cs)
		{}

Then declare and initialize interfaces to be required.

private:	
	//Example of a required interface initialization
	IRequiredInterface1 *myRequiredInterface1;
	void initializeRequiredInterfaces() {	
		requestRequiredInterface<IRequiredInterface1>(&myRequiredInterface1);
	}

Virtual methods init and quit are declared in Component base class so they must be declared and implemented here. Provided interface's methods must also be declared and implemented.

public:
	//Mandatory
	void init(void);
	void quit(void);
	//Example of interface implementation declaration
	//IProvidedInterface1 implementation
	void interfaceProcedure1();
	void interfaceProcedure2();
private:
	//Put class attributes here
private:
	//Put class private methods here
};
#endif

Then in the source file you need to include the class header and the following implementation of createComponent procedure:

ExampleSpark.cpp

#include "ExampleSpark.h"
#ifdef _WIN32
#else
extern "C"
Component *createComponent(
		char *componentInstanceName,
		char *componentType,
		ComponentSystem *componentSystem
	)
{
	if (!strcmp(componentType, "ExampleSpark")) {
			return new ExampleSpark(componentInstanceName, componentSystem);
		}
	else {
			return NULL;
		}
}
#endif

Lastly you need to write the initialization of your spark within the init procedure and give an implementation to the quit and rest of interface's functions you want to provide. Remember it is mandatory to implement all the methods of a provided interface. You should use the -> operator to call required interface's methods.

/// Initializes ExampleSpark component.
void ExampleSpark::init(void){
}
/// Unitializes the ExampleSpark component.
void ExampleSpark::quit(void){
}
//Example of provided interface implementation
//IProvidedInterface1 implementation
void ExampleSpark::interfaceProcedure1(void){
	//do something
	//Interface's methods example of use
	//myInterface->interfaceMethod();
}
void ExampleSpark::interfaceProcedure2(void){
	//do something
	//Interface's methods example of use
	//myRequiredInterface1->interfaceMethod();
}

Example of a Spark

This Spark is a very simple one that loads some text from the configuration file when startSpeaking is called and thanks to sayThis method, offered by IVoice required interface, sends that data to a TTS to synthesize.
It also implements methods to start and stop moving the avatar lips and to refresh the face expression with appropriate values.

VoiceStartSpark.cpp

/// @file VoiceStartSpark.cpp
/// @brief VoiceStartSpark class implementation.
#include "stdAfx.h"
#include <stdlib.h>
#include "VoiceStartSpark.h"
#ifdef _WIN32
#else
extern "C"
Component *createComponent(
		char *componentInstanceName,
		char *componentType,
		ComponentSystem *componentSystem
	)
{
	if (!strcmp(componentType, "VoiceStartSpark")) {
			return new VoiceStartSpark(componentInstanceName, componentSystem);
		}
	else {
			return NULL;
		}
}
#endif
/// Initializes VoiceStartSpark component.
void VoiceStartSpark::init(void){	
	movingLips=false;
	alfa=0.0;
        //Subscribe to the publisher, since VoiceStartSpark component could modify avatar's face expressions
        myFrameEventPublisher->addFrameEventSubscriber(this);
}
/// Unitializes the VoiceStartComponent component.
void VoiceStartSpark::quit(void){
}
void VoiceStartSpark::startSpeaking(void){
	char sayConfigurationText[1024];
	getComponentConfiguration()->getString("Tts_Text", sayConfigurationText, 1024);
	movingLips=true;
	myFlow->processData(sayConfigurationText);
}
void VoiceStartSpark::startVoice(void){
	movingLips=true;
}
void VoiceStartSpark::stopSpeaking(void){
	movingLips=false;
}
//FrameEventSubscriber implementation
void VoiceStartSpark::notifyFrameEvent()
{
	if (movingLips){
		//updateVisemeMorphTargets();
		myFaceExpression->setFaceExpression("vis-eh_ShapeKey",alfa);
		if (alfa == 0.0){
			alfa=0.5;
		}
		else if (alfa == 0.5){
			alfa=1;
		}
		else if (alfa == 1){
			alfa=0;
		}
	}
	else{
		myFaceExpression->setFaceExpression("vis-eh_ShapeKey",0.0);
	}
}

VoiceStartSpark.h

/// @file VoiceStartSpark.h
/// @brief Component VoiceStartSpark main class.
#ifndef __VoiceStartSpark_H
#define __VoiceStartSpark_H
#include "Component.h"
#include "FrameEventSubscriber.h"
#include "IControlVoice.h"
#include "IFaceExpression.h"
#include "IFlow.h"
#include "IFrameEventPublisher.h"
/// @brief This is the main class for component VoiceStartSpark.
///
/// It controls when the voice and the movements of lips start and finish
class VoiceStartSpark :
	public Component,
	public IControlVoice,
	public FrameEventSubscriber
{
public:
	VoiceStartSpark(
		char *instanceName, 
		ComponentSystem *componentSystem
	) : Component(instanceName, componentSystem)
	{}
private:
	IFlow<char *> *myFlow;
	IFaceExpression *myFaceExpression;
	IFrameEventPublisher *myFrameEventPublisher;
	void initializeRequiredInterfaces() {
		requestRequiredInterface<IFlow<char *> >(&myFlow);
		requestRequiredInterface<IFaceExpression>(&myFaceExpression);
		requestRequiredInterface<IFrameEventPublisher>(&myFrameEventPublisher);
	}
public:
        //Mandatory
	void init(void);
	void quit(void);
	//IControlVoice
	void startSpeaking();
	void stopSpeaking();
	void startVoice();
        //FrameEventSubscriber implementation
	void notifyFrameEvent();
private:
	bool movingLips;
	float alfa;
};
#endif

Motion Control Sparks: publisher-subscriber architecture

Some sparks control avatar's movements,which are related to some specific parts of the avatar's body, such us eyes, mouth, nose, ... These sparks stablish some parameters which transform parts of the avatar's body. RemoteCharacterEmbodiment3DSpark (RCE3D) is the spark which renders the avatar, and it needs to know when it has to do it. A way to do that is using publisher-subscriber architecture, where motion controller sparks are the subscribers and RCE3D is the publisher. There are two basic steps: A controller spark has to subscribe to the publisher. The publisher has to know what to do when it updates the frame.

First step: subscribing to the publisher

The subscriber will always have an IFrameEventPublisher subscriber, to tell the publisher that it wants to be a subscriber. The publisher will save it in an array.

So we have to add to the header file of controller Spark this piece of code:

#include "IFrameEventPublisher.h"
...
private:
...
void initializeRequiredInterfaces() {	
	requestRequiredInterface<IFrameEventPublisher>(&myFrameEventPublisher);
	}

We have to add to the cpp file of controller Spark this piece of code too:

void NameOfControllerSpark::init()
{
...
    myFrameEventPublisher->addFrameEventSubscriber(this);
...
}

Second step: telling the publisher what to do

We have to tell the publisher what it has to do when it updates avatar. Subscriber inherits from the class FrameEventSubscriber.

class FrameEventSubscriber 
{
public:
    virtual void notifyFrameEvent() = 0;
};

So we have to add to header file of controller spark this piece of code:

#include "FrameEventSubscriber.h"
...
class NameOfControlSpark:
      ...
      public FrameEventSubscriber,
      ...
public: 
      void notifyFrameEvent();

We have to add to cpp file of controller spark this piece of code too:

      void notifyFrameEvent()
      {
           //do things to update, for example calling interfaces to transform avatar's body
      }

How to upload a Spark

Once your code is ready, it’s time to upload your work to our platform. This is done through the “SparkUp” page which is available after you login using your Fiona account. But first you have to prepare your files to comply our requisites.

Files needed

The file types accepted are .zip, .rar and tar.gz (that is, only a compressed file). This file should include:

Only a folder with the same name as the Spark to upload. This folder must contain:

A folder named "bin" with your binary (.so), or a "src" and "include" folders, in case you want to upload your source code and leave the compilation process in our hands. You have to choose only one of the two possibilities.
A "config" folder with the Spark's configuration file.
An "icon" folder with the Sparks's icon. This must be a jpg or png image and will be displayed in the SparkStore listings.
Optionally, a "dependencies" folder will be needed.
A "README" file with a detailed explanation of how the Spark works. Additionally, you will have to provide some info about the compilation (flags, requirements, etc) or dependencies of your Spark.

Only the compressed files that match this structure will be accepted!

Fill-in information

Spark information

You will have to fill out several fields with the Spark information such as:
-"Name", the name of your developed Spark. It must end in 'Spark'.
-"Version", the Spark's version, with the format x.y where 'x' and 'y' are digits .
-"Description", a detailed description of what the Spark does.
-"Short description", a brief description of the Spark. This will appear in result listings and SparkStore.
-"Support email address", a notification mail.

Spark interfaces

Add the interfaces your Spark is using to the appropiate box ('provided' or 'required'). See more info about interfaces.

Spark configuration

Add the parameters of the Spark's configuration file that you want to be configurable by the user.
Explore the section 'Own list types' where you can create new list of values, e.g. if your Spark has a different behavior depending on the day of the week and the user will be able to edit that parameter, create your own list "days of the week", and the user will see a drop-down list with the selectable values.

The upload process

Once you have completed the upload process using the SparkUp page, Fiona administrators will be instantly notified. The Spark will be tested with our core and then three things could happen:

If the Spark works, it will be made available through the SparkStore and you will be notified so you can check how your Spark works.

If the Spark has some problems during compilation and we can fix it, we will notify you to tell you what happened and what we did, then you can choose between checking the changes and re-submit the Spark or just give the ok to publish the first files with changes made.

If the Spark has some problems that we are unable to fix, we will contact you with a brief report on what happened so you can fix it and re-submit the Spark.

Eclipse CDT templates

C++ Spark header template

/// @file ${file_base}.h
/// @brief Component ${file_base} main class.
#ifndef __${file_base}_H
#define __${file_base}_H
#include "Component.h"
//**To change for convenience**
//Example of provided interface
#include "IProvidedInterface1.h"
//**To change for convenience**
//Example of required interface
#include "IRequiredInterface1.h"
/// @brief This is the main class for component ${file_base}.
///
/// 
class ${file_base} :
	public Component,
	//**Spark class must derive from provided interfaces**
	public IProvidedInterface1	
{
public:
		${file_base}(
				char *instanceName,
				ComponentSystem *cs
		) : Component(instanceName, cs)
		{}
private:
	//**To change for convenience**
	//Example of a required interface initialization
	IRequiredInterface1 *myRequiredInterface1;
	void initializeRequiredInterfaces() {	
		requestRequiredInterface<IRequiredInterface1>(&myRequiredInterface1);
	}
public:
	//Mandatory
	void init(void);
	void quit(void);
	//**To change for convenience**
	//Example of interface implementation declaration
	//IProvidedInterface1 implementation
	void interfaceProcedure1();
	void interfaceProcedure2();
private:
	//Put class attributes here
private:
	//Put class private methods here
};
#endif

C++ Spark source template

/// @file ${file_base}.cpp
/// @brief ${file_base} class implementation.
#include "stdAfx.h"
#include "${file_base}.h"
#ifdef _WIN32
#else
extern "C"
Component *createComponent(
		char *componentInstanceName,
		char *componentType,
		ComponentSystem *componentSystem
	)
{
	if (!strcmp(componentType, "${file_base}")) {
			return new ${file_base}(componentInstanceName, componentSystem);
		}
	else {
			return NULL;
		}
}
#endif
/// Initializes ${file_base} component.
void ${file_base}::init(void){
}
/// Unitializes the ${file_base} component.
void ${file_base}::quit(void){
}
//**To change for your convenience**
//Example of provided interface implementation
//IProvidedInterface1 implementation
void ${file_base}::interfaceProcedure1(void){
	//do something
	//Interface's methods example of use
	//myInterface->interfaceMethod();
}
void ${file_base}::interfaceProcedure2(void){
	//do something
	//Interface's methods example of use
	//myRequiredInterface1->interfaceMethod();
}

Spark Developer Guide

Contents