This paper presents an Assistive social robots architecture designed for social interaction with humans, mediated by a humanoid robot. The architecture has been designed for being used by people with severe paralysis and the architecture has been tested by a user affected by amyotrophic lateral sclerosis (ALS) in a locked-in state. The system allows the patient to communicate with the stakeholders using a Brain Controlled Interface, based on Evoked Response Potentials (ERP), to express needing, feelings or writing phases. Stakeholders visualize messages sent by the patient on a GUI and use a tele operated humanoid robot as an avatar of them to extend their physical presence to interact with the patient even when they are away from him. This research demonstrates that is possible to insert humanoid robot into the conventional home management of ALS patients and a communication system that trough a Brain-computer interface (BCI) and a humanoid robot as avatar, can help increase the quality of life of the locked-in patients.

Keywords: brain computer interface, social robotics, assistive robotics, architecture, EEG, ERP, human-humanoid interaction

BCI, brain computer interface; ALS, amyotrophic lateral sclerosis; EEG, electro encephalon graphy; ERP, evoked response potentials; VEP, visual evoked potentials

The amyotrophic lateral sclerosis is a neurodegenerative disease characterized by progressive muscular paralysis reflecting degeneration of motor neurons in the primary motor cortex, corticospinal tracts, brainstem and spinal cord. It’ incidence (average 1.89 per 100,000/year) and prevalence (average 5.2 per100,000) are relatively uniform in Western countries. At the moment there is no treatment for this disease and the only management of ALS is palliative, for this reason it is required a supportive and multidisciplinary approach.¹ In ALS, paralysis is progressive and leads to death due to respiratory failure within 2–3 years. During the latter stage of the disease, patients suffer of a locked-in condition, where their brain maintains all the cognitive functions operating, in a paralyzed body.² Brain Computer Interface allows people in locked-in states to communicate with the external world and to control the environment using brain waves, rather than muscles.³ Among the paradigm presented in literature, we used the P300 oddball paradigm for our architecture. Accordingly to this paradigm, it is possible to elicit a particular wave, named P300 (P indicates a positive deflection, 300 represents the latency in milliseconds on stimulus onset) by showing an unattended stimulus during a sequence of expected stimuli.⁴ Assistive robotics is a new field of Human Robotic Interaction that is a new subfield of robotics that bridges HRI, rehabilitation robotics, social robotics, and service robotics.⁵ These robots can serve a variety of therapeutically relevant functions, including providing feedback,⁶ accomplishing tasks,^7,8 extending user presence⁹ and help in the management of the patient.¹⁰ Telenoid Robot is a small-size Japanese robot developed from the idea of transferring human's "presence". Telenoid was developed to appear and behave as a minimal design of human features. Its name derived from “Tele” as tele operation and “noides” as twin, to reassemble, even in the name, how the robot is an extension of the user himself.¹¹ The architecture has been designed to provide new communication channel for people affected by ALS. The use of the telenoid robot as communication mediator enables the operator the feeling of sharing a space with the user.

In Figure 1 is reported the general architecture structure. The system is based on the UnipaBCI framework, a modular architecture entirely developed by the authors and presented in a previous paper,¹² easily scalable and flexible to allow the development of new features or the removal of the older ones, maintaining the general structure unchanged. The framework has been developed using Microsoft Visual Studio 2012 to provide a complete IDE for the code writing. The operating system used to implement UnipaBCI is Windows 10 since it provides more support to the API of the g.usbAmp hardware device used to test the framework. C++ was selected as language, because it introduces a low performance overhead at run-time. GUI and other visual components are implemented in OpenGL. It is composed by a GUI which provides feedback to the patient (from now on defined user) who wears a bio-signal amplifier. The amplifier acquires EEG signal in real time from user’s brain converting in messages. A cloud based network system, interlaces user with robot controller, where the stakeholder (from now on defined operator) visualizes the messages. The operator controls the Telenoid robot in tele-operation mode, in such operating mode; the telenoid robot repeats each movement of the operator and speaks reproducing his voice.

Figure 1 The Architectural Schema.

The EEG signals are recorded by the acquisition module, from gel based electrodes placed at four locations of the scalp, according to the 10-20 standard system: P3, Pz, and P4 and Cz. The ground is set at FPz in the forehead and the mass is set on the left earlobe. The signal is digitized at 256 Hz using the g.usb amp, developed by the Guger Technologies. The signal is then filtered using a Butterworth filter 6-th order between 1-40 Hz to reduce artifacts and notched at 50 Hz with a fourth-order notch filter to suppress power line noise. User GUI, has shown in Figure 2(a), is a 4x4 matrix containing feelings (Happy, Sad, Content, Embarrassed, Surprised, Angry, Tired, Annoyed) and phrases (Thank you, Turn the light off, feeling cold, Move me, Turn PC on, I’m hungry, I need you, I love you). Both feelings and phrases has been chosen by the user during a preliminary screening, selecting them from a list, as the most frequent phrase and feelings he wanted to express. GUI icons are highlighted 20 times (trials) using the oddball paradigm and synchronized with the sampled EEG signals. User is request to mentally count each time the icon of his interest is highlighted. At the end of each trial, the feature classification module is used to calculate user’s intention. To use the system user is required to perform a training phase, where he must mentally count each time known symbols is highlighted to train the classifier over user’s P300 response.

Figure 2 (a) On the left the User Interface. (b) On the right the Operator Interface.

After the training phase, user can operate the system: for each icon, the portion of EEG signals are averaged over trials and channels obtaining a time-locked signal over a window of 600 ms. The obtained windows are labeled with the corresponding icon and one versus all stepwise linear Discriminant analysis¹³ is iteratively applied to each window, considering on window as target class and the others as no-target. At the end of this phase, the class with the highest value is selected as target and the corresponding message is sent via the network module to the Operator GUI. The Network module implements a Control Protocol/Internet Protocol (TCP/IP) and it acts as a middle ware between the BCI system and the Robot System. The Operator Interface, described in Figure 2(b) shows the messages sent by the user. The robot system is natively equipped with a facial tracking system for tele operation, whose details can be found in,¹¹ and for each operator’s head movement, the tracking system calculates robot joints movements to replicate them over the Telenoid. In addition, the camera module shows the scenario captured in real time from the Telenoid camera to the operator.

The operator console is used by the operator to have a feedback from the user. The operator console, performs the speech synthesis allows the operator to modify the settings of the speech synthesis, such as tonality, speed of reading and the language used. A button has also been inserted inside the window to delete the chat history. This module implements a web server for real time communication between user and operator. Each command, selected via the User Interface it is sent over a Java Servlet connected to a web socket. Every time a message is received on the Servlet, it verifies if the client is connected and forwards the message. The message is vocally reproduced through the Google Speech API. The described architecture has been tested in a real-world scenario with an ALS male patient of 31 years in Palermo with no prior knowledge of Brain Computer Interface and robotics. In Figure 3 is reported the experimental setting. User lies in this room (Room2) with a BCI monitor in front of him and wears a G.USB Amplifier, connected to a BCI server. User can see the Stakeholder room by a cam connected to a secondary monitor. User controls a BCI system to send messages to the Telenoid Tele operation Controller, in Room 1, where Operator sits. The Telenoid Tele operation Controller is equipped with tracking software and audio-video connection with the Telenoid robot, sets in room 2. Operator answer to the user speaking on his tele operation console and Telenoid reacts repeating his head movement and speaking with his voice, giving user the feelings to be with the operator.

Figure 3 (a) The experimental setting and (b) Some snapshots from the experiment.

A total of 6 sessions of approximately 45 minutes for each session has been run with one subject.

We evaluated system precision and latency, expressed as time between the end of a BCI selection and the appearance of the corresponding message over the operator console. In Table 1 are reported the corresponding results. To evaluate system precision, we asked to the participants to freely use the User Interface to select the most appropriate item during the conversation with the operator. After each selection, we asked user if the selected command $Se{l}_{comm}$ was the expected command $Ex{p}_{Comm}$ . The precision has been calculated as the ratio of correct classified selection over the total selections.

$\mathrm{Pr}ecision=\frac{Se{l}_{Comm}}{Ex{p}_{Comm}}=\frac{110}{126}=87.30\%$

The average precisions of 87.195% (+-0.37%) demonstrate the user ability to mentally select the desired command. No statistical difference can be found on the precision of the system along all session; from this it is possible to conclude that even a novel user can properly control the system. The system latency is calculated as the composition of the classification time with the network latency. Since the system has been tested over a local area Network, the latter could be considered not significant. The observed average latency over the whole sessions is 2.734 seconds (+- 0.174). From the latency it is possible to calculate an average of approximately 22 commands/minutes. The symbolic interface permits to optimize the time for spelling each command, since using a symbolic interface, user doesn’t need to spell each letter of the message he wishes to send but he needs only to select one command.

The proposed architecture describes a novel interaction system for human–humanoid interaction mediated by a humanoid robot. The brain Computer Interface system enables ALS patient to communicate using a symbolic interface which enhance the command rate because user can select a message with a single command and a very slow latency. On the other hand, the operator using the humanoid robot can communicate, naturally speaking with the user. The Using the robot provides an embodiment which helps to maintain a sense of co-presence. Further study will test the system with a broader number of users, exploring user’s acceptance and satisfaction. The User Interface will be enriched with new commands and a predictive text modality to give to the user more expression power. Finally the humanoid robot will be connected to the user to use it as a physic extension of the patient himself, giving him the opportunity to surmount the limit imposed by his disease giving him an embodiment in places far from him.

Authors wish to thanks Marcello Giardina for providing the conceptual idea. Vincenzo La Bella and Rossella Spataro for their support during the interaction with the patient. The patient Fulvio Lopez and his family for the patience and passion they put in participating in the experimentation. This research project was partially sponsored by g.tec medical engineering who provided financial support providing a g.usb bio signal amplifier.

Authors declare that no financial interest or any conflict of interest exists.