Arm Orientation Estimation Method with Multiple Devices for NUI/NUX

1. Introduction

Given the recent various applications of virtual reality and mixed reality [ 1 ], the importance of the Natural User interface/Natural User Experience (NUI/NUX) has increased for improved realism of games and control accuracy. Motions in games can be utilized as commands. Therefore, motion estimation techniques are a key NUI/NUX technology.

Traditional motion estimation approaches utilize Kinect data [ 2 ], anatomic information [ 3 ], and vision-based calibration [ 4 , 5 ] to support and improve the accuracy of signals for motion estimation. For example, the arm movements of users can be estimated by using optical markers such as the Perception Neuron and Xsens MVN or several tangible devices such as the HTC VIVE and Thalmic Lab Myo. However, such devices are inconvenient because users have to wear one or more devices. To solve this problem, the Bayesian probability is utilized to estimate unmeasured motions based on data from a Myo, which can reduce the number of tangible devices that should be attached to users for accurate motion measurement. However, motion estimation approaches based on heterogeneous devices are required to increase the motion estimation accuracy.

This paper proposes an orientation estimation method based on the positions of hands and orientations of arms by using multiple devices and the Bayesian probability. Estimating the arm orientations should make it possible to realize more realistic virtual characters. This paper is constructed as follows. Section 2 describes traditional motion estimation approaches, and Section 3 describes the proposed method. Section 4 describes the experimental results, and Section 5 concludes the paper.

2. Traditional Motion Estimation Approaches

Wang et al. [ 4 ] developed a computer vision-based marker-less motion capture method that uses two optical cameras. The joint kinematics data of a person estimated by this method are almost like that estimated by a marker-based motion capture method. Mehrizi et al. [ 5 ] developed a vision-based calibration method for dual remote center-of-motion (RCM)-based robot arms in a human–robot collaborative minimally invasive surgery (MIS) scenario.

Kim et al. [ 6 ] introduced a method for estimating upper arm motions by forearm motions. This allow the number of Myos attached to a user to be reduced. Given that motions are expressed by orientations, they utilized two Myos to collect the orientations of a forearm and upper arm. They calculated the Bayesian probability based on the two kinds of measured orientations. The motions corresponding to the orientations with the maximum probabilities were selected as the estimated motions. Thus, only one Myo is required to estimate the forearm and upper arm based on the calculated Bayesian probability. However, they classified orientations with a fixed interval considering the possible ranges of orientations and then calculated the Bayesian probability with the classified orientations. Therefore, some Bayesian probabilities could not be calculated because the associated orientations were not measured. Lee et al. [ 7 ] introduced an approach to optimize the intervals of the motion orientations. Instead of considering the possible range of orientations, they considered the range between the minimum and maximum orientations among the measured orientations. By reducing the size of the considered range, they increased the accuracy of the Bayesian probability. Chae et al. [ 8 ] improved the accuracy of the estimated motions by utilizing a genetic algorithm to calculate the weights for each type of orientation. They utilized the products of the estimated orientations and calculated weights as the final estimated orientations. By considering the importance of each type of orientations, the motion estimation accuracy was increased. In another study, Chae et al. [ 9 ] used not only orientations but also electromyography (EMG) to improve the motion estimation accuracy. They calculated the Bayesian probability for each motion by utilizing both signals. The motions with a higher probability according to the estimated final motions of the two signals were utilized. By referencing motions with a higher probability, the motion estimation accuracy was increased.

Kim et al. [ 10 ] estimated the motions of forearms based on EMG of the biceps and triceps and inertial measurement unit (IMU) values of the upper arm. Especially, they focused on the mean, sum, and velocity of sequences based on the measured signals of both types.

3. Orientation Estimation Based on Positions

The proposed method estimates the orientations of an arm based on the positions of a hand using Bayesian probability. Dependent orientations are estimated by independent positions. Set the independent positions [TeX:] $$p _ { t } ^ { I } = \left[ x _ { t } ^ { I } , y _ { t } ^ { I } , z _ { t } ^ { I } \right]$$ and the dependent orientation [TeX:] $$o _ { t } ^ { D } = \left[ x ^ { D } _ { t } , y ^ { D } _ { t } , z _ { t } ^ { D } \right]$$ at time t .

Set [TeX:] $$\delta ^ { x , \zeta } = \frac { \left( \operatorname { Max } \left( x ^ { I } _ { t } \right) - \operatorname { Min } \left( x ^ { I } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\delta ^ { y , \zeta } = \frac { \left( \operatorname { Max } \left( y ^ { I } _ { t } \right) - \operatorname { Min } \left( y ^ { I } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\delta ^ { z , \zeta } = \frac { \left( \operatorname { Max } \left( z _ { t } ^ { I } \right) - \operatorname { Min } \left( z ^ { I } _ { t } \right) \right) } { \zeta }$$

where ζ is the number of sections.

Set [TeX:] $$\theta ^ { x , \zeta } = \frac { \left( \operatorname { Max } \left( x ^ { D } _ { t } \right) - \operatorname { Min } \left( x ^ { D } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\theta ^ { y , \zeta } = \frac { \left( \operatorname { Max } \left( y ^ { D } _ { t } \right) - \operatorname { Min } \left( y ^ { D } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\theta ^ { z , \zeta } = \frac { \left( \operatorname { Max } \left( z ^ { D } _ { t } \right) - \operatorname { Min } \left( z ^ { D } _ { t } \right) \right) } { \zeta }.$$

Set [TeX:] $$x ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { x ^ { D } _ { i } } { \theta ^ { x , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ [TeX:] $$y ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { y ^ { D } _ { i } } { \theta ^ { y , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ and [TeX:] $$z ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { z ^ { D } _ { i } } { \theta ^ { z , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right)$$

where i is the index of dependent orientations, probability [TeX:] $$\mathrm { p } \left( x ^ { D } t , \zeta \right) = \max p \left( \left[ \frac { x ^ { D } _ { i } } { \theta ^ { x , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ probability [TeX:] $$p \left( y ^ { D } _ { t , \zeta } \right) = \max p \left( \left[ \frac { y ^ { D } _ { i } } { \theta ^ { y , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ and probability [TeX:] $$\mathrm { p } \left( z ^ { D } _ { t , \zeta } \right) = \max p \left( \left[ \frac { z ^ { D } _ { i } } { \theta ^ { z , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right).$$

The error rate is effected by the section ζ. Therefore, dependent orientations are determined considering multiple orientations calculated by multiple section ζs. Set [TeX:] $$x ^ { D } _ { t } = \underset { \arg \max } {x ^ { D }_{t,\zeta} } p \left( x^{D} _ { t, \zeta }\right)$$ , [TeX:] $$y ^ { D } _ { t } = \underset { \arg \max }{ y ^ { D }_{t,\zeta} } p \left( y^ { D }_{ t , \zeta } \right) $$ , [TeX:] $$z ^ { D } _ { t } = \underset { \arg \max }{ z ^ { D }_{t,\zeta} } p \left( z^ { D }_{ t , \zeta } \right)$$

4. Experiments

The proposed method was validated on a computer equipped with Windows 10 Pro, Intel i7-7700 (3.6 GHz), NVIDIA GeForce GTX 1050 2G, and RAM 16G (DDR4). One VIVE and one Myo were utilized for motion learning and motion estimation, as shown in Fig. 1. In the experiment, the VIVE headset was only used to view the boxing contents, not to measure the location of the headset. First, seven boxing motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard) were defined, as shown in Fig. 1.

Each motion was measured three times at 30 fps for motion learning. The motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard) were measured one by one. To train the orientations with the Bayesian probability, the minimum and maximum of the x and y positions for the hands and x, y, and z orientations for the arms were obtained. The differences were divided into five increments for orientation and position quantization. Fig. 2 shows the obtained positions and orientations of each motion, and Fig. 3 shows the quantized positions and orientations of each motion.

The quantized positions and orientations of each motion were utilized to calculate the Bayesian probability. To verify the proposed method, each boxing motion were measured twice more and then quantized. The measured and quantized positions of the hands are shown in Figs. 4 and 5, respectively. The quantized orientations of the arms were estimated according to the Bayesian probability, as shown in Fig. 6.

To verify the estimated orientations, the estimated orientations were compared with the actual orientations of the arms as measured by a Myo, as shown in Figs. 7 and 8.

Fig. 9 shows the accumulated sum of the measured orientations. Fig. 10 shows the cumulative error rate, which was calculated by the rate of the difference between the estimated and quantized orientations and the measured and quantized orientations.

Figs. 11 and 12 show the error rate for each type of orientation. The average error rate was about 26% with the section ζ by 10. The average error rate was effected by the section ζ. Therefore, the section ζ was set by 5. In the results, the average error rate with the section ζ by 5was about 20%. When both sections were considered to validate the proposed method, the average error rate was about 19%.

5. Conclusion

This paper proposes a method to estimate the arm orientations based on the hand positions by using the Bayesian probability. One VIVE and one Myo were utilized to collect the hand positions and arm orientations. The Bayesian probability was then calculated based on the collected data during orientation learning. The unmeasured arm orientations were estimated using the positions measured with VIVE based on the calculated Bayesian probability during the orientation estimation.

In the experiments, seven boxing motions were defined to validate the proposed method. Each boxing motion was performed three times for orientation learning and two times for orientation estimation. The estimated boxing orientations were compared with the orientations measured by a Myo and quantized. The average error rate was 19%, but the possibility of estimating the orientation of any body part without additional devices was demonstrated.

Acknowledgement

This research was supported by the Ministry of Science, ICT and Future Planning (MSIP), Korea, under the Information Technology Research Center support program (No. IITP-2017-2013-0-00684) supervised by the Institute for Information & Communications Technology Promotion (IITP).