# Arm Orientation Estimation Method with Multiple Devices for NUI/NUX

Yunsick Sung* , Ryong Choi* and Young-Sik Jeong*

## Abstract

Abstract: Motion estimation is a key Natural User Interface/Natural User Experience (NUI/NUX) technology to utilize motions as commands. HTC VIVE is an excellent device for estimating motions but only considers the positions of hands, not the orientations of arms. Even if the positions of the hands are the same, the meaning of motions can differ according to the orientations of the arms. Therefore, when the positions of arms are measured and utilized, their orientations should be estimated as well. This paper proposes a method for estimating the arm orientations based on the Bayesian probability of the hand positions measured in advance. In experiments, the proposed method was used to measure the hand positions with HTC VIVE. The results showed that the proposed method estimated orientations with an error rate of about 19%, but the possibility of estimating the orientation of any body part without additional devices was demonstrated.

Keywords: Bayesian Probability , HTC VIVE , Motion Estimation , NUI/NUX , Thalmic Myo

## 1. Introduction

Given the recent various applications of virtual reality and mixed reality [ 1 ], the importance of the Natural User interface/Natural User Experience (NUI/NUX) has increased for improved realism of games and control accuracy. Motions in games can be utilized as commands. Therefore, motion estimation techniques are a key NUI/NUX technology.

Traditional motion estimation approaches utilize Kinect data [ 2 ], anatomic information [ 3 ], and vision-based calibration [ 4 , 5 ] to support and improve the accuracy of signals for motion estimation. For example, the arm movements of users can be estimated by using optical markers such as the Perception Neuron and Xsens MVN or several tangible devices such as the HTC VIVE and Thalmic Lab Myo. However, such devices are inconvenient because users have to wear one or more devices. To solve this problem, the Bayesian probability is utilized to estimate unmeasured motions based on data from a Myo, which can reduce the number of tangible devices that should be attached to users for accurate motion measurement. However, motion estimation approaches based on heterogeneous devices are required to increase the motion estimation accuracy.

This paper proposes an orientation estimation method based on the positions of hands and orientations of arms by using multiple devices and the Bayesian probability. Estimating the arm orientations should make it possible to realize more realistic virtual characters. This paper is constructed as follows. Section 2 describes traditional motion estimation approaches, and Section 3 describes the proposed method. Section 4 describes the experimental results, and Section 5 concludes the paper.

## 2. Traditional Motion Estimation Approaches

Wang et al. [ 4 ] developed a computer vision-based marker-less motion capture method that uses two optical cameras. The joint kinematics data of a person estimated by this method are almost like that estimated by a marker-based motion capture method. Mehrizi et al. [ 5 ] developed a vision-based calibration method for dual remote center-of-motion (RCM)-based robot arms in a human–robot collaborative minimally invasive surgery (MIS) scenario.

Kim et al. [ 6 ] introduced a method for estimating upper arm motions by forearm motions. This allow the number of Myos attached to a user to be reduced. Given that motions are expressed by orientations, they utilized two Myos to collect the orientations of a forearm and upper arm. They calculated the Bayesian probability based on the two kinds of measured orientations. The motions corresponding to the orientations with the maximum probabilities were selected as the estimated motions. Thus, only one Myo is required to estimate the forearm and upper arm based on the calculated Bayesian probability. However, they classified orientations with a fixed interval considering the possible ranges of orientations and then calculated the Bayesian probability with the classified orientations. Therefore, some Bayesian probabilities could not be calculated because the associated orientations were not measured. Lee et al. [ 7 ] introduced an approach to optimize the intervals of the motion orientations. Instead of considering the possible range of orientations, they considered the range between the minimum and maximum orientations among the measured orientations. By reducing the size of the considered range, they increased the accuracy of the Bayesian probability. Chae et al. [ 8 ] improved the accuracy of the estimated motions by utilizing a genetic algorithm to calculate the weights for each type of orientation. They utilized the products of the estimated orientations and calculated weights as the final estimated orientations. By considering the importance of each type of orientations, the motion estimation accuracy was increased. In another study, Chae et al. [ 9 ] used not only orientations but also electromyography (EMG) to improve the motion estimation accuracy. They calculated the Bayesian probability for each motion by utilizing both signals. The motions with a higher probability according to the estimated final motions of the two signals were utilized. By referencing motions with a higher probability, the motion estimation accuracy was increased.

Kim et al. [ 10 ] estimated the motions of forearms based on EMG of the biceps and triceps and inertial measurement unit (IMU) values of the upper arm. Especially, they focused on the mean, sum, and velocity of sequences based on the measured signals of both types.

## 3. Orientation Estimation Based on Positions

The proposed method estimates the orientations of an arm based on the positions of a hand using Bayesian probability. Dependent orientations are estimated by independent positions. Set the independent positions [TeX:] $$p _ { t } ^ { I } = \left[ x _ { t } ^ { I } , y _ { t } ^ { I } , z _ { t } ^ { I } \right]$$ and the dependent orientation [TeX:] $$o _ { t } ^ { D } = \left[ x ^ { D } _ { t } , y ^ { D } _ { t } , z _ { t } ^ { D } \right]$$ at time t .

Set [TeX:] $$\delta ^ { x , \zeta } = \frac { \left( \operatorname { Max } \left( x ^ { I } _ { t } \right) - \operatorname { Min } \left( x ^ { I } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\delta ^ { y , \zeta } = \frac { \left( \operatorname { Max } \left( y ^ { I } _ { t } \right) - \operatorname { Min } \left( y ^ { I } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\delta ^ { z , \zeta } = \frac { \left( \operatorname { Max } \left( z _ { t } ^ { I } \right) - \operatorname { Min } \left( z ^ { I } _ { t } \right) \right) } { \zeta }$$

where ζ is the number of sections.

Set [TeX:] $$\theta ^ { x , \zeta } = \frac { \left( \operatorname { Max } \left( x ^ { D } _ { t } \right) - \operatorname { Min } \left( x ^ { D } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\theta ^ { y , \zeta } = \frac { \left( \operatorname { Max } \left( y ^ { D } _ { t } \right) - \operatorname { Min } \left( y ^ { D } _ { t } \right) \right) } { \zeta },$$ [TeX:] $$\theta ^ { z , \zeta } = \frac { \left( \operatorname { Max } \left( z ^ { D } _ { t } \right) - \operatorname { Min } \left( z ^ { D } _ { t } \right) \right) } { \zeta }.$$

Set [TeX:] $$x ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { x ^ { D } _ { i } } { \theta ^ { x , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ [TeX:] $$y ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { y ^ { D } _ { i } } { \theta ^ { y , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ and [TeX:] $$z ^ { D } _ { t , \zeta } = \underset { \arg \max }{ \left| \frac { x ^ { D } } { \theta ^ { x , \zeta } } \right| } p \left( \left[ \frac { z ^ { D } _ { i } } { \theta ^ { z , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right)$$

where i is the index of dependent orientations, probability [TeX:] $$\mathrm { p } \left( x ^ { D } t , \zeta \right) = \max p \left( \left[ \frac { x ^ { D } _ { i } } { \theta ^ { x , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ probability [TeX:] $$p \left( y ^ { D } _ { t , \zeta } \right) = \max p \left( \left[ \frac { y ^ { D } _ { i } } { \theta ^ { y , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right),$$ and probability [TeX:] $$\mathrm { p } \left( z ^ { D } _ { t , \zeta } \right) = \max p \left( \left[ \frac { z ^ { D } _ { i } } { \theta ^ { z , \zeta } } \right] | \left[ \frac { x ^ { I } _ { t } } { \delta ^ { x , \zeta } } \right] , \left[ \frac { y ^ { I } _ { t } } { \delta ^ { y , \zeta } } \right] , \left[ \frac { z ^ { I } _ { t } } { \delta ^ { z , \zeta } } \right] \right).$$

The error rate is effected by the section ζ. Therefore, dependent orientations are determined considering multiple orientations calculated by multiple section ζs. Set [TeX:] $$x ^ { D } _ { t } = \underset { \arg \max } {x ^ { D }_{t,\zeta} } p \left( x^{D} _ { t, \zeta }\right)$$ , [TeX:] $$y ^ { D } _ { t } = \underset { \arg \max }{ y ^ { D }_{t,\zeta} } p \left( y^ { D }_{ t , \zeta } \right)$$ , [TeX:] $$z ^ { D } _ { t } = \underset { \arg \max }{ z ^ { D }_{t,\zeta} } p \left( z^ { D }_{ t , \zeta } \right)$$

## 4. Experiments

The proposed method was validated on a computer equipped with Windows 10 Pro, Intel i7-7700 (3.6 GHz), NVIDIA GeForce GTX 1050 2G, and RAM 16G (DDR4). One VIVE and one Myo were utilized for motion learning and motion estimation, as shown in Fig. 1. In the experiment, the VIVE headset was only used to view the boxing contents, not to measure the location of the headset. First, seven boxing motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard) were defined, as shown in Fig. 1.

Seven boxing motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard).

Each motion was measured three times at 30 fps for motion learning. The motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard) were measured one by one. To train the orientations with the Bayesian probability, the minimum and maximum of the x and y positions for the hands and x, y, and z orientations for the arms were obtained. The differences were divided into five increments for orientation and position quantization. Fig. 2 shows the obtained positions and orientations of each motion, and Fig. 3 shows the quantized positions and orientations of each motion.

The quantized positions and orientations of each motion were utilized to calculate the Bayesian probability. To verify the proposed method, each boxing motion were measured twice more and then quantized. The measured and quantized positions of the hands are shown in Figs. 4 and 5, respectively. The quantized orientations of the arms were estimated according to the Bayesian probability, as shown in Fig. 6.

Measured positions and orientations.
Quantized positions and orientations.
Measured positions for validation.
Quantized positions for validation.
Quantized and estimated orientations for verification.
Measured orientation with the Myo.
Quantized orientation with the Myo.
Sums of the measured orientations.
Cumulative error rates.

To verify the estimated orientations, the estimated orientations were compared with the actual orientations of the arms as measured by a Myo, as shown in Figs. 7 and 8.

Fig. 9 shows the accumulated sum of the measured orientations. Fig. 10 shows the cumulative error rate, which was calculated by the rate of the difference between the estimated and quantized orientations and the measured and quantized orientations.

Error rate of the orientation with 5 steps.
Error rate of the orientation with 10 steps.

Figs. 11 and 12 show the error rate for each type of orientation. The average error rate was about 26% with the section ζ by 10. The average error rate was effected by the section ζ. Therefore, the section ζ was set by 5. In the results, the average error rate with the section ζ by 5was about 20%. When both sections were considered to validate the proposed method, the average error rate was about 19%.

## 5. Conclusion

This paper proposes a method to estimate the arm orientations based on the hand positions by using the Bayesian probability. One VIVE and one Myo were utilized to collect the hand positions and arm orientations. The Bayesian probability was then calculated based on the collected data during orientation learning. The unmeasured arm orientations were estimated using the positions measured with VIVE based on the calculated Bayesian probability during the orientation estimation.

In the experiments, seven boxing motions were defined to validate the proposed method. Each boxing motion was performed three times for orientation learning and two times for orientation estimation. The estimated boxing orientations were compared with the orientations measured by a Myo and quantized. The average error rate was 19%, but the possibility of estimating the orientation of any body part without additional devices was demonstrated.

## Acknowledgement

This research was supported by the Ministry of Science, ICT and Future Planning (MSIP), Korea, under the Information Technology Research Center support program (No. IITP-2017-2013-0-00684) supervised by the Institute for Information & Communications Technology Promotion (IITP).

## Biography

##### Yunsick Sung
https://orcid.org/0000-0003-3732-5346

He is currently an Assistant Professor in the department of Multimedia Engineering at Dongguk University, Seoul, Korea. He received the B.S. degree in Division of Electrical and Computer Engineering from Pusan National University, Busan, Korea, in 2004, the M.S. degree in Computer Engineering from Dongguk University, Seoul, Korea, in 2006, and the Ph.D. degree in Game Engineering from Dongguk University, Seoul, Korea, in 2012. He was employed as a Member of the Researcher at Samsung Electronics in Korea, between 2006 and 2009.

## Biography

##### Ryong Choi
https://orcid.org/0000-0003-3044-7877

He received B.S. degrees in Department of Computer Engineering from Soongsil University in 2017. Since August 2017, he is with the Department of Multimedia Engineering from Dongguk University, Seoul as a MS candidate. His current research interests include Artificial intelligence and Immersive SW.

## Biography

##### Young-Sik Jeong
https://orcid.org/0000-0002-7421-1105

He is a professor in the Department of Multimedia Engineering at Dongguk University in Korea. His research interests include multimedia cloud computing, information security of cloud computing, mobile computing, IoT (Internet of Things), and wireless sensor network applications. He received his B.S. degree in Mathematics and his M.S. and Ph.D. degrees in Computer Science and Engineering from Korea University in Seoul, Korea in 1987, 1989, and 1993, respectively. He was a professor in the Department of Computer Engineering at Wonkwang University in Korea from 1993 to 2012. He worked and researched to Michigan State University and Wayne State University as visiting professor in 1997 and 2004, respectively. He is also an Executive Editor of Journal of Information Processing Systems, an Associate Editor of Journal of Supercomputing (JoS) and International Journal of Communication Systems (IJCS), and an Editor of Journal of Internet Technology (JIT), finally an Associate Editor of Journal of Human-centric Computing (HCIS). In addition, he has been serving as a Guest Editor for international journals by some publishers: Springer, Elsevier, John Wiley, Oxford University Press, Hindawi, Emerald, Inderscience, and so on. He is also a member of the IEEE.

## References

• 1 G. Bang, J. Yang, K. Oh, I. Ko, "Interactive experience room using infrared sensors and user’s poses," Journal of Information Processing Systems, 2017, vol. 13, no. 4, pp. 876-892. doi:[[[10.3745/jips.02.0068]]]
• 2 W . Song, L. Liu, Y . Tian, G. Sun, S. Fong, K. Cho, "A 3D localisation method in indoor environments for virtual reality applications," Human-centric Computing and Information Sciencesarticle no. 39, 2017, vol. 7, no. article 39. doi:[[[10.1186/s13673-017-0120-7]]]
• 3 J. Kim, D. Chung, I. Ko, "A climbing motion recognition method using anatomical information for screen climbing games," Human-centric Computing and Information Sciencesarticle no. 25, 2017, vol. 7, no. article 25. doi:[[[10.1186/s13673-017-0106-5]]]
• 4 Z. Wang, Z. Liu, Q. Ma, A. Cheng, Y. H. Liu, S. Kim, et al., "Vision-based calibration of dual RCM-based robot arms in human-robot collaborative minimally invasive surgery," IEEE Robotics and Automation Letters, 2017, vol. 3, no. 3, pp. 672-679. doi:[[[10.1109/LRA.2017.2737485]]]
• 5 R. Mehrizi, X. Peng, X. Xu, S. Zhang, D. Metaxas, K. Li, "A computer vision based method for 3D posture estimation of symmetrical lifting," Journal of Biomechanics, 2018, vol. 69, pp. 40-46. doi:[[[10.1016/j.jbiomech.2018.01.012]]]
• 6 P. Y. Kim, Y. Sung, J. H. Park, "Bayesian probability-based motion estimation method in ubiquitous computing environments," in Advances in Computer Science and Ubiquitous Computing. Singapore: Springer2015,, pp. 593-598. doi:[[[10.1007/978-981-10-0281-6_84]]]
• 7 S. G. Lee, Y. Sung, and J. H. Park, Multimedia Tools and Applications, 2016., https://doi.org/10.1007/s11042-016-3843-y
• 8 J. Chae, Y . Jin, Y . Sung, K. Cho, "Genetic algorithm-based motion estimation method using orientations and EMGs for robot control," Sensorsarticle no. 183, 2018, vol. 18, no. article 183. doi:[[[10.3390/s18010183]]]
• 9 J. Chae, Y. Jin, M. Wen, W . Zhang, Y. Sung, K. Cho, "Genetic algorithm-based adaptive weight decision method for motion estimation framework," The Journal of Supercomputing2018. https://doi.org/10.1007/ s11227-018-2247-3. doi:[[[10.1007/s11227-018-2247-3]]]
• 10 H. J. Kim, Y . S. Lee, D. Kim, "Arm motion estimation algorithm using MYO armband," in Proceedings of IEEE International Conference on Robotic Computing, Taichung, Taiwan, 2017;pp. 376-381. doi:[[[10.1109/IRC.2017.32]]]
Seven boxing motions (jab, straight, hook, uppercut, left weaving, right weaving, and guard).
Measured positions and orientations.
Quantized positions and orientations.
Measured positions for validation.
Quantized positions for validation.
Quantized and estimated orientations for verification.
Measured orientation with the Myo.
Quantized orientation with the Myo.
Sums of the measured orientations.
Cumulative error rates.
Error rate of the orientation with 5 steps.
Error rate of the orientation with 10 steps.