Designing a Vibrotactile Reading System for Mobile Phones

1. Introduction

With the miniaturization of touch screen mobile devices, the design and development of multichannel sensing and non-visual text reading methods have become a popular issue in the research on human–computer interaction [1, 2] . Vibrotactile perception assists people in non-visual information transfer. It allows for the design of blind use interfaces in specific conditions and enables visually impaired individuals to read [3-5] .

The design of vibrotactile patterns and their use in encoding text information are two of the key problems in tactile perception interfaces [6] . The key problems in design include making the interface easy to feel and identify and generating a reasonable coding of tactile perception for reading text information. At present, the preferred method for tactile reading involves utilizing Braille character encoding to design tactile perception. Examples include determining the different positions of the fingers on the touch screen through feedback vibrations to read Braille information [6] , using rhythmic coding by vibration to allude to Braille spot information [7] , and Morse code [1]. The main challenge of these methods is that they are difficult to learn. The user must remember numerous of codes, from alluding tactile sensing information to Braille code, and then identify text information. These methods impose a large burden on memory and recognition [1]. In addition, improving the reading accuracy and speed of alphabetic characters is difficult when using these methods.

In this work, we propose new vibrotactile codes and utilize known index characters in the form of alphabet rows and columns to enable reading. The 26 English alphabet letters are divided into four rows and seven columns in an index table. The vibrotactile feedback design of vibration on and off states allows the user to perceive the number of vibrations and index the characters in specific rows and columns from the index table. This method offers new vibrotactile codes, which are distinct and easy to memorize. In addition, the reading speed is faster than those in conventional methods. Fig. 1 shows an overview of the system interaction.

Fig. 1.

Overview of system interaction. (a) The phone graphical interface shows the character index table and vibrotactile codes, which are used to introduce the concept of the system. (b) The system produces a vibration pattern to the user. (c) The user perceives the vibrotactile code and reads the character “J”.

To improve the speed and accuracy of user perception of the number of vibrations, two user experiments were conducted. In the first experiment, we studied the human perception of continuous vibrations, namely, of 2, 3, 4, and 5 vibrations in an individual time, to determine the accuracy with which the users can perceive continuous vibrations. From this experiment, we learned that two continuous vibrations were reasonable. In the second experiment, we studied different parameters of minimum interval between continuous vibrations, namely, 90, 100, 110, 120, 130, and 140 ms. The results show that the appropriate parameter was 110 ms, with an accuracy of 92%. On the basis of these results, we designed the vibrotactile codes and reading system prototype. In the system evaluation test, we achieved 91.7% in terms of character reading recognition rate and approximately 617.8 ms in reading speed. These results were better than those obtained using traditional encoding methods that are based on Braille and Morse code [1, 6-8] and provide insights into further vibrotactile interaction design.

2. Related Work

Numerous approaches have been proposed to address the problem of tactile reading.

A study [6] proposed three tactile reading schemes that are based on a frictional tactile technique, namely, scan, sweep, and rhythm. Characters are encoded according to six-spot Braille. For scan and sweep, the finger is perceived as sliding on the screen on the basis of tactile feedback to read the characters that correspond to the Braille spots. For rhythm, a certain tactile feedback coding that is produced on the basis of the position of finger press in a timed series is used to represent the concaveconvex six-spot Braille. Rhythm provides better reading efficiency than scan and sweep and generates a high reading speed of up to 1,250 ms. The study initially proposed solutions to encoding tactile feedback with characters. However, it adopted the Braille code as a bridge to link tactile and character reading. This increased the mental burden of reading, and ordinary users are generally unfamiliar with the Braille code.

Several studies [1,7] extended the rhythm method using vibration tactile feedback of different lengths to simulate Morse code for interpreting Braille characters while improving the reading speed. This method divides the six-spot Braille into two rows. Each row has eight kinds of concave-convex permutations and combinations of three Braille spots. Combinations of different vibration lengths are used to show the corresponding semantic meaning. The reading speed of this method reaches a maximum of 797.7 ms, and the highest character recognition accuracy is 76%. Although the reading performance is improved, similar to the proposed method in another study [6], the system requires users to remember eight kinds of codes through long-term learning and skillfully grasp Braille coding.

Jayant et al. [2] showed that a mobile phone screen can be divided into six regions corresponding to the six spots of Braille and the vibration feedback of a finger touching different regions can represent the corresponding Braille spots. The study presented an easy-to-learn spatial mapping tactile feedback with Braille dots. However, the character reading performance is low, the speed of this method ranges from 4.3 to 26.6 seconds, and its recognition accuracy is 90%.

Similarly, in UbiBraille [8], a Braille-reading vibrotactile prototype was proposed. It mounts six rings that are augmented with vibrotactile capabilities on both hands to present the same coding used in a Braille typewriter. It encodes a character in a single point in time by actuating simultaneously in different fingers, and the recognition accuracy is 82% on average. However, the greatest disadvantage of UbiBraille is that it adds six wired rings for users to wear.

In this research, we present new vibrotactile codes and a character reading concept for mainstream mobile phones. We aim to implement a fast character reading system and propose a simple concept that is easy to learn on mobile phones. Vibration motor has become the standard accessory of smart phones and is currently the main means of design and development of tactile interaction on mobile phones [9-12]. Thus, the vibration tactile feedback is used as the method for designing the reading system. We conduct user experiments on vibrotactile code parameter tuning and evaluate the character reading performance of the proposed system. We also discuss and compare the reading performance of the system with that of conventional systems.

3.System Design

The 26 English alphabet characters are divided into four rows and seven columns, as shown in Table 1. The order of the alphabet is the most common arrangement and an easily memorable form for learning the characters. Each character can be positioned by using the numbers of the rows and columns. For example, character k is in row 2, column 4. Therefore, tactile design can encode rows and columns through the number of vibrations. For a character in row n, vibration will occur n-1 times. The number of vibrations of a column is based on the location of the character in the alphabet.

The required vibration tactile design is represented by the number of vibrations (e.g., character k is in row 2, column 4; thus, the number of vibrations will be 1 + 4 = 5). The key problems in its design are the consideration of time efficiency, in which shorter vibration and interval times should be achieved, and that the need for the user to clearly distinguish between the numbers of vibrations to guarantee the accuracy of information. Therefore, the parameters must be studied via a user experiment.

Our pilot study shows that a user can clearly perceive vibrations of 40 ms or longer and distinguish between different numbers of vibrations with a continuous vibration time interval of 70 ms or longer. However, accurately confirming the number of vibrations is difficult when the continuous vibration time is greater than 3. Therefore, single and two incidences of continuous vibration are used in the design scheme. After two incidences of continuous vibration, a 110-ms pulse is used as the time interval between the two incidences of continuous vibration, which results in a high recognition rate. The user obtains semantic information by identifying the number of vibrations, e.g., a single vibration is 1, whereas two incidences of vibration is 2. Furthermore, the linear accumulation of 1 and 2 can express any value.

To read a character, a single vibration is first generated according to the row of the character. Then, two incidences of continuous vibration or a single vibration is generated according to the column. Row 1 starts to vibrate the column directly, and 220 ms is used as the pulse time interval. For example, character “J” is in row 2, column 3. Hence, the vibration scheme comprises an initial 1-time vibration that represents row 2, and the 220 ms pulse represents the end of the row vibration. Next, two incidences of continuous vibration with a pulse of 110 ms and one incidence of vibration gain represent column 3. The duration of the entire process is row 40 + 220 + 40 + 70 + 40 + 110 + 40 = 560 ms.

The two following equations express the time for presenting a character in mathematics, where T1 refers to the time spent on row vibration codes, and T2 refers to the time spent on column vibration codes. r and c refer to the row and column values in the index table, respectively. i1 and i2 refer to the time intervals between two incidences of continuous vibration and between two incidences of continuous vibration and the next vibration, respectively (Fig. 2). The final time (ms) for presenting a character can be calculated as T = T1 + 220 + T2.

Therefore, the time spent on vibration tactile feedback of all characters can be calculated on the basis of this scheme. The shortest character is “A”, which requires 40 ms, whereas the longest character is “Z”, which requires 1,150 ms. On the basis of the frequency of a character in a word, the time spent on the vibration tactile feedback of a character is approximately 617.8 ms. This result is better than the theoretical finding of a previous study [1], which was 797.7 ms.

According to the design scheme above, the key problem is determining the better time interval on vibration tactile feedback between vibrations through the user experiment, as shown in Fig. 2. A highlevel segment indicates the vibration time, a low-level segment represents the stopping vibration, i1 refers to the time interval between two incidences of continuous vibrations, and i2 refers to the time interval between two incidences of continuous vibration and the next vibration. To determine the optimal values of i1 and i2 under the premise of recognition accuracy, attempting to reduce the value results in the best reading efficiency. Then, we can verify the rationality of i1 and i2 values through the user experiment.

Fig. 2.

Time intervals between vibrations. High-level segment represents vibration; low-level segment represents stopping vibration. i1 and i2 represent the short and long intervals between vibrations, respectively.

4. Experiment 1

In this experiment, we determined the time interval between continuous vibrations (i1) and the recognition accuracy of the user for the number of vibrations. We could check the recognition accuracy of the user for the number of vibrations under the conditions that the time intervals between vibrations were 40, 50, 60, 70, 80, and 90 ms and the numbers of vibrations were 2, 3, 4, and 5.

4.1 Apparatus

An HTC S720e mobile phone was used for the experiment. The phone has a 4.7-inch screen, weighs 130 g, is equipped with an internal vibration motor, and runs on the Android 4.0 OS.

4.2 Participants

Twelve college students, six females and six males with normal eyesight, were invited as experimental participants. Their average age was 20.5 years. All participants used their right hands to hold the phone in portrait mode. After the experiment, the participants were compensated with 30 RMB.

4.3 Procedure

A combination of time interval and the number of vibrations was generated randomly by the mobile phone application. The user clicked on the screen to begin to feel the vibration tactile feedback. After the vibration, the experimental participants stated the number of vibrations they felt. The participants spent approximately 5 minutes to practice and familiarize themselves with the vibrotactile codes. During the experiment, the participants had only one opportunity to state the number of vibrations. The generated vibration time interval and the number of vibrations that the participants felt were used to analyze i1. To prevent noise interference, the participants were asked to wear sound insulation earplugs. The experiment environment is shown in Fig. 3. The experiment lasted for approximately 30 minutes.

The total number of test tasks was as follow: 6 kinds of time intervals × 4 kinds of different numbers of vibrations × 5 experiment runs × 12 users = 1,440.

Fig. 3.

Experiment environment.

4.4 Results

The experimental result is shown in Figs. 4 and 5. When the numbers of vibrations were 2 and 3, the mean recognition accuracies were 93.6% and 86.4%, respectively. When the number of vibrations was greater than 3, recognition accuracy decreased rapidly. Therefore, 2 or 3 vibrations were appropriate for use as the recognizable tactile indicator in the design. For improved recognition accuracy, this study adopted two vibrations, together with one incidence of vibration, as the vibration tactile coding to be used as the design scheme of vibration codes.

Fig. 5 shows that when the vibration time interval was 70 ms or longer, the recognition accuracy of the user for the number of vibrations reached 91.7%. When the interval was less than 70 ms, the recognition accuracy decreased rapidly. Therefore, the time interval adopted in this work was 70 ms, that is, i1 = 70 ms

Fig. 4.

Recognition accuracy of different numbers of vibrations.

Fig. 5.

Impact of different time intervals on the recognition accuracy of vibrations.

5. Experiment 2

After determining two incidences of continuous vibrations and the vibration time interval, we further determined the influence of the time interval between one or two incidences of continuous vibrations (i2) after two incidences of continuous vibrations on recognition accuracy. The experiment detected the recognition accuracy of the user for the numbers of vibrations under different i2 values: 90, 100, 110, 120, 130, and 140 ms.

5.1 Apparatus, Participants and Procedure

The apparatus, participants, and procedure were identical to those used in Experiment 1.

The number of test tasks throughout the experiment was as follows: 6 kinds of time intervals × 5 times of repeated experiments × 12 users = 360 permutations.

5.2 Results

The experimental result is shown in Fig. 6. When the vibration time interval was 110 ms or longer, the user’s recognition accuracy of the vibration time reached 92%. The recognition accuracy decreased rapidly when the interval was shorter than 110 ms. Therefore, the time interval adopted after a continuous vibration was 110 ms, that is, i2 = 110 ms.

Fig. 6.

Recognition accuracy on different time intervals ( i2).

6. System Evaluation

The tactile perception reading system was designed according to the evaluation of reading accuracy, reading efficiency, and user satisfaction based on user experience.

6.1 Apparatus and Participants

The apparatus was identical to that in the previous section. The same group of participants was invited another day to participate in the experiment. After the experiment, the participants were compensated 30 RMB.

6.2 Procedure

In the experiment, an English alphabet character was provided randomly by the system. The user clicked on the mobile phone screen to begin feeling the vibration tactile feedback of this character. After the vibration was over, the experimental participants mentioned to the tester the row and column of the character they felt. In each experiment, the users had approximately 5 minutes to practice the test. During the experiment only one chance was given to state the number of vibrations they felt. The experiment lasted approximately 20 minutes.

The total number of test tasks in the entire experiment was as follows: 15 characters × 3 blocks of testing × 12 users = 540.

6.3 Results

The experimental results are shown in Fig. 7. The reading accuracies of the three blocks were 89%, 93%, and 93%, with a mean of 91.7%. The reading accuracy improved with the enhancement of user proficiency.

Fig. 7.

Reading accuracy.

7. Discussion

For the scheme proposed in this work, the vibration tactile feedback of each character has a fixed time. Calculations show that the mean vibration tactile time of the 26 characters is 656.9 ms. According to the probability of the emergence of English alphabet characters in the vocabulary [13], the mean time is 617.8 ms. Table 2 lists several design schemes proposed for tactile reading in recent years. The proposed method generates better reading efficiency and reading accuracy.

Using tactile feedback for reading is a challenging task, as reported by a previous study [1,7], where the user’s response time to character recognition was not considered in the conclusion. The follow-up study evaluated the users’ learning cost for different coding methods, their response time to the recognition, and their multi-character continuous reading. Meanwhile, high-frequency vocabulary can be considered to appear in front of the alphabet to increase the reading efficiency, on the basis of the frequency distribution of characters in the vocabulary. However, this technique increases the burden on learning and memory [1].

In the experiment, we empirically studied the interval parameters of vibrations, i1 = 70 ms and i2 = 110 ms with the efficiency and accuracy trade-offs. These parameters are useful in the distinguishability design of vibrotactile codes. In addition, two experimental participants could skillfully use 40 and 50 ms time intervals to recognize the number of vibrations and accurately read five incidences of continuous vibrations through touch. In the future design, the difference between individuals will be considered to design the personalized parameter customization. The users can choose between vibration, time interval, and continuous vibration according to their own preferences to achieve better reading efficiency. Meanwhile, a new semanteme with a longer vibration than the current 40 ms vibration should be considered in future designs to shorten the character reading time.

The participants were generally satisfied with the scheme proposed in this study. They gave it an average score of 4.2 out of 5. However, the participants needed to focus their attention while reading; otherwise, they could miss a vibration and commit an error. This result is similar to the findings of a past study [1]. Unlike the methods proposed in several works [6,7], that in this research does not require users to grasp Braille coding; instead, the users simply need to be familiar with the character of the alphabet. This requirement is less demanding on learning and memory.

8. Conclusions and Future Work

In this study, we proposed the design of vibrotactile codes and a character index table method to implement non-visual text reading on mobile phones. The experiment results indicated that the character reading accuracy reached 91.7% and the character reading speed was approximately 617.8 ms. These figures are better than those of conventional methods, such as the Braille coding method. In addition, we presented the experimental evaluation of the design of significant parameters of the vibrotactile feedback and found that (1) users could accurately identify the number of vibrations in an actuator’s on and off state interval at 70 ms or longer in perceiving continuous vibrations; (2) users could recognize two incidences of vibrations at high accuracy, that is, 93.6%, but the recognition accuracy decreased rapidly when the number of vibrations exceeded 3; and (3) the time interval between continuous vibrations should be at least 110 ms to achieve high recognition accuracy, that is, better than 92%.

In future work, we will consider the universal application with different kinds of users, such as the young, the old, the handicapped, and the blind, which might require additional experiments to identify the usability and optimize the vibrotactile code parameters during design. In addition, personalized parameter customization where users can choose the vibration, time interval, and continuous vibration according to their own preferences can be a direction for improved reading efficiency. Furthermore, a new semanteme with different vibration patterns should be considered to expand the vibrotactile coding dimensions.

Acknowledgement

This paper is supported by National Natural Science Foundation of China (No. 61502415) and Public Projects of Zhejiang Province (No. 2016C31G2240012).