Web-Based Behavioral Tracking Management System for Elderly Care Automation

1. Introduction

The proportion of the elderly continues to increase every year, caused by a low birthrate compared to the increase in life expectancy with the development of medical technology [1]. Hence, social interest in the health and safety of the elderly is increasing. The number of nursing homes for the elderly continues to increase every year. However, the quality of service is poor due to the lack of elderly caregivers. To improve the quality-of-care centers for the elderly, various studies such as the U-Care [2], the Internetof- Things (IoT) Care Robot [3] and other care services are under development [4-10]. These studies are based on the convergence between IoT services, wearable devices, and cloud platforms. Existing studies have been conducted to measure and analyze signals generated according to user actions based on wearable devices or sensors.

“IoT Care Robot” provides services based on ARTIK, an Internet-of-Things platform. Data that affects the health and cleanliness of patients is collected through sensors for each environmental information installed in the hospital room and wearable equipment worn by the patient. The collected data can be viewed and managed through an AI-based IoT Care Robot using cloud services. For the above system are high cost to building initial system. Because sensors must be installed, and wearable equipment needs to provide for each patient [4]. The “U-Care” service installs gas sensors, fire detection sensors, and motion detection sensors to cope with emergencies in elderly households living alone. It is a risk management system for emergency reporting through interworking with local centers and the National Emergency Management Agency if the sensor detects an emergency call of the elderly living alone or a gas leak. However, the above systems did not consider the general home environment, so there are limitations in the installation of the system depending on the structure of the house. It is also highly misdetected due to the equipment's battery, humidity, and the water vapor generated during cooking [5]. However, there is a problem that the cost of building the system is high and can be checked only the sensing data for abnormal symptoms.

To solve these problems, we propose a web-based on-demand streaming system for senior behavioral management services. In the existing system, when an abnormal pattern occurs through equipment that can measure behavioral information, such as a wearable device, it is reported to the manager. In this way, the actual occurrence information can confirm the behavior as a signal pattern. The proposed system manages a large amount of data by recording behavioral information over the web-based on the image information collected in the nursing home for the elderly and provides a management interface to collect datasets using the tags made by managers. Behavioral information can be confirmed as an image through this, and behavioral singularities that occur can track over time through the video. The system acquires videos of the elderly using CCTV (closed-circuit television) at the nursing home. Managers of the nursing home can annotate and manage behavioral information of actions of the elderly when they find out certain actions on the screen. This result makes it easy to find an annotation of behavior information recorded by a manager in a large amount of data. Elderly behavior information and the segmented video taken through CCTV should be available for inquiry according to the manager’s request. To guarantee the quality-of-service (QoS) of CCTV video streaming for behavior annotation, we analyze the correlation of streaming latency with the HTTP live streaming (HLS) protocol-based segmented image playback length and the number of video files.

2. Related Research

2.1 Communication Protocol

In this system, CCTV video files are segmented into pre-defined time lengths. To provide streaming of the segmented files, we compare and analyze various streaming protocols.

Real-time streaming protocol (RTSP) is a protocol to control real-time video-on-demand (VOD) services [11]. The real-time messaging protocol (RTMP) is designed for high-performance transmission of audio, video, and data in Adobe Flash platform systems [12]. It provides a stable connection with the server with QoS to users. However, the Adobe Flash player is not supported anymore since 2020. Apple develops HLS protocol for transmitting video and audio over a web server on iOS 3.0 and QuickTime X [13].

RTSP and RTMP have under 2 seconds of initial streaming delay, but HLS has over 2 seconds [14]. However, an open-source media player, VLC, and HTML5-based media players do not provide random playback when RTSP or RTMP is applied.

We use the HLS protocol for the proposed system to reduce external dependencies and to maintain them simply. To use the HLS protocol, the video file type must be a transport stream (TS). For streaming multiple TS files, m3u8 type files, which are lists for TS files to continuous play, are required.

2.2 Streaming Service Platform

Red5, ffserver, Live555, and mkvserver_mk2 has been studied for streaming service platform over the web. ffserver is an FFmpeg-based streaming server that uses HTTP and RTSP [14]. However, it has dependency problems and difficulties and no more updates [14].

Live555 can use WebM, H.264, and MPEG-1 formats, and various protocols such as RTSP, RTMP, and HLS. However, it is difficult to get information to develop and research [15]. mkvserver_mk2 can be a substitute instead of ffserver [16]. But there are no implemented services and documents.

Red5 is a streaming server based on Tomcat and Java servlet [17]. It provides RTMP essentially, and RTSP and HLS optionally with plug-ins. FLV, F4V, MP4, 3GP, and TS formats can be streamed on Red5. We use Red5 as a streaming platform for the proposed system.

3. Proposal System

Fig. 1 shows the system architecture diagram of the proposed system. The system consists of a frontend module and a streaming server with storage and CCTVs. The front-end module is implemented using the Bootstrap library and JSP. Fig. 2 is the main interface of the front-end module. The main interface comprises a calendar that provides a function of selecting recorded videos by date, a video player that outputs real-time or recorded videos, a tagging table for recording elderly behavior, and an output table of elderly behavior record information. In this interface, supervisors can record their requests and patients' or elderly’s behavioral information at the repository. Live-streamed CCTV videos and recorded videos are also played on this page. A manager navigates videos with seek bar and tags clicks.

Fig. 3 shows define the behavior information of the elderly in advance and constitute a table to add consistency among managers who record behaviors. In the case of the behavioral information table, the primary behavioral information occurring in the elderly care center, such as daily life, eating, and moving, was composed. In addition, in case of unusual matters other than the previously defined behavioral patterns, the user can directly add tags to the user tag table and share them with other managers.

Fig. 1.

System architecture of the proposed system.

Fig. 2.

Main interface of front-end module.

Fig. 3.

Tag table by predefined behaviors.

A streaming server provides CCTV selection, storage management, and streaming control. The user selects the recorded date of time and CCTVs for annotation or behavior analysis. The annotation tags are predefined in the database for seeking video and indexing. For saving annotated information and tags, it extracts a thumbnail screenshot of the selected time using FFmpeg and stores the manager ID, the selected tag, location, time of date, patient or elderly ID, and further information. When the user moves the mouse cursor over the custom seek bar, thumbnails are immediately displayed over the custom seek bar. When a manager selects a start time and an end time, the streaming server generates an HLS protocolbased .m3u8 file for streaming video files according to selected times. Furthermore, annotated behavioral information between the selected start time and the selected end time is also provided on a web page. Streaming videos are streamed and played according to an index file. An extension plug-in or hls.js [18] is needed for HLS protocol-based video playback on the user's browser. We deploy the Flowplayer [19] for video playback on the user’s browser. Flowplayer has a built-in hls.js plug-in and provides multiple functions like thumbnail and playback speed control.

4. Experimental Environment and Results

CCTVs installed in a nursing home for the elderly consist of divided videos at regular intervals for reasons such as setting in CCTVs. The proposed system is when requesting information on behavioral information and videos by the nursing home manager according to the date, the server returns videos and tagged data. In this case, when providing the divided original videos, the manager who checks each video has high fatigue. To solve this problem, in this section, the segmented videos are composed of a single stream. In addition, to guarantee the video streaming quality, the following experiment is conducted to minimize the delay during information requests and playback and random access to the video stream.

To provide seamless streaming, the initial delay and playback delay should be minimized. To reduce the initial delay and the playback delay of the proposed system, we try to find the optimal time length of a segmented video file. For experiments, we make segmented test video clips such as 1-second long, 10 s, 20 s, 30 s, 40 s, 50 s, and 60 s, from 10-minute-long videos. To evaluate the delays, random playback is applied. In each random playback, we measure playback delays. Testing system was Windows 10 Pro 64bit, i7- 7700, 64 GB RAM, Red5 1.0.9, Java 1.8.0.201, Chrome 73.0.3683.86, Flowplayer 7.2.6, Mysql 8.0.15.

Fig. 4 shows the initial delay time when loading on the web page by video segmentation size. The blue line is the left Y-axis, presenting index file creation, and the orange line is the right Y-axis, showing the initial delay in image load time. We can see that the delay time of navigating is reduced depending on the size of the video segmentation.

Fig. 5(a) shows 1-, 10-, 20-, and 30-second segment video clips, and Fig. 5(b) shows 40-, 50-, 60- second segment video clips and unsegmented 10-minute video. The experiment was in a localhost environment, and the results measured through five repeated experiments were averaged.

Segmented video clips of 1 second have the minimum and maximum delay times a cycle of about 3 seconds when random access playback. Segmented video clips of about 1 second have a minimum delay of 6.6 milliseconds and a maximum delay of 17.5 milliseconds.

Fig. 4.

Delay time occurs when navigating the image position by video segmentation size.

Fig. 5.

Delay in video position navigation according to video segmentation size: (a) 1- to 10-second segment video clip, (b) 40- to 60-second segment video clip and origin video (10 minutes).

In the case of segmented video clips of 10 seconds, it has the minimum and maximum delay times a cycle of about 3 seconds when random access playback like a 1 second segmented video clip. Segmented video clips of about 10 seconds have a minimum delay of 5.5 milliseconds and a maximum delay of 17.7 milliseconds. The segmented video clips of 20, 30, 40, and 50 seconds have the minimum and maximum delay times a cycle of about 3 seconds when random access playback. Segmented video clips of a minimum delay are 6.6, 6.7, 6.9, and 7.4 milliseconds and a maximum delay is 27.8, 35.1, 49.2, and 59.5 milliseconds. Segmented video clips in 60 seconds and 10 minutes unsegmented video have minimum and maximum delay times a cycle of about 60 seconds when random access playback. Random access delay times of a minimum of 4.2 and 3.6 milliseconds and a maximum delay of 246 and 242 milliseconds, respectively.

5. Conclusion

In this paper, we proposed a system for elderly behavior annotation and management services using CCTV streaming and VOD at nursing homes for the elderly. The proposed system provides annotation and management of elderly behavior information with predefined tags, date of time, and video clips. The streaming server provides live streaming and VOD of CCTVs with random access playback. When streaming video using the HLS protocol, the initial delay depends on the time length of segmented files. To provide seamless streaming, we evaluated the performance of the proposed system. By the experimental results, the number of segmented files in an index file and the time length of segmented video files affect the playback delay and the initial delay. We confirmed that the playback delay of the proposed system is less than 50 milliseconds, and the initial delay is less than 5 seconds. In addition, in the case of existing research, behavioral tracking is performed by purchasing additional devices, such as wearables, to check behavioral patterns. Although the proposed system is limited to the facility where the video recording device is built, there is an advantage that can use the service without a separate construction cost. In future works, we plan to study automated annotation of elderly behavior based on deep learning. In addition, we will be implementing an interplanetary file system (IPFS)-based information management system to store and manage videos generated continuously through CCTVs.

Acknowledgement

This research was supported by research grants from Daegu Catholic University in 2020.