Virtual Guide Dog: Next-generation pedestrian signal for the visually impaired

Abstract

Accessible pedestrian signal was proposed as a mean to achieve the same level of service that is set forth by the Americans with Disabilities Act for the visually impaired. One of the major issues of existing accessible pedestrian signals is the failure to deliver adequate crossing information for the visually impaired. This article presents a mobile-based accessible pedestrian signal application, namely, Virtual Guide Dog. Integrating intersection information and onboard sensors (e.g. GPS, compass, accelerometer, and gyroscope sensor) of modern smartphones, the Virtual Guide Dog application can notify the visually impaired: (1) the close proximity of an intersection and (2) the street information for crossing. By employing a screen tapping interface, Virtual Guide Dog can remotely place a pedestrian crossing call to the controller, without the need of using a pushbutton. In addition, Virtual Guide Dog informs VIs the start of a crossing phase using text-to-speech technology. The proof-of-concept test shows that Virtual Guide Dog keeps the users informed about the remaining distance as they are approaching the intersection. It was also found that the GPS-only mode is accompanied by greater distance deviation compared to the mode jointly operating with both GPS and cellular positioning.

Introduction

The Americans with Disabilities Act (ADA) requires the same level of service to be provided to the visually impaired as to others. ¹ Despite the significant improvements for accessible pedestrian signal (APS) systems over the years, APSs are still facing many challenges. For example, the visually impaired have to deviate from their travel path in order to reach the pushbutton to actuate the pedestrian phase. A study showed that less than 16% of the participating visually impaired persons looked for a pushbutton and successfully found one. ² The APS also brings externality to the surrounding environment: the repeating tone from APS adds 5 dB of noise within a 12 ft radius. Besides, the installation cost for each APS system is approximately US$6000 per intersection with the labor cost included. ³

Even with APSs, the visually impaired are still experiencing difficulties in crossing signalized intersection due to the lack of safe crossing information. Research showed that only 49% of them started crossing during the walk interval and 27% of all independent crossings ended after the onset of the conflicting traffic. ² More seriously, when a crossing push button is unavailable, the walk time of the visually impaired become shorter than necessary for crossing under such circumstances. Admittedly, there are several state-of-the-art solutions that have been designed specifically for the visually impaired, including electronic white cane ⁴ with integrated camera, ⁵ wearable belt with GPS and compass, ⁶ BrailleNote GPS, ⁷ radio frequency identification (RFID), and ultrasonic-to-audio guidance system. ⁸ However, the majority of these devices have limited market penetration and they are still inadequate in one of the crucial aspects of a signalized intersection, which is the communications between the users and the signal controllers. Owing to the advancement of the information and communication technologies, a new generation of APS with a user-friendly interface can be developed. This study proposes a mobile-based APS, namely, the Virtual Guide Dog (VGD), which provides the visually impaired users with personalized intersection crossing instructions and improves the safety of the crossing. Not only does VGD provide a more interactive, yet higher usable application to the visually impaired, but it is also able to make full use of the existing “infrastructure,” as smartphones have evolved to an integral part of the modern society.

The remainder of the article is organized as follows. In section “Literature review,” the challenges of APS for the visually impaired as well as relevant research efforts are summarized. Then the proposed application architecture, test bed, and application logic are presented in section “System architecture.” Results of the preliminary proof-of-concept field test are discussed in section “Evaluation,” followed by section “Conclusion.”

Literature review

Accurate construction of a representation of the world is challenging for a person with loss of vision. ⁹ Besides physical inability, the “inaccessibility” was also characterized by the difficulty in acquiring information and spatial knowledge. ¹⁰ The visually impaired primarily rely on the auditory, olfactory, and somatosensory (haptic and temperature) cues to acquire the aforementioned information for navigation. ¹¹

Wayfinding systems have been developing to aid the visually impaired to travel independently. Rodriguez-Sanchez and colleagues^12,13 proposed an accessible wayfinding service system that focuses on people with a disability in both outdoor and indoor environments. Apostolopoulos et al. ¹⁴ developed an online localization system which helped the visually impaired navigate in an indoor environment by utilizing inexpensive sensors which were readily available in current smartphones. The system tracked the location of the user by sensor readings, knowledge of the indoor environment (e.g. architectural blueprints), and possible landmark confirmed by users. The system employed text-to-speech technology for interfacing with users. One of the drawbacks of the system was the inaccurate localization which was caused by electromagnetic noise in an indoor environment and mediocre quality of the built-in sensors in certain smartphones.

Crossing an intersection introduces an extra layer of complexity to the typical wayfinding for the visually impaired. The visually impaired are more vulnerable to collision, especially when crossing an intersection, due to insufficient information concerning traffic, walk phase, and intersection geometry. ⁷ A seemingly effortless crossing at intersection for sighted pedestrians has been proven considerably difficult for the visually impaired. The sub-tasks involving in crossing a street can be broken down to

1. Detecting the existence of an intersection;

2. Determining the street or direction (heading) to cross;

3. Locating the crosswalk;

4. Aligning the heading with the destination;

5. Maintaining a straight path during crossing within a crosswalk.

The visually impaired mainly relies on the sound created by the parallel movement of traffic as cue for starting the crossing at signal crossings that are not equipped with APSs. ¹⁵ With the presence of APSs, the beeps and chirps from APS provided directional crossing cues. However, the persistent noise from the street can mask the target signal (APS) and consequentially affect orientation and mobility for directional travel for the visually impaired. ¹¹

Furthermore, intersection geometric features (e.g. dedicated right-turning lanes, multiple lanes, rounded corners), which are designed to move vehicles as efficiently as possible, inevitably increase the difficulties for the crossing of the visually impaired. The National Cooperative Highway Research Program (NCHRP) conducted an extensive study ¹⁶ regarding the accessibility and crossing solutions at roundabouts and channelized turn lanes for pedestrians with vision disability. It was found that the surveyed intersections failed to provide reliable information for the following principle tasks: (1) determining crosswalk location, (2) aligning to cross, and (3) maintaining heading during crossing.

Not until the year of 2000 were audible pedestrian signals included in the Manual on Uniform Traffic Control Devices (MUTCD). ¹⁷ Chapter 4E in the MUTCD regulates the pedestrian control feature, including APSs and detectors. The manual specifies the standards in terms of location, walk indication, tactile arrows and locator tones, and extended pushbutton press feature from Section 4E.09 to 4E.13. Under the standards set forth by the MUTCD, the required features for an APS system include the following:

1. Tone for the pushbutton location;

2. Vibrotactile arrow indicating the crossing direction on the pushbutton;

3. Vibrotactile indication of beginning of the “walk” phase;

4. Audible “walk” phase indication tone (e.g. percussive tone or speech walk message);

5. Automatic volume adjustment function which maintains volume above the ambient sound level.

Guth et al. ¹⁸ developed an intersection database that provided intersection and crossing information to the visually impaired via a braille-based personal data assistant (PDA) device. Four categories of intersection information were available: (1) intersection shape and size, (2) crosswalks and curb ramps, (3) traffic signals and control, and (4) APSs. In the field experiments, they found significant positive effects for database description on crossing subtasks, including crossing initiation and use of the pushbutton. The positive effect, however, was not shown in the other subtasks, such as maintaining a straight path in a crosswalk. They also found that the likelihood of completing crossing was nine times higher in the presence of APS. With APS, a higher percentage of crossing completion before the end of the crossing signal was observed. ¹⁹

Bentzen et al. ²⁰ conducted a before-and-after study to examine the impact of APS equipped with beacon features in two US cites. The performance of the APS was evaluated based on two main categories: timing measure and wayfinding measure. Results showed that numerous improvements were achieved, including reduced stating delay, higher rates in finding the starting location, and better crosswalk alignment. Despite the improvements thus far, APS still faces challenges. First of all, the current APS requires users to know their travel direction in advance. Second, the actuation of the pedestrian phase (via a pushbutton) requires the visually impaired pedestrians deviate from their path of travel in most instances. Third, its above-ambient-volume requirement increases 5 dB of noise on average in the surrounding area; finally, ongoing maintenance for the APS is necessary and could be costly.

Wilson et al. ²¹ developed an assistance device for the visually impaired, called system for wearable audio navigation (SWAN). It addressed the limitations of previous speech-based navigation aids by adapting non-speech audio presentation of navigation information when possible. It also enabled the ability to track users’ current location and heading. In addition, it provided guidance for a near-optimal and safe walking path to destinations and the awareness of salient features of the path.

Ramadhan ²² proposed a wearable smart system that integrated a microcontroller (Arduino Uno), cellular communication, GPS module, and a solar panel. Three types of sensors (i.e. ultrasonic sensor, accelerometer, and voice recognition sensor) were used to track the path and alert the users of obstacles. The system was also equipped with the remote monitoring capability which could be made available to family and caregivers. Lan et al. ²³ developed a smart glass system, the main function of which was to see the world for the visually impaired, with a focus on public sign recognition. A mobile-based personal APS, named mobile accessible pedestrian signals (MAPS), was proposed by Liao. ²⁴ The main function of the MAPS system was to provide the visually impaired with the available intersection geometry condition as well as signal timing information through a smartphone application. Using built-in sensors of a smartphone (e.g. GPS and digital compass) along with signal phasing and timing plans, the MAPS can inform pedestrians not only when to cross, but also how to align with the crosswalk. Since the signal plans are centrally stored within the Minnesota Department of Transportation, necessary cyber-security infrastructures had to be placed including virtual private network tunneling, firewall, and request authentication.

The VGD application was motivated by the fact that the majority of the wayfinding applications were passive systems, which means that there was little, if any at all, communication between the visually impaired and the traffic signal controllers. In this article, we proposed a pedestrian crossing mobile application which could send crossing requests to the signal controller via The National Transportation Communication for Intelligent Transportation System Protocol (NTCIP) without the need for pushing the conventional actuation pushbutton. NTCIP ²⁵ is a family of standards intended for use in all types of management systems concerning the transportation environment (e.g. traffic signals, transit, emergency management, data archiving). NTCIP was developed to provide communication standards which ensure the interoperability and interchangeability among traffic controllers and intelligent transportation system (ITS) devices. Owing to NTCIP, a user-oriented and personalized APS could be developed.

System architecture

The high-level architecture for the VGD system is shown in Figure 1. The VGD mobile application continuously accesses the built-in GPS and compass units to determine the location and heading of a user, respectively. When a user reaches the proximity of an intersection, the VGD informs the user the proximity of an intersection and ask the user about the direction or street he or she wants to go. When a user arrives at the intersection, the application retrieves the available crossing information (e.g. direction, street names) for the user. A dedicated database containing the geometry information of intersections can be established in advance.

Figure 1.

High-level architecture for VGD.

Alternatively, open source GIS (e.g. Google Maps API) ²⁶ could also be utilized. Once the user has selected the path and stated the street name of choice, the VGD application verbally instructs the user to turn to the heading of crossing and, at the same time, sends the crossing request to the controller to actuate the pedestrian crossing phase. Once the crossing phase starts, the application informs the user that it is safe to cross the roadway and the remaining time for the green phase.

Figure 2 shows an overall procedure implemented by the VGD mobile application. The VGD mobile application plays a crucial role in enabling the VGD system to communicate with the users of the information that is necessary for safe and prompt crossing. The primary functions of the applications include the following:

• Constant GPS-based localization for a pedestrian on a roadway and crossing information based on proximity to the intersection;

• Touchable and audible user interface for users to exchange information;

• Pedestrian phase actuation without the need to press the pushbutton;

• Instructions, if necessary, for pedestrians to ensure alignment with the crosswalk;

• Wireless communication with a traffic signal controller via Bluetooth.

Figure 2.

VGD mobile application logic.

Currently, the VGD application is available on any Android-based mobile devices. Figure 3 shows the main screen of the VGD application running on an Android smartphone. Note that the VGD application is fully expected to be made available on iOS-based devices (e.g. iPhone, iPad).

Figure 3.

VGD Android application interface.

Proof-of-concept test

The use of VGD enables the visually impaired to save certain steps which would have been necessary. The steps that could be saved using the VGD application include (1) determining if an intersection is within the proximity, (2) determination of the location of a crossing pushbutton, if it exists, (3) excursion to the pushbutton from the path of crossing, and (4) determination of the start of a crossing phase. All of the steps saved above potentially result in a short crossing cycle (i.e. from the beginning to the completion of crossing activity at an intersection) by providing additional information (e.g. street name and heading information) to reduce users’ uncertainties. A shorter crossing time also enhances user’s satisfaction.

Simulation-based test

The application was first tested in a hardware-in-the-loop simulation (HILS) test bed. The primary purpose of the simulation was to test whether the application could successfully place the crossing request at the controller using NTCIP. As shown in Figure 4, the proposed HILS test bed is composed of (1) Vissim (a microscopic traffic simulator), (2) Vissim Component Object Model (COM) Interface, (3) a hardware signal controller, and (4) the VGD application. A production signal controller was used to test potential integration issues. A simulation manager program written in Vissim COM interface is used to coordinate all the simulation components. During the simulation, pedestrian crossing requests were generated and sent to the signal controller for actuation. The Vissim COM Interface reads the real-time signal data via NTCIP and updates the Vissim simulation. The simulation test showed that the VGD can successfully place requests to the controller.

Figure 4.

Hardware-in-the-loop simulation framework.

Field test

The second stage of the test is to examine the VGD application in the field. An intersection around the New Jersey Institute of Technology (NJIT) was selected for the field demonstration. Located at the intersection of Central Avenue and Lock Street in Newark, NJ, the selected intersection has a high pedestrian crossing demand due to its proximity to the university campus. A microcomputer which was capable of communicating with the mobile application and relaying crossing request remotely was added.

For the field evaluation, an Econolite Cobalt ATC controller situated in the Intelligent Transportation System Laboratory (ITSL) at NJIT, which was retrofitted with a microcomputer and Bluetooth scanner as shown in Figure 5, was used to test the controller integration. The pedestrian crossing calls were placed remotely from the field intersection via the NJIT campus wireless network or 4G/LTE cellular network. In order to monitor the real-time status on the controller remote location, as one would see the overhead signal, a virtual signal program which independently synchronizes and displays the controller signal head status was developed (Figure 6). The ground station for the field test is shown in Figure 7.

Figure 5.

Controller retrofitted with microcomputer and Bluetooth scanner.

Figure 6.

Virtual traffic signal head.

Figure 7.

Field test setup.

During the test, volunteer users were assigned with Android smartphones installed with the VGD application. They were instructed to walk to the intersection as shown in Figure 8. After being initialized, the application continuously updates a user’s location and the distance to the intersection. If the distance is greater than 500 m, the application does not produce any verbal notification. When the user reaches within 500 m radius of an intersection, the application notifies the user of the distance in every 100 m. Upon arrival of the intersection, the application announces the street and heading information. The user can loop through the available crossing directions by a short tapping on the smartphone screen. The crossing call is placed by a longer tapping on the screen after the intended crossing direction is announced. The evaluation results showed that the application is able to remotely communicate with the controller via NTCIP and place crossing requests successfully. More importantly, the application can reliability actuate the controller pedestrian phase. Only the Cobalt ATC controller was used during the test, but it is expected that the VGD can be easily integrated with other signal controllers which are NTCIP compatible.

Figure 8.

Field test participant.

Results and findings

The accuracy of position based on either onboard GPS or cellular-enhanced GPS was tested and the results are shown in Figure 9. The distance to the intersection from the starting point was 58.5 m away from the intersection and four intermediate reference points were recorded separately during the walk. The application showed 18 m closer than the ground truth for the enhanced GPS mode, whereas the distance was measured 14.8 m closer to the intersection than the ground truth for the GPS mode. The distance deviations exhibited a decreasing trend as the user is getting closer to the intersection. A 1.7 m closer (5%) distance was recorded for the enhanced GPS mode at reference point #2 and a 2.9 m closer (8%) distance deviation was measured at reference point #3. For the GPS-only mode, fluctuations were more obvious than the cellular-enhanced mode: the deviation was ranging between negative and positive values among the reference points, for instance, the deviation changed from 5% further at reference point #2 to 12% closer at reference point #3. When the user reached close proximity to the intersection (reference point #4), the distance deviation increased slightly for both modes. Overall, the distance measured by the enhanced GPS mode is more consistent than that of the GPS mode.

Figure 9.

Comparison of distance information.

The field test demonstrated that the VGD application was able to inform users the proximity of an intersection and provide intersection. Crossing requests were promptly relayed by the microcomputer and placed via NTCIP. However, there were some issues discovered during the field experiment:

• Sometimes, the built-in GPS experienced difficulty to pinpoint the location of the users. It could be attributed to the quality of the built-in GPS transponder as well as the default GPS filtering algorithm.

• As experienced in all GPS-based applications, the environment could impact the accuracy of the GPS. For instance, the GPS signal strength is weaker during cloudy days or under the shade of trees.

• Compass deviation was observed when an unaware user tilted the smartphone at an angle which exceeded certain threshold.

Conclusion

In this study, we proposed a mobile VDG application for the visually impaired users’ safe intersection crossing. Using NTCIP to communicate with the signal controller, the VGD application could be an attractive alternative for conventional APSs. The cost of implementing VGD is only a fraction of the conventional APS: the average cost including labor for installing one APS system for one intersection is US$6000, ³ whereas the estimated cost for implementing VGD is approximately US$100–US$500 depending on the price of the mobile device and the data plan. A remote signal controller was adopted for the field test, which demonstrated the success of using NTCIP, in conjunction with the VGD application, to make crossing request for the visually impaired persons using their smartphones.

The field test revealed possible future improvements as well. First, more signal controllers of different vendors should be tested for VGD interoperability. In addition, approaches to make VGD work with non-NTCIP-compliant signal controllers need to be developed. In such extreme cases, additional software/equipment may be needed for integrating the VGD system into incompatible signal controllers. Second, more advanced GPS filtering algorithms that are specifically designed for low-speed (i.e. walking speed) conditions could be adapted to enhance the positioning accuracy. For example, the pedestrian dead reckoning is an effective way to obtain the pedestrian’s location by the estimated distance traveled via inertial sensors ²⁷ and it can be used as a supplement for localization. Finally, feedbacks from the visually impaired users are desired for further improvements. Focus groups of experienced and inexperienced eligible users could be conducted to further investigate the user preferences.