Sage Journals: Discover world-class research

Abstract

This study investigates the influence of smartphone use on the embodied experiences of pedestrians in urban public spaces. Participants in this study engaged in leisure walks in a bustling urban environment. The study employed a multi-sensory, multi-modal data collection approach, which incorporated mobile eye tracking, screen capture, think-aloud, and participant data review. Findings revealed dynamic relationships between the urban pedestrians’ embodied experiences (“streets as experienced through the body”), digital content (“streets on the screen”), and spatial knowledge (“streets in the mind”). The study explored the “unfolding” practice between multiple versions of the surrounding environment and sheds light on the complex interplay of cognitive, experiential, and digital inputs in shaping pedestrian actions. Moreover, the study uncovers the paradoxical effects of smartphone usage and introduces both serendipity and familiarity into the pedestrian's journey through public spaces. Further, the implications of this research highlight the need for mobile media studies to embrace “messy” and “noisy” data for a more comprehensive understanding of the interconnections between minds, bodies, places, and mobilities.

Keywords

Pedestrians mobilities smartphones urban public space eye-tracking

Walking in a public space is a fundamental way of place-sensing and place-making, especially in a walkable, city environment: “The act of walking is to the urban system what the speech act is to language or to the statements uttered” (de Certeau, 1984, p. 27). Walking is essential for developing a sense of place (Adams, 2001; Wunderlich, 2008). Adams argued that walking affords the multi-sensory experience necessary for developing an organic sense of place, which he calls the peripatetic sense of place, that could not be developed through media (Adams, 2001). However, Laurier et al. (2016) noted that walking has historically been a mediated practice that involves various forms of media, such as maps and guidebooks. Due to the widespread use of mobile phones, digitally mediated walking practices have emerged over the past two decades. Initially centered around phone calls and text messaging, the last decade witnessed a broader range of mediated activities with smartphones and location-aware applications. These new practices are gradually shaping city dwellers’ sense of place in nuanced ways. Research has found that the rate of pedestrians whose attention is fixated on a smartphone screen while walking in public spaces has increased in recent years (Argin et al., 2020a; Fernández et al., 2020).

Meanwhile, scholars have become aware that urban spaces have been transformed into hybrid spaces that consist of multiple layers of physical spaces, socially constructed spaces, social relationships, and digital data (Bentley et al., 2012; de Souza e Silva, 2006; Gordon & de Souza e Silva, 2011). In the mobile media literature, the interest in the relationship between mobile media, spaces, and places peaked in the recent decade, with the emergence of extensive studies on mobile locative media. Researchers found that mobile locative media were associated with activity coordination (Frith, 2014; Humphreys, 2007; Licoppe, 2014; Xie, 2020), identity expression (Cramer et al., 2011; Evans, 2015; Rost et al., 2013), chance encounters (Licoppe, 2016; Licoppe et al., 2016), improving spatial knowledge (Bentley et al., 2012, 2015; Farrelly, 2012; Özkul, 2015), navigating through unfamiliar urban spaces (Kim & Lingel, 2016), and fostering a sense of place (Farrelly, 2014; Schwartz, 2015). There have also been a few concerns with these smartphone-mediated socio-spatial practices. Some scholars have made arguments about algorithms that are embedded in mobile applications that may have the potential to reduce urban chance encounters and serendipity (Foth, 2016; Zuckerman, 2011). Others have maintained that user-generated contents reinforce social biases about local communities (Frith, 2017; Zukin et al., 2017).

The above-mentioned opportunities and challenges that emerged from smartphone use may have a profound impact on users’ socio-spatial practices and their sense of place. Although existing research is abundant in reporting the general uses of smartphone apps in relation to place-making and place-sensing, more empirical research is needed to understand how these apps are used in situ.

Theoretical framework

Mediated social interactions in public spaces

Erving Goffman's (1963, 1971) works on social interactions in public spaces have been the basis of many research studies of mobile media use in public spaces. One specific research focus is related to the reconfiguration of co-presence in public spaces. Goffman perceived two types of individuals in public spaces, the “Withs” and the “Singles.” In Goffman's time, Withs and Singles were defined by the physical co-location of a company. In early ethnographic research of mobile phone uses in public, Humphreys (2005) found that Withs can be transformed into Singles, when one person in a co-located dyad answers the phone. Lasen (2006) found that Singles in public spaces who used their mobile phones did not simply disengage from their immediate surroundings. Instead, they and co-located others (strangers or companies) often collaboratively maintained interactions of various orders. On the other hand, researchers noticed that mobile media could create mediated Withs, thus turning Singles into Withs and creating multiple fronts of social interactions (Hampton et al., 2015; Hampton & Gupta, 2008; Humphreys & Hardeman, 2021).

In addition, Goffman (1971) differentiated between two things a person in a public space could be, a vehicular unit and a participation unit. Whereas much more attention has been paid to the individual-as-a-participation-unit in his writing (e.g., the Singles and the Withs), the less discussed individual-as-vehicular-unit is indeed crucial in mobile media and mobilities research (Jensen, 2006; Murtagh, 2001). An individual-as-a-vehicular-unit is a pedestrian who participates in the flow of street traffic. They do not simply walk. The pedestrian navigates through the public space by following the unspoken traffic “codes” and by constantly scanning their surroundings not only to avoid collision, but also to manage social encounters. Goffman also recognized that the pedestrian-as-a-vehicular-unit often partakes in other activities; that is, getting from point A to point B is seldom the sole purpose (Jensen, 2006). Jensen (2010) conceptualized that individuals traversing through urban spaces frequently slip in and out of “mobile withs” through “temporary congregations” with other co-present individuals. Further, he recognized that mobile media extends this notion of mobile withs to include remote others with whom the mobile user could interact while moving.

Smartphone apps, especially locative media apps, further complicate the meaning of co-presence in public spaces. Mobile media users are no longer interacting only with other acquainted users, but also with unacquainted users who share(d) the same space. Licoppe (2016, 2017) argued that mobile locative media could provide the affordance of chance encounters with nearby strangers on smartphone screens. He called these encounters pseudonymous strangers in contrast to the nameless strangers whom people could physically encounter in public spaces. As Goffman (1971) argued, a pedestrian in the public space constantly monitors their surroundings and performs various rituals to exhibit their indifference to strangers occupying the same space. Smartphone apps, especially those with locative features, can help the user to access the digital fronts of local places and form mobile withs with pseudonymous strangers on the screens. By placing the individual both “here” and “there” at the same time, this use of the smartphone may complicate the ways in which a pedestrian manages their encounters with others and local places.

Walking with smartphone as a multi-sensory, multi-modal experience

Walking in a public space is a multi-sensory, embodied experience. The pedestrian takes in information about their immediate surroundings through sight, hearing, and smell, among other senses. Farman (2012) argued that, with mobile media, our embodied engagement with the world is simultaneously through the mode of sensory and the mode of inscription. Argin et al. (2020b) found that, when performing a flânerie-type, walking-wandering task in an urban public space, participants who used their smartphones exhibited different bodily rhythms and visual attentions to their surroundings than their non-smartphone-user counterparts. By observing the use of a mobile map app by a pair of tourists, Laurier et al. (2016), found that the map app did not simply provide the users a “you-are-here dot.” The use and ongoing reconfiguration of the map app was reflexively tied to the tourists’ walking actions. Drawing on assemblage theory, Holton (2019) examined the mobility-technology assemblage that emerged in a city tour aided by a digital app. His findings revealed a complex web of relationships between the walkers’ mobilities, the mobile app, the environment, co-present others, and the embodied experience.

To understand the complex relationship between the pedestrian's embodied experience, digital information, the built environment, and social encounters in public spaces, this study draws on Christian Licoppe's concepts of the reflexive hybrid ecology and the unfolding practice afforded by mobile media (Licoppe, 2016; Licoppe & Figeac, 2018). These concepts stem from the notions of seams and hybrid ecology from the field of human–computer interaction (HCI). These two concepts are used to describe the way in which various sociotechnical infrastructures can create distinct domains of activities, as opposed to the notion of “seamless” integration of technologies in the lived experience. Licoppe (2016) argued that a core affordance of mobile locative media is that the “surroundings become simultaneously accessible through the embodied sensorium and through the mobile terminal, with both versions being somehow referred to an ego-centered ‘here and now’ ” (p. 101). A reflexive hybrid ecology, thus, is a sociotechnical environment in which “the digital experience and the live embodied experience of place are reflexively tied through proximity or location awareness” (p. 105). He further argued that, when using mobile locative media, the user's direct sense of the place and their digitally augmented sensing of proximal entities are often “unfolded” (similar to “unfolding” a paper map), in that the mismatches between the two articulated versions of the surroundings become salient to the user. This conceptualization offers methodological clarity for studying smartphone use in public spaces. It offers a feasible way to analyze the messy, interconnected nature of the complex sociotechnical system that is associated with mobile phone use in public spaces.

In Licoppe's theorization, unfolding is associated with the mismatch between two versions of the surrounding environment – one on the screen and the other though embodied experience. However, the everyday socio-spatial practices of urban pedestrians are often embedded in a complex web of sociotechnical systems, which may involve street grid, signage, digital displays, co-located people, and so on, as well as the mobile user's past experience and existing knowledge of the public space. As such, the goal of this study is to explore the unfolding practice between multiple versions of the surrounding environment.

Informed by this theoretical framework, this study aims to answer the following question. How does the use of a smartphone transform the urban pedestrian as a vehicular unit, in relation to making sense of their surroundings, as well as managing nearby encounters?

Research design

A multi-sensory, multi-modal data collection approach

The theoretical framework detailed in the previous sections requires a research design that captures mobile media users’ embodied experience and situated actions. Therefore, an ethnomethodology-informed field study was conducted in which participants engaged in a flânerie-type leisure walk in a public space. (A detailed description of the research procedure is in the next section.) An ethnomethodology-informed approach allows for a moment-by-moment examination of how pedestrian/mobile users’ actions are integrated with elements in the surroundings and on the screen. Ethnomethodological studies of movements often involve closely examining video/audio recordings to understand how participants’ actions are shaped in the situation by pondering the “why that (action) here?” type of questions (Laurier et al., 2016).

This methodological approach requires the collection of detailed data on the participants’ sensory experiences, smartphone use, and social encounters. Therefore, I used a method similar to subjective evidence based ethnography (SEBE; Lahlou, 2011; Lahlou et al., 2015), in which I combined mobile eye tracking (MET), screen capture, think-aloud, and participants’ own reflections. SEBE is a research method that involves two key steps. First, researchers capture video data from a first-person perspective using a wearable camera that is positioned at eye level. In this study, Pupil Labs mobile eye-trackers (Kassner et al., 2014) were used to capture the first-person-perspective videos and eye gazes. Wearable cameras and eye-trackers have been used in mobilities and mobile media studies that examined micro-level behaviors in various contexts (e.g., Argin et al., 2020b; Figeac & Chaulet, 2018; Guntarik et al., 2018; Heitmayer, 2021; Kiefer et al., 2014; Laurier et al., 2016; Licoppe & Figeac, 2018; Wilhoit & Kisselburgh, 2016). Licoppe and Figeac (2018) argued that video-recorded data are essential for making sense of situated technological practices. However, they warned that the video data obtained from the wearable cameras are only a proximation of what the participant actually sees. As Figure 1 shows, the participant wore the Pupil Lab eye-tracking headset (a) that was connected to a MacBook Air laptop (d) in a backpack, which they carried during the walk. The second key step in SEBE is a “replay interview” in which participants watch the recorded video and provide explanations. In this study, the replay interview was conducted immediately after the video data were collected. The combination of these two steps allows researchers to gather both objective data from the video footage and subjective insights from the participants.

Figure 1.
A participant in this study walking in a shopping mall. Image of the eye-tracker (a) was taken from pupil lab’s official website.

Participants used their own smartphones in this study. The smartphone was connected to the research laptop through a USB cable (see Figure 1 (c)). During the study, all screen activities on the mobile phones were recorded through QuickTime (iPhones) or Android Debug Bridge (ADB) shell scripts (Android phones). At the time of the study, these methods of screen recording were the most unobtrusive and efficient, because they did not require the participants to install any third-party app on their phones. In a previous pilot study, I discovered that participants’ awareness of screen capture could influence their behavior and cause them to use their smartphones more frequently to “contribute” data for the study. To avoid this priming effect, I emphasized that the participants should use their smartphones as they normally would and reminded them that an artificial increase in their smartphone usage would not produce the data that I needed for my research.

I used the “think-aloud” method (Ericsson & Simon, 1993) to elicit the participants’ thought processes. Participants wore a lavalier microphone that was connected to the research laptop (Figure 1 (b)); they were instructed to say their thoughts aloud while walking. The microphone captured not only the participants’ in-situ reflections of their actions, but also the external audio from the environment. Lastly, throughout the study process, I shadowed each participant as they walked around in the public space. The main purpose was to assist the participants when necessary and to reduce their anxiety walking in a public space as “singles” with research equipment. This shadowing protocol was also an important data source, because it allowed me to gain first-hand, immersed experience of the participants’ mobilities (Lingel, 2013).

Participants and research procedure

This study was a part of a research project that examined city residents’ use of digital media and place-making practices in the city of Philadelphia. All participants were recruited through door-to-door visits in selected neighborhoods. Neighborhoods of various racial compositions and newcomer/long-term-resident ratios were selected to diversify the pool of participants. A subset of the participants completed the field study portion of the research. The findings in this report are based on data collected from 17 research sessions. The participants in this study came from different socioeconomic backgrounds and lived in neighborhoods of varying racial and cultural characteristics. As Table 1 indicates, the demographic characteristics of these participants were diverse.

Table 1.
List of participants, ordered by date of the study sessions.

Alias Gender Race/ethnicity Age Education Years of residency

Dan M East Asian 36 Post-graduate 5

Chad M Caucasian 27 Post-graduate 1

Chris M African American 23 High school 1

Matt M African American 36 High school 36

Leah F Caucasian 19 High school 1

Stephanie F East Asian 26 College 4

Lily F East Asian 23 College 23

Todd M African American 50 Post-graduate 50

Ral M South Asian 19 High School 1

Brian M Caucasian 39 Post-graduate 7

Ashley F Caucasian 41 Post-graduate 5

Lisa F Caucasian 34 Post-graduate 10

Kelly F East Asian 44 Post-graduate 10

Nick M African American 21 High school 11

Lindsey F Black Latina 46 Post-graduate 22

Annie F African American 32 High school 10

Natalie F African American 41 High school <1

Many previous field studies of the spatial behaviors of pedestrians have preferred experimental methods that rely on participants who do not have familiarity with the research site (e.g., Bertel et al., 2017; Kiefer et al., 2014; Münzer et al., 2006; Willis et al., 2009). Even in studies in which researchers conducted naturalistic observations (e.g., Holton, 2019; Laurier et al., 2016), researchers chose to study tourists, because tourists were more likely to use mobile media to explore their surroundings. However, research has shown that participants who are unfamiliar with the research location tend to look more at their surroundings and use their smartphones more for exploration (Argin et al., 2020b). In other words, using participants who are unfamiliar with the research location may produce data that reflect the perspective of a tourist or outsider. In contrast, the current study involved local residents, whose actions were driven by their everyday interests, needs, and socio-spatial practices as actual residents of the city.

Each participant met with the researcher near Rittenhouse Square in downtown Philadelphia. I selected this space as the research site, because it is known to be a pedestrian-friendly area that is often filled with diverse people, activities, and places of interest. Figure 2 shows a map of the core part of the research site, as well as selected street views taken from the video data. At the center is the Rittenhouse Square Park, often described by locals as the place for “people-watching.” To the north of the park is a busy, commercial area with tall buildings, big-brand stores, and a high volume of pedestrian traffic. To the south of the park is a quiet, residential area with small stores and restaurants.

Figure 2.
A map of the research site, with screenshots from the video data, showing the views and traffic volume of different areas. Map tiles by Stamen Design (maps.stamen.com), licensed under CC-BY 3.0. Data by OpenStreetMap, licensed under CC-BY-SA.

Each participant was then given a hypothetical scenario in which they moved to a nearby neighborhood: “On a Sunday afternoon, you have some time to kill and decide to check out the surroundings.” They were instructed to explore freely the surroundings in any way they wished and to terminate the walk at their own discretion. This instruction was designed to give the participant an exploratory walking experience in a public space. This task was supplemented by two purposive walking tasks. All participants were instructed to visit two places before they completed their leisure walk: a T-Mobile store and a Chinese restaurant. Participants were instructed to act as if they had to run some errands on the same day. The names and addresses of the two places were sent via text messages to the participants’ mobile phones. The locations of these two places are shown in Figure 2. The two places were selected for this study for two reasons. First, they are generic places of which the participants—despite being locals —were unaware prior to this study. Their locations also guided the participants to traverse varying city spaces (both the busy/commercial areas and the quiet/residential areas). After the walk was completed, I sat down with the participant in a nearby coffee shop to review the video data together. The participant provided detailed comments about their actions observed in the video.

Data analysis

For each research session, eye-gaze visualization, first-person-perspective video, and audio recording were analyzed in Pupil Lab's Pupil Player computer program. Smartphone uses were analyzed by matching each on-screen activity to the corresponding video footage. In addition, the replay interview recording was transcribed and matched to the eye-tracking video data and the smartphone screen recordings. Data analysis in this study used an ethnomethodological approach and focused on the sequential relationships between a participant's in-situ embodied actions, sensory inputs, cognitive reflections, and mobile media use.

Data validity

Initially, concerns emerged about two factors that could possibly compromise the quality of the data:
The participants might rush to complete the study by visiting the two required destinations as quickly as possible (completing the purposive walking with no regard to the exploratory walking instruction).

The eye-tracking device might cause the participants to act unnaturally in public.

However, the results were encouraging. The shortest walk took approximately 28 minutes, and the longest approximately 115 minutes. In contrast, if a participant wished to end the study as quickly as possible, it would take only 10 to 15 minutes to visit both places (approximately 900 meters of walking distance in total). As can be seen from the aggregated walking paths shown in Figure 3, many participants wandered quite far from where they began.

Figure 3.
Aggregated paths from all research sessions.

The research equipment did not appear to cause the participants to feel awkward or behave unnaturally. Each participant was asked how they felt about wearing the eye-tracking headset in public. All participants maintained that they no longer minded it after walking for only a few minutes. These claims were supported by what many participants did during the walks. They felt comfortable entering shops and restaurants, making purchases, or talking with people on the street and in the stores. During all research sessions, there were only three instances in which other people (one pedestrian and two store employees) took notice of and briefly inquired about the eye-tracking device.

It should be noted that some participants reported that the awareness that their smartphone screen was being recorded affected (but not to a great extent) their use of interpersonal communication on the phone, such as reading/sending text messages and checking social media. However, all participants said that other types of smartphone uses were consistent with what they would do in their everyday lives.

Findings

Overview

During this study, most participants used their mobile phones for a variety of activities. Table 2 provides a summary of the activities that the participants performed on their phones. This table excludes the use of the mobile phone for checking the addresses of the two destinations sent to the participants through a text message. Most participants used mobile maps (Google Maps or Apple Maps) for the purposive walking tasks. Fourteen participants used map apps at least once. More than half of the participants used their phones for other purposes, such as exploring local places on Yelp, taking pictures, searching on Google, and taking notes.

Table 2.
Participants’ use of their mobile phone during the walk.

Map Navigation Search engine Yelp Phone call &message Takingpictures Taking notes Pokémon GO

Annie

Ashley ✓ ✓

Brian ✓ ✓ ✓ ✓ ✓ ✓ ✓

Chad ✓ ✓ ✓ ✓

Chris ✓ ✓ ✓

Dan ✓ ✓ ✓

Kelly ✓ ✓ ✓

Leah ✓ ✓

Lily ✓

Lindsey ✓ ✓

Lisa ✓

Matt ✓ ✓ ✓

Natalie ✓

Nick ✓

Ral ✓ ✓ ✓ ✓ ✓

Stephanie ✓

Todd ✓ ✓ ✓

Streets in the eyes, streets on the screen, and streets in the mind

The purposive walking component revealed dynamic relationships between the visual input (“streets in the eyes”), the digital content (“streets on the screen”), and participants’ existing spatial knowledge (“streets in the mind”), among other situated factors.

Four participants (Leah, Matt, Chris, and Todd) used the navigation feature in the map apps to guide them to at least one of the two destinations. Using the turn-by-turn navigation app demanded that the participants monitor and compare the information on their phones, the immediate surroundings, and their existing knowledge of the area. Leah frequently checked her phone to make sure that she did not deviate from the instructed path. For example, when she was walking toward the park, her visual attention was initially on the buildings and store fronts (Figure 4, panes 1 & 2). Upon hearing the voice instruction given by the navigation app, she held up her phone and began to compare what she saw as the park (3, visual attention on the trees in the park) and what she saw on the screen (4). She said: “OK. Looks like I’m going to walk straight through … Rittenhouse (Square).” She then followed the navigation app's instruction and walked through the park. Throughout the navigation-guided part of the walk, she had to frequently perform this type of unfolding to compare what she was seeing, what the app was instructing her to do, and her knowledge of the area.

Figure 4.
Leah approaching the park, following the instructions from the navigation app. Blue circles highlight where the eye gazes (red crosses) concentrate, for visual clarity.

Participants not only “unfolded” their mobile apps to compare digital information with the immediate surroundings, but also frequently “unfolded” their survey knowledge of the area. This occasionally caused frustration for some participants. Several times, Chris had to stop at various intersections to manage the gaps between the instructions given by the navigation app, the surroundings, and his memory. Figure 5 shows one instance in which he was frustratingly trying to figure out the right direction at an intersection. When asked about this during the replay interview, Chris explained:
I was confused about which way to go, because the GPS was telling me to turn, but I knew, just from experience, that I could go straight. So, I just took that chance and took Google Maps’ way and kept that turn. . . . ’cause I’m like, it might save me a minute. It might be the best route for me, even though I had my own route.

Figure 5.
Chris getting confused at an intersection. The arrow indicates north. The pedestrian icon indicates moving (walking). Eye gazes highlighted with blue circles. Some gaze points are out of bounds (e.g., when he looks down at the phone).

The majority of the participants used map apps only briefly to check the locations of the destinations. Four participants used no map app at all. This was because the city's easy-to-navigate street grid made wayfinding easier for local residents who understood the rules of the street numbers. Nevertheless, street addresses alone did not give the participants sufficient information to pinpoint the two locations accurately. The map app became useful in these cases. Figure 6 shows a series of on-screen actions performed by Ashley in an attempt to locate the Chinese restaurant. Moving continuously through the park, she looked down at the text messages and said: “So the first is … Szechuan Hunan Chinese Restaurant. I don’t know where that one is. I don’t know where (No.) 271 is on 20th Street. So I’m gonna go ahead and Google the restaurant.” She opened the Google app on her phone and typed in “schezchuan hunan.” The app gave her a few autocomplete suggestions. She tapped on “Szechuan hunan restaurant phildelphia pa,” which showed her the information about the restaurant. She clicked on the map to bring up a full-screen view of the map. She then zoomed in on the restaurant. The screen showed the restaurant's location as well as her current location (blue dot): “Oh! It's on Spruce. I kind of went the wrong way. I always walk this way because I live in this direction. So I guess I just started going this way anyway,” she said when gesturing in the direction in which she was walking. She then put her phone away and continued her walk to find the restaurant. Note that figuring out the route to the restaurant was not the result of seeing the blue dot on the screen, but it came after the realization that the restaurant was at the intersection of Spruce Street and 20th Street (“Oh! It's on Spruce”).

Figure 6.
Ashley trying to locate the Chinese restaurant while walking in the park.

Although most participants planned the routes in their heads rather than in the map app, the quick glances at the map interface may have had an effect on their paths. When using mobile mapping apps, the users were exposed to information that allowed them to obtain a quick preview of the local places they might encounter. For example, when Chad searched for the Chinese restaurant in Apple Maps, he noticed the icons displayed on the map that represented businesses along South 20th Street (Figure 7). Glancing at the map had an effect on his later behaviors, as he explained during the data review:
I just quickly glanced down a list of places I was going to pass, saw all the juiceries, and I guess the Food & Friends. That's what got to me. It definitely made me aware of what was along the road. And I guess, have them being listed vertically here and the way the street was—I kind of targeted my approach to walk down the street that way instead of coming down from the side, where there's really nothing on the screen there.

Figure 7.
Screenshot of Chad checking out the location of the Chinese restaurant.

The mobile store front: various ways to interact with local places

When looking at the instances in which the participants used their phones for exploratory purposes, they often used smartphone apps to break the physical barriers in order to interact with local places. As Goffman (1963, 1971) noted, communication boundaries in public spaces are maintained by various physical and social barriers. Physical barriers, such as walls and closed doors, often prevent access. Glass windows allow partial access. Although many places in the public space are accessible, a pedestrian does not randomly enter any spaces that are open to them. For example, during this study, multiple participants entered local shops and restaurants. However, they were more likely to enter large shops or grocery stores simply to browse goods. And when they entered small shops where they had to interact with the employees, there were often clear purposes for doing so. For example, when Lindsey noticed a small apothecary, she only looked into the store through the glass, but only a few moments later she walked into a nearby grocery store and browsed the goods (Figure 8). Similarly, Natalie looked into a bakery through the glass, but did not enter the store. Later, she entered an optical shop to inquire about prescription glasses and entered a pizza shop and a coffee shop to inquire about and purchase foods. In these instances, she had conversations with the store employees, while sustaining eye contact with them. This shows that various public places have various levels of “openness” to them. As such, shops and restaurants often have window displays or menus posted at the entrance to allow for limited engagement.

Figure 8.
Two participants engaging with various local stores. Eye-gaze areas highlighted with blue circles for clarity.

Smartphones allow alternative ways to access local places. Two types of smartphone uses for exploratory purposes were noted. The first was to use the smartphone to explore immediate places without physically accessing them. In one instance, Ral, a young student, walked past a bakery/restaurant called Spread Bagelry (Figure 9, 1). Approximately 16 seconds later, he paused his walking and began to search for “spread bagelry” in Google Search. He not only read the reviews in the “critic reviews” section of the result page but also clicked on the link that opened the menu on the bakery's website. When reviewing the data, he explained his actions: “If I walked by something, I might check the ratings and reviews and make a mental note. There are a lot of food and restaurant places in the city. I’ll make a note if I’m interested.” During his 50-minute walk, Ral had checked the information on 2 other restaurants after he walked past them.

Figure 9.
Ral looking up Spread Bagelry on his phone.

This was a common practice performed by many participants. When walking across an intersection, Dan made some sniffing noises with his nose, saying: “Something smells really nice!” He then noticed a restaurant on the street corner (Figure 10, 1). After crossing the street, he took out his phone and opened the Yelp app to look up information about this restaurant (2). He kept reading detailed information about the restaurant while walking the entire block, approximately 150 meters in distance (3–6). During this time, he only occasionally looked up from his phone to avoid collision with people in the street (4 & 6). In this instance, mobile window shopping was triggered not by visual cues, but by olfactory cues in the surroundings. In this case, we also see how mobile window shopping could take place while the individual was constantly moving, managing his interactions both with people around him and the store front on the screen. In Goffman's terminology, the individual simultaneously acted as a mobile, vehicular unit through the physical space and a part of a participation unit on the screen.

Figure 10.
Dan looking up information about a restaurant without stopping. Eye-gaze areas highlighted with blue circles.

In addition to using smartphones to access nearby places, some participants used their smartphones to explore remote places and events. For example, when Ral walked past a 7-Eleven convenience store, he glanced at the store (Figure 11, 1) but kept walking (2). Although he did not stop walking, he quickly pulled out his phone and searched “wawa near me” in Google (3). The app showed him a map and a list of three Wawa convenience stores. He scrolled down to check all three locations. He later explained this sequence of actions: “So, I have passed a 7-Eleven. I’m not a huge fan of 7-Eleven. So, I was just curious to see if there's any Wawas nearby—for reasons I grew up back home with a Wawa.”

Figure 11.
Ral searching for Wawa. Eye gazes are highlighted with blue circles.

In a similar case, when Lindsey was walking in a busy, commercial area, she took out her phone and searched on Google for “running shoes.” The app showed several images of Nike running shoes and some ads. Dissatisfied with the result, she added “rittenhouse square” to the search keywords. This time, her phone showed a map and a list of relevant shops nearby, which included “Philadelphia Runner,” “New Balance,” and “Lululemon.” She tapped on “Philadelphia Runner” and carefully examined the description of the store while continuing to walk. Later, she explained her behavior in the interview:
Because I’ve seen Modell's (a sporting-goods store) at the corner of my eye. So, I wanted to see where there was a legitimate runner store . . . . Really, I was trying to remember the name of the store. I knew specifically there was a runner store, like a technical runner store. But I don’t remember the name of the store. I remember the general vicinity of the store.

Figure 12 shows this sequence of actions. Lindsey saw the sporting-goods store (1), walked for three seconds (2), and then decided to search for the running shoes store while walking (3).

Figure 12.
Lindsey searching for running shoes stores. Eye gazes are highlighted using blue circles.

It may have become evident that using the smartphone to access nearby or remote store fronts was not simply triggered by environmental cues. The individual's personal interest and experience played important roles. This is evident in an instance that is illustrated in Figure 13. Chad, a medical student, first walked past “Yogorino,” a frozen yogurt shop (1). Seeing that the shop was not open, he kept on walking, during which he visited both the Chinese restaurant and T-Mobile (2). Sixteen minutes later, standing next to T-Mobile, he noticed “Sweet Café” across the street (3a). He said: “It says ‘Sweet Café,’ which might have frozen yogurt.” He took out his phone and searched “frozen yogurt” in Yelp (3b). Both Sweet Café and Yogorino appeared on the screen. He read a few reviews of both places while saying: “Yogorino … Oh! It's got pretty good reviews … Looks pretty good … ‘Hands down the best frozen yogurt I’ve ever had.’ I plan to head back there later.” Later during the data review session, he reflected on this sequence of actions:
I had seen the other frozen yogurt place (Yogorino) earlier and I was going to compare the two and see if there were any other places around that might be different. And the Sweet Café, the aesthetic of it was kind of generic and didn't seem to be like something I would be interested in. It was more like a commercial frozen yogurt, not really pertinent to the neighborhood. Whereas Yogorino was more unique, artisanal, which appealed to me. So, I guess I just wanted to see what else there was around before committing to just walk into a random place.

Figure 13.
Chad looking for frozen yogurt.

Although Sweet Café was a serendipitous discovery that pertained to Chad's interest, using the smartphone to compare it to a remote place that he had previously encountered reduced the possibility of his engagement with a “generic” dessert shop, which—to him—did not fit into the ideal image of the Rittenhouse Square neighborhood. In the previous three cases, the use of a smartphone was paradoxically connected to a city resident embracing one serendipitous discovery but rejecting another.

Discussion and conclusion

Drawing on Goffman (1963, 1971) and existing theories of mobile locative media (Farman, 2012; Holton, 2019; Licoppe, 2016), this study examined the various ways smartphone apps are associated with reconfiguring the space–time relationships between a pedestrian and their surroundings. Findings from this study show that the in-situ actions of participants were shaped by a complex, contingent, and intertwined set of cognitive, habitual, experiential, environmental, and digital inputs. Various uses of smartphone apps could enable the individual simultaneously to traverse the urban space while engaging with local places. This was achieved through various “unfolding” practices (Licoppe, 2016). Further, findings from this study suggest that individuals do not simply unfold their smartphone apps to compare two different versions of the “here and now.” Rather, they need to manage multiple versions of the “here and now” that are shaped by various sensory inputs, memories/lived experiences, spatial knowledge, urban design, habits, personal interests, and preferences, as well as digital information. The individual, therefore, often needs to make sense of the discrepancies between these different versions of surroundings. The study also uncovered the paradoxical effect of smartphone use, wherein it simultaneously introduced elements of serendipity and uncertainty as well as familiarity and certainty into a pedestrian's journey through public spaces. These findings extend Licoppe's (2016) observation that digital mobilities reshaped the way in which city pedestrians managed engagement, disengagement, and detachment (“the right to preserve one's tranquility”, p. 114) in urban public spaces.

In the past decade, the mainstream mobile media research has largely abandoned the “zero-sum” view of mobile-phone-mediated interactions and actions in public spaces. Instead, the present paradigm recognizes the new opportunities that smartphones have introduced to existing actions and interactions pertaining to local places and spaces (Campbell, 2019). This study contributes to this research program by providing a comprehensive analysis of smartphone uses within an intricate setting, closely mirroring the real-life environments urban dwellers encounter in their daily lives. The insights gleaned from this investigation suggest that mobile media research may stand to gain from transcending the confines of the interplay between on-screen and off-screen elements. Moores (2017) posited that media uses represent merely a single facet of place-making practices, coexisting among a multitude of others. Consequently, he championed a non-media-centric approach to media studies, encompassing a holistic perspective on the interconnections between media and space. Similarly, although it may appear counterintuitive to decenter mobile media within mobile media research, this approach could potentially unveil a more intricate interplay between minds, bodies, places, and mobilities.

A related methodological implication of this study is that mobile media research could benefit from embracing “messy” and “noisy” data generated in less controlled field studies. In the current study, many “noises” were permitted into the data by giving the participants freedom to determine how or whether they should use their phones. However, this research design yielded data of high ecological validity. This study also captured multi-sensory inputs ranging from eye gazes, audio, think-aloud data, and participants’ own reflections of their actions. Many ethnographic studies in the mobile media literature have been influenced by phenomenology and ethnomethodology, in which the embodied, multi-sensory experience is emphasized. However, video data collected using body cameras or eye-glass cameras (which are often used in these studies) may not offer as accurate insights into the participants’ perspective as eye-tracking devices would. Although far from perfect, the think-aloud method provided this study with valuable data regarding sensory inputs beyond sight and sound. The amount of information and noise embedded in the dataset presented tremendous challenges, both for data collection and data analysis, but they also yielded exciting findings.

Alias	Gender	Race/ethnicity	Age	Education	Years of residency
Dan	M	East Asian	36	Post-graduate	5
Chad	M	Caucasian	27	Post-graduate	1
Chris	M	African American	23	High school	1
Matt	M	African American	36	High school	36
Leah	F	Caucasian	19	High school	1
Stephanie	F	East Asian	26	College	4
Lily	F	East Asian	23	College	23
Todd	M	African American	50	Post-graduate	50
Ral	M	South Asian	19	High School	1
Brian	M	Caucasian	39	Post-graduate	7
Ashley	F	Caucasian	41	Post-graduate	5
Lisa	F	Caucasian	34	Post-graduate	10
Kelly	F	East Asian	44	Post-graduate	10
Nick	M	African American	21	High school	11
Lindsey	F	Black Latina	46	Post-graduate	22
Annie	F	African American	32	High school	10
Natalie	F	African American	41	High school	<1

Footnotes

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Weixu Lu

Author biography

Weixu Lu is an assistant professor in the Department of Communication Studies at the University of Wisconsin–La Crosse. His research interest is digital media use and social networks.

References

Adams

P. C.

(2001). Peripatetic imagery and peripatetic sense of place. In P. C. Adams, S. Hoelscher, & K. E. Till (Eds.), Textures of place: Exploring humanist geographies (pp. 186–206). University of Minnesota Press.

Argin

Pak

Turkoglu

(2020a). Between post-flâneur and smartphone zombie: Smartphone users’ altering visual attention and walking behavior in public space. ISPRS International Journal of Geo-Information, 9(12), 700. https://doi.org/10.3390/ijgi9120700

Argin

Pak

Turkoglu

(2020b). Through the eyes of (post-)flâneurs—altering rhythm and visual attention in public space in the era of smartphones. Anthropologic: Architecture and fabrication in the cognitive age—Proceedings of the 38th ECAADe conference, 1, 239–248. https://doi.org/10.52842/conf.ecaade.2020.1.239

Bentley

Cramer

Hamilton

Basapur

(2012). Drawing the city. CHI ’12: Proceedings of the 2012 ACM annual conference on human factors in computing systems, 1603–1606. https://doi.org/10.1145/2207676.2208282

Bentley

Cramer

Müller

(2015). Beyond the bar: The places where location-based services are used in the city. Personal and Ubiquitous Computing, 19(1), 217–223. https://doi.org/10.1007/s00779-014-0772-5

Bertel

Dressel

Kohlberg

von Jan

(2017). Spatial knowledge acquired from pedestrian urban navigation systems. Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, 1–6. https://doi.org/10.1145/3098279.3098543

Campbell

S. W.

(2019). From frontier to field: Old and new theoretical directions in mobile communication studies. Communication Theory, 29(1), 46–65. https://doi.org/10.1093/ct/qty021

Cramer

Rost

Holmquist

L. E.

(2011). Performing a check-in: Emerging practices, norms and “conflicts” in location-sharing using foursquare. Proceedings of the 13th international conference on human–computer interaction with mobile devices and services, 57–66. https://doi.org/10.1145/2037373.2037384

de Certeau

(1984). The practice of everyday life. University of California Press.

10.

de Souza e Silva

(2006). From cyber to hybrid: Mobile technologies as interfaces of hybrid spaces. Space and Culture, 9(3), 261–278. https://doi.org/10.1177/1206331206289022

11.

Ericsson

K. A.

Simon

H. A.

(1993). Protocol analysis: Verbal reports as data (Rev. ed.). MIT Press.

12.

Evans

(2015). Being-towards the social: Mood and orientation to location-based social media, computational things and applications. New Media & Society, 17(6), 845–860. https://doi.org/10.1177/1461444813518183

13.

Farman

(2012). Mobile interface theory: Embodied space and locative media. Routledge.

14.

Farrelly

(2012). The role of location-based services in shaping sense of place: The role of location-based services in shaping sense of place. Proceedings of the American Society for Information Science and Technology, 49(1), 1–3. https://doi.org/10.1002/meet.14504901251

15.

Farrelly

(2014). Irreplaceable: The role of place information in a location based service. Journal of Location Based Services, 8(2), 123–132. https://doi.org/10.1080/17489725.2013.879217

16.

Fernández

Vicente

M. A.

Carrillo

Guilabert

Mira

J. J.

(2020). Factors influencing the smartphone usage behavior of pedestrians: Observational study on “Spanish smombies”. Journal of Medical Internet Research, 22(8), e19350. https://doi.org/10.2196/19350

17.

Figeac

Chaulet

(2018). Video-ethnography of social media apps’ connection cues in public settings. Mobile Media & Communication, 6(3), 407–427. https://doi.org/10.1177/2050157917747642

18.

Foth

(2016). Why we should design smart cities for getting lost. In J. Watson (Ed.), The conversation yearbook 2016: 50 standout articles from Australia’s top thinkers (pp. 109–113). Melbourne University Press.

19.

Frith

(2014). Communicating through location: The understood meaning of the foursquare check-in: Foursquare and mobile coordination. Journal of Computer-Mediated Communication, 19(4), 890–905. https://doi.org/10.1111/jcc4.12087

20.

Frith

(2017). Invisibility through the interface: The social consequences of spatial search. Media, Culture & Society, 39(4), 536–551. https://doi.org/10.1177/0163443717698871

21.

Goffman

(1963). Behavior in public places: Notes on the social organization of gatherings. Free Press.

22.

Goffman

(1971). Relations in public: Microstudies of the public order. Basic Books.

23.

Gordon

de Souza e Silva

(2011). Net locality: Why location matters in a networked world. Wiley-Blackwell.

24.

Guntarik

Garcia

J. E.

Howard

S. R.

Dyer

A. G.

(2018). TRACES: Mobile eye tracking captures user sensory experience in an outdoor walking tour environment. LEONARDO, 51(2), 163–164. https://doi.org/10.1162/LEON_a_01571

25.

Hampton

K. N.

Goulet

L. S.

Albanesius

(2015). Change in the social life of urban public spaces: The rise of mobile phones and women, and the decline of aloneness over 30 years. Urban Studies, 52(8), 1489–1504. https://doi.org/10.1177/0042098014534905

26.

Hampton

K. N.

Gupta

(2008). Community and social interaction in the wireless city: Wi-fi use in public and semi-public spaces. New Media & Society, 10(6), 831–850. https://doi.org/10.1177/1461444808096247

27.

Heitmayer

(2021). “It’s like being gone for a second”: Using subjective evidence-based ethnography to understand locked smartphone use among young adults. Proceedings of the 23rd international conference on mobile human–computer interaction, 1–17. https://doi.org/10.1145/3447526.3472026

28.

Holton

(2019). Walking with technology: Understanding mobility-technology assemblages. Mobilities, 14(4), 435–451. https://doi.org/10.1080/17450101.2019.1580866

29.

Humphreys

(2005). Cellphones in public: Social interactions in a wireless era. New Media & Society, 7(6), 810–833. https://doi.org/10.1177/1461444805058164

30.

Humphreys

(2007). Mobile social networks and social practice: A case study of dodgeball. Journal of Computer-Mediated Communication, 13(1), 341–360. https://doi.org/10.1111/j.1083-6101.2007.00399.x

31.

Humphreys

Hardeman

(2021). Mobiles in public: Social interaction in a smartphone era. Mobile Media & Communication, 9(1), 103–127. https://doi.org/10.1177/2050157920927062

32.

Jensen

O. B.

(2006). “Facework”, flow and the city: Simmel, Goffman, and mobility in the contemporary city. Mobilities, 1(2), 143–165. https://doi.org/10.1080/17450100600726506

33.

Jensen

O. B.

(2010). Negotiation in motion: Unpacking a geography of mobility. Space and Culture, 13(4), 389–402. https://doi.org/10.1177/1206331210374149

34.

Kassner

Patera

Bulling

(2014). Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction. Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing: Adjunct publication, 1151–1160. https://doi.org/10.1145/2638728.2641695

35.

Kiefer

Giannopoulos

Kremer

Schlieder

Raubal

(2014). Starting to get bored: An outdoor eye tracking study of tourists exploring a city panorama. Proceedings of the symposium on eye tracking research and applications, 315–318. https://doi.org/10.1145/2578153.2578216

36.

Kim

Lingel

(2016). Working through paradoxes: Transnational migrants’ urban learning tactics using locative technology. Mobile Media & Communication, 4(2), 221–236. https://doi.org/10.1177/2050157915619650

37.

Lahlou

(2011). How can we capture the subject’s perspective? An evidence-based approach for the social scientist. Social Science Information, 50(3–4), 607–655. https://doi.org/10.1177/0539018411411033

38.

Lahlou

Le Bellu

Boesen-Mariani

(2015). Subjective evidence based ethnography: Method and applications. Integrative Psychological & Behavioral Science, 49(2), 216–238. https://doi.org/10.1007/s12124-014-9288-9

39.

Lasen

(2006). How to be in two places at the same time? Mobile phone use in public places. In J. Höflich & M. Hartmann (Eds.), Mobile communication in everyday life: Ethnographic views, observations and reflections (pp. 227–251). Frank & Timme.

40.

Laurier

Brown

McGregor

(2016). Mediated pedestrian mobility: Walking and the map app. Mobilities, 11(1), 117–134. https://doi.org/10.1080/17450101.2015.1099900

41.

Licoppe

(2014). Living inside location-aware mobile social information, the pragmatics of foursquare notifications. In J. Katz (Ed.), Living inside mobile social information (pp. 109–130). Greyden Press.

42.

Licoppe

(2016). Mobilities and urban encounters in public places in the age of locative media: Seams, folds, and encounters with ‘pseudonymous strangers.’ Mobilities, 11(1), 99–116. https://doi.org/10.1080/17450101.2015.1097035

43.

Licoppe

(2017). An attempt at exhausting an augmented place in Paris: Georges Perec, observer-writer of urban life, as a mobile locative media user. In R. Wilken & J. Clemens (Eds.), The afterlives of Georges Perec (pp. 205–224). Edinburgh University Press.

44.

Licoppe

Figeac

(2018). Gaze patterns and the temporal organization of multiple activities in mobile smartphone uses. Human–Computer Interaction, 33(5–6), 311–334. https://doi.org/10.1080/07370024.2017.1326008

45.

Licoppe

Rivière

C. A.

Morel

(2016). Grindr casual hook-ups as interactional achievements. New Media & Society, 18(11), 2540–2558. https://doi.org/10.1177/1461444815589702

46.

Lingel

(2013). Information practices and urban spaces: Portraits of transnational new-comers [Unpublished doctoral dissertation]. Rutgers, State University of New Jersey.

47.

Moores

(2017). Arguments for a non-media-centric, non-representational approach to media and place. In P. C. Adams, J. Cupples, K. Glynn, J. André, & S. Moores (Eds.), Communications/media/geographies (pp. 132–160). Routledge.

48.

Münzer

Zimmer

H. D.

Schwalm

Baus

Aslan

(2006). Computer-assisted navigation and the acquisition of route and survey knowledge. Journal of Environmental Psychology, 26(4), 300–308. https://doi.org/10.1016/j.jenvp.2006.08.001

49.

Murtagh

G. M.

(2001). Seeing the “rules”: Preliminary observations of action, interaction and mobile phone use. In Wireless world: Social and interactional aspects of the mobile age (pp. 81–91). Springer-Verlag.

50.

Özkul

(2015). Mobile communication technologies and spatial perception: Mapping London. In R. Wilken & G. Goggin (Eds.), Locative media (pp. 39–51). Routledge.

51.

Rost

Barkhuus

Cramer

Brown

(2013). Representation and communication: Challenges in interpreting large social media datasets. Proceedings of the 2013 conference on computer supported cooperative work—CSCW, 13, 357. https://doi.org/10.1145/2441776.2441817

52.

Schwartz

(2015). Online place attachment: Exploring technological ties to physical places. In A. de Souza e Silva & S. Mini (Eds.), Mobility and locative media: Mobile communication in hybrid spaces (pp. 85–100). Routledge.

53.

Wilhoit

E. D.

Kisselburgh

L. G.

(2016). Through the eyes of the participant: Making connections between researcher and subject with participant viewpoint ethnography. Field Methods, 28(2), 208–226. https://doi.org/10.1177/1525822X15601950

54.

Willis

K. S.

Hölscher

Wilbertz

(2009). A comparison of spatial knowledge acquisition with maps and mobile maps. Computers, Environment and Urban Systems, 33(2), 100–110. https://doi.org/10.1016/j.compenvurbsys.2009.01.004

55.

Wunderlich

F. M.

(2008). Walking and rhythmicity: Sensing urban space. Journal of Urban Design, 13(1), 125–139. https://doi.org/10.1080/13574800701803472

56.

Xie

(2020). Mobile communicating place and place-inscribed communicative mobilities: Shaping alternative consumer cultures in mobile media communication. Mobile Media & Communication, 9(1). https://doi.org/10.1177/2050157920927451

57.

Zuckerman

(2011, May 7). Desperately seeking serendipity. In CHI ’11: Proceedings of the SIGCHI conference on human factors in computing systems. https://doi.org/10.1145/1978942.2167183

58.

Zukin

Lindeman

Hurson

(2017). The omnivore’s neighborhood? Online restaurant reviews, race, and gentrification. Journal of Consumer Culture, 17(3), 459–479. https://doi.org/10.1177/1469540515611203

Streets as experienced through the body,mind,and screen: The smartphone and the pedestrian's engagement with an urban public space

Abstract

Keywords

Theoretical framework

Mediated social interactions in public spaces

Walking with smartphone as a multi-sensory, multi-modal experience

Research design

A multi-sensory, multi-modal data collection approach

Participants and research procedure

Data analysis

Data validity

Findings

Overview

Streets in the eyes, streets on the screen, and streets in the mind

The mobile store front: various ways to interact with local places

Discussion and conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

Author biography

References