Sage Journals: Discover world-class research

Abstract

State-of-the-art robots show promise in supporting but not completely replacing human work in most precision agriculture applications. For many potential agricultural robot applications, there are no comparable systems nor readily available information on the human operator activities to guide the systems engineering process. Such is the situation for Medjool date thinning, a tedious and hazardous manual operation for which technological assistance has yet to be developed. Here we describe using cognitive system engineering methods to develop operational concepts and human-robot coordination requirements for a pioneer system, a Robotic Medjool Date Thinning System (RDTS). We leveraged the abstraction hierarchy to characterize the RDTS’s envisioned goals and functionality. We developed alternative functional allocations to explore the design space based on the availability of different enabling technologies. After downselecting to the function allocation, including the technologies expected to be developed, we created operational event sequence diagrams to visualize the operation flow and to identify requirements related to the human operator and the joint human-robot system. Applying these methods in the early design stages helped to refine the human-robot coordination requirements and to identify gaps in the operational concept—they show great potential to support the introduction of agricultural robots and bring them to fruition.

Keywords

abstraction hierarchy agricultural robots function allocation human-robot interaction operational event sequence diagram

Introduction

Robots are beneficial in processes where tasks are repetitive, workload is high, and/or work-related hazards exist (Vasconez et al., 2019). In agriculture, robots are being developed to address the emerging shortage in the human labor pool—a challenge globally faced by the agricultural industry (Bechar & Vigneault, 2016, 2017; Salzer et al., 2021; Vasconez et al., 2019). Potential benefits include reduced operational hazards, increased agricultural productivity, increased product quality, reduced production costs, and reduced environmental impact (Nair et al., 2021).

Incorporating robot systems into agricultural work is challenging because, unlike industrial product lines, the agricultural environment and its products are unstructured; they are complex and highly variable, characterized by rapid changes in time and space (Bechar & Vigneault, 2016). Applying cognitive systems engineering (CSE) methods during system conception holds promise for advancing a successful application of agricultural robotics (Bechar & Vigneault, 2016). Various CSE approaches (Cuevas et al., 2007; Effken et al., 2001; Ernst et al., 2019; Huddlestone et al., 2017), among which are cognitive work analysis methodologies (Lintern et al., 1999; Naikar et al., 2006; Salmon et al., 2010; Stanton & Jenkins, 2017), have supported the incorporation of automation in various work domains. They provide information about the fundamental characteristics of the envisioned system’s work domain and the expected cognitive demands they impose (Bisantz & Burns, 2016; Bisantz & Roth, 2007; Stanton & Jenkins, 2017) and address perspectives with which the researchers gain an understanding of the factors affecting the operation performance in the particular domain (Bisantz & Roth, 2007).

However, human-robot collaboration with agricultural robots has been understudied. The majority of the reports on agricultural technological developments detail the technical aspects of the invention (De-An et al., 2011; Hohimer et al., 2019). Fewer details are provided concerning the features associated with the human operator, such as how, when, and under what conditions the operator collaborates with the new agricultural system and what are the expected consequences of that collaboration (Fei & Vougioukas, 2021; Reina et al., 2016; Silwal et al., 2017). Some agricultural system studies have evaluated the implications of a limited number of ways to couple the human with the robotic system’s autonomy (Berenstein & Edan, 2012; Bergerman et al., 2015; Tkach et al., 2011); others explored the consequences of a particular user interface design on the operator’s situation awareness and performance in the agricultural work environment (Adamides et al., 2017; Anagnostis et al., 2021; Cullen et al., 2012; Scholtz et al., 2005). Common to these studies, the researchers addressed the human operator’s needs with an already-developed robotic system.

Unfortunately, the lack of human-robot collaboration considerations in the early design stages is one of the critical factors impeding successful commercial assimilation of newly developed complex agricultural systems (Marinoudi et al., 2019). Joint human-robot systems in agriculture generally fall into the category of first-of-a-kind systems, that is, without comparable existing systems. First-of-a-kind systems pose multiple work analysis challenges (Bisantz & Burns, 2016); their conceptual stage holds no readily available information on the human operator’s activities and associated models and analyses. Work procedures, behavior, and strategies are more likely to be defined only after the operators gain experience with the new system. Because futurist agricultural robots are expected to operate in an unstructured, complex agricultural environment, the challenge of defining human roles and human-robot interface requirements increases (Marinoudi et al., 2019; Vasconez et al., 2019).

Medjool date farming is a prime target for robotic systems. Dates are an important crop in arid, warm regions of the world, and more than nine million tons are produced annually (FAOSTAT, 2020). Medjool is an elite cultivar praised for its quality. More than 750,000 Medjool trees in Israel are cultivated on approximately 6000 hectares (Cohen & Glasner, 2022). For Israeli growers in the desert and arid regions, dates are the main cultivated fruit crop and thus are a significant source of income (Cohen & Glasner, 2016).

Manual date thinning is one of the most labor-intensive tasks in Medjool cultivation (Cohen & Glasner, 2016; El-Assar & Refaat, 2013; Moustafa, 1998), incurring high costs and dependence on a large seasonal semi-skilled workforce that is becoming less available. Farmers invest approximately 3.5 labor hours to thin a single mature tree. Depending on the technique and the workers’ proficiency, it can take 40 labor days to manually thin one ha of 125 Medjool date trees. The workforce shortage and the time constraints force many large plantation growers to start thinning before fruit are visible and before the early natural abscission. Otherwise, thinning is completed later in the season when the fruitlets are too large. Robot systems could significantly enhance the thinning process’s efficiency, safety, and quality and reduce dependency on the limited seasonal workforce.

A venture was established to develop a first-of-a-kind Robotic Medjool Date Thinning System (RDTS). The RDTS’ goals were to estimate the number of fruitlets per fruit bunch and execute the thinning operation. This work presents the use of CSE methods to address the human-robot system-related requirements for this first-of-a-kind system. In the following sections, we describe the date cultivation work domain and the challenges that motivated the development of a first-of-a-kind RDTS. We apply an abstraction hierarchy (AH) framework (extended from Salzer & Bass, 2020). We then detail potential function allocations that consider the feasibility of associated enabling technologies. The selected function allocation (FA) takes advantage of enabling technologies currently under development. With the selected functional allocation, we generate an Operational Event Sequence Diagram (OESD) to visualize the envisioned operational flow. Finally, we leverage the OESD to derive information and human-robot collaboration requirements. We suggest that the approach described here is applicable across various work domains where novel, joint human-robotic system concepts are under investigation. Insights from the process are presented so that other cognitive system engineers can consider the approach for related application domains.

Date Cultivation: Domain Description

Medjool cultivation is labor-intensive and hazardous. In Israel, the annual cultivation cycle starts with removing unnecessary leaves and the spines from the bases of the remaining leaves to ensure a safe working environment. Growers need to pollinate female inflorescences (February and March, in Israel), conduct fruit surveys to estimate fruit settings (i.e., the number of rapidly developing fruits vs. degenerated or abscised ones) (April–May), thin the fruit bunches (March–May), tie up and cover the fruit bunches with net sacks (July), and selectively and repeatedly harvest the fruit (August–October) (Cohen & Glasner, 2016). The crown at the top of each tree is approached 8–10 times annually. The crowns may be over 25 m high. While in some parts of the world, workers climb with or without high ladders and use a securing belt, Israeli workers access the crowns of trees using specially adapted high platforms (Figure 1). When trees grow extremely high, they are cut down.

Figure 1.

High platforms support access to inflorescences and fruit bunches (clusters) for pollination, thinning, and other horticultural tasks. The platform closes around and completely surrounds the tree’s trunk just below the crown to enable a convenient approach for workers (Medjool date plantations in southern Arava, Israel).

Unlike most date cultivars, Medjool dates are usually marketed as a premium product. Size and fruit qualities are critical pricing factors that depend on good management practices and accurate and timely fruitlet thinning. On every fruit bunch, several thousand fruitlets can develop into fruit. They are arranged on 50–100 spikelets (strands), each carrying 50–100 fruitlets (Figure 2). Without thinning, 2000–3000 fruits may remain on each fruit bunch, but they will be too small for market. Therefore, farmers need to thin each fruit bunch, leaving only several hundred fruitlets. The thinning level depends on the tree’s age, the number of fruit bunches, environmental conditions, expected yields, and the specific thinning protocol employed. Typically, plots are homogenous in age and thus are managed uniformly; the horticultural protocol, particularly the thinning decision, is made per plot.

Figure 2.

Medjool date fruit bunch at the early thinning stage (approximately five weeks after pollination). During spring, large inflorescences emerge from the axils of the leaves that grew in the previous year. Each inflorescence comprises a fruit stalk (inflorescence rachis) at its base, with 50–150 spikelets (strands), each with 50–100 female flowers. Following pollination and fruit settings, the fruit stalk extends up to 2 m. A mature date palm tree may carry about 20–30 fruit bunches (photo adapted with modifications from Shoshan & Berman, 2021).

The ideal time for thinning is after fruit setting and natural early abscission; in southern Israel, this is usually in the middle of April. Date thinning is an effortful and hazardous operation. Hazards include dehydration, overheating, falling from a great height, cutting tool injuries, and harmful and painful palm spine stabs (Shapiro et al., 2006). On the platform, the workers reach through the canopy of the palm tree, interleaving fruit bunches and palm fronds, to reach for the fruit bunch they wish to thin. Workers sometimes climb on the safety rail and use the tree’s leaf bases as footholds to reach the fruit bunch (Figure 1).

The standard Medjool date thinning practice involves removing the central strands and shortening the spikelets. The workers use one hand to grab the inner part of the fruit stalk and secateurs in the other hand for cutting. Then the workers use one hand to hold a group of spikelets; with the secateurs in the other hand, they cut the spikelets to the desired length. Typically, the human hand cannot bind the entire fruit bunch’s spikelets at once; thus, these steps are repeated until all the spikelets in the fruit bunch are thinned as planned. The workers continue to the next fruit bunch until all fruit bunches are thinned. This process is usually performed in two to three rounds.

Method

Abstraction Hierarchy

The AH is a multileveled analysis that provides information about the system structure independent of any specific architecture or design (Bisantz & Burns, 2016; Rasmussen, 1985; Salmon et al., 2010). The higher levels of abstraction represent the system in terms of goals and functions, whereas the lower levels represent physical and informational resources. The modeling captures the overall purpose of the system, the value priorities, purpose-related functions and objects, the physical functions, the physical and informational resources, and the constraints imposed by the environment (Bisantz & Burns, 2016; IJtsma et al., 2019; Salmon et al., 2010; Stanton & Jenkins, 2017). Here, with AH, the cognitive systems engineers defined the system, independent of any autonomy, technology, or specific human-robot interaction.

Function Allocation

AH comprises a finite number of physical functions, physical resources, and informational resources. FA refers to exploring different human and machine work function assignments (Feigh & Pritchett, 2014; IJtsma et al., 2019; Pritchett et al., 2014). Theoretically, we could allocate each function to the human operator, the robotic system, or both; hence, in theory, the upper limit of combinations is 3ⁿ, where n is the number of physical functions. Each defined FA highlights individual tasks and collaborative human-robot system activities. IJtsma and colleagues proposed that an efficient FA should comply with one of three possible coherencies based on the association to the general function, coherency in the resources, or a FA that supports parallel work (IJtsma et al., 2019). Other considerations, such as cost, development time, or technical feasibility, may influence the selected FA. The FA under consideration here was limited by the development of enabling technologies coupled with research funding limitations. One could argue that this strategy is a form of “automate what you can and leave the rest to the operator,” a process that is not human-centered. However, in this case, it is a technical risk reduction strategy. So many of the system components are under research and development. In order to meet the system goals such as reducing hazards to the workers, the team also needed to address the available resources.

Operational Event Sequence Diagrams

The OESD is a time-based sequence schematic layout of information-decision-action tasks used in the dynamic description of human-machine systems (Brooks, 1960; Kurke, 1961). The OESD represents processes, procedures, decisions, outcomes, information flow, and other elements. Each subsystem and human agents are presented as vertical “swim lanes.” A set of predefined symbols describe the functions and processes. Directional connecting arrows within and between lanes denote informational flow and their relative timing. As such, the OESD visualizes the envisioned activities, decisions, and information needs (Harris et al., 2015; Huddlestone et al., 2017) and graphically represents the essential interactions between humans and machines (Stanton et al., 2021).

At the outset, there were no procedures, protocols, or operation concepts of how the operator was to work in coordination with the RDTS subsystems. To develop the OESD, the cognitive systems engineers conducted multiple meetings with each of the four enabling technologies research and development teams (each responsible for developing a subset of the RDTS (i.e., a subsystem)) and two date horticulture experts involved in the project. The first meeting yielded information about how the envisioned subsystems could be used by the human operator. The teams described inputs and outputs: what information the subsystem requires from the human and possibly other subsystems, and what information the subsystem will provide. After developing a draft OESD, the cognitive systems engineers met with each development team, so they could provide feedback. The resulting OESD summarizes the envisioned sequence of activities performed by the human or robot (task work), the activities performed to coordinate between the human and robot (teamwork), and the informational flow between them.

Information and Interface Requirements

Visualization of sequences of operations and interaction between the operator and subsystems supports developing requirements for individual and collaborative activities. An information presentation requirement refers to the information that the operator should receive. A control requirement refers to what the operator needs to control or manipulate. An input data requirement refers to the information that the operator enters or shares with the system. An information flow requirement refers to the content of information necessary to transfer between agents. A polygon positioned in the human’s swim lane maps to information representation or control requirement. An arrow crossing a swim lane maps to informational flow between the operator and the subsystem(s).

Results

Abstraction Hierarchy

In this work, the highest abstraction level describes the RDTS’s functional purpose (goal of the system): to reduce the number of fruitlets on each fruit bunch (Figure 3). In the next abstraction level, the system’s goal is decomposed into the system’s priorities and values:

1. Safety: The operational context and work environment are unsafe for humans; heavy equipment and cutting tools are used in a dense vegetative environment high above the ground. The climate is sweltering.

2. Efficiency: The objective is to reduce single-tree thinning time and the required number of human workers.

3. Accuracy: To achieve the expected date quality, spikelets trimming need to be performed.

Figure 3.

Hierarchical abstraction of the RDTS.

The operation is translated into three generalized functions: management of the desired yield, managing the robotic arm, and the trimming process. The next level describes the generalized functions and decomposes them into physical functions. At the lowest abstraction level are the physical and information resources that define the RDTS’s state of work and the environment.

The AH provides a high-level abstraction of the envisioned RDTS operation (Figure 3). For desired fruit management, the fruit bunch needs to be identified, the number of fruitlets currently on the fruit bunch needs to be estimated, and the desired amount of fruit to leave on the bunch needs to be determined. Based on the existing and desired fruitlets' load, the length of the spikelets needs to be determined. Thus, the location of the thinning cut needs to be set. For successful management of the robotic arm, the robotic arm needs to move close to the fruit bunch to grab the fruit stalk. For thinning, the robotic arm must position the trimming apparatus at the desired cut location. The trimming apparatus must grab the spikelets, cut them, and then release them. After trimming, the robotic arm needs to withdraw from the bunch.

Function Allocation

Table 1 presents seven function allocation strategies ordered by increasing need for enabling technologies. FA1 in Table 1 allocates most functions to the human operator but still supports reducing hazards to the operator (falling, dehydration, and injuries from the vegetation) as well as reducing physical effort, single-tree thinning time, and the number of required human workers. The human has complete control over the apparatus. The human needs to be able to move the robotic arm in three dimensions, in and out through the tree canopy, inspect the bunch, estimate fruit load, guide the gripper to the desired cutting location, and operate the trimming operation.

Table 1.

Function Allocations are Presented in Terms of the Increased RDTS Capabilities. Op = human operator, RDTS = Robotic Date Thinning System, N/A = not applicable.¹

Physical Function	FA1	FA2	FA3	FA4	FA5	FA6	FA7
Identify a fruit bunch	Op.	RDTS	RDTS	RDTS	RDTS	RDTS	RDTS
Estimate fruitlets load	Op.	Op.	Op.	Op.	RDTS	RDTS	RDTS
Determine the desired after-thinning fruitlets load	Op.	Op.	Op.	Op.	Op.	Op.	RDTS
Determine the location of the thinning cut	Op.	Op.	Op.	Op.	Op.	RDTS	RDTS
Reach for fruit bunch	Op.	Op.	Op.	RDTS	RDTS	RDTS	RDTS
Grasp/release a fruit bunch	Op.	Op.	Op.	RDTS	RDTS	RDTS	RDTS
Move to trim location	Op.	Op.	RDTS	RDTS	RDTS	RDTS	RDTS
Cut fruit bunch	Op.	Op.	RDTS	RDTS	RDTS	RDTS	RDTS
Withdraw from fruit bunch	Op.	Op.	Op.	RDTS	RDTS	RDTS	RDTS
Physical resources
Fruit bunch	Op.	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS
Means of vision	Op.	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS
Robotic arm	Op.	Op.	Op	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS
Trimming apparatus	Op.	Op.	RDTS	RDTS	RDTS	RDTS	RDTS
Fruitlets load estimation apparatus	N/A	N/A	N/A	N/A	RDTS	RDTS	RDTS
Informational resources
Position of fruit bunch	Op.	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS	Op./RDTS
Position of the robotic arm	Op.	Op.	Op.	Op.	Op./RDTS	Op./RDTS	Op./RDTS
After-thinning fruitlets load	Op.	Op.	Op.	Op.	Op.	Op.	Op./RDTS
Fruitlets load on a fruit bunch	Op.	Op.	Op.	Op.	Op./RDTS	Op./RDTS	Op./RDTS
Thinning cut location	Op.	Op.	Op.	Op.	Op./RDTS	Op./RDTS	Op./RDTS
Structure and orientation of fruit bunch	Op.	Op.	Op.	Op.	Op./RDTS	Op./RDTS	Op./RDTS

¹FA1 – The operator is responsible for all physical functions. All resources are assigned to the operator. FA2 – The RDTS identifies a bunch and recognizes its features. FA3 – The RDTS controls the trimming apparatus. It moves along the fruit stalk to the trimming location and executes the cut. All other functions and resources are assigned to the operator. FA4 – In addition to FA3, the human operator and RDTS share control of the robotic arm. The RDTS reaches for the fruit stalk, grabs it, and finally releases the bunch. FA5 – In addition to FA4, the RDTS controls the apparatus that provides the fruitlets load estimation. FA6 – In addition to FA5, once the desire fruitlets load is determined, the RDTS identifies the correct plane for cutting the spikelets. FA7 – In addition to FA6, the RDTS determines the desired fruitlets load.

With FA2 through FA4, the human must collaborate with the robotic arm and means of vision. With FA2, an automatic system identifies the fruit bunch and recognizes its structure. With FA3, approaching the thinning location and executing the cut are automated. With FA4, reaching for the fruit bunch and grasping are associated with robotic arm management.

With FA5 through FA7, the human and robotic systems communicate and share information associated with thinning decisions and execution. With FA5, the RDTS estimates the fruitlets load. Determining the location of the thinning cut is assigned to the RDTS with FA6. Determining desired fruitlets load is part of FA7. Assigning fruitlets load assessment and setting cut location to the RDTS can support the grower’s thinning decision and potentially advance the desired yield while maintaining crop quality.

The venture established to develop the RDTS initially envisioned a fully automated system, as shown in FA7. Availability of resources to develop enabling technologies led to the downselect to FA6. Automation of determining desired fruitlets load would entail stakeholder analyses, algorithm development and other research left for future work. Hence, an informed decision was made not to pursue this trajectory. The RDTS will identify a fruit bunch and execute the mechanics of the thinning process—a robotic arm will reach out to the fruit bunch, grab it, and then, with some cutting component, cut the spikelets at the human’s chosen length—leaving the desired number of fruitlets. The decision of how many fruitlets should be left on the fruit bunch after thinning is derived from one’s expertise, additional horticulture and environmental factors, and date market capitalization estimates. Developing decision support of this type was not prioritized in the research project but rather assigned to human operator. Hence, the grower will determine the desired yield per tree or fruit bunch and provide the information to the system.

Operational Event Sequence Diagrams

Four teams are conducting research and development of the robotic system’s enabling technologies. The robotic arm, grippers & trimming controller unit (RGU) team is developing a robotic manipulator (arm) for an environment with soft obstacles. The team explored hardware alternatives for the grippers manipulating the spikelets and alternatives for the cutting devices. The team evaluated commercial applications and incorporated candidate approaches into a custom robotic arm. The visual processing unit (VPU) team is developing a subsystem that ensures the gripper and end effector cutting system can be located at the thinning cut location. Image processing is used to identify the fruit bunches and features (structure, shape, tilt, and fruit stalk localization) (Bar-Shira, 2019; Shoshan et al., 2021). The motion control unit (MCU) team is responsible for a subsystem that controls the robotic arm moving toward the fruit stalk and placing it at the appropriate location and orientation along the spikelets. Following, the RGU can cut. The fruitlets load estimation unit (FEU) team is responsible for a subsystem that estimates the fruit bunch fruitlets load. The approach includes an echo acquisition system and a custom multi-channel sonar sensor.

In the OESD, one column is assigned to a human operator, and four others to the RGU, VPU, MCU, and FEU. Figures 4 and 5 present the OESD as well as highlight how the resulting requirements (discussed in the next section) emerged.

Figure 4.

Operational Event Sequence Diagram (OESD) for the Robotic Date Thinning System (RDTS). Numbered boxes and labled arrows are associated with Table 2 (OESD required interfaces).

Figure 5.

(Continued). Operational Event Sequence Diagram (OESD) for the Robotic Date Thinning System (RDTS). Numbered boxes and labled arrows are associated with Table 2 (OESD required interfaces).

Information and Interface Requirements

Human-robot interface requirements (information representation, input data, and control requirements) and information flow requirements are labelled in Figures 4 and 5 and detailed, respectively, in Table 2. We labeled and numbered the human-robot interface requirements with “HRI” in colored squares pinned to the polygon elements. We labeled and numbered the information flow with “Info” hovering over the arrows crossing the operator’s swim lane boundary.

Table 2.

Human-Robot Interface and Information Requirements. The Numbered Labels are Associated With Figures 4 and 5.

Label on Figures 4 and 5	Requirement (requirement category)
HRI.1	The operator should be able to select the source of visual feed (control)
	The operator needs to view visual data of the tree crown parts – leaves, fruit bunches (information representation)
	The operator needs to be able to move, zoom and manipulate the visual source (control)
Info.1	The operator needs to inform the VPU that the current position of the visual feed is valid for activating the visual processing procedure (information flow)
Info.2	The VPU needs to present the operator with the output of its processing procedure—detected fruit bunches and stalks (information flow)
HRI.2	The visual processing system will feature the detected fruit stalks and fruit bunch to allow the operator to interpret whether the VPU has successfully detected fruit bunches and stalks (information representation)
HRI.3	(Further to HRI.2) the system will allow the operator to determine whether the fruit bunch and fruit stalks can be associated in pairs (information representation)
HRI.4	The operator needs to associate the fruit bunch with its fruit stalk (control, input data)
Info.3	The operator needs to provide the MCU with the current positioning (location and orientation) of the robotic arm (starting position) (information flow)
Info.4	The operator needs to present the VPU with the fruit stalk and spikelets coupling (information flow)
Info.5	The system units will provide the operator with entrance configuration and fruit bunch base position associated with the selected fruit bunch, that is, the selected fruit bunch-stalk pair (information flow)
HRI.5	The visual display needs to support the operator in grasping the location of the fruit bunch base position concerning the selected fruit bunch, that is, fruit bunch-stalk pair (information representation)
	The visual display needs to support the operator in judging the practicability of the suggested entrance configuration in relation to the selected fruit bunch, that is, fruit bunch-stalk pair (information representation)
	The operator needs to inform the system that the base position and entrance configuration are acceptable (input data)
Info.6	The RGU needs to receive the operator’s approval of the entrance configuration (information flow)
HRI.6	The operator should be able to select the source of visual feed (control)
	The operator needs to view the selected fruit bunch and its surroundings—
	other leaves and fruit bunches (information representation)
	The operator needs to monitor the robotic arm’s gripper and end effector trimming system (information representation)
	The operator needs to be able to move, zoom, and manipulate the visual source (control)
	The operator needs to be able to move and manipulate the robotic arm (control)
	The operator needs to understand where the robotic arm is located in relation to the location of the selected fruit bunch (information representation)
Info.7	The system should inform the operator that the robotic arm has arrived at the fruit bunch base position (information flow)
HRI.7	The operator should be able to select the source of visual feed (control)
	The operator needs to view the selected fruit bunch and its surroundings—such as other leaves and fruit bunches (information representation)
	The operator needs to have a view of the robotic arm’s gripper and cutting device (information representation)
	The system needs to provide the operator with the ability to evaluate whether the robotic arm is correctly located at the base of the selected fruit bunch (information representation)
HRI.8	The operator needs to be able to inform the system that the robotic arm is correctly located at the base of the selected fruit bunch (input data)
Info.8	The operator needs to be able to inform the system that the robotic arm is correctly located at the base of the selected fruit bunch (information flow)
Info.9	The system units will provide the operator with the estimated fruitlet load number associated with the selected fruit bunch (information flow)
HRI.9	The human needs to be able to read the estimated fruitlet load
	The operator needs to view the selected fruit bunch (information representation)
	The system needs to provide the operator with the means to decide, with regards to the selected fruit bunch, where the thinning cut should take place (input data)
Info.10	The system needs to process the human decision to extract the length of the after-thinning spikelets (no owner)
Info.10	The system needs to forward the length of the after-thinning spikelet to the MCU (information flow)

The operational flow begins with the robotic system stationed at the tree crown’s height, up and running, facing a fruit bunch. For the VPU to operate successfully, a fruit bunch must be in sight. Therefore, it will be the role of the operator to teleoperate the robotic arm (labeled HRI.1 and labeled Info.1), changing its orientation and positioning until the VPU successfully identifies the fruit bunch and can differentiate the upper and lower parts of the fruit bunch (the spikelets wreath and the fruit stalk) from the rest (the palm fronds and tree trunk).

Correctly matching a fruit stalk to its spikelets is critical for the MCU to adequately calculate the position and orientation in which the robotic arm can approach the fruit bunch, dubbed here entrance configuration. The VPU development team reported that the unit is expected to detect several fruit bunches and stalks in one frame (labeled Info.2). However, the published algorithm cannot match a specific fruit stalk to its fruit bunch, that is, a cluster of spikelets (Shoshan et al., 2021). Hence, the OESD has assisted teams in highlighting that the human operator must match the fruit stalk to its fruit bunch (labeled HRI.2, HRI.3, and HRI.4).

The fruit stalk and spikelets coupling is an information resource that needs to be communicated to the VPU (labeled Info.4). The VPU will extract the reference location, where the spikelets come out from the fruit stalk, dubbed here fruit bunch base position, and transfer the coordinates to the MCU. The human will provide the MCU with the current positioning (location and orientation) of the robotic arm; information resource dubbed starting position (labeled Info.3). The location and orientation will be stored for future use when the robotic arm withdraws from the fruit bunch. Obstacles (leaves and other fruit bunches) will likely be present between the starting position and the fruit bunch base position. The MCU will manage an active vision procedure to cope with that challenge, that is, a planned movement that scans the target fruit bunch from different angles. The outcome of the active vision procedure is a movement plan: how to navigate through the obstacles to arrive at the desired fruit stalk. Acquisition of the optimal entrance configuration is mandatory to reach for the fruit bunch successfully. The operator needs to approve the entrance configuration. The human operator will be required to estimate whether the MCU’s planned motion is feasible (labeled HRI.5). The cost of poor planning might involve damaged crops or a disabled robotic arm. The MCU and VPU will provide this information (labeled Info.5) and await the operator’s approval (labeled Info.6). If not approved, the human operator will take control of the robotic arm and manipulate it toward the desired fruit stalk location at the base of the fruit bunch (labeled HRI.6). If approved (labeled Info.6), the MCU will execute its motion plan and inform the operator when completed (Info.7). Again, the human operator will be required to confirm that the robotic arm is correctly positioned at the base of the fruit bunch (labeled HRI.7 and Info.8). The cost of failed transition or a misplaced arm might be cutting at the wrong location of the fruit bunch, cutting a leaf, or potentially harming other valuable fruit bunches. In cases of no confirmation, the operator will take control of the robotic arm and manually move it toward the desired fruit stalk location at the base of the fruit bunch.

A second concurrent sub-process involves the FEU echo acquisition system. The MCU will move the robotic arm to provide the FEU’s multi-channel sonar sensor with optimal information gathering angles for fruitlets load estimation. The outcome of the FEU estimation will be communicated to the human operator (labeled Info.9). The operator must decide how much fruit to leave on the fruit bunch (labeled HRI.9) and translate it to setting a trimming cut location. The length of the trimmed spikelets dictates the amount of fruit left on the fruit bunch. The after-thinning spikelet length, measured from the base of the fruit bunch to the location of the trimming cut, needs to be calculated and communicated to the VPU and MCU (labeled Info.10). The OESD highlighted that the length calculation task had yet to be assigned to any team; this work is to be completed.

The RGU will first close the grippers to support penetration. When near the fruit stalk, the RGU will open the grippers, move to the fruit stalks, and close the grippers around them. At the same time, the MCU will ascend the robot arm from the “fruit bunch base position” along the spikelets until traveling desired after-thinning spikelet length distance. Once the arm arrives at the desired trimming position, the RGU will activate the cutting apparatus to trim the spikelets. The RGU will release the gripper, the VPU will communicate the starting position coordinates to the RGU, and the RGU will move the robotic arm back to its starting position. When completed, the next fruit bunch should be addressed.

Discussion

Agricultural robots are being developed to support humans where the tasks are repetitive, demand high workload and increased quality, or endanger workers; however, humans cannot be completely replaced because the agricultural work environment is complex and varied. For successful commercial assimilation of newly developed complex agricultural systems, the needs of the human operator, namely, how and when the human operator interacts with the agricultural robot, should be considered during system research and development stages.

The work described herein illustrates the use of CSE methods for a first-of-a-kind agricultural robot system in its early design stage. We employed the AH (Lind, 1999; Rasmussen, 1985) framework to define the system goals and functions, considering “what” needs to be done but not “how” it will be accomplished. The AH framework supported the development of a shared understanding of the purposes of the system and its structure independent of any specific event, technical ability, or design (Bisantz et al., 2003). The AH provided the basis for exploring the range of who-does-what possibilities, that is, function allocation. The seven FAs ordered by increasing need for enabling technologies allowed consideration of the enabling technologies (i.e., what can be automated in practice) and associated joint human-robot system interaction. With the selected FA, the OESD supported deriving human-robot collaboration requirements.

Previously, cognitive system engineers used OESD models to explore the operator’s behavior, workload, and support situation awareness in complicated aviation and transportation systems (Revell et al., 2016; Sorensen et al., 2011; Stanton et al., 2021). Huddlestone and colleagues (Huddlestone et al., 2017) applied OESD to infer the tasks and supporting technologies needed for the desired modification in an existing system. Harris and colleagues (Harris et al., 2015) compared the OESD of an existing system with a newer system configuration; the comparison allowed the analysts to reconsider the allocation of functions and associated technical consequence. Here the OESD was used to identify what information needs to flow between the operator and system units, to specify the monitor and control requirements, and to identify in greater detail the tasks dictated by the integration of the enabling technologies under development. The research and development teams made assumptions about the human-robot system interaction (e.g., the human will determine the cutting location). Gaps created from such assumptions informed some of the requirements in Table 2. The outcome of the OESD analysis, in the form of a list of information and interface requirements, as seen in Table 2, supports characterizing the research and development roadmap for future work. As the project progresses, technological capabilities outside of the venture will improve and technical gaps associated with the venture may narrow. Similar to the work of Huddlestone and colleagues (Huddlestone et al., 2017), the research and development teams will be able to use the original and any modified OESDs to compare, review, and discuss the emerging implications of the future technological advances.

With the OESD, we revealed the changing nature of the thinning operation; major differences exist between how the task is performed manually and how it is expected to be using the RDTS. We unveiled the teamwork by inspecting where lines cross swim lanes and the human task work by inspecting the processes within the human’s swim lane. Comparing the envisioned teamwork and task work to the current manual operation reveals fundamental changes in the future operation. The operator must teleoperate the robotic arm in a dense vegetative environment and manipulate distance cameras to see the robot or the fruit bunch in its surroundings. Because the technology cannot guarantee that the spikelets and fruit stalk are correctly matched or that the robot’s fine-movement plan toward the fruit bunch base is feasible, the operator will be required to master new tasks: match fruit bunch to fruit stalk, evaluate the reference location, identify where the spikelets come out from the fruit stalk, and assess the practicability of the position and orientation in which the robotic arm can approach the fruit bunch. The original thinning decision, which used to take place with the worker close to the fruit bunch while holding the spikelets, will be conducted in an entirely different manner—observing the fruit bunch through a visual feed. It is still unclear how accurate the operator’s decision will be in these new work settings and how the decision will translate for the system to be executed.

The RDTS will fundamentally change the strategy for the Medjool date thinning operation. Currently, thinning decisions are made per plot—a manageable plantation section. The RDTS is expected to substantially shorten thinning times, eliminate the risk and the worker’s physical effort, and reduce the number of workers involved in single tree thinning. Successful assimilation of this novel technology can advance precision agriculture exploration and change operational practices. With an accurate estimate of fruitlets load on a specific fruit bunch, the system is anticipated to assist the stationed-on-ground human operator in making a tree or fruit bunch-level thinning decisions based on a real-time fruit bunch-level survey. However, implementing the RDTS radically changes the kinds of tasks operators will be required to execute.

The OESD provided a structured representation to help refine the chosen FA, share it with others, get feedback, and update the representation. We then used the OESD to further review the capabilities of the system in greater detail, identify technological gaps, and define tasks and teamwork. Using the CSE methods detailed in this paper, we have a better understanding of the expected human operator’s activities. The analyses uncovered the need to implement support for new tasks and interfaces. These in turn will help to identify the required skill sets and training needs. Further study is required to validate whether the future workforce can do these envisioned tasks and, if so, how they will be trained.

While the enabling technology researchers provided ideas regarding their research goals and the predicted end states of their work, the cognitive system engineers strove to ensure that their results are translated into a fielded system—a system that will successfully team with the human operator. Here we employed analytical methods and were able to introduce the human-robot interaction problem to the multidisciplinary research and development team, thus providing a research and development roadmap that addresses the needs of the human operator in a new system. The AH and then FA phases allowed the collaborating teams to downselect to a feasible FA. The OESD further supported consensus of the operation concept, future research and development goals, and gaps to be addressed. The OESD phase assisted in clarifying how each subsystem or operator is dependent on the environment provided by the other system units to function. The case of the RDTS emphasizes the potential for these methods to support the introduction of sophisticated agricultural robot concepts. We believe applying these methods in the early design stages is critical to bringing state-of-the-art agricultural and other robot concepts to fruition.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Chief Scientist of the Israeli Ministry of Agriculture and Rural Development and the Israeli Date Grower’s Board in the Plant Council (Project 20-07-0018). This work was partly supported by the Louis and Bessie Stein Family Fellowship at Drexel University.

ORCID iDs

Yael Salzer

Yuval Cohen

Appendix

Yael Salzer is a research scientist at the Institute of Agricultural Engineering, Agricultural Research Organization, Volcani Center. Salzer completed B.Sc. in Electronic Engineering at Tel-Aviv University, M.Sc. and PhD in Industrial Engineering and Management at Ben-Gurion University of the Negev. Salzer was a post-doctorate in the Faculty of Social and Behavioural Sciences, University of Amsterdam. Her research is focused on the challenges humans meet in the complexity of the technologically emerging agricultural work environment.

Noy Saraf is a human factors engineer and UX researcher at the Agricultural Research Organization, Volcani Center. Noy completed a B.A. in Behavior Science from Ariel University of Samaria, and an M.Sc. in Industrial Engineering and Management from Ben-Gurion University of the Negev. Saraf is currently focused on bridging the gap between the challenges encountered in daily agricultural work, the technologies available, and those that will be available in the future.

Avital Bechar is a senior research scientist and the Institute of Agriculture Engineering director at the Agricultural Research Organization, Volcani Center. Prof. Bechar conducts fundamental and applied research in human-robot collaborative systems, proximal sensing, and developing new concepts and approaches for the operation and development of agricultural robots. Prof. Bechar is the former chairman of the Israeli Society of Agricultural Engineering, a co-founder of the Israeli Robotics Association, and an IEEE senior member.

Yuval Cohen is a research scientist at the Volcani Center in Israel. His research focuses on date palms and mangos. He combines horticultural studies with physiological and molecular biology approaches. Cohen has studied different aspects of date palm physiology, including fertilization and fruit settings, fruit quality, and the effects of plant bio-regulators on vegetative growth. He collaborated with other research groups to promote solutions for precision agriculture and plant protection of date palms.

Ze’ev Schmilovitch is an emeritus senior research scientist at the Institute of Agriculture Engineering at the Agricultural Research Organization, Volcani Center. Schmilovitch received his B.Sc., M.Sc., and PhD in Agricultural Engineering from the Technion, Israel. Schmilovitch has expertise in post-harvest machinery and Non-Destructive Testing (NDT) methods for produce quality sorting.

Sigal Berman is an associate professor in the Department of Industrial Engineering and Management at Ben-Gurion University of the Negev, Beer-Sheva. She received a B.Sc. in Electrical and Computer Engineering from the Technion, an M.Sc. in Electrical and Computer Engineering, and a PhD in Industrial Engineering from Ben-Gurion University. Berman leads the Intelligent Systems Engineering Laboratory (ISEL) where her research focuses on analyzing and engineering intelligent systems capable of dexterous motion.

Yossi Yovel is the head of the NeuroEcology lab in the School of Zoology and the Sagol School of Neuroscience at Tel Aviv University. He received a B.Sc. in Biology and another in physics from Tel Aviv University, an M.Sc. in Neuroscience from Tel Aviv University, and a PhD in Biology and Machine Learning from the University of Tuebingen. He completed a post-doc in the neuroscience department at the Weizmann Institute.

Avraham Sadowsky is a research scientist at southern Arava R&D. Sadowsky specializes in palm date cultivation.

Ellen J. Bass is Interim Senior Associate Dean for Research and Professor in the Department of Information Science at Drexel University’s College of Computing and Informatics. She also holds affiliate status in Drexel University’s College of Nursing and Health Professions and in the School of Biomedical Engineering, Science and Health Systems. Bass has over 40 years of human-centered systems engineering research and design experience. Bass is a fellow of the Human Factors and Ergonomics Society and former Secretary-Treasurer.

References

Adamides

Katsanos

Constantinou

Christou

Xenos

Hadzilacos

Edan

(2017). Design and development of a semi-autonomous agricultural vineyard sprayer: Human-robot interaction aspects. Journal of Field Robotics. 34(8), 1407–1426. https://doi.org/10.1002/rob.21721

Anagnostis

Benos

Tsaopoulos

Tagarakis

Tsolakis

Bochtis

(2021). Human activity recognition through recurrent neural networks for human–robot interaction in agriculture. Applied Sciences, 11(5), 1–21. https://doi.org/10.3390/app11052188

Bar-Shira

(2019). Ben-Gurion University of the Negev Faculty of Engineering Sciences. (In Thesis) Issue September.

Bechar

Vigneault

(2016). Agricultural robots for field operations: Concepts and components. Biosystems Engineering, 149(September), 94–111. https://doi.org/10.1016/j.biosystemseng.2016.06.014

Bechar

Vigneault

(2017). Agricultural robots for field operations. Part 2: Operations and systems. Biosystems Engineering, 153(January), 110–128. https://doi.org/10.1016/j.biosystemseng.2016.11.004

Berenstein

Edan

(2012). Human-robot cooperative precision spraying: Collaboration levels and optimization function. IFAC Proceedings Volumes, 45(22), 799–804. https://doi.org/10.3182/20120905-3-HR-2030.00084

Bergerman

Maeta

S. M.

Zhang

Freitas

G. M.

Hamner

Singh

Kantor

(2015). Robot farmers: Autonomous orchard vehicles help tree fruit production. IEEE Robotics and Automation Magazine, 22(1), 54–63. https://doi.org/10.1109/MRA.2014.2369292

Bisantz

A. M.

Burns

C. M.

(2016). Applications of cognitive work analysis. In Bisantz

A. M.

Burns

C. M.

(eds.), Applications of cognitive work analysis. CRC Press. https://doi.org/10.1201/9781420063059

Bisantz

A. M.

Roth

(2007). Analysis of cognitive work. Reviews of Human Factors and Ergonomics, 3(1), 1–43. https://doi.org/10.1518/155723408x299825

10.

Bisantz

A. M.

Roth

Brickman

Gosbee

L. L.

Hettinger

McKinney

(2003). Integrating cognitive analyses in a large-scale system design process. International Journal of Human-Computer Studies, 58(2), 177–206. https://doi.org/10.1016/S1071-5819(02)00130-1

11.

Brooks

F. A..

(1960). Operational sequence diagrams. IRE Transactions on Human Factors in Electronics, HFE-1(1), 33–34. https://doi.org/10.1109/THFE2.1960.4503264

12.

Cohen

Glasner

(2016). Date palm status and perspective in Israel. In Al-Khayri

J. M.

Jain

S. M.

Johnson

D. V.

(eds.), Date Palm Genetic Resources and Utilization (vol. 2). Springer. https://doi.org/10.1007/978-94-017-9707-8

13.

Cohen

Glasner

(2022). Agricultural robots for field operations: Concepts and components. Biosystems Engineering. In Zaid

Oihabi

(eds.), Mejhoul variety – the jewel of dates – origin, distribution and international markets (pp. 141–144). Khalifa International Award for Date Palm and Agricultural Innovation.

14.

Cuevas

H. M.

Fiore

S. M.

Caldwell

B. S.

Strater

(2007). Augmenting team cognition in human-automation teams performing in complex operational environments. Aviation Space & Environmental Medicine 78(5), B63–B70.

15.

Cullen

R. H.

Smarr

C. A.

Serrano-Baquero

McBride

S. E.

Beer

J. M.

Rogers

W. A.

(2012). The smooth (tractor) operator: Insights of knowledge engineering. Applied Ergonomics, 43(6), 1122–1130. https://doi.org/10.1016/j.apergo.2012.04.002

16.

De-An

Jidong

Wei

Ying

(2011). Design and control of an apple harvesting robot. Biosystems Engineering, 110(2), 112–122. https://doi.org/10.1016/j.biosystemseng.2011.07.005

17.

Effken

Loeb

Johnson

Reyna

(2001). Using cognitive work analysis to design clinical displays. Studies in Health Technology and Informatics, 84(1), 127–131. https://doi.org/10.3233/978-1-60750-928-8-127

18.

El-Assar

A. M.

Refaat

A. A.

(2013). The economics effect for some operating of fruit thinning and bunch covvering treatments on “Sewi cv.” date palm productivity in Siwa Oasis. Journal of Agricultural Economy and Social Sciences, 4(1), 65–81.

19.

Ernst

Roth

Militello

Sushereba

DiIulio

Wonderly

Scheff

Klein

Taylor

(2019). A strategy for determining optimal crewing in future vertical lift: Human-automation function allocation. Vertical flight society’s 75th annual forum & technology display. The Vertical Flight Society.

20.

Faostat (2020). http://www.fao.org/faostat/en/#data/QC

21.

Fei

Vougioukas

S. G.

(2021). Co-robotic harvest-aid platforms: Real-time control of picker lift heights to maximize harvesting efficiency. Computers and Electronics in Agriculture, 180, 105894. https://doi.org/10.1016/j.compag.2020.105894

22.

Feigh

K. M.

Pritchett

A. R.

(2014). Requirements for effective function allocation. Journal of Cognitive Engineering and Decision Making, 8(1), 23–32. https://doi.org/10.1177/1555343413490945

23.

Harris

Stanton

N. A.

Starr

(2015). Spot the difference: Operational event sequence diagrams as a formal method for work allocation in the development of single-pilot operations for commercial aircraft. Ergonomics, 58(11), 1773–1791. https://doi.org/10.1080/00140139.2015.1044574

24.

Hohimer

C. J.

Wang

Bhusal

Miller

Karkee

(2019). Design and field evaluation of a robotic apple harvesting system with a 3d-printed soft-robotic end-effector. Transactions of the ASABE, 62(2), 405–414. https://doi.org/10.13031/trans.12986

25.

Huddlestone

Sears

Harris

(2017). The use of operational event sequence diagrams and work domain analysis techniques for the specification of the crewing configuration of a single-pilot commercial aircraft. Cognition, Technology & Work, 19(2–3), 289–302. https://doi.org/10.1007/s10111-017-0423-5

26.

Ijtsma

L. M.

Pritchett

A. R.

Feigh

K. M.

(2019). Computational methodology for the allocation of work and interaction in human-robot teams. Journal of Cognitive Engineering and Decision Making, 13(4), 221–241. https://doi.org/10.1177/1555343419869484

27.

Kurke

M. I.

(1961). Operational sequence diagrams in system design. Human Factors: The Journal of the Human Factors and Ergonomics Society, 3(1), 66–73. https://doi.org/10.1177/001872086100300107

28.

Lind

(1999). Making sense of the abstraction hierarchy. Cognition, Technology & Work, 5(2), 67–81.

29.

Lintern

Waite

Talleur

A. D.

(1999). Functional interface design for the modern aircraft cockpit. The International Journal of Aviation Psychology, 9(3), 225–240. https://doi.org/10.1207/s15327108ijap0903_3

30.

Marinoudi

Sørensen

C. G.

Pearson

Bochtis

(2019). Robotics and labour in agriculture. A context consideration. Biosystems Engineering, 184, 111–121. https://doi.org/10.1016/j.biosystemseng.2019.06.013

31.

Moustafa

A. A.

(1998). Studies on fruit thinning of date palms. First International Conference on Date Palms, March. 354–363.

32.

Naikar

Moylan

Pearce

(2006). Analysing activity in complex systems with cognitive work analysis: Concepts, guidelines and case study for control task analysis. Theoretical Issues in Ergonomics Science, 7(4), 371–394. https://doi.org/10.1080/14639220500098821

33.

Nair

A. S.

Nof

S. Y.

Bechar

(2021). Emerging directions of precision agriculture and agricultural robotics. In Innovation in agricultural robotics for precision agriculture: A roadmap for integrating robots in precision agriculture (pp. 177–210). Springer. https://doi.org/10.1007/978-3-030-77036-5_8.

34.

Pritchett

A. R.

Kim

S. Y.

Feigh

K. M.

(2014). Modeling human-automation function allocation. Journal of Cognitive Engineering and Decision Making, 8(1), 33–51. https://doi.org/10.1177/1555343413490944

35.

Rasmussen

(1985). The role of hierarchical knowledge representation in decisionmaking and system management. IEEE Transactions on Systems, Man, and Cybernetics, 15(2), 234–243. https://doi.org/10.1109/TSMC.1985.6313353

36.

Reina

Milella

Rouveure

Nielsen

Worst

Blas

M. R.

(2016). Ambient awareness for agricultural robotic vehicles. Biosystems Engineering, 146, 114–132. https://doi.org/10.1016/j.biosystemseng.2015.12.010

37.

Revell

K. M.

Stanton

N. A.

Kelleher

(2016). Exploring changes in pilot behaviour during distributed crewing. In The Institute of Ergonomics and Human Factors Annual Conference. Daventry, UK.

38.

Salmon

Jenkins

Stanton

Walker

(2010). Hierarchical task analysis vs. cognitive work analysis: Comparison of theory, methodology and contribution to system design. Theoretical Issues in Ergonomics Science, 11(6), 504–531. https://doi.org/10.1080/14639220903165169

39.

Salzer

Bass

E. J.

(2020). Using function allocation methods to inform research roadmaps: A human-robot interaction case study for Medjool date thinning. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 64(1), 354–358. https://doi.org/10.1177/1071181320641080

40.

Salzer

Naor

Yonai

Yechezkely

Kfir

Zait

(2021). Time study for improving outputs and apple quality using integrated harvest platform conveyors. In Bochtis

Achillas

Banias

Lampridi

(eds.), Bio-economy and agri-production (pp. 221–233). Academic Press/Elsevier. https://doi.org/10.1016/C2018-0-05596-6

41.

Scholtz

J. C.

Antonishek

Young

J. D.

(2005). Implementation of a situation awareness assessment tool for evaluation of human-robot interfaces. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 35(4), 450–459. https://doi.org/10.1109/TSMCA.2005.850589

42.

Shapiro

Korkidi

Demri

Ben-Shahar

Riemer

Edan

(2006). Toward elevated agrobotics: Development of a scaled-down prototype for visually guided date palm tree sprayer. Journal of Field Robotics, 26(6–7), 572–590. https://doi.org/10.1002/rob.20299

43.

Shoshan

Bechar

Cohen

Sadowsky

Berman

(2021). Segmentation and motion parameter estimation for robotic Medjoul-date thinning. Precision Agriculture, 23, 0123456789. https://doi.org/10.1007/s11119-021-09847-2

44.

Shoshan

Berman

(2021). Medjool-date images. https://doi.org/10.17632/k7xk2nwgrh.2

45.

Silwal

Davidson

J. R.

Karkee

Zhang

Lewis

(2017). Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 34(6), 1140–1159. https://doi.org/10.1002/rob.21715

46.

Sorensen

L. J.

Stanton

N. A.

Banks

A. P.

(2011). Back to SA school: Contrasting three approaches to situation awareness in the cockpit. Theoretical Issues in Ergonomics Science, 12(6), 451–471. https://doi.org/10.1080/1463922X.2010.491874

47.

Stanton

N. A.

Brown

Revell

K. M. A.

Langdon

Bradley

Politis

Mouzakitis

Skrypchuk

Thompson

(2021). Validating operator event sequence diagrams: the case of an automated vehicle to human driver handovers. Human Factors and Ergonomics Methods 2020, 32(1), 1–13. https://doi.org/10.1002/hfm.20887

48.

Stanton

N. A.

Jenkins

D. P.

(2017). Application of cognitive work analysis to system analysis and design. In NevilleStanton

D. P. J. A.

Salmon

P. M.

Walker

G. H.

(eds.), Cognitive work analysis (pp. 3–72). CRC Press. https://doi.org/10.1201/9781315572536-2.

49.

Tkach

Bechar

Edan

(2011). Switching between collaboration levels in a human-robot target recognition system. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6), 955–967. https://doi.org/10.1109/TSMCC.2011.2119480

50.

Vasconez

J. P.

Kantor

G. A.

Auat Cheein

F. A.

(2019). Human-robot interaction in agriculture: A survey and current challenges. Biosystems Engineering, 179, 35–48. https://doi.org/10.1016/j.biosystemseng.2018.12.005

Integrating Function Allocation and Operational Event Sequence Diagrams to Support Human-Robot Coordination: Case Study of a Robotic Date Thinning System

Abstract

Keywords

Introduction

Date Cultivation: Domain Description

Method

Abstraction Hierarchy

Function Allocation

Operational Event Sequence Diagrams

Information and Interface Requirements

Results

Abstraction Hierarchy

Function Allocation

Operational Event Sequence Diagrams

Information and Interface Requirements

Discussion

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

Appendix

References