The effects of selected object features on a pick-and-place task: A human multimodal dataset

Abstract

We propose a dataset to study the influence of object-specific characteristics on human pick-and-place movements and compare the quality of the motion kinematics extracted by various sensors. This dataset is also suitable for promoting a broader discussion on general learning problems in the hand-object interaction domain, such as intention recognition or motion generation with applications in the Robotics field. The dataset consists of the recordings of 15 subjects performing 80 repetitions of a pick-and-place action under various experimental conditions, for a total of 1200 pick-and-places. The data has been collected thanks to a multimodal setup composed of multiple cameras, observing the actions from different perspectives, a motion capture system, and a wrist-worn inertial measurement unit. All the objects manipulated in the experiments are identical in shape, size, and appearance but differ in weight and liquid filling, which influences the carefulness required for their handling.

Keywords

pick-and-place manipulation carefulness hand-object interaction object weight multisensory dataset human kinematics

1. Introduction

The physical characteristics of objects can play a significant role in how humans handle them. It has been shown that the weight of an object directly relates to the speed adopted to lift and transport it, therefore influencing the kinematics of the movement (Bingham 1987; Hamilton et al. 2007). Similarly, when engaging with an item difficult to transport without damage, we approach it carefully and move it with particular attention, reducing the speed and prolonging the deceleration phase (Duarte et al. 2020; Lastrico et al. 2021).. Human-to-human communication heavily relies on implicit non-verbal signals for coordination and intention understanding. Therefore, how an object is handled conveys relevant information to the observers.

When dealing with robots, humans tend to attribute human-like abilities to them. For example, we expect a robot to focus its attention on where its cameras are looking or be immediately ready to hand over an object we need. When our expectations are disappointed, the whole interaction can be compromised (Breazeal 2003; Chaminade and Cheng 2009; Sandini et al. 2019). Furthermore, for robots to collaborate with humans, it is fundamental to understand what the partner is doing and when it is the right moment to act. In this context, it is necessary to study how humans communicate through implicit clues to enhance the interaction capabilities of robots (Pezzulo et al. 2013; Dragan et al. 2013).

Recently, there has been a growth of research interest in solutions to estimate the physical characteristics of objects when humans manipulate them. The main insight is that a correct estimate of object properties would allow a robot to interact with a human more appropriately, especially when physical interactions are involved, for example, a handover. Researchers explored the usage of deep neural networks to estimate containers capacity, dimension, and weight while observing human manipulations with RGB-D (Wang et al. 2022; Apicella et al. 2022) or even simple RGB cameras (Pang et al. 2021). However, a robot should also consider if an object requires particular care for being handled to avoid changing irreparably some of its properties. The state-of-the-art describes this problem as a binary classification, whereby the robot should distinguish whether or not the object manipulation requires carefulness. In this context, proposed solutions include template matching approaches relying on Gaussian Mixture Models (Duarte et al. 2020) or deep neural network classifiers (Lastrico et al. 2021). The definition of carefulness is not straightforward, since many factors may solicit cautious actions: from the physical context where the action takes place, to the properties of the object involved such as its fragility, precarious balance, or content about to spill (Lastrico et al. 2022). Given the difficulty in framing this feature, not many studies explicitly refer to it, although it is addressed, for example, in the handling of filled containers (Xompero et al. 2022; Mottaghi et al. 2017; Pang et al. 2021; Sanchez-Matilla et al. 2020; Yu et al. 2015). From the same perspective, we define carefulness as the modulation of arm motion that minimizes liquid spilling during the manipulation of containers. Therefore, in the following, we will investigate the concept of carefulness in scenarios where humans transport containers and adapt their motion to the presence or absence of liquid inside.

Since most of these studies rely on data-driven approaches, datasets must ground the learning process. The literature presents different examples of datasets for the study of object manipulations (Huang et al. 2016), most of the time focusing on the interaction between the humans and the objects in specific applications, for example, activities of daily living (Huang and Sun 2019), kitchen-related actions (Tenorth et al. 2009; Stein and Mckenna 2013; Nicora et al. 2020), or handovers (Carfì et al. 2019). A recently published dataset, the CORSMAL Container Manipulation by Xompero et al. (2022), collects actions such as pouring and handover initiations with containers with various shapes, materials, and content recorded with RGB-D cameras and microphones. However, the datasets currently available are more oriented to classical object recognition problems, proposing a high variability in the shapes and sizes of the objects examined, considering fewer sensors or a limited pool of participants; moreover, no one takes into consideration nor is aimed at modeling carefulness.

Therefore, the contribution of this paper is twofold. First, we introduce a novel dataset describing careful and non-careful manipulations and narrow the focus on human pick-and-place actions of transparent cups, with or without water filling and balanced weights. Limiting the variability of objects allows the effect of features on human motion to be studied in detail. Data are recorded with a synchronized multisensory setup, that is, motion capture system (MoCap), cameras, wrist-worn inertial measurement units (IMUs) and the robot point of observation, allowing for a complementary description of the scene from different perspectives. We then present an example of usage of the dataset by training Long Short-Term Memory Neural Networks, using the different sources of information provided by the dataset, to discriminate if carefulness was adopted during the manipulation and if the cup transported was light or heavy. The novelty of the approach lies in using the modulations of human kinematics to foster the inference process and understand the objects’ latent features, completely overlooking their appearance. Moreover, the controlled design of the setup allows for deeply studying human strategies during the pick-and-place of objects with specific properties. By identifying the general rules to act on a set of objects, independently from their use, shape, or material, they can be used to design a robot’s behavior appropriately.

The article is organized as follows. Section 2 describes, in detail, the study design and the data acquisition process. In Section 3 we present a description of the dataset and its organization. Section 4 describes the code provided to inspect and visualize the dataset. Finally, Section 5 presents an example of how to use the dataset. Conclusions follow.

2. Experimental setup

This section describes the study design and its main technical characteristics. Liguria Regional Ethical Committee approved the research protocol for this study (protocol 396REG2016 of July 25th, 2019), and all participants provided written informed consent to publish the collected data.

2.1. Study design

An object may require careful manipulation for different reasons. A glass full of water requires carefulness to avoid spilling, while a ceramic vase requires carefulness to avoid breaking it. Likewise, the different reasons inducing carefulness also influence its physical manifestation. The careful manipulation of a glass of water would manifest in slow motions with constant orientation. Instead, carefully manipulating a ceramic vase would maximize the distance to nearby objects to avoid collisions.

Given the limitations of previous research in the field, we decided to narrow the study limiting the notion of carefulness to the one induced by the need to move a container filled with a liquid while avoiding spills. In particular, the human actions recorded in the dataset consist of reaching, transportation, and departing movements involving four possible objects. In order to allow for simple reproduction of the experiments, we chose plastic glasses, which are easy to manipulate and of everyday use. The glasses, identical in shape and material, differed in their contents. In order to induce careful behavior in the recorded actions, we filled two of the glasses with water to the brim so that they required a high level of carefulness. Two different weights are considered: light (W1: 167 g) and heavy (W2: 667 g). Such values were determined by the fact that we wanted the light and heavy objects to be consistently different (500 g) while balancing the presence or the absence of water in containers with the same volume. The desired weights were obtained by adding screws and coins inside the glasses, until reaching W1 or W2, balancing the presence of water. In this way, we defined four classes of actions, depending on the properties of the manipulated object, namely, light and not careful (W1-NC), light and careful (W1-C), heavy and not careful (W2-NC), heavy and careful (W2-C).

The sequence of performed actions is the same for every participant. It is designed to alternate the manipulation of the four categories of objects together with the direction of the movements.

At the beginning of the experiment, the volunteer sits at a table, with their hands resting on it. On the table, covered with a black cloth, there is one shelf at each end, a scale right in front of the subject, and a keyboard on the left side, see Figures 1 and 2. Each shelf has four possible positions, denoted by a letter marked on the frontal edge of the shelf, where the glasses could be positioned, that is, two on the bottom level and two on the upper one, see Figure 1(d). The shelves measure 36 × 23 cm, the top level is 36 cm above the bottom one and there is a border of 6 cm, delimiting each shelf and constituting an obstacle. The two positions on each shelf level are indicatively 18 cm apart. A blue cross on the table marks the resting position that the participants’ right hand should reach after each movement, see Figure 1(a). The distance between the starting position and the shelves is indicatively 50 cm. The humanoid robot iCub (Metta et al. 2008) is placed in front of the table and passively records the scene with its left camera.

Figure 1.

The three points of view of the experiment: the resting position as seen by the iCub robot (a), a transportation movement towards the right shelf (b), and the positioning of a glass on the scale (c). In (d) the labels identify the 8 positions on the shelves. (a) Robot view. (b) Lateral view. (c) Back view. (d) Frontal view.

Figure 2.

Top view of the experimental setup. In dark gray, the cameras from the Optotrak motion capture system used to detect the active markers, while in black, the two high resolution cameras.

As instructed by a synthetic voice, the participants perform a series of reaching, transportation, and departing movements of the four glasses. The volunteers interacted with the items with their right hand and received instructions on the next movement to perform by pressing a key on the keyboard with their left hand. The experiment is set up as summarized in Figure 3:

(1) The experiment starts with the volunteer in the resting position and with the four objects distributed on the shelves.

(2) When a key of the keyboard is pressed, a synthetic voice indicates the position on the shelf of the object to be transported. The position is referred to using the corresponding letter.

(3) The volunteer reaches for the glass and grasps it in the specified location (reaching phase), as in Figure 1(b).

(4) In the transportation phase, the volunteer moves the glass from the shelf (shelf initial position) to the scale.

(5) The volunteer releases the glass and returns with the dominant hand to the resting position.

(6) The volunteer presses the key a second time, and the synthetic voice suggests where the glass should be placed on the other shelf. The shelf spot chosen this time is vacant.

(7) The volunteer reaches for the scale and takes the glass, see Figure 1(c).

(8) The volunteer moves the glass from the scale to its final location on the shelf, performing a transportation action.

(9) The volunteer places the glass down and returns to the resting position.

Figure 3.

The structure of each trial falls within two possibilities, whereby the main action is the glass transportation: in (a) are shown the steps for taking a glass from the shelves and placing it on the scale, while in (b) those for putting back the glass from the scale to the shelf. (a) Transport from shelf to scale. (b) Transport from scale to shelf.

The order in which the volunteers performed the experiment is detailed in Table 1. The first sixteen trials were used as a practice before the main experiment started. In the main experiment, each volunteer performed 64 reaching movements, 64 transportation movements, and 64 departing movements to the resting position. After the 16th and the 48th trial, the objects’ position is changed by an experimenter to maintain the properties of the manipulated objects and the initial position of the glasses equally balanced. To prevent fatigue and gesture automatism, at the end of each sequence of 16 pick-and-place actions, participants could rest as much as they wanted, both physically and mentally.

Table 1.

Movements sequence performed by each participant. The first column refers to the ID used to identify each trial. The second and third columns specify the characteristics of the object handled in the trial, which could be light (W1) or heavy (W2) and require carefulness (c) or not (NC). The last two columns refer to object positions: they indicate whether the glass to grab was placed on the shelf or on the scale, and the shelf position where the glass needed to be taken from or put down.

	Trial ID	Weight	Care	Object initial position	Shelf position
Practice	1	W1	NC	Shelf	Z
	2	W1	NC	Scale	K
	3	W2	NC	Shelf	H
	4	W2	NC	Scale	U
	5	W1	C	Shelf	S
	6	W1	C	Scale	M
	7	W2	C	Shelf	C
	8	W2	C	Scale	T
	9	W1	NC	Shelf	K
	10	W1	NC	Scale	Z
	11	W2	NC	Shelf	U
	12	W2	NC	Scale	H
	13	W1	C	Shelf	M
	14	W1	C	Scale	S
	15	W2	C	Shelf	T
	16	W2	C	Scale	C
1 ^st Session	17, 33	W1	NC	Shelf	H
	18, 34	W1	NC	Scale	U
	19, 35	W2	NC	Shelf	Z
	20, 36	W2	NC	Scale	K
	21, 37	W1	C	Shelf	M
	22, 38	W1	C	Scale	S
	23, 39	W2	C	Shelf	T
	24, 40	W2	C	Scale	C
	25, 41	W1	NC	Shelf	U
	26, 42	W1	NC	Scale	H
	27, 43	W2	NC	Shelf	K
	28, 44	W2	NC	Scale	Z
	29, 45	W1	C	Shelf	S
	30, 46	W1	C	Scale	M
	31, 47	W2	C	Shelf	C
	32, 48	W2	C	Scale	T
2 ^nd Session	49, 65	W1	NC	Shelf	S
	50, 66	W1	NC	Scale	M
	51, 67	W2	NC	Shelf	C
	52, 68	W2	NC	Scale	T
	53, 69	W1	C	Shelf	U
	54, 70	W1	C	Scale	H
	55, 71	W2	C	Shelf	K
	56, 72	W2	C	Scale	Z
	57, 73	W1	NC	Shelf	M
	58, 74	W1	NC	Scale	S
	59, 75	W2	NC	Shelf	T
	60, 76	W2	NC	Scale	C
	61, 77	W1	C	Shelf	H
	62, 78	W1	C	Scale	U
	63, 79	W2	C	Shelf	Z
	64, 80	W2	C	Scale	K

2.2. Data acquisition

The data collection process involved 15 healthy right-handed participants (8 males, 7 females, age: 28.6 ± 3.9). The participants are part of our research organizations, but none of them is directly involved in this research. Each subject performed 80 trials, ensuring 300 interactions for each of the 4 objects. Such participants’ numerosity is generally higher than that of other object manipulation datasets in the literature, as highlighted by the review of Huang et al. (2016) and, as recent examples, the works by Kratzer et al. (2021); Xompero et al. (2022) and Fan et al. (2023).

The framework adopted during the experiment was designed to ensure a high degree of automation in the acquisition phase. Most sensors were directly interfaced with the YARP middleware (Metta et al. 2006), allowing timestamp synchronization, whereas the wrist-worn IMU was using a Robot Operating System (ROS)-YARP interface.

The data have been segmented to separate each action presented in Figure 3. The MoCap and the IMU segmentation have been performed automatically, exploiting the participants’ pressures on the key. Instead, the camera images were organized offline into separate folders according to the saved timestamps. For each participant, we saved a log file containing information on the experiment. These log files contain the YARP timestamp of each instruction communicated by the synthetic voice and the time for each key’s pressures.

2.2.1. Motion capture system

As a motion capture system (MoCap), we used the Optotrak Certus^®, NDI, with active infrared markers. In total, we recorded the signal from 15 markers. As pictured in Figure 4, the five markers on the hand were placed, respectively, on the metacarpophalangeal joints of the index and the little finger, on the diaphysis of the third metacarpal, and on the smartwatch in correspondence to the radial and ulnar styloid. Additionally, two markers were positioned on the watch strap, one per side, to better characterize wrist movements. Even though the main focus of the recording was to acquire hand and wrist motions, we decided to position a few markers on the participants’ arm and forearm. We used two rigid cardboards where we put four markers each. See Figure 4 for reference. The frequency of the acquisition is 100 Hz. For every frame, the three-dimensional coordinates of every marker (in millimeters) were saved into the file associated with the trial. Moreover, the timestamp at the beginning and the end of each trial was saved. This was used to retrieve the timestamp corresponding to each frame by applying a linear interpolation.

Figure 4.

Marker positions on the participant’s right arm and hand. The smartwatch equipped with inertial sensors is visible on the wrist.

2.2.2. Inertial sensors

On the right wrist of the volunteers, we mounted an LG G Watch R smartwatch equipped with a 6-axis IMU. The sampling rate was 71 Hz. As for the MoCap data, a separate file was created for each trial, whenever the key on the keyboard was pressed at the end of the departing movement. The file was saved in json format containing the ROS and YARP timestamps at each sample, the internal Android timestamp, and the three components of the linear acceleration in m/s² and those of the angular velocity in rad/s. The data were published by the Android app on ROS for the smartwatch acquisition and then saved. Through the ROS-YARP interface, the corresponding YARP timestamp was sent to ROS at each key’s pressure and written on the json file.

2.2.3. Cameras

Two cameras with a resolution of 1920 × 1080 pixels were positioned in the room where the experiments took place. The former was placed at the back of the participant’s chair and recorded the scene from an overhead viewpoint, remaining elevated by 130 cm with respect to the table. The latter was on the left side of the participants, with an oblique point of view, 65 cm higher than the table with a distance from the hand starting position of circa 140 cm. The reader is referred to the scheme in Figure 2 for reference. The frame rate was set to 30 Hz. The last sensor used in the experiment was iCub’s left camera. The robot was located opposite the table, in front of the volunteer, with a complete perspective of the table and the shelves. The camera’s resolution was 320 × 240 pixels, and the frame rate was 22 Hz. Even though such resolution is particularly low, this third camera offers a complementary point of view to the other two, which is relevant for a possible deployment of the data acquired in a human-robot interaction context. However, we suggest future researchers interested in acquiring a similar setup, to use a standard webcam for the frontal view in addition to the robot one, to grant the best quality view from all perspectives.

As previously mentioned, the images acquired with the cameras were not automatically segmented during the acquisition. Segmentation occurred afterward using the YARP timestamps, which were saved for each image from each camera in a log file. Indeed, each saved camera frame was associated with a YARP timestamp, making it possible to relate the acquired images with the events triggered by the key’s pressure.

3. Data records

The dataset is available on Kaggle,¹ while the software utilities can be found on GitHub.² While we used MATLAB as a reference software to create the utilities functions, we chose to make sensory data available in a non-proprietary format (log, csv, json, or jpg, depending on the sensor). A comprehensive table with a summary of performed movements, with the details about their direction and the properties of the object involved is available in the utilities, also with a machine readable structure. Table 1 summarizes the characteristics of each trial. The same sequence was performed by each participant.

In the repository, the data are organized into separate folders on a sensor basis. Inertial and MoCap recordings can be found in folders data/inertial and data/mocap, respectively. While cameras recording are divided in three folders according to the following naming:

• data/cam_0: low resolution frontal images from the iCub left camera,

• data/cam_1: high resolution lateral camera,

• data/cam_2: high resolution images from behind.

The main folder also includes one folder data/log, which contains the experiments log files. They report the YARP timestamp corresponding to the key’s pressures at the end of each trial, together with the information relative to the transport movement: the instruction given by the synthetic voice is reported as “sending speech: Prendi/Metti in {Position letter}.” The verbs “Prendi” and “Metti” mean, respectively, “Take from” and “Put in” in Italian, and were used to tell participants where to take the glass from or where to put it back on the shelf, using the letter corresponding to the position, as in Figure 1(d).

Each one of the described folders is organized into sub-folders named from P001 to P015, containing the files associated to each participant. In the case of MoCap and inertial sensors, the files are sequentially named from trial 1 to 80.

Instead, the camera folders are divided, for each subject, into 80 sub-folders (one for each trial), for example, the folder data/cam_1/P001/P001_Trial_001 contains the images from the lateral high resolution camera for the first trial of the first subject. The same sub-folders also contain a data.txt file reporting the correspondence between the YARP timestamps and images names for the specific trial.

The MoCap raw data are organized in csv files, with 62 columns and as many rows as the samples in the trial. The first column contains a progressive ID, the second one the YARP timestamp for each sample. As previously mentioned, this timestamp was computed after the acquisition by linearly interpolating the timestamps indicating the start and the end of the trial. The remaining columns represent triplets of three-dimensional trajectories (x, y, z coordinates) followed by an additional column for each marker containing 0, if the marker was visible in that particular frame, or 1 if not. The order in which the markers appear in the files follows the numbering introduced in Figure 4, therefore the first triplet being the coordinates of the marker on the metacarpophalangeal joint of the index, the second triplet the marker on the joint of the little finger, and so on.

Regarding the inertial sensor raw files, they are saved in json format. For each sample, the available information is the ROS timestamp, the YARP timestamp, the Android one, and the three components of linear acceleration and angular velocity.

3.1. Data assessment

A first evaluation on the quality of the provided data is related to their frequency. As already mentioned, the acquisition frequency depends on the specific sensor, and it is 100 Hz for the MoCap, 71 Hz for the inertial sensor, 30 Hz for the cameras, and 22 Hz for the robot camera. Figure 5 represents the frequencies for the sensors considering the whole experiment for all the participants. Even though some outliers are present, down sampling the data to the lowest frame rate (22 Hz) allows for automatically cleaning them and keeping a frequency well above the range of human motion, that is, generally below 10 Hz (Khusainov et al. 2013) and lies in the interval [0.3, 4.5] Hz for hand motion (Xiong and Quek 2006).

Figure 5.

Box plots of each sensor recording frequency. The red lines represent the medians and the blue rectangles limit the 25th and 75th percentiles.

An in-depth kinematics analysis can be carried out to study how the properties of the different objects affect the movement of the arm and condition how the transport action is completed. Our dataset allows not only to calculate different kinematic parameters, but also to compare the information which can be retrieved from the different sensors. In Figure 6 can be found an example, which compares kinematics parameters extracted from the synchronized MoCap and inertial sensors during the transportation of the four glasses. Figure 6(a) illustrates, as a proof of concept, how the hand mean velocity during the transport phases evolves depending on the number of participants considered. As the sample becomes larger, the measure stabilizes, and from the point when the dataset has 11 subjects, the velocity is almost completely constant. This analysis hints that the sample size of 15 participants (for a total of 1200 trials) is suitable for a reliable description of the proposed pick-and-place scenario; indeed, the acquired data have an internal coherency and well represents the diversity and the natural variance of human motion in this context. In detail, in Figure 6(b) are shown the median, the 25th and the 75th percentiles of the hand velocity calculated deriving the 3D position of one of the markers placed on the volunteer’s hand (markers 1 to 4, see Figure 4 for reference). According to the Kruskal–Wallis test for non-normal distributions, we found a significant difference in the velocity adopted for transporting the glasses between those filled with water (W1-C and W2-C) and those empty, with simply a weight difference (W1-NC and W2-NC). Even though a trend is visible, no significant difference was found concerning the weight. The same results emerge in Figure 6(c) as well, where the wrist’s mean angular velocities are recorded with the inertial sensor in the smartwatch. Again, a significant difference in the magnitude of the angular velocity appears between the Careful and Not Careful transport motions.

Figure 6.

Kinematics parameters during the transport movements for the four different classes, retrieved for motion capture data (6b) and inertial sensor data (6c). Figure 6(a) shows as a proof of concept how the mean velocity measured with the moCap stabilizes when considering a properly size pool of participants. (a) Mean velocity during the transport motion considering an increasing subset of participants. When considering the whole dataset, the values are the same as in Figure 6(b) below. The colored dots represent the median, while the bars are the standard deviation. (b) Mean velocity of the participants hand while manipulating the four different glasses, acquired through active infrared markers. The black lines represent the medians, the colored rectangles limit the 25th and 75th percentiles, and * indicates a significant difference according to the Kruskal–Wallis test. (c) Mean angular velocity during the glasses transportation as recorded by the smartwatch on the participants right wrist. Same graphical conventions as in Figure 6(b) above.

Finally, Figure 7 provides an insightful qualitative overview of the hand velocity profile acquired with the MoCap system during the reaching towards the cups (first peak) and the transport phase (second peak). To create such a global representation, the mean of the hand velocities of all the trials, separated into the four classes of motion, was computed for every time instant, together with its standard deviation. It can be noted how not only the maximum velocity decreases when transporting the full cups, but also the peak is anticipated, especially for the full, heavy container. Participants tended to be especially cautious in the final phase of the movement, gently leaning the glass so as not to spill the content.

Figure 7.

Hand mean velocities and standard deviation, in transparency, associated with the reaching (first peak) and transportation movements (second peak) of the four different glasses.

4. Code availability

The Github repository² includes a number of MATLAB scripts allowing users to load and process the data. Further details on how to use the available functions are contained in each of them.

• loadMocap and loadInertial allow users to load and save, respectively, the motion capture and the inertial data coming from the smartwatch in an easy-to-use data structure in the form of a 15 × 80 cell array (number of subjects × number of trials).

• loadTimestampsCameras saves in a data structure the YARP timestamp for each one of the three cameras during the experiment.

• visualizeHandTrajectory produces a 3D plot with the visible markers positioned on the hand for specified trial and subject combo.

• createVideo creates a video from the images, specifying the desired participant, trial and camera (one of the two high resolution cameras or the robot camera). It also saves the YARP timestamps corresponding to the considered frames.

• animatedPlot renders the video previously created with createVideo together with the trajectories of the markers on the hand during the specified trial and the three components of the acceleration recorded with the smartwatch. The 3D trajectories, the acceleration and the video are reproduced simultaneously; the number of the markers to visualize can be easily modified.

5. Example of usage

In this section, we provide insights into the dataset usage. In particular, we compare the performances of the same classifier, a Long Short-Term Memory Neural Network (LSTM-NN), applied to the data sources present in the dataset, that is, MoCap, Robot Camera, and IMU. In this example, we focus on transportation motions because of the direct influence of the objects’ physical characteristics given by the contact presence. After being segmented, the transportation phase of each trial was appropriately labeled according to the features of the cup involved; thus, we trained binary classifiers to discriminate its weight or the presence of carefulness in the motion. This study’s objective is to provide insight on which sensing approach better fits the problem and if fusing information from different sensing modalities could help.

5.1. Data preprocessing

Before using them to train and test the LSTM-NNs, it is necessary to preprocess the data. This preprocessing involves segmentation, feature extraction, data filtering, resampling, data normalization, and padding.

As shown in Figure 3, each trial contains the whole pick-and-place action divided into reaching, transportation, and departing. At the beginning and the end of the transportation, the subject hand stops to grasp and release the object. This characteristic can be leveraged to isolate the transportation motion. We did it by computing the norm of the hand velocity, using the MoCap data, and applying a thresholding mechanism to identify the timestamps related to the start and end of the transportation. We used the resulting timestamps to segment the IMU and the robot camera data.

Given the different nature of the three sources of data selected, to adopt the same classification pipeline, it is necessary to extract consistent features. From the robot camera images, we computed the Optical Flow (OF) (Vignolo et al. 2017), and we used it to extract the components of the motion velocity on the image plane. From these data, we then calculated the norm velocity, the angular velocity, the curvature, and the radius of curvature (Lastrico et al. 2021). We selected these four features since, in the past, they have been successfully adopted to characterize biological motions (Vignolo et al. 2017) and discriminate between careful and non-careful transportations (Lastrico et al. 2021). Instead, we selected six features both for the MoCap and the IMU. For the MoCap, we computed the hand triaxial linear acceleration and velocity. For the IMU, we used the raw data, that is, linear acceleration and angular velocity.

We suggest filtering the data for noise reduction as part of the data preprocessing. In this case, we applied to every temporal sequence a first-order Butterworth filter with a threshold frequency equal to the original sampling rate of the sensor (i.e., 71 Hz for the IMU, 100 Hz for the MoCap, and 22 Hz for the camera). To simplify the comparison and easily combine information from the different sources, we resampled the IMU and MoCap data to match the camera sampling frequency. This choice also finds support in previous research suggesting that 20 Hz is an ideal sampling frequency for the perception of human daily activities (Antonsson and Mann 1985). To perform the resampling, we interpolated the data and used the camera timestamp to extrapolate the new data.

The resulting sequences are then scaled using min-max normalization to decrease the difference in scale between the features. For the IMU, we selected as maximum and minimum values the full-scale range of the sensors, that is, ± 2g for accelerations and ±8.73 rad/s (±500 deg/s) for the angular velocities. Finally, since all the trials have a different temporal length, we padded the data with zeros to be able to use batch training. In particular, we used pre-padding since it is considered more robust to the noise introduced by the zeros (Dwarampudi and Reddy 2019).

5.2. Model training and validation

As mentioned before, our experiment aimed to compare the results of different classifiers for distinguishing, separately, whether carefulness was adopted during the transportation and if the object involved was heavy or light. The evaluation focuses on four classifiers that differ from each other for the used data source. The study comprehends one classifier for each sensor (i.e., camera, IMU, and MoCap) and a classifier using both IMU and camera data. We explore the combination of IMU and camera data to determine if an autonomous robot could leverage it for more reliable perception. We chose a simple LSTM model followed by fully connected layers for each of the four models. The networks were implemented in Python using the sequential layers provided by Keras.³ The LSTM layer has 64 hidden units and an input shape equal to [sequence_length × n_features], where the sequence length is fixed to 134 samples (maximum sequence length after resampling), and the number of features varies according to the data source (i.e., 6 for MoCap and IMU, 4 for the camera and 10 for IMU plus camera). The next layer is fully connected, with 32 neurons, and it is preceded and followed by dropout layers with a value of 0.5. The output layer is another fully connected one with two output neurons corresponding to the two classes: careful and not careful. Given the double output, we chose a softmax function for the activation and the categorical cross-entropy to evaluate the loss. An L1-L2 kernel regularization was added to the last layer to prevent the model from overfitting with L1 = 0.001 and L2 = 0.001 as parameter values. The chosen optimization algorithm was AdamOptimizer, with a learning rate of 0.0002 and batch size of 16.

For each of the four models, we carried out the training and testing phases by adopting the Cross-Validation with a Leave-One-Out approach to test the ability of the model to generalize over different participants. Therefore, to split the 1200 sequences (15 participants × 80 sequences) into training, validation and test sets, we adopted the following procedure. One at a time, the data of each participant were used as a test set, and the remaining 14 were further divided, 80% for the training and 20% for the validation. The validation set has been picked randomly from the 14 volunteers. The models have been trained for 100 epochs using an early stopping on the validation loss with patience set to 5 epochs to avoid overfitting.

5.3. Results

The classification results of the four models are reported in Figure 8 using boxplots showing the median, the average, and the distribution of the values for each data source. Regarding the carefulness, the overall performances of the models are comparable, that is, 91.3% for the IMU, 91.4% for the camera, 91.6% for the MoCap, and 91.1% for IMU plus camera (see Figure 8(a)). Therefore, considering this evidence, the 4 data sources are almost equivalent for classification purposes. On the other hand, if we exclude the outliers, the minimum values displayed in the chart appear to reflect some differences between the models. The lowest accuracy achieved while using camera and MoCap data is around 84%, for the IMU reaches approximately 82%, and for the combination of IMU and camera achieves 75%. This result suggests that combining IMU and camera data may not be particularly effective for estimating motion carefulness. However, since all the single classifiers reached high performances, it is possible to imagine an autonomous system combining the results of different classifiers to obtain more stable and reliable inferences.

Figure 8.

The results of the four classifiers trained with different data sources in terms of accuracy expressed as boxplots for carefulness (8a) and weight (8b) discrimination. The yellow line represents the median, while the green arrow indicates the average. (a) Carefulness classification. (b) Weight classification.

With the same approach, the accuracy of the weight classification is instead not as satisfying as the one achieved for carefulness. The mean accuracy values, as reported in Figure 8(b), are 55% for the IMU, 56% for the camera, 54% for the MoCap, and 59% for IMU plus camera. This result is particularly interesting, as the inference process for the weight is not as simple as expected. Our hypothesis is that the greatest challenge for the volunteers during the experiment was to safely handle the filled glasses and that the difference in weight between the objects did not have an impact as strong as the water filling on the kinematics. This observation is confirmed by Figures 6 and 7, where the significant difference in the considered kinematics features emerged only between full and empty cups. Objects can have multiple concomitant features, which do not always have the same effect on kinematics as their interaction may lead to the attenuation of their effect as compared to when considered individually. Our dataset, by attentively balancing the combinations of water filling and weight along multiple directions, can be particularly useful to deepen the understanding of human motor strategies in this context.

5.4. Discussion

Based on the results presented in the previous section, it appears to be inconsequential which sensing modality we adopt, regardless of the object feature under study. However, we should make a few observations to provide a more comprehensive view of the matter. Firstly, we should note that, in our approach, we processed RGB images with the optical flow to extract features that describe only human motion. We proceeded in this way to isolate human kinematics, but this procedure reduces the richness of information of the sensing modality. In fact, we would expect a classifier trained on raw images to achieve better accuracies since it can leverage the visual features of objects as well. Secondly, the presented results have been achieved in a specific experimental setting designed to be as fair as possible for each sensing modality. Therefore, future studies could benefit more from specific sensing modalities depending on the constraints and characteristics of the application and human actions under consideration. For example, in a scenario where humans interact with crowded environments or use different motor actions, the frequent occlusions with other objects could impair visual sensing. On the other hand, wearable sensors such as IMUs are not affected by occlusions, but it is not always possible or convenient to sensorize humans.

6. Conclusions

This article provides a multimodal dataset of human object pick-and-place tasks under different experimental conditions. In particular, the dataset describes the effect of object weight and associated carefulness on human motions during a pick-and-place. The dataset is collected in a controlled environment with 15 subjects performing 80 different pick-and-place actions. The dataset contains multiple camera views, IMU, and MoCap data providing the possibility of integrating and comparing the diverse sensing modalities. The data collection was prompted by the need to create reliable models of human strategies in the context of object manipulation, which could be used by robots to understand the scene and facilitate interaction. In this article, as an example of usage, we propose a simple comparison of the performances of classifiers using different sensing modalities to distinguish between careful and non-careful transportations or between heavy and light cups.

The motions which constitute the dataset are also strictly controlled concerning their direction (left, right, up, down, adduction, and receding) and the different phases of the actions can be easily split, distinguishing between the reach-to-grasp, the transport, and the departing motions (see schema in Figure 3). This makes the dataset particularly suitable for studying human intention recognition; understanding what is going to be grasped, and where, can lead to significant improvements in the HRI experience, allowing robots to anticipate our goals and adapt accordingly. This kind of studies can also be conducted by comparing the different sensors available, so as to build a framework that can be adapted to different contexts: for instance, by relying more on inertial data in case of obstructions to the cameras.

Other applications to Robotics fall under the theme of motion generation. The presented dataset, however, is not aimed at classical robot learning from human demonstration; indeed, useful sensors for such use, such as grip and force sensors, are not included. Still, the acquired human velocity profiles have been used to build on top of standard robot trajectories (pick-and-place and collaborative handovers) by adding a communicative layer to the motions. Generative Adversarial Networks were trained on human examples to produce velocity profiles falling within the desired careful/not careful attitude and such time-series used to control the end-effector trajectories. Such an approach has been successfully deployed on humanoids and robot manipulators producing gestures adapted to cups’ content and improving the interaction efficiency (see Garello et al. (2021); Lastrico et al. (2023) for reference).

This dataset represents a contribution to the study of how specific object characteristics influence human motion. In future works, it could be interesting to expand the dataset to consider different levels of carefulness and study which factors (such as context, value, fragility, potential danger, and so on) impact human motion going beyond the spillability of the water content.

Footnotes

Acknowledgments

AS is supported by a Starting Grant from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program. GA No 804388, wHiSPER. This work is partially supported by the CHIST-ERA (2014-2020) project InDex, and received funding from the Italian Ministry of Education and Research (MIUR). This research is partially supported by the Italian government under the National Recovery and Resilience Plan (NRRP), Mission 4, Component 2 Investment 1.5, funded by the European Union NextGenerationEU and awarded by the Italian Ministry of University and Research.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by H2020 European Research Council (wHiSPER, No. 804388), CHIST-ERA (InDex (2014-2020)) and European Union (RAISE, CUP J33C22001220001).

ORCID iD

Alessandro Carfi

Notes

References

Antonsson

Mann

(1985) The frequency content of gait. Journal of Biomechanics 18(1): 39–47.

Apicella

Slavic

Ragusa

, et al. (2022) Container localisation and mass estimation with an RGB-D camera. In: The 32nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Singapore, 9152–9155.

Bingham

(1987) Kinematic form and scaling: further investigations on the visual perception of lifted weight. Journal of Experimental Psychology: Human Perception and Performance 13(2): 155–177.

Breazeal

(2003) Toward sociable robots. Robotics and Autonomous Systems 42(3): 167–175. Socially Interactive Robots.

Carfì

Foglino

Bruno

, et al. (2019) A multi-sensor dataset of human-human handover. Data in Brief 22: 109–117.

Chaminade

Cheng

(2009) Social cognitive neuroscience and humanoid robotics. Journal of Physiology Paris 103(3): 286–295. Neurorobotics.

Dragan

Lee

KCT

Srinivasa

(2013) Legibility and predictability of robot motion. In: Proceedings of the 8th ACM/IEEE International Conference on Human-Robot Interaction. Tokyo, Japan, 301–308.

Duarte

Chatzilygeroudis

Santos-Victor

, et al. (2020) From human action understanding to robot action execution: how the physical properties of handled objects modulate non-verbal cues. In: 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 1–6.

Dwarampudi

Reddy

NVS

(2019) Effects of Padding on LSTMS and CNNS, 07288. ArXiv abs/1903.

10.

Fan

Taheri

Tzionas

, et al. (2023) ARCTIC: a dataset for dexterous bimanual hand-object manipulation. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

11.

Garello

Lastrico

Rea

, et al. (2021) Property-aware robot object manipulation: a generative approach. In: 2021 IEEE International Conference on Development and Learning (ICDL), 1–7.

12.

Hamilton

Joyce

Flanagan

, et al. (2007) Kinematic cues in perceptual weight judgment and their origins in box lifting. Psychological Research 71: 13–21.

13.

Huang

Sun

(2019) A dataset of daily interactive manipulation. The International Journal of Robotics Research 38(8): 879–886.

14.

Huang

Bianchi

Liarokapis

, et al. (2016) Recent data sets on object manipulation: a survey. Big Data 4(4): 197–216.

15.

Khusainov

Azzi

Achumba

, et al. (2013) Real-time human ambulation, activity, and physiological monitoring: taxonomy of issues, techniques, applications, challenges and limitations. Sensors 13(10): 12852–12902.

16.

Kratzer

Bihlmaier

Midlagajni

, et al. (2021) Mogaze: a dataset of full-body motions that includes workspace geometry and eye-gaze. IEEE Robotics and Automation Letters 6(2): 367–373.

17.

Lastrico

Carfì

Vignolo

, et al. (2021) Careful with that! observation of human movements to estimate objects properties. In: Proceedings of the 13th International Workshop of Human-Friendly Robotics (HFR). Innsbruck, Austria: Springer International Publishing, 127–141.

18.

Lastrico

Garello

Rea

, et al. (2022) Robots with different embodiments can express and influence carefulness in object manipulation. In: 2022 IEEE International Conference on Development and Learning (ICDL), 280–286.

19.

Lastrico

Ferreira Duarte

Carfì

, et al. (2023) Expressing and inferring action carefulness in human-to-robot handovers. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Accepted for publication.

20.

Metta

Fitzpatrick

Natale

(2006) Yarp: yet another robot platform. International Journal of Advanced Robotic Systems 3: 43–48.

21.

Metta

Sandini

Vernon

, et al. (2008) The icub humanoid robot: an open platform for research in embodied cognition. In: Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems (PerMIS). Gaithersburg, Maryland, USA, 50–56.

22.

Mottaghi

Schenck

Fox

, et al. (2017) See the glass half full: reasoning about liquid containers, their volume and content. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 1889–1898.

23.

Nicora

Goyal

Noceti

, et al. (2020) The MoCA dataset, kinematic and multi-view visual streams of fine-grained cooking actions. Scientific Data 7(1): 432.

24.

Pang

Xompero

, et al. (2021) Towards safe human-to-robot handovers of unknown containers. In: 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN). Virtual, 51–58.

25.

Pezzulo

Donnarumma

Dindo

(2013) Human sensorimotor communication: a theory of signaling in online social interactions. PLoS One 8: 1–11.

26.

Sanchez-Matilla

Chatzilygeroudis

Modas

, et al. (2020) Benchmark for human-to-robot handovers of unseen containers with unknown filling. IEEE Robotics and Automation Letters 5(2): 1642–1649.

27.

Sandini

Sciutti

Rea

(2019) Movement-based communication for humanoid-human interaction. In: Humanoid Robotics: A Reference. Dordrecht: Springer Netherlands, 2169–2197.

28.

Stein

Mckenna

(2013) Combining embedded accelerometers with computer vision for recognizing food preparation activities. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 729–738.

29.

Tenorth

Bandouch

Beetz

(2009) The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: 2009 IEEE 12th International Conference on Computer Vision Workshops. ICCV Workshops, 1089–1096.

30.

Vignolo

Noceti

Rea

, et al. (2017) Detecting biological motion for human–robot interaction: a link between perception and action. Frontiers in Robotics and AI 4.

31.

Wang

Zhu

, et al. (2022) Improving generalization of deep networks for estimating physical properties of containers and fillings. In: The 32nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Singapore, 9147–9151.

32.

Xiong

Quek

FKH

(2006) Hand motion gesture frequency properties and multimodal discourse analysis. International Journal of Computer Vision 69: 353–371.

33.

Xompero

Donaher

Iashin

, et al. (2022) The Corsmal Benchmark for the Prediction of the Properties of Containers. IEEE Access.

34.

Duncan

Yeung

(2015) Fill and transfer: a simple physics-based approach for containability reasoning. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 711–719.