A feasibility study of ballet education using measurement and analysis on partial features of still scenes

Abstract

There have been a number of dance and ballet education systems using different multimedia devices. One of the well-known multimedia devices is Kinect which uses multiple built-in sensors. We focus on Kinect-based dance and ballet education systems. Existing systems that use Kinect cannot properly recognize the turnout movement. We propose the use of a lower body joint point estimation algorithm and a closest foot points estimation algorithm that can efficiently perform image localization for recognition of basic ballet movements. In addition, in order to evaluate correct ballet movements, we propose a method that extracts partial features from still scenes and performs measurements for knee and foot positions. The proposed method is the first ballet education system that properly measures movements of a ballet dancer.

Introduction

Ballet is a well-known type of dance that uses the human body to express inner emotions and feelings. In addition to requiring physical development, ballet can also stimulate mental and emotional human development. ¹ Traditionally, a ballet instructor is responsible for development of correct ballet techniques of a dancer. However, it is difficult for a ballet instructor to determine how much of the body movements of a dancer are incorrect because, in most cases, the instructor relies on visual experience. ²

There have been a number of dance and ballet education systems that use different multimedia devices. However, most of these devices are expensive and uncomfortable to use. As a result, alternative methods that use inexpensive multimedia devices, such as Kinect, have been proposed. However, the drawback of existing methods is that in certain cases, Kinect produces errors in recognition of certain joint positions. For example, Kinect cannot properly detect the turnout movement in which rotation of the leg from the hips causes the knee and foot to turn outward, away from the center of the body. ³

We propose an efficient ballet education method that focuses on lower body joints. For this, we propose the two algorithms: lower body joint point estimation (LJE) and closest foot point estimation (CFE). On one hand, LJE analyzes joint skeletal data and is used for recognition of joints of the first and second foot positions among the five foot positions of ballet. On the other hand, CFE is used for processing data obtained from a side optic sensor and measures lower body joints based on the analysis of still scenes from the side. CFE is mainly applied for recognition of the fourth and fifth foot positions.

More specifically, we make the following contributions:

• The disadvantage of Kinect for ballet education is that the turnout movement cannot be detected. In order to solve this problem, we propose the LJE algorithm, which can properly detect the knees and feet of a ballet dancer. This is the first research to address detection of the turnout movement.

• Another problem of using Kinect in ballet education is movements where a ballet dancer crosses the feet cannot be detected at some foot positions. In order to solve this problem, we propose the CFE algorithm that estimates foot joint positions using a side optic sensor;

• We propose a ballet basic education interface (BEI) for efficient ballet education. BEI embraces both LJE and CFE algorithms and provides accuracy of visualization for certain ballet movements. In addition, instructional feedbacks are provided in real time.

This article proceeds as follows: In section “Ballet education,” we explain ballet education and terminology. In section “Related work,” we introduce related work for ballet education. In section “New alternatives for restrictions of Kinect ballet education,” we introduce a measuring function for ballet education, the LJE and CFE algorithms, and the ballet BEI. In section “Performance evaluations,” we explain experiment results. In section “Conclusion,” conclusions are presented.

Ballet education

This section describes essential elements of ballet education. We first describe basic ballet education movements in section “Basic ballet movements” and then present terminology that is used in this article in section “Terminology.”

Basic ballet movements

Ballet is a combination of movements. It is important to educate a ballet dancer in correct movements as it leads to good performance. Generally, there are two types of training methods in ballet education, such as bar work and center work. In bar work, ballet dancer holds a bar with one hand and practices the basic movements. In center work, ballet dancer performs the same ballet movements without bar. The center work is an appropriate method to evaluate the movements trained using bar work. In most cases, the movements that are not correct in center work are practiced again using bar work. Traditionally, ballet education focuses on training of lower body parts due to the fact that ballet is based on foot movements and, thus, requires development of muscles in both feet. For example, there are nine types of ballet movements that we can perform using bar work, such as plies, battement tendus, battement tendu jete, ronds de jambe a terre, battement fondus, ronds de jambe en l’ air, petits battement sur le cou-de-pied, battement developes, and grand battement jetes. ⁴ These nine basic ballet movements are based on training of lower body parts. Since lower body parts are essential for all ballet movements, it is important to develop a system that scientifically analyzes and determines correctness of a certain movement.

On the other hand, the five foot positions in ballet are first, second, third, fourth, and fifth positions for which it is important to spread the pelvis, distribute the weight equally on both legs, and not to move the hips downward. ⁵ In addition, the legs must be straightened, and the direction of both knees and feet must be the same. These foot positions are shown in Figure 1.

Figure 1.

Five foot positions in ballet: (a) position 1, (b) position 2, (c) position 3, (d) position 4, and (e) position 5. ⁶

Plié is one of the basic ballet movements and has two variations such as demi plié and grand plié. Here, the direction of both knees and feet must be identical, while hips must not move downward. ³ Figure 2 demonstrates these movements. Here, the movement illustrated in Figure 2(a) and (b) is called demi plié, in which a dancer partially bends the knees in foot positions 1 and 2, respectively. Figure 2(c) and (d) indicates a grand plié, in which a dancer fully bends the knees in foot positions 1 and 2, respectively. It is also appropriate to practice demi plié and grand plié movements in foot positions 3, 4, and 5. When practicing grand plié in position 2, the feet must not be lifted from the floor, as shown in Figure 2(d).

Figure 2.

Ballet plié: (a) demi plié in foot position 1, (b) demi plié in foot position 2, (c) grand plié in foot position 1, and (d) demi plié in foot position 2.^7,8

Terminology

The terminology that is used throughout this article includes the turnout, in which movement is rotation of the leg from the hips, causing the knee and foot to turn outward, away from the center of the body. ⁴

• Definition 1 (lower body joint points). Joint points ${\mathbb{P}}_{\mathrm{joint}}$ obtained from skeletal data.

• Definition 2 (LJE). The algorithm estimates lower body joint points ${\mathbb{P}}_{\mathrm{joint}}$ based on still scenes. Let ${\mathbb{P}}_{\mathrm{joint}}$ be a set of joint positions ${\mathbb{P}}_{\mathrm{joint}}=\{p_{\mathrm{left}\mathrm{knee}},p_{\mathrm{right}\mathrm{knee}},p_{\mathrm{left}\mathrm{tiptoe}},p_{\mathrm{right}\mathrm{tiptoe}},p_{\mathrm{left}\mathrm{heel}},p_{\mathrm{right}\mathrm{heel}}\}$ where $p_{leftknee},p_{rightknee},p_{lefttiptoe},p_{righttiptoe},p_{leftheel},\mathrm{and}{p}_{rightheel}$ indicate the left and right knees, the left and right tiptoes, and the left and right heels, respectively.

• Definition 3 (estimated lower body joint points). Lower body joint points ${\tilde{\mathbb{P}}}_{joint}$ are estimated based on still scenes. Let ${\tilde{\mathbb{P}}}_{joint}$ be a set of joint positions such that ${\tilde{\mathbb{P}}}_{joint}=\{{\tilde{p}}_{leftknee},{\tilde{p}}_{rightknee},{\tilde{p}}_{lefttiptoe},{\tilde{p}}_{righttiptoe},{\tilde{p}}_{leftheel},{\tilde{p}}_{rightheel}\}$ where ${\tilde{p}}_{\mathrm{left}\mathrm{knee}},{\tilde{p}}_{\mathrm{right}\mathrm{knee}},{\tilde{p}}_{\mathrm{left}\mathrm{tiptoe}},{\tilde{p}}_{\mathrm{right}\mathrm{tiptoe}},{\tilde{p}}_{\mathrm{left}\mathrm{heel}},\mathrm{and}{\tilde{p}}_{\mathrm{right}\mathrm{heel}}$ indicate the left and right knees, the left and right tiptoes, and the left and right heels, respectively.

• Definition 4 (knee inner degree). The angle of the knee ${\mathrm{\theta }}_{\mathrm{knees}}$ for measurement of the accuracy of movement based on still scenes. Let ${\mathrm{\theta }}_{\mathrm{knees}}$ be ${\mathrm{\theta }}_{\mathrm{knee}}=\{{\mathrm{\theta }}_{\mathrm{left}\mathrm{knee}},{\mathrm{\theta }}_{\mathrm{right}\mathrm{knee}}\}$ , where ${\mathrm{\theta }}_{\mathrm{left}\mathrm{knee}}$ and ${\mathrm{\theta }}_{\mathrm{right}\mathrm{knee}}$ indicate the left knee inner degree and the right knee inner degree, respectively. ${\mathrm{\theta }}_{\mathrm{left}\mathrm{knee}}$ and ${\theta }_{rightknee}$ are defined as $\angle p_{lefthip},{\tilde{p}}_{leftknee},{\tilde{p}}_{leftheel}$ and $\angle p_{\mathrm{right}\mathrm{hip}},{\tilde{p}}_{\mathrm{right}\mathrm{knee}},{\tilde{p}}_{\mathrm{right}\mathrm{heel}}$ .

• Definition 5 (foot degree). The angle of the feet ${\mathrm{\theta }}_{\mathrm{feet}}$ for measurement of the accuracy of movement based on still scenes. Let ${\mathrm{\theta }}_{\mathrm{feet}}$ be ${\mathrm{\theta }}_{\mathrm{feet}}=\{{\mathrm{\theta }}_{\mathrm{left}\mathrm{foot}},{\mathrm{\theta }}_{\mathrm{right}\mathrm{foot}}\}$ , where ${\mathrm{\theta }}_{\mathrm{left}\mathrm{foot}}$ and ${\mathrm{\theta }}_{\mathrm{right}\mathrm{foot}}$ are defined as $\angle {\tilde{p}}_{\mathrm{left}\mathrm{tiptoe}},{\tilde{p}}_{\mathrm{left}\mathrm{heel}},\tilde{p}{\prime }_{\mathrm{left}\mathrm{tiptoe}}$ and $\angle {\tilde{p}}_{\mathrm{right}\mathrm{tiptoe}},{\tilde{p}}_{\mathrm{right}\mathrm{heel}},\tilde{p}{\prime }_{\mathrm{right}\mathrm{tiptoe}}$ . $\tilde{p}{\prime }_{\mathrm{tiptoe}}$ is a virtual point from ${\tilde{p}}_{\mathrm{tiptoe}}$ to the y-axis symmetry of ${\tilde{p}}_{\mathrm{heel}}$ .

• Definition 6 (closest foot points). Detection of the closest position of the feet ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ based on optic sensor. Let ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ be ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}=\{{\tilde{p}}_{\mathrm{front}\mathrm{closest}\mathrm{foot}},{\tilde{p}}_{\mathrm{rear}\mathrm{closest}\mathrm{foot}}\}$ , where ${\tilde{p}}_{\mathrm{front}\mathrm{closest}\mathrm{foot}}$ and ${\tilde{p}}_{\mathrm{rear}\mathrm{closest}\mathrm{foot}}$ indicate the front and rear closest foot points based on side optic sensor, respectively.

• Definition 7 (CFE). The algorithm measures a lower body ballet posture from a side view as described in section “LJE.”

Related work

A large body of work for dance and ballet education systems using different multimedia devices is briefly described. Chan et al. ⁹ proposed a dance education system based on motion capture and virtual reality (VR) technologies. The proposed system provides a virtual interface where a dancer can repeat movements instructed by a virtual teacher. The proposed system captures motion data of the dancer and, after analysis, provides instructional feedback based on dancer performance. A similar system using mixed reality technology has also been proposed. ¹⁰ Sukel et al. ¹¹ proposed a process for design of an interface for a nonconventional computational system that teaches ballet. The proposed system provides interaction and communication features where a student can ask for repetition and feedback, make specific queries, and practice at different speeds. The aforementioned systems^9–11 use different wearable devices for recognition of joint positions. However, most of these devices are expensive and uncomfortable to use. In addition, in certain cases, wearable devices require a complicated recognition procedure for joint position and, thus, are difficult for application in real life.

An alternative multimedia device, which is relatively cheap and easy to use, is Kinect. Recently, there have been a number of dances and ballet education systems proposed based on Kinect. For example, Marquardt et al. ¹² proposed a Kinect-based ballet education system called Super Mirror that captures motion data of a ballet dancer, provides analysis of technique, and provides instructional feedback in real time. Trajkova and Ferati ¹³ evaluated the usability of the Super Mirror for ballet education. Sun et al. ¹⁴ and Muneesawang et al. ¹⁵ proposed a ballet dance education system in a cave automatic virtual environment (CAVE). Similar to the Super Mirror, the proposed method also provides instructional feedback based on an interface for visualization of instructions and feedback in a CAVE. The drawback of aforementioned systems^12,14 using Kinect is that in certain cases, the systems are not suitable for evaluation of correct ballet movements. For example, the existing methods cannot properly detect the turnout movement, which is one of the main movements in ballet. To overcome these drawbacks, we propose the LJE and CFE algorithms that can increase the detection accuracy of ballet movements.

New alternatives for restrictions of Kinect ballet education

In this section, we describe a proposed method for efficient ballet education. Specifically, in section “LJE,” we first explain the LJE algorithm for detection of lower body joint points and measurement of the turnout movement and knee inner degree using Kinect. In section “CFE,” we describe the CFE algorithm for estimation of closest foot points using a side optic sensor. In section “Ballet BEI,” we present the BEI that includes both the LJE and CFE algorithms.

LJE

Kinect recognizes up to 20 human joints, which enables easy estimation of body parts. However, as mentioned in section “Introduction,” one of the drawbacks of using Kinect for ballet education is a low degree of accuracy for recognition of lower body joints, such as knees and ankles. In addition, Kinect cannot recognize the heel, which is one of the essential lower body parts in ballet. Thus, the turnout movement cannot be detected by Kinect. In order to overcome the aforementioned drawbacks of Kinect, we propose the LJE algorithm that can properly measure plié movements. Figure 3 shows the process of extracting features from lower body joints when a ballet performance is observed from the front.

Figure 3.

Partial feature extraction using LJE.

The LJE algorithm shown in Figure 5 contains the following steps: A color image and skeletal data from Kinect are used without depth data because the feet in plié are always on the floor, and thus, it is difficult to distinguish the floor and the feet based on Kinect depth data for estimation of lower body joint points. Accordingly, we first obtain a color image and lower body joint points ${\mathbb{P}}^T$ as inputs and then perform (1) binarization of the red, green, and blue (RGB) image and (2) contour extraction. Furthermore, we perform a union (3) set from a convex hull and polygonal approximation for detection of interest points from data obtained in step 2. Finally, we use a clustering algorithm based on mean shift ¹⁶ (i.e. an initial seed indicated with a yellow circle, final estimated points indicated by a red circle, and seed movements indicated by a green arrow) that enables estimation of lower body joint points, including heel positions. Skeletal data from Kinect provide useful information for estimation of accurate joint points within region of interest (ROI). Generally, initial seed is set manually. LJE algorithm can detect accurate joint points using skeletal data as initial seed. As a result, we obtain lower body joint points ${\tilde{\mathbb{P}}}_{\mathrm{front}\mathrm{joint}}$ estimated in step 4.

More specifically, the LJE algorithm proceeds as follows: We first generate a binary image using Otsu’s algorithm ¹⁷ for extraction of a human shape from a color image. Otsu’s algorithm divides pixels into the two classes of intraclass and interclass based on the T threshold value for separation of background and foreground and then calculates the maximum T value ratio between variances of the intraclass and interclass

$$\mathrm{TH}=\underset{T\in [0,255]}{\mathrm{argmax}}\left(\frac{{\sigma }_{\mathrm{Inter}}^2\left(T\right)}{{\sigma }_{\mathrm{Intra}\left(T\right)}^2}\right)$$

(1)

$$\mathrm{BI}\left(p\right)=\{\begin{array}{c}0,\mathrm{TH}>I\left(p\right)\hfill \\ \multicolumn{1}{c}{255,\mathrm{otherwise}}\end{array}$$

(2)

where TH in equation (1) is an adaptive threshold for binarization, and ${\sigma }_{\mathrm{Inter}}^2(T)$ and ${\sigma }_{\mathrm{Intra}}^2(T)$ are variances of the intraclass and the interclass at threshold T, respectively. $\mathrm{BI}(p)$ in equation (2) indicates the binary image from the adaptive threshold TH. The binary image shows the human shape based on reducing the level of noise from the external environment. Afterward, we perform contour extraction ¹⁸ for the analysis of a human shape. Contours are boundary lines of geometric shapes. Thus, identification of contours is crucial for analysis of image contents.

Figure 4 shows the detection process of interest points when a ballet performance is observed from the front. We extract interest points for accurate estimation of joint points. In human visual perception, human knee, toe, and heel positions have a convex shape. Based on this evidence, interest points are constructed based on ${\mathbb{P}}_{\mathrm{convex}\mathrm{hull}}$ values that are calculated using the convex hull algorithm ¹⁹ and ${\mathbb{P}}_{\mathrm{polygonal}}$ values using the polygonal approximation algorithm. ²⁰ A union set of ${\mathbb{P}}_{\mathrm{union}}$ is constructed based on ${\mathbb{P}}_{\mathrm{convex}\mathrm{hull}}$ and ${\mathbb{P}}_{\mathrm{polygonal}}$ values. In Figure 4, we can observe interest points that are located close to coordinate values of knees, tiptoes, and heels.

Figure 4.

Extraction of interest points and connectivity between them (white line): (a) still image of a plié movement, (b) interest points (white circles) based on the convex hull algorithm, ¹⁸ and (c) interest points based on polygonal approximation ¹⁹ (white circles).

Finally, we use a mean-shift algorithm ¹⁶ for extraction of lower body joint points from interest points. The mean-shift algorithm is used to find the centroid of the data distribution. Figure 5 demonstrates the shifting process of the knee joint position using mean-shift algorithm. Here, we use an initial seed as lower body joint points ${\mathbb{P}}_{\mathrm{joint}}$ from a skeletal data (yellow circle). The goal of the mean-shift algorithm is to find lower body joint points ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$ (red circle) from the data distribution of interest points (blue circle).

Figure 5.

For estimation of accurate joint positions, movement of initial seeds using the mean-shift algorithm ¹⁶ of a still scene is used.

Specifically, the estimation process of lower body joint points using the mean shift is as follows:

1. Set up initial seed points ${\mathbb{P}}_{\mathrm{joint}}$ (yellow circle) from skeletal data;

2. Extract interest points $\{p_0,p_1,\dots ,p_n\}$ as a data distribution within a search window r (red rectangle);

3. Move to the current position based on calculation of a coordinate value $\sum _{i=0}^n{p}_i/n$ of the centroid from interest points within a search window r, where n is the number of interest points;

4. Re-iterate steps 2 and 3 until no permitted move $(m<0.5)$ is achieved, and the distance of the centroid of each seed of current points is d < 30.

Algorithm 1 shows pseudo-code of the LJE algorithm. The input is an RGB image $I_{\mathrm{RGB}}$ and lower body joint points ${\mathbb{P}}_{\mathrm{joint}}$ . Lower body joint points ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$ are extracted as output from Algorithm 1.

Algorithm 1: LJE.
Input: RGB image as $I_{\mathrm{RGB}}$ and ${\mathbb{P}}_{\mathrm{joint}}$ from Kinect
Output: ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$
Algorithm:
1: procedure LJE ( $I_{\mathrm{RGB}},{\mathbb{P}}_{\mathrm{joint}}$ )
2: Igray scale = convertRGBtoGrayscale ( $I_{\mathrm{RGB}}$ ) as conversion from RGB to gray scale.
3: Ibinary = Threshold ( $I_{\mathrm{gray}\mathrm{scale}}$ ,Otsu( $I_{\mathrm{gray}\mathrm{scale}}$ )) from equations (1) and (2)
4: ${\mathbb{P}}_{\mathrm{contour}}=\mathbf{ContourExtraction}(I_{\mathrm{binary}})$ 17
5: ${\mathbb{P}}_{\mathrm{convex}\mathrm{hull}}=\mathbf{ConvexhullApproximation}(P_{\mathrm{contour}})$ 18
6: ${\mathbb{P}}_{\mathrm{polygonal}}=\mathbf{PolygonalApproximation}(P_{\mathrm{contour}})$ 19
7: ${\mathbb{P}}_{\mathrm{union}}={\mathbb{P}}_{\mathrm{convex}\mathrm{hull}}\cup {\mathbb{P}}_{\mathrm{polygonal}}$
8: ${\tilde{\mathbb{P}}}_{\mathrm{joint}}=\mathbf{Meanshift}({\mathbb{P}}_{\mathrm{union}})$ 20
9: end procedure

In ballet, most movements are based on a turnout movement that has association with ${\mathrm{\theta }}_{\mathrm{feet}}$ . Thus, on one hand, ${\mathrm{\theta }}_{\mathrm{feet}}$ is the accuracy of foot position and indicates the level of turnout movement. On the other hand, ${\mathrm{\theta }}_{\mathrm{knees}}$ is a criterion for classification of plié ballet movements into demi plié and grand plié. Thus, it is important to practice ${\mathrm{\theta }}_{\mathrm{knees}}$ .

Figure 6 shows measurements of lower body ballet postures that are used to achieve accurate plié movement measurements. We measure ${\mathrm{\theta }}_{\mathrm{knees}}$ and ${\mathrm{\theta }}_{\mathrm{feet}}$ values when a ballet dancer performs different movements. Specifically, Figure 6(a) demonstrates measuring positions of ${\mathrm{\theta }}_{\mathrm{left}\mathrm{knee}}$ and ${\mathrm{\theta }}_{\mathrm{right}\mathrm{knee}}$ from still scenes when a ballet dancer performs foot position 1. Figure 6(b) shows measurement positions of ${\mathrm{\theta }}_{\mathrm{left}\mathrm{foot}}$ and ${\mathrm{\theta }}_{\mathrm{left}\mathrm{foot}}$ when a ballet dancer performs foot position 2. ${\mathrm{\theta }}_{\mathrm{knees}}$ and ${\mathrm{\theta }}_{\mathrm{feet}}$ are calculated from lower body joint points using LJE algorithm. An accurate estimation of lower body joint points enables accurate measurements of ballet movements.

Figure 6.

Measurement of lower body ballet postures: (a) ${\mathrm{\theta }}_{\mathrm{knees}}$ and (b) ${\mathrm{\theta }}_{\mathrm{feet}}$ .

Figure 7 shows crossed feet for measurement of the turnout movement. Figure 7(a) shows that measurement of the angle in the crossed foot state is difficult. Thus, we propose a method for measurement of the lower ballet posture from a side view using an optic sensor and detection of foot joint points using a horizontal line (white dotted line) between the closest foot points based on an optic sensor in Figure 7(b). The proposed method measures ballet performance not only from the front view but also from the side view.

Figure 7.

The new alternative for detection of ${\mathrm{\theta }}_{\mathrm{feet}}$ of foot positions 4 and 5: (a) incorrect prediction of joint points and (b) application of CFE for detection of feet.

We propose a measuring function for evaluation of accuracy of ballet movements. Specifically, we propose a scoring criterion for evaluation of ${\mathrm{\theta }}_{\mathrm{knees}}$ and ${\mathrm{\theta }}_{\mathrm{feet}}$ . Tables 1 and 2 show score criteria for ${\mathrm{\theta }}_{\mathrm{knees}}$ and ${\mathrm{\theta }}_{\mathrm{feet}}$ according to ballet type and foot position. Here, the measurement of ${\mathrm{\theta }}_{\mathrm{knees}}$ is performed at foot positions 1, 2, 4, and 5. Scoring criteria for knees in demi plié shown in Table 1 are independent of foot positions. However, it is important to note that the values of ${\mathrm{\theta }}_{\mathrm{knees}}$ for position 2 and positions 1, 4, and 5 are not the same. The scoring criterion for knees in grand plié shown in Table 1 is different between foot positions 1, 4, and 5 and 2.

Table 1.

The scoring criteria for ${\mathrm{\theta }}_{\mathrm{knees}}$ according to the ballet type and foot position.

Ballet movement type	Foot position	${\mathrm{\theta }}_{\mathrm{knees}}$ (°)	Score
Demi plié	Positions 1, 2, 4, and 5	$85<{\mathrm{\theta }}_{\mathrm{knees}}\le 100$	100
		$75<{\mathrm{\theta }}_{\mathrm{knees}}\le 85$ or $100<{\mathrm{\theta }}_{\mathrm{knees}}\le 110$	90
		${\mathrm{\theta }}_{\mathrm{knees}}\le 75$ or $110\le {\mathrm{\theta }}_{\mathrm{knees}}$	80
Grand plié	Positions 1, 4, and 5	$50\ge {\mathrm{\theta }}_{\mathrm{knees}}$	100
		$50<{\mathrm{\theta }}_{\mathrm{knees}}\le 60$	90
		$60<{\mathrm{\theta }}_{\mathrm{knees}}$	80
	Position 2	${\mathrm{\theta }}_{\mathrm{knees}}\le 80$	100
		$80<{\mathrm{\theta }}_{\mathrm{knees}}\le 90$	90
		$90<{\mathrm{\theta }}_{\mathrm{knees}}$	80

Table 2.

The scoring criteria for ${\mathrm{\theta }}_{\mathrm{feet}}$ according to the ballet type and foot position.

Plié type	Foot position	${\mathrm{\theta }}_{\mathrm{feet}}$ (°)	Score
Demi plié and grand plié	Positions 1 and 2	$165<{\mathrm{\theta }}_{\mathrm{feet}}\le 180$	100
		$155<{\mathrm{\theta }}_{\mathrm{feet}}\le 165$	90
		$155\le {\mathrm{\theta }}_{\mathrm{feet}}$	80

CFE

In a previous section, we explained how lower body ballet postures are measured using the LJE algorithm. However, in order to improve the efficiency of the proposed ballet education system, another alternative measurement of lower body ballet movements from the side when a ballet dancer is in an aligned foot position for a particular ballet foot position is needed. We propose the CFE algorithm that is able to measure lower body ballet posture from a side view. The process for detection of closest foot points is shown in Figure 8.

Figure 8.

The process of the CFE algorithm using side optic sensor.

In order to measure lower body ballet posture from a side view, in contrast with the LJE algorithm, the CFE algorithm uses an optic sensor instead of Kinect. The optic sensor can capture an RGB image and detect closest foot points based on symmetric information using polynomial approximation. Thus, (1) a binarization process is performed on an RGB image obtained from an optic sensor, and (2) contours are extracted from the binary image. ¹⁷ Furthermore, we detect (3) interest points using polygonal approximation. ¹⁸ Then, we identify (4) symmetric points by finding a centroid of interest points and (5) detect the closest foot points based on selection of the largest y values on the y-axis for each point set ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ . The formula to find the centroid of interest points and symmetric point is

$$p_{\mathrm{centeroid}}=\frac{\sum _{i=0}^{N\left(P_{\mathrm{polygonal}}\right)}{P}_{\mathrm{polygonal}}\left(i\right)}{N\left(P_{\mathrm{polygonal}}\right)},$$

(3)

$$p_{\mathrm{symmetric}}=\underset{p\in P_{\mathrm{polygonal}}}{\mathrm{argmin}}{p}_{\mathrm{centeroid}}-p_{\mathrm{Euclidian}}$$

(4)

where $p_{\mathrm{centeroid}}$ represents centroid of interest points, $N(P_{\mathrm{polygonal}})$ indicates the number of points $P_{\mathrm{polygonal}}$ , and $p_{\mathrm{symmetric}}$ is the nearest point to the $p_{\mathrm{centeroid}}$ calculated based on Euclidean distance between $p_{\mathrm{centeroid}}$ and a point in $P_{\mathrm{polygonal}}$ . The formula to predict the closest foot points ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ using a symmetric feature is as follows

$$({\mathbb{P}}_{\mathrm{front}},{\mathbb{P}}_{\mathrm{rear}})=\mathbf{Div}(p_{\mathrm{symmetric}},{\mathbb{P}}_{\mathrm{polygonal}})$$

(5)

$$p_{\mathrm{closest}\mathrm{front}\mathrm{foot}}=\underset{p\in \mathbb{P}}{\mathrm{argmax}}Y\left(p_{\mathrm{front}}\right)$$

$$p_{\mathrm{closest}\mathrm{rear}\mathrm{foot}}=\underset{p\in \mathbb{P}}{\mathrm{argmax}}Y\left(p_{\mathrm{rear}}\right)$$

(6)

where ${\mathbb{P}}_{\mathrm{front}}$ and ${\mathbb{P}}_{\mathrm{rear}}$ are two sets of points that are divided by the symmetric point $p_{\mathrm{symmetric}}$ based on x-axis from ${\mathbb{P}}_{\mathrm{polygonal}}$ . $Y(p_{\mathrm{front}})$ indicates y-axis value of a point among ${\mathbb{P}}_{\mathrm{front}}$ . In this case, points with the largest values on y-axis represent toes. Algorithm 2 shows CFE algorithm for horizontal alignment from side optic sensor where input is a RGB image from an optic sensor and output is closest foot points ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ .

Algorithm 2: CFE.
Input: RGB image $I_{\mathrm{RGB}}$
Output: closest foot points ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}=\{{\tilde{p}}_{\mathrm{closest}\mathrm{front}\mathrm{foot}},{\tilde{p}}_{\mathrm{closest}\mathrm{rear}\mathrm{foot}}\}$
Algorithm:
1: procedure CFE ( $I_{\mathrm{RGB}}$ )
2: Igray scale = convertRGBtoGrayscale ( $I_{\mathrm{RGB}}$ ) as conversion from RGB to gray scale image
3: Ibinary = Threshold ( $I_{\mathrm{gray}\mathrm{scale}}$ , Otsu( $\mathrm{\gamma }$ )) from equations (2) and (3)
4: ${\mathbb{P}}_{\mathrm{contour}}=\mathbf{ContourExtraction}(I_{\mathrm{binary}})$ 17
6: ${\mathbb{P}}_{\mathrm{polygonal}}=\mathbf{PolygonalApproximation}({\mathbb{P}}_{\mathrm{contour}})$ 18
7: $p_{\mathrm{symmetric}}=\mathbf{SymmetricPoint}({\mathbb{P}}_{\mathrm{polygonal}})$ from equations (3) and (4)
8: $({\mathbb{P}}_{\mathrm{left}},{\mathbb{P}}_{\mathrm{right}})=\mathbf{Div}(p_{\mathrm{symmetric}},{\mathbb{P}}_{\mathrm{polygonal}})$ from equation (5)
9: ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}=(\underset{p\in \mathbb{P}}{\mathrm{argmax}}Y(p_{\mathrm{left}}),\underset{p\in \mathbb{P}}{\mathrm{argmax}}Y(p_{\mathrm{right}}))$ from equation (6)
10: end procedure

Ballet BEI

A ballet BEI is presented for efficient ballet education. The BEI includes software using the LJE and CFE algorithms for measurement of ballet movements. The BEI provides visualization of the degree and scores for plié ballet movements and provides real-time feedback messages to users. Figure 9 displays the user interface (UI) of BEI, and Table 3 shows the function and the specification equivalent to Figure 9. More specifically, the proposed interface has the following functions:

• Provides a function for training of core ballet movements and helps a user in injury prevention and improvement of physical strength via training for ballet movements;

• Provides real-time feedback and visualization of measurements from the LJE and CFE algorithms and real-time feedback messages and guides direction of correct ballet movements;

• Provides a self-training system for study of ballet movements and helps learning by providing a comparison between an expert and user ballet movements.

Figure 9.

User interface (UI) of BEI.

Table 3.

The function and the specification of BEI.

Index	Functions	Specification
A	Front scene from Kinect	Display of learner front ballet movements for self-training
B	Side scene from optic sensor	Display of learner side ballet movements for self-training
C	Display of knee inner degree	Real-time visualization of each knee degree
D	Display of foot degree	Real-time visualization of each foot degree
E	Introduce learning time for ballet movements	Introduction of learning time meter within a limited time for efficient ballet training
F	Repeat time	Repeat time in each foot position and plié movement
G	Feedback message	Provide real-time feedback as learner message for correct ballet movements (e.g. “band your left knee” when ${\mathrm{\theta }}_{\mathrm{left}}<{\mathrm{\theta }}_{\mathrm{right}\mathrm{knee}}$ )
H	Select the foot position and plié movement	Selection of foot position and basic plié movements
I	Display score of plié movements	Provide visualization of score of plié movements

BEI: basic education interface.

Figure 10 shows a flowchart for the BEI program. A detailed description of the BEI flowchart is as follows:

1. A learner can select the ballet foot position (1, 2, 4, and 5) and ballet plié movement (demi and grand plié);

2. Watching the expert video from the front and side views continuously;

3. Learn ballet movements;

4. Measurement extraction for plié movements of the feet for positions 1, 2, 4, and 5 using LJE and CFE algorithms;

5. Display the measurements and feedback message in the real time;

6. Repeat steps 3–5 within a limited time;

7. Comparison between learner and expert ballet movements.

Figure 10.

Flowchart of the BEI program.

Performance evaluations

This section describes experimental results. In section “Experimental environment,” we describe the experimental environment. In section “Experiment 1: comparison of the accuracy of the LJE and CFE algorithms,” we compare the accuracy of lower body joint points ( ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$ from LJE and ${\mathbb{P}}_{\mathrm{joint}}$ from Kinect skeletal data) for validation of the proposed algorithm. In section “Experiment 2: a feasibility test using collaborated sensors for ballet education,” we discuss experiments for feasibility of BEI application in a real ballet education system.

Experimental environment

This section introduces the experimental environment for experiments 1 and 2. Figure 11 illustrates the experimental environment. For experiment 1, we set the Kinect in front and an optic sensor at the side. The learner can train for ballet movements using the BEI program projected on a screen. We performed experiments on plié movements at each foot position. For experiment 2, we selected five female students. We performed experiments on plié movements at the two foot positions such as foot positions 1 and 4. It is important to note that this experimental environment is used in experiment 2.

Figure 11.

Environment for experiment.

Experiment 1: comparison of the accuracy of the LJE and CFE algorithms

For the LJE algorithm, we performed experiments for comparison of accuracy between estimated lower body joint points and skeletal data from Kinect. However, Kinect has a limitation because skeletal data cannot provide heel points. Moreover, Kinect cannot accurately detect lower body joint points for correct evaluation of lower body ballet movements. Thus, for comparison of the accuracy of estimated joint points, heel points were generated using ankle data, and tiptoe points were derived from skeletal data. For fair comparison, we used derived heel points based on a relationship between joint points from skeletal data. Figure 12 shows the relationship between joint points (ankle, tiptoe, and heel points). Let $l_1=\overline{p_{\mathrm{tiptoe}}{p}_{\mathrm{ankle}}},l_2=\overline{p_{\mathrm{ankle}}{p}_{\mathrm{heel}}}$ , and $l_3=\overline{p_{\mathrm{tiptoe}}{p}_{\mathrm{heel}}}$ , where l, $l_2$ , and $l_3$ are lines. Line $l_3$ can be calculated based on a rotational transformation of line $l_1$ . We set angle $\mathrm{\theta }=\mathrm{\pi }/9$ for rotational transform. To derive the heel point $p_{\mathrm{heel}}$ value, we used the intersection point of two lines ( $l_1$ and $l_2$ ) as the heel point.

Figure 12.

Relationship between joint points.

Table 4 shows measurable lower body joint points for the LJE algorithm at different ballet foot positions and for different plié movements. The value of ${\mathrm{\theta }}_{\mathrm{knees}}$ is calculated at all ballet positions and plié movements using values for estimated joint points ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$ . However, ${\mathrm{\theta }}_{\mathrm{feet}}$ is only calculated at foot positions 1 and 2. Heel points cannot be detected in foot positions 4 and 5 because of crossed feet.

Table 4.

Measurable lower body joint points for the LJE algorithm at different ballet foot positions and for different plié movements.

Plié movements	Foot position	Estimated joint points
Demi plié and grand plié	Positions 1 and 2	${\tilde{p}}_{\mathrm{left}\mathrm{knee}}$ ; ${\tilde{p}}_{\mathrm{right}\mathrm{knee}}$ ; ${\tilde{p}}_{\mathrm{left}\mathrm{tiptoe}}$ ; ${\tilde{p}}_{\mathrm{right}\mathrm{tiptoe}}$ ; ${\tilde{p}}_{\mathrm{left}\mathrm{heel}}$ ; ${\tilde{p}}_{\mathrm{right}\mathrm{heel}}$
	Positions 4 and 5	${\tilde{p}}_{\mathrm{left}\mathrm{knee}}$ ; ${\tilde{p}}_{\mathrm{right}\mathrm{knee}}$ ; ${\tilde{p}}_{\mathrm{left}\mathrm{tiptoe}}$ ; ${\tilde{p}}_{\mathrm{right}\mathrm{tiptoe}}$

LJE: lower body joint point estimation.

Figure 13 shows the lower body joint points from ground truth (blue circle), Kinect skeletal data (red circle), and the LJE algorithm (white circle) for positions 1 and 2. Estimated lower body joint points are near real joint points since the proposed algorithm uses shape information for estimation of joint points.

Figure 13.

Lower body joint points from ground truth (blue circle), Kinect skeletal data (red circle), and the LJE algorithm (white circle) at positions 1 and 2: (a) demi plié at position 1, (b) grand plié at position 1, (c) demiplié at position 2, and (d) grand plié at position 2.

However, the proposed algorithm has two limitations. First, we cannot recognize heel position because of crossed feet. This limitation is shown in positions 4 and 5. Second, we cannot recognize knee position relative to feet because shape information for knees is not presented. Figure 14 shows the lower body joint points from ground truth (blue circle), Kinect skeletal data (red circle), and the LJE algorithm (white circle) for the fourth and fifth positions.

Figure 14.

Lower body joint points from ground truth (blue circle), Kinect skeletal data (red circle), and the LJE algorithm (white circle) for positions (a) 4 and (b) 5.

To verify the quantitative accuracy of ${\tilde{\mathbb{P}}}_{\mathrm{joint}}$ , calculate the error between real points and estimated lower body joint points and between real points and skeletal data. We use Euclidean distance as an error term between the same joint points. Figure 15 displays error values of joint points for the LJE algorithm and Kinect skeletal data. As a result of error, the LJE algorithm provides a higher degree of accuracy for estimated knee points than Kinect skeletal data of approximately 2.4, 3.1, 4.3, and 1.7 times in Figure 15(a) for estimated tiptoe points than Kinect skeletal data of approximately 19.5, 14.7, 36.9, and 26.4 times in Figure 15(b) and of estimated heel points than Kinect skeletal data of approximately 6.3, 1.9, 2.8, and 3.9 times in Figure 15(c).

Figure 15.

Error values of the LJE algorithm and Kinect skeletal data for foot positions 1 and 2: (a) error values for knee points, (b) error values for tiptoe points, and (c) error values for heel points.

Figure 16 shows the closest foot points from ground truth (blue circle), Kinect skeletal data (red circle), and the CFE algorithm (white circle) for each foot in position 4. Estimated closest foot points are closer than skeletal data from real joint points.

Figure 16.

Closest foot points from ground truth (blue circle), Kinect skeletal data (red circle), and the CFE algorithm (white circle) at foot position 4: (a) left foot in front and (b) right foot in front.

To verify the quantitative accuracy of ${\tilde{\mathbb{P}}}_{\mathrm{closest}\mathrm{foot}}$ , we calculated the error between real points and estimated the closest foot points between real points and skeletal data. We used Euclidean distance as an error term as for the LJE algorithm. Table 5 shows the error of joint points for the CFE algorithm and Kinect skeletal data. As a result of error, the CFE algorithm exhibited higher accuracy of estimated closest foot points than Kinect skeletal data by approximately 7.2 times.

Table 5.

Error values for the CFE algorithm and Kinect skeletal data at foot position 4.

Methods	Estimated closest foot points
Kinect	$24.71$
CFE algorithm	$3.42$

CFE: closest foot point estimation.

Experiment 2: a feasibility test using collaborated sensors for ballet education

The procedure for experiment 2 was as follows: First, a learner selects a plié type and foot position. Second, the learner watches an expert educational guide video prior to practice. In this level, the learner can check the angles of knee and tiptoes in real time. Repetitive practice must produce a score of at least 95 based on scoring criteria as shown in Tables 1 and 2. In this level, the learner practices ballet without an angle view of the knee and tiptoe. After finishing practice, the learner can identify accuracy and watch side motion and front motion for comparison with expert motion. The learner can confirm the closest foot horizontal alignment from the side and the accuracy of ballet posture from the front. Figure 17 shows real plié movements for ballet education.

Figure 17.

A learner ballet movement using the BEI program: (a) front image and (b) side image.

Based on the experimental results, the average repeat time to achieve a score of 95 is 1.4 with demi and position 1, 2.8 with demi and position 4 for the right foot in front, 1.6 with demi and position 4 for the left foot in front, 2.2 with grand and position 1, 1.8 with grand and position 4 for the right foot in front, and 2.4 with grand and position 4 for the left foot in front as shown in Figure 18. Basic learner timing of skill improvement has a mean score of 95. These are useful data in future work for setting of an average time for practice.

Figure 18.

Average repeat time for plié movements and foot positions for learners.

Conclusion

We have focused on ballet education based on Kinect. We have solved the Kinect problem of nonrecognition of the turnout movement based on the use of the two new algorithms LJE and CFE. The proposed method is the first for measurement of the accuracy of lower body joints. In addition, we have proposed a ballet interface and performed experiments based on ballet scenarios. Experimental results show that the LJE algorithm has a high degree of accuracy for recognition of knees, feet, and heels, in comparison with skeletal data of Kinect. Specifically, on one hand, the accuracy of detecting knees when performing demi plié is increased by 2.4 times in foot position 1; 3.1 times in position 2; and when performing grand plié, the accuracy of foot position 1 is increased 4.3 times and position 2 is increased 1.7 times. On the other hand, the accuracy for detection of tiptoe when performing demi plié is increased 19.5 times in foot position 1 and 14.7 times in position 2. For grand plié, the accuracy is increased 36.9 times in foot position 1 and 26.4 times in position 2. CFE algorithm also provides a higher degree of accuracy for estimating the closest foot points comparing to Kinect skeletal data by approximately 7.2 times. Proposed algorithm can be applied for other applications that require the estimation of lower body joint points such as movement analysis in sports and dances.

However, the limitation of the proposed method is that we could not detect accurate lower joint points in the foot positions 4 and 5 using LJE algorithm due to hidden joint points. Thus, in future work, we will solve this problem and demonstrate application of Kinect in ballet education.