Keywords

1 Introduction

Evaluation sport action of sporter is an increasingly importance topic that should be achieved by video and acquisition of body parts features. With the development of 2D human pose estimation technology, the precise determination of body coordinates from images has become feasible, which is gradually introduced into sport fields. Atima et al. [1] applied 2D human pose estimation to extract human skeleton of coaches and trainees during Taijiquan, and evaluated actions by comparing the average angle difference between the shoulders, hips, arms and legs. Moritz et al. [2] obtained the human pose sequence of athletes during the long jump by the variation structure of Mask R-CNN network, and further got angles of body parts to analyze each action. A skeleton diagram of human actions can be obtained by human pose estimation for visual analysis, and body coordinates can extract more body features, signifying significant implications for sports action assessment.

Sport action evaluation evolved from previous manual evaluation to visual evaluation through human pose estimation technology, which greatly improves the evaluation efficiency and accuracy. However, the above mentioned action assessments are merely an evaluation analysis of each posture during the action process and does not provide a quantitative assessment of the overall action.

Hence, this paper introduces the fuzzy comprehensive evaluation method (FCE), which combines fuzzy mathematics with multi-index decision-making, and is suitable for the object system with many evaluation factors and complex structure. In 1965, the concept of fuzzy set was initially introduced by Zadeh L A et al. [3], which is the pioneer use of mathematical methods to describe the fuzzy concept, since then FCE has been widely applied. FCE emerges as a potent tool for quantitative analysis of objects containing multiple complex evaluation factors such as hazard prediction [4] and software system [5], whose results are more intuitive, accurate and reliable. In this paper, several key postures and body joint angles in sport action process are taken as the indicators sets, the weight sets are determined by the Analytic Hierarchy Process (AHP), the membership degree is computed by employing appropriate membership function to yield the evaluation result.

Furthermore, this paper proposes a sport action evaluation method based on human pose estimation. Firstly, professional coaches specify key postures and features for standard action videos. Then, key feature extraction is performed on test action videos. Lastly, the final score is obtained through a evaluation algorithm based FCE.

The rest of the paper is organized as follows. The following section provides a brief literature review of human pose estimation. Section 3 introduces our sport action approach based on human pose estimation. Section 4 presents a case study that uses this approach to evaluate 100 golf swing actions. Finally, conclusions and ideas for future research are offered.

2 Human Pose Estimation

Human pose estimation involves using deep learning networks to recognize key body parts in images and obtain the coordinates of key body parts. Riza Alp Guler et al. [6] were the first to apply deep learning to human pose estimation. The network is divided into multiple stages, with the key points detected in the first stage used as input for subsequent stages to further improve the detection effect. Subsequent researchers [7,8,9,10,11] have made improvements to the network to enhance detection accuracy. Zhe Cao et al. [12] introduced the OpenPose framework, utilizing Part Affinity Fields for rapid keypoint connections. This framework achieves remarkable results on the COCO dataset, providing coordinates for 18 body keypoints and marking a qualitative leap in accuracy and precision for human pose estimation. Hao-Shu Fang et al. [13] proposed a spatial transformation network called AlphaPose, ensuring that the results of target detection yield better outcomes after transformation. In 2021, Google introduced the lightweight pose estimation model MoveNet [14], capable of detecting 17 key points on the body as shown in Fig. 1:

Fig. 1.
figure 1

Human body key points estimated by MoveNet.

Full size image

MoveNet is an estimation model, using heatmaps to accurately localize human key points, which can estimate 17 body points. The accuracy reached 96% on datasets with intense motion, making it an ideal model for sports applications.

3 Sport Action Evaluation Approach Based on Human Pose Estimation

Based on sports knowledge and human pose estimation, this paper designed a sport action evaluation approach, as illustrated in Fig. 2:

Fig. 2.
figure 2

Sport action evaluation approach based on human pose estimation.

Full size image

Given a standard single sporter action video, firstly, coaches designate the key postures and features of each key posture. Then, with the use of human pose estimation, key posture images and feature values are extracted from the input action video. At last, the feature values are input into the action evaluation approach to obtain the action score.

Our approach consists of 3 parts: designating key postures and features, extracting key postures and features and action evaluation, as follows:

3.1 Designate Key Postures and Features of Each Key Posture

Combining expertise from professional coaches and sports knowledge, for a standard sport action video, we specify the most crucial postures among all postures. Through analyzing these postures, we proceed to conduct a comprehensive assessment of the entire action. Clearly, the most intuitive metric for assessing a posture is the angular relationship between body parts, which are as key features. Therefore, in this step, following coaches advice, this approach provides descriptions of key postures, body part angles, and ideal angles values for the sport action under evaluation.

3.2 Extract Key Postures and Features of Each Key Posture

For extracting key postures, this paper uses OpenCV to break down the input video into frames. Then, MoveNet is utilized to extract body points coordinates for each frame. These features are compared to the key postures defined by coaches, and the frame most closely resembling the expert-specified key posture is identified as the key posture frame. Regarding the extraction of actual angle features, cosine calculations are performed on the coordinates of 3 adjacent body parts which are obtained by MoveNet to get angles values by Eq. 1 and Eq. 2, which takes the calculation of the right elbow angle as an example.

$$ A_{relbow \, } = \left\{ {\begin{array}{*{20}c} {360^{^\circ } + \alpha_{relbow \, } ,} & { - 360^{^\circ } \le \alpha_{relbow \, } < - 180^{^\circ } } \\ { - \alpha_{relbow \, } ,} & { - 180^{^\circ } \le \alpha_{relbow \, } < 0^{^\circ } } \\ {\alpha_{relbow \, } ,} & {0^{^\circ } \le \alpha_{relbow \, } < 180^{^\circ } } \\ {360^{^\circ } - \alpha_{relbow \, } ,} & {180^{^\circ } \le \alpha_{relbow \, } \le 360^{^\circ } } \\ \end{array} } \right. $$
(1)
$$ \alpha_{relbow} = \arctan (\frac{{y_{rshoulder} - y_{relbow} }}{{x_{rshoulder} - x_{relbow} }}) - \arctan (\frac{{y_{rwrist} - y_{relbow} }}{{x_{rwrist} - x_{relbow} }}) $$
(2)

Where \((x_{relbow \, } ,y_{relbow \, } )\) and \((x_{rwrist \, } ,y_{rwrist \, } )\) are right elbow and right wrist coordinates.

3.3 Action Evaluation Approach

To achieve the purpose of the comprehensive quantitative evaluation, FCE is employed to propose our action evaluation approach, which are shown in Fig. 3.

Fig. 3.
figure 3

Action evaluation approach.

Full size image

Firstly, the indicator sets (U), evaluation set (V) and weight sets (ω) are designed. Next, the fuzzy relationship matrix (R) is builded based on U and V by the membership function (r). Then, the comprehensive evaluation set (B) is obtained by the synthesis operation between ω and R. Finally, an action score is output by the score calculation algorithm.

Below, each of the steps is discussed in turn, through a case study.

4 Case Study

This case study was conducted to evaluation 100 golf swing video actions collected from publicly available golf swing videos on the internet, which also includes the corresponding hitting distances for each swing action. The hitting distance serves as a criterion to judge the credibility of our scores. As, without considering the influence of other factors (such as swing force, environmental conditions, etc.), the golf swing action should be positively correlated with the hitting distance.

The following section provides a detailed overview of the evaluation process.

4.1 Key Postures and Angles Designation

Under the guidance of golf coaches, we designated 7 key postures: Stand (ST), Up (U), Top (T), Down (D), Hit (H), Swing (SW), and End (E), as shown in Fig. 4. At the same time, for every posture, ideal angle values \(A_{i}\) are listed in Table 1.

Fig. 4.
figure 4

7 key postures of the golf swing action.

Full size image

(a): The preparation posture for starting the swing. (b): The moment when the left arm is parallel to the ground during a counterclockwise back swing. (c): The moment of transition from back swing to down swing. (d): The moment when the left arm is parallel to the ground during the clockwise down swing. (e): The moment the club touches the ball. (f): The moment when the right arm is parallel to the ground during a clockwise forward swing. (g): The end of the entire swing process.

Table 1. The ideal angles values of each posture.
Full size table

4.2 Key Postures and Angles Extraction

Firstly, traverse through the test videos and each frame image is extracted by OpenCV. The second-level indicators corresponding to each first-level indicator are shown in Table 3, the body key coordinates for each frame are obtained by MoveNet. Then, based on the changes in body key coordinates of key postures specified by coaches from the standard action video, the frames most similar to the standard postures images are extracted as the key pose image for the test video. Finally, the real angle values can be computed through Eq. 1.

4.3 Action Evaluation

Based on FCE, a golf swing action evaluation algorithm is implemented in Table 2.

Table 2. Golf Swing Action evaluation algorithm.
Full size table

The detailed process is as follows:

  1. (1)

    Indicator sets

This paper establishes a golf swing action evaluation indicator system with 7 first-level indicators and 27 second-level indicators. The second-level indicators corresponding to each first-level indicator are shown in Table 1. Therefore, the first-level indicators set of golf swing action evaluation is

$$ U = \{ U_{1} ,U_{2} , \cdots ,U_{7} \} . $$

The second-level indicators sets are

$$ U_{1} = \{ u_{11} ,u_{12} ,u_{13} ,u_{14} ,u_{15} \} , $$
$$ U_{2} = \{ u_{21} ,u_{22} ,u_{23} \} , $$
$$ U_{3} = \{ u_{31} ,u_{32} ,u_{33} ,u_{34} ,u_{35} \} , $$
$$ U_{4} = \{ u_{41} ,u_{42} ,u_{43} \} , $$
$$ U_{5} = \{ u_{51} ,u_{52} ,u_{53} ,u_{54} ,u_{55} \} , $$
$$ U_{6} = \{ u_{61} ,u_{62} ,u_{63} \} , $$
$$ U_{7} = \{ u_{71} ,u_{72} ,u_{73} \} . $$
  1. (2)

    Evaluation set

Based on coaches opinions, the evaluation set \(\left( V \right)\) is divided into four levels: Failing \(\left( {V_{1} } \right)\), Moderate \(\left( {V_{2} } \right)\), Good \(\left( {V_{3} } \right)\), and Excellent \(\left( {V_{4} } \right)\), which is represented as:

$$ V = \{ V_{1} ,V_{2} ,V_{3} ,V_{4} \} = {\text{\{ [0,60],[60,80],[80,90],[90,100]\} }}. $$
  1. (3)

    Weight sets

We establish the weight set \(\left( \omega \right)\) using the Analytic Hierarchy Process (AHP). Its establishment is carried out in 3 steps:

  • ①Construction of judgment matrix

The judgement matrix reflects the importance relationship between each two factors in the same indicator set. The importance scale of factors is shown in Table 3:

Table 3. The importance scale of factors.
Full size table

Where \(a_{ij}\) denotes the importance of indicator i relative to factor j in the same indicator set.

The judgment matrix \(\left( A \right)\) is determined by the expert survey method. Therefore, the judgement matrix of the first-level index set U is determined as follows:

$$ A_{U} = \left[ {\begin{array}{*{20}c} 1 & 3 & 2 & 3 & 1 & 6 & 6 \\ {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & 1 & {{1 \mathord{\left/ {\vphantom {1 2}} \right. \kern-0pt} 2}} & 1 & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & 3 & 3 \\ {{1 \mathord{\left/ {\vphantom {1 2}} \right. \kern-0pt} 2}} & 2 & 1 & 2 & {{1 \mathord{\left/ {\vphantom {1 2}} \right. \kern-0pt} 2}} & 5 & 5 \\ {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & 1 & {{1 \mathord{\left/ {\vphantom {1 2}} \right. \kern-0pt} 2}} & 1 & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & 3 & 3 \\ 1 & 3 & 2 & 3 & 1 & 6 & 6 \\ {{1 \mathord{\left/ {\vphantom {1 6}} \right. \kern-0pt} 6}} & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & {{1 \mathord{\left/ {\vphantom {1 5}} \right. \kern-0pt} 5}} & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & {{1 \mathord{\left/ {\vphantom {1 6}} \right. \kern-0pt} 6}} & 1 & 1 \\ {{1 \mathord{\left/ {\vphantom {1 6}} \right. \kern-0pt} 6}} & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & {{1 \mathord{\left/ {\vphantom {1 5}} \right. \kern-0pt} 5}} & {{1 \mathord{\left/ {\vphantom {1 3}} \right. \kern-0pt} 3}} & {{1 \mathord{\left/ {\vphantom {1 6}} \right. \kern-0pt} 6}} & 1 & 1 \\ \end{array} } \right] $$

The judgement matrix of the second-level indicators sets \(U_{i}\) can be obtained using the same method as described above.

  • ②Compatibility test

Compatibility test is to assess whether the judgement matrix is valid or not by obtaining the consistency ratio (CR) and if CR ≤ 0.1, the compatibility of the judgment matrix is acceptable and the eigenvectors of the matrix can be used as the weigh set. Otherwise, the judgement matrix is readjusted. The CR is computed by:

$$ CR = \frac{{\lambda_{\max } - m}}{m - 1} $$
(3)

Where \(\lambda_{\max }\) denotes the maximum eigenvalue of the matrix, m represents the numbers of corresponding indicators. The CR of each judgment matrix built is as shown in Table 4.

Table 4. \(\lambda_{{{\text{max}}}}\) and CR of the judgment matrixes A
Full size table

It can be seen that the CR are all less than 0.1, then the weigh set of each indicator set is the eigenvector of the corresponding judgement matrix.

  • ③Construction of weight set.

The weight set of first-level indicators is

$$ \omega_{U} = \{ 0.277,0.095,0.174,0.099,0.277,0.039,0.039\} , $$

The weight sets of second-level indicators are

$$ \omega_{{U_{1} }} = \{ 0.088,0.298,0.158,0.158,0.298\} , $$
$$ \omega_{{U_{2} }} = \omega_{{U_{4} }} = \omega_{{U_{6} }} = \{ 0.540,0.163,0.297\} , $$
$$ \omega_{{U_{3} }} = \omega_{{U_{5} }} = \{ 0.370,0.206,0.109,0.109\} , $$
$$ \omega_{{U_{7} }} = \{ 0.547,0.190,0.263\} , $$
  1. (4)

    The fuzzy relationship matrix

For golf swing actions, the assessment is conducted based on body angles, where the evaluation involves determining the membership degree of an actual angle to different evaluation levels within given angle interval for each evaluation level. However, for cases where the actual angle falls just outside a evaluation level's interval but is very close, direct assignment to another evaluation level is not appropriate, which lacks continuity. Hence, this paper opts for a trapezoidal membership function (Eq. 4) to calculate membership, for obtaining fuzzy relationship matrix(R).

$$ \begin{array}{*{20}l} {r_{1} (x) = \left\{ {\begin{array}{*{20}c} 1 & {x < x_{0} } \\ {\frac{{x_{1} - x}}{{x_{1} - x_{0} }}} & {x_{0} \le x \le x_{1} } \\ 0 & {x \ge x_{1} } \\ \end{array} } \right.} \hfill & {r_{2} (x) = \left\{ {\begin{array}{*{20}c} 0 & {x \le 2x_{0} - x_{1} } \\ {\frac{{x - x_{0} }}{{x_{1} - x_{0} }} + 1} & {2x_{0} - x_{1} < x < x_{0} } \\ 1 & {x_{0} \le x \le x_{1} } \\ {\frac{{x_{1} - x}}{{x_{1} - x_{0} }}} & {x_{1} < x < 2x_{1} - x_{0} } \\ 0 & {x_{1} < x < 2x_{1} - x_{0} } \\ \end{array} } \right.} \hfill \\ {r_{2} (x) = \left\{ {\begin{array}{*{20}c} 0 & {x \le 2x_{1} - x_{2} } \\ {\frac{{x - x_{1} }}{{x_{2} - x_{1} }} + 1} & {2x_{1} - x_{2} < x < x_{1} } \\ 1 & {x_{1} \le x \le x_{2} } \\ {\frac{{x_{2} - x}}{{x_{2} - x_{1} }}} & {x_{2} < x < 2x_{2} - x_{1} } \\ 0 & {x \ge 2x_{2} - x_{1} } \\ \end{array} } \right.} \hfill & {r_{4} (x) = \left\{ {\begin{array}{*{20}c} 0 & {x < x_{1} } \\ {\frac{{x - x_{1} }}{{x_{2} - x_{1} }}} & {x_{1} \le x \le x_{2} } \\ 1 & {x \ge x_{2} } \\ \end{array} } \right.} \hfill \\ \end{array} $$
(4)

Where x represents actual values of second-level indicator. x0, x1, and x2 represent the critical value of the evaluation level of each indicator, as shown in Table 5.

Table 5. The angle interval of each evaluation level.
Full size table

The membership matrix of each second-level indicators constitutes a 27 × 4 fuzzy relationship matrix \(\left( {R_{2} } \right)\):

figure a
  1. (5)

    The comprehensive evaluation set

The synthesis operation is carried out and the FCE set of every first-level indicators is obtained, which is combined to a 7 × 4 matrix \(\left( {B_{2} } \right)\). At the same time, the fuzzy relationship matrix \(\left( {R_{1} } \right)\) is equal to \(B_{2}\), as following:

$$ R_{1} = B_{2} = \left[ {\begin{array}{*{20}c} {0.3} & {0.1} & {0.6} & {0.4} & {0.4} & {0.3} & {0.4} \\ {0.3} & {0.3} & {0.7} & {0.5} & {0.3} & {0.4} & {0.3} \\ {0.4} & {0.8} & {0.6} & {0.8} & {0.5} & {0.8} & {0.6} \\ {0.7} & {0.8} & {0.2} & {0.5} & {0.6} & {0.6} & {0.5} \\ \end{array} } \right]^{T} $$

Then, the ultimate fuzzy comprehensive evaluation set \(B_{1}\) can be derived by fuzzy synthesis operation between \(R_{1}\) and \(\omega_{U}\), which is below:

$$ B_{1} = R_{1} \circ \omega_{U} = \left( {b_{1} ,b_{2} ,b_{3} ,b_{4} } \right) = \left( {0.40,0.41,0.59,0.50} \right) $$

Where \(\circ\) fuzzy operation is \(M\left( { \bullet , \oplus } \right)\). Its specific separation is as follows:

$$ b_{i} = \min \{ 1,\sum\limits_{j = 0}^{m} {\omega \cdot r_{ji} } \} ,(i = 1,2,3,4) $$
(5)

Where \(b_{i}\) denotes the likelihood that the evaluation subject belongs to the corresponding evaluation level.

  1. (6)

    The action score

Apparently, the evaluation level “Good” \(V_{3}\) corresponding to \(b_{3}\) represents the assessment result for this swing action. According to steps 18 to 20 outlined in Table 2, the final score is derived \(S = 85.9\).

Additionally, to verify the effectiveness of indicators sets, weight sets and membership function specified, based on the parameters of the fuzzy comprehensive evaluation model mentioned above, this paper conducted 8 comparative experiments by modifying the weights of first-level indicators, the numbers and weights of second-level indicators, and the membership function. The experimental results are shown in Table 6:

Table 6. Comparison of different fuzzy comprehensive evaluation model
Full size table

In Table 6, U2-27 represents the model mentioned above, where U2-17, U2-21, and U2-23 denote models with 17 second-level indicators, 21 second-level indicators, and 23 second-level indicators, respectively. U1-2 and U1-3 are models with two modifications for the weight sets of first-level indicators. U2-E represents the model with modification to the weight sets of second-level indicators. R2 indicates the model with modification to the membership function. Abnormity refers to the percentage of grade which is less than 50. Accuracy denotes the proportion of which is consistent with the hitting distance after removing the abnormal data. Abnormity of U2-27 is 7% higher than that of U2-23, but accuracy is 16.5% higher than that of U2-23. The U2-27 model has the highest accuracy when abnormity is not too high. On the principle of high accuracy priority, this paper chooses U2-27 as the second-level index set of the model.

Finally, the evaluation outputs of part different distances based on the U2-27 model are shown in Table 7:

Table 7. The evaluation outputs of part different distances
Full size table

As shown in Table 7, it can be observed that the final score is basically positively correlated with the hitting distance. The higher the evaluation score of the swing action, the farther the hitting distance. Considering that factors such as swing force and natural wind speed also affect the hitting distance, this paper does not explore the accurate proportional relationship between swing action and hitting distance.

5 Conclusion

In this paper, to achieve the comprehensive quantitative assessment of sport action, we propose a sport action evaluation method based on human pose estimation. This method requires initial professional analysis by coaches to specify key postures and angle features of the standard sports action. Subsequently, based on human pose estimation, key postures are extracted from test action videos by motion trajectories, and actual angle features are extracted by our proposed angle calculation algorithm. Finally, the actual angle features are input into our proposed action evaluation approach based on FCE to obtain the final score. A dataset comprising 100 golf swing videos labeled with the hitting distances is established as a case study to validate the effectiveness of our proposed method. The case results on golf swing actions indicate a basically proportional relationship between the final action score and the hitting distance, demonstrating the rationality of our approach. The sports action evaluation method presented in this paper is applicable to all sports emphasizing action standardization and proves beneficial for sports teaching.

In sports teaching, this approach can be applied to online learning platforms. For the evaluation of a specific sports action, it only requires inviting professional coaches to specify the key postures of the action, the key angle values corresponding to each key posture, and their respective weight sets. Then, the platform can implement accurate quantitative evaluation of student actions using the above approach, eliminating the need for teachers to assess visually. Moreover, students can engage in sports actions learning at any time, significantly enhancing teaching quality and efficiency.