XBox Kinect for teaching on-ice skills

[lindafmb]

I mentioned in another post that I'm working on an off-ice app for the XBox Kinect, based on some of the stuff I do for strength and conditioning for myself and with clients, as well as what some of our coaches teach in off-ice jump class. When I was talking with my husband last night about my testing experience, he said it was too bad that skating couldn't be scored more accurately based on biomechanics. And that got me thinking...you CAN use the Kinect system with a laptop too, meaning that, at least theoretcially, you can use it ON THE ICE if you have access to the Kinect system/camera and a laptop. Don't know how many of you may be familiar with the biomechanics program and setup at the University of Delaware. I've only seen it at presentations at Nationals and when Adam Rippon was rigged up with it for some jump inconsistency issues he was having. The setup with the sensors seems a bit cumbersome, and from what I've seen it seems like that itself may throw off especially an elite skater with finely honed technique.

In contrast, the Kinect doesn't use on-body sensors; instead, the camera visually "maps" you and matches you relative to your own segment lengths with a model (I've been using Evgeni Plushenko as the model for jumps, since I have a ton of footage of him from my MS days, and hey, that dude is a jumping MACHINE when it comes to his biomechanical consistency). So far, I can't think of any reason why (1) this couldn't work on the ice, and (2) more coaches wouldn't want want to use such a system, since it's pretty uncumbersome, can be relocated to ANY area of the ice/rink, and only requires a Kinect and a laptop, rather than a huge, in-house system and an awkward body sensor suit.

What do others think? Especially coaches and those with an interest in biomechanics?

L

[icefrog]

Isn't that what Dartfish is? I think all you need is a laptop, camera, and the program. A coach at my rink uses it, but I've never taken from him and done it.

[lindafmb]

Nope, Dartfish is more of a video capture/overlay type tool so that you can see your own technique relative to yourself at other instances or relative to a video of someone else. There's a free version of a very similar tool, but the name of that escapes me at the moment. Both are good for reviewing video. Just recording video and reviewing it is good for that purpose too, especially if the coach has a good understanding of biomechanics.

What Kinect does is a bit different. It captures a visual image of you, and then that image adjusts with your movement. Essentially, your body becomes a controller that feeds information into the computer. The computer then can perform VISUAL matching to sample biomechanical statistics and immediately feedback how accurate each segment of your body is in achieving the desired skill. Your body is represented biomechanically segmented, and shows how close you are to matching a model performing the skill accurately--and here's the important part--as the skater is moving, so that you can even watch (you the coach, probably not the skater, unless it's something where they can watch their own image, like a dancer would watch herself in a mirror) and make adjustments on the fly, or through review.

In this example, the skills being taught are simple t'ai chi movements. Notice what component is scored, and the degree of accuracy. The programs I've developed are a little more sophisticated in terms of the speed of capture and complexity of movement, and the biomechanical representation looks a lot more like the person who's being captured than the "shadow guy" shown here. But it gives you a pretty good idea of how Kinect can be used to teach sophisticated movement skills:

http://www.youtube.com/watch?v=UXhoh2yE6bI

I'd want to be able to feed in your own models as well, so that it would be possible to review entire programs and elements specifically in your own programs.

To me, the off-ice application is a total no-brianer, because you can do that at home in front of a big screen TV. But when my husband pointed out the value of using Kinect on-ice, I selfishly thought about how much that would help ME out personally 

[jjane45]

Wow very interesting, thank you for sharing your expertise. When someone's multi-rotational jump is compared to Plushenko's, does the program adjust for difference in rotation speed?

(I know nothing about Dartfish / Kinect, guessing that jump height is not too relevant and and number of rotations can be matched...?)

[lindafmb]

Wow very interesting, thank you for sharing your expertise. When someone's multi-rotational jump is compared to Plushenko's, does the program adjust for difference in rotation speed?

It would actually SHOW and could TELL (audibly and/or on-screen text) you the difference in every dimension of movement--speed, trajectory, vertical, horizontal, and rotational changes in velocity, and so forth--on all aspects of the jump--preparation, take-off, flight position, and landing. The technology is very cool and can do a lot, that's why I'm so excited about the possibilities. 

(I know nothing about Dartfish / Kinect, guessing that jump height is not too relevant and and number of rotations can be matched...?)

Oh, it's relevant, that's why I think having the capability to input your own data in addition to loaded models is an important feature. So for a complete app, I'd need to have models doing accurate representations of ALL jumps (single, doubles, triples, quads), spins, footwork, turns, artistic movements etc.

Here's a link to some info on Dartfish, imho also an excellent biomechanical analysis tool (I've used it pretty extensively): http://www.dartfish.com/en/software/index.htm

[Query]

I think it would be a fun toy!

See if I have this right: Kinect has a very short range (e.g., 10-15'?), so it cannot trace a program on the ice?

----

Based on the resolution you show, it would be hard to do the full equivalent of Dartfish.

There are better ways to do this, with wearable arrays of coded transponders, but you might have to engineer your own hardware. I wish I knew enough electrical engineering to do it.

In another thread, I talked about using something along this line with 3D goggles (semi-transparent, so you don't skate into other people!), so you could see yourself moving inside the image of someone doing it "right".

With 3D goggles, you can see in real time how your motion differs from that of the model - you don't even need Kinect or equivalent. But to compare your motions to the model after the fact, you do need Kinect or equivalent.

Skating style has so many different "right" ways to move, that it could be confusing if you have  learned/are being taught by your coach to move one way, and the demo uses another. There have been complaints about that with Dartfish.

But I still think it would be neat. Expensive, though.

[lindafmb]

I think it would be a fun toy!

See if I have this right: Kinect has a very short range (e.g., 10-15'?), so it cannot trace a program on the ice?

This is one of the aspects (the range of the capture area) I've been working on with the developer's kit, so that it can capture the entire rink, have zoom capability, etc. BTW, Dartfish was built with Silverlight technology, also another Microsoft product, but one that's meant to handle video primarily.

I did a fair amount of VR (virtual reality) work back in the day too. It is the googles and sensors and capture apparatus that I've always found to be the issue with most biomechanical analysis tools, and it was actually the set-up I saw, and the evolution of it at U of Delaware, that got me thinking of "the body as the controller." The encoders do tend to throw the athlete off or impede movement for various reasons. That's why the Kinect has such an advantage, provided that the content can be captured appropriately in space (the range, as you mentioned, being the main issue for on-ice application for say, a complete program).

The issue with what's consider the "right way" or having many "right ways" is not necessarily something a software program with one "correct" model can accomplish. That's why having the added functionality of using your own content as a model makes the app even more valuable.

Thanks for your thoughts. Makes me think and compare technologies and capabilities, which is always helpful.
L

[Query]

Uses multiple cameras to detect dots on clothing might well be the best way to do this, because you don't need any transponders or transmitters. Just bright reflective dots with unique color signatures. If there aren't any patents on it, it could be very cheap, except for the cameras, and use off-the-shelf hardware. (Maybe it doesn't even need an engineer.)

Is that what Dartfish uses?

I bet the U of Delaware has a real fun lab.

[JimStanmore]

Kinovea is one open source motion analysis program.  It allows comparisons between two videos and trajectory tracking and I consider it similar to Dartfish in many ways.  i use it to slow motion analyze skating moves.  I last used it to correct falling over on my left inside three turn (found my hip popping out of the circle at the last moment) and to dissect the exact positions of a Lambiel footwork sequence I want to eventually master.

The one I really like for motion analysis is Video4Coach SkillSpector/SkillCapture (freeware) and it is similar to the full fledged university software.  I take video of a move or jump and load it into the software.  Then I identify the joints on my body in the first frame (the program sequences you through head, shoulder, elbow, hand, hip, knee, ankle down one side and up the other.)  Then you click through the frames.  The software predicts where the joints have moved and you can fine tune the guess in each frame in case it is wrong.

After you are through, you have a wire frame figure that you can observe through the maneuver.  I find it very useful for tracking the motion of particular body parts.  So, it is just like the sensor version, but I spend about 15 minutes of data entry for a 10 second move - very cost effective for me (free.)  In effect, it does what you wanted by evaluating the body based on the initial digitizing.  It doesn't generate any comments on its own, though.  Keep in mind that a triple rotation jump requires one second in the air so, a complete jump may be about 3 or 4 seconds tops.  I have wireframed several Olympic competitors to use to correct my body positions and timing.

With SkillCapture the camera can automatically start when the skater enters the frame and stop after a predetermined number of seconds.  Then it will allow immediate playback in slow motion.  To wireframe, you have to identify the joints just like an imported video.