If you were in charge of creating a perfect judging system...

[irenar5]

What would you do to improve the current judging system?  How can it be made better?

[jbruced]

Get rid of the thing where they randomly drop two of the scores.

[AgnesNitt]

Get rid of the thing where they randomly drop two of the scores.

+1000

I don't like the emphasis on crotch exposing spins and spirals. Yeah a spin where you can wrap your leg around your neck is hard, but the line has to be drawn someplace.  And you would be shocked at how many pervs there are that come to my blog for this one post Safe. I mean during the Olympics that one post was getting a 100 or 200 page views a day. I know there will always be pervs out there, but maybe we shouldn't have aspects of the sport that entices them.

Now that that rant's over, I'd like to say I'm sick of the quad. Yeah, I know it's hard to discriminate the top male skaters, but tough beans. Yeah, the quad should get big points, but a fall should lose a lot bigger than it loses now.

Finally, ISU does a crap job on explaining the system. Unless you're *in* figure skating and you really, really care, no way of figuring it out. And Scott Hamilton's comments never indicate to me that he has a grasp of the new system. Get some commentators in and have them start talking about the scores. Get some youtube videos out there to explain the system. ISU had some good videos that have popped up on skating elements, I haven't seen one for scoring. NBC also did a crap job of talking about skating. Remember the ABC educational minutes about skating during the olympics? I didn't see any this time.

The two heads of the ISU since 1980 have both been speed skaters. If a speed skater is elected to replaced Ottavio Cinquatta when he retires in 2016, figure skating will be dead. It may struggle along, but it.will.die.

Then maybe we can get our sport back and stop licking the boots of the ISU just because of the Olympics.

[eillie]

And you would be shocked at how many pervs there are that come to my blog for this one post Safe. I mean during the Olympics that one post was getting a 100 or 200 page views a day. I know there will always be pervs out there, but maybe we shouldn't have aspects of the sport that entices them.

I had to click out of curiosity and add to the page view count, and the deformed bottoms did amuse greatly!  (Yes, I can be childish.)


But seriously, I can't comment on how to improve the judging system because I really don't know much about it.  It could be because I've never competed, but on the other hand, I've been skating for over 2 years now and spend a sizable chunk of my free time poking around online skating-related sites, and I haven't come across anything more than a table of the different types of jumps listed in order of what their (base?) point value is. 


I've learned a lot more in just the last week though, thanks to the whole Yuna/Adelina controversy ...

[icedancer]

I think it would be great if the quad were not emphasized so much - you can fall on a quad and still get more points than if you land a perfect triple axel.  That is just not right.  The skaters are killing their bodies trying to achieve this jump.

In fact I think a fall on a jump should be 0 points.  Just as though you had not done the element at all.

The system is pretty complex but not unlearn-able (I know that is not a word) - I have a generally good understanding of it - the rules change every year as they continue to tweak it and since I am not a competition judge (as in, I don't judge at Regionals, Sectionals or Nationals, etc., ) I know the IJS on a "need to know" basis - generally it is MUCH easier than the 6.0 system but for those that have been judging with the 6.0 system for decades have a much harder time with it - I started judging just when the 6.0 system was being phased out for major competitions - anyway, I digress...

So I think that the commentators should go to judging school to learn about the system and learn how to explain it concisely so that the general public might understand.

I say might because as I speak with non-skating friends about the skating and the Olympics trying to explain the difference between dance and pairs - I guess the commentators could start there...

[Loops]

I agree with previous posts on crotch visibility and the jumps.  I did not realise that a fall did not negate the element.  It should.  A fallen quad should be worth nothing.  Definitely less than a clean 3A.   I've also been thinking that perhaps putting a cap on the number/percentage of points that can be earned by jumps (and perhaps spins, too???) might be a good idea.  That might take some of the pressure off and we can see more inter-element skating like great foot work, or maybe a 10-second kwan-esque spiral. 

I would like to see skaters held responsible for aesthetics (how, I haven't thought through yet).  Some of you know I loathe and detest the A-frame spin.  I don't care how difficult a position is, if you're sticking your butt (or crotch) in the air that violates standards of aesthetics.  And skating is about the grace and beauty as much as the athletics (right?).   I did notice that at least for the ladies, it was mostly the bottom flight including the A-frame, so that to me says something.

Having said all that, I am torn about limits.  How do we decide and set them?  Julia Lipnitskaya should be rewarded for her flexibility and the work she has done to achieve that. 

Anonymity should be removed, I think it creates more issues than it solves.  We all know some judges are high-scorers and some low-scorers.  That kind of thing should be accepted and OK, but they should be consistent. They don't have to be now.  Drop the high and low scores, but not random scores that someone mentioned in another thread.

I'm still learning the system, but I do feel like it is accessible if one puts in an effort.  I have some more effort to spend, but now that I've one competition under my belt and more to come (so more score sheets to decipher) I feel like I'll get there.

Since the USFSA and ISU seem to be blowing it on outreach/public education-perhaps we could put together a quick and dirty "dummies guide to skating judging." That can be posted on Agnes' and others blogs or somewhere where people who aren't skaters and have no investment in the system outside of the olympics/nationals can get a basic understanding.

[fsk8r]

So I think that the commentators should go to judging school to learn about the system and learn how to explain it concisely so that the general public might understand.

I think that may just be a problem with the US commentators. The BBC commentators were very knowledgeable and explained why jumps were getting downgraded (especially on the replays highlighting what the judges/Tech Specs are looking for) and explained how the spins were getting the levels. They didn't go into why one step sequence was harder than another, but given that the general public goes jump, spin ooh pretty, they aren't ready for step sequence nuances.
But having commentators who work with the judging system means that there's more acceptance of it over here.
But the UK has also switched to IJS based testing and IJS competitions for all levels. The system is far from perfect (-1 for a fall is a very large percentage of my score when I'm not getting much more than mid teens and it's peanuts for someone getting ten times that), but I think there are a lot of positives.
Most sports with subjective judging throw out the highest and lowest scores, so I'm quite happy to keep doing that as it prevents bias having an effect on the results. The annonymous judging however does no one any favours.

And I think they need to have a serious discussion at the ISU congress about speed skating. You can't penalise one athlete for doing something in one race and then penalise the one being affected in the second race for near identical incidents. But I doubt anyone in the US has heard what has happening to one of the British speed skaters. And there's no right of appeal of why the referee made the decision.

[PhysicsOnIce]

Personally, I agree with almost everything that has been said.
I am getting really tired of the "lets reward jumping jelly beans" mentality, and really missing the artistry that was skating back in the 6.0 system. Okay I agree the 6.0 system was not as objective as the new IJS but skating is turning into a fight for points with very little rewards for a clean beautifully skated program. I have always preferred a clean simple program than an over the top program full of mistakes. In my little world, which I think most of you will agree, the a clean quad should be very well rewarded but a missed jump (whether or not it is a quad) should be either not counted at all of severally downgraded. Maybe this is a bit of the statistician in me coming out, but falls should reduce the value of the jump to 10-20% of the base value and the percentage should be fixed. So, 10% of a single flip is 0.05  but 10% of a quad sal is 1.05. That way skaters trying a quad do get something, but it is still significantly comparable to a clean double toe or double sal. Okay maybe 10% is too little, but the idea holds.

Also, from a statistical point of view throwing out the low and high scores only makes sense if they are at least one sigma away from the mean value, and I do think that scores should be normalized to the mean value. A simple summation does not help if you are throwing out scores randomly. Skater with all relatively high scores is actually being punished. 

As for spins, I want spins to be rewarded for class and style not flexibility alone. That being said, if you are Lipnitskaya and can keep your rotation speed up, while getting into beautiful positions, then go for it. But if you are struggling to get into nice positions, losing your speed and grace, DON'T try it.

[jbruced]

Even in my lowly skating mind I'm sensing from this thread that there are some simple tweaks that can be made to the current scoring system. Then I wonder if anyone from the ISU ever reads any of these message boards and the many blogs that make similar comments?

[JSM]

While there is no perfect judging system, I do think falls should be penalized differently than they are.

At my level (single and double jumps), falling on a jump essentially means you have negative points for that element.  Between lost points in GOE and the 1 point deduction, you'd be better off not trying the jump at all.  A very severe penalty - I'd rather see the jump be worth 0.

However, at the elite level, a 1 point deduction per fall, when scores are over 100 points, seems like it's not severe enough. 

The REAL incentive for the quad, IMO, is the HUGE increase in base value that it gives you.  Because competitors are limited to repeating only 2 triple jumps in the program, adding a quad, even one that you fall on, raises that BV significantly.  If a program without a quad repeats the axel and lutz, that means you can only do 1 flip, loop, salchow, and toeloop.  Then you are filling in the rest of your jumping passes with double axels, which has a BV of 3.3 points.

A quad toe loop has a value of 10.3 points.  Fall on it, loose 3 points in GOE plus 1 point for the fall, you are still at at 6.3 points.  A bit less than a 3A (8.5), but more then any other triple.  It's a no brainer.

I'd like fall deductions to be proportional to the value of the jump, I'm at a loss of how much though.  Or, maybe, there should be an automatic deduction in PCS due to falls instead?  They do really disrupt the flow of a program.

[fsk8r]

If you were wanting to take a fall deduction as a percentage of the base mark of a jump, why not make it like the underrotation downgrade. And then we'd need to have another fall deduction for when you trip over your feet.
It's quite difficult to come up with a deduction which is fair at the bottom end of the sport (doing single jumps) and at the top end (doing quads).

[Loops]

Well, if we make the fall be a percentage of the jump, that could help, but still would penalize more at the lower levels than the elite.  Would it be better to just have the missed jump count as 0 (since, heck, they didn't actually "do" it anyway),  Thereby lowering the base value of the program?  PLUS there can be some dinging for the actual fall in the artistic section?  Or would that count as double-jeopardy?  I like the idea of considering the under-rotated value.  How would that work? If you underrotate a 3A, is it then counted as a 2A?  How would that be scored (assuming no fall)?

In terms of what PhysicsOnIce said about throwing out highs and lows......would you recommend then just taking a basic average of the scores?  Do we know how often the highs and lows are greater than one sigma value away?  My stats are rusty, but sigma is standard deviation, right? 

[icedancer]

A quad toe loop has a value of 10.3 points.  Fall on it, loose 3 points in GOE plus 1 point for the fall, you are still at at 6.3 points.  A bit less than a 3A (8.5), but more then any other triple.  It's a no brainer.

This is correct - I said in my original post on this topic that a fall on a quad was more points than a triple axel - I meant it was more than some of the triple jumps.

As far as anonymity goes I think most people have forgotten that the reason for the anonymity was that a judge at the 2002 SLC Olympics got a death threat - I believe it was a Canadian judge.  As a judge I can see where one might want to be anonymous if that is what was happening!

[techskater]

Get rid of the thing where they randomly drop two of the scores.

That is long gone.  They drop the highest and lowest (trimmed mean method)

[Query]

Make it like a reality TV series. Viewers call or add their votes by computer. Pay $1/vote.



It is objective in that the system is completely open!!! It's the ultimate free market approach to judging figure skating.



This time around, Russia spent about $50 billion to bring the Olympics to Russia. They could have spent a few billion outspending the rest of us on votes too. So the Russian skaters would still have done well.



(Please don't take this suggestion too seriously.)

For those of you who think that figure skating is more subjective than, say, hockey - did you watch the hockey? The referees have a lot of power. A lot of penalties weren't called that could have been, perhaps because the players were too sneaky to get caught. Plus, the referees are actively involved in play, in that players often direct the puck at the referees (probably because none of the players wants to knock down the referee, so bouncing it off the referee limits interceptions), or through the referees' legs, and the referees often get in the way. Mostly not on purpose, but any referee who wanted to influence the results, could.

[techskater]

Changes:
1) find a method of judging the PCS mark so that the whole >> than the sum of parts.  Basically, the sum of parts under IJS >> than the whole.  I don't know how to fix this in the current system, this is what 6.0 was good for
2) spread the wealth of the tech panel.  In the ladies event at the most recent Olympics, Alexander Lakernik (RUS) was the TC and Olga Baranova (FIN  ) was the ATS.  Olga Baranova is orignally from RUS (and, according to some stories, was Sotnikova's coach when she was younger).  I think whoever the expected top 5 countries are in an event should have 3/5 of the tech panel from random draw and a fourth the data operator.  This spreads the wealth/responsibility/politiking around and doesn't give power to just one country (where Lakernik and Baranova could outvote Guesmeroli on reviews, even though one was "Finnish").  That puts double pressure on your skaters to do well to get a shot at a tech panel from your country
3) get rid of anonymous judging
4) if we are to maintain this current system, release the video the tech panel saw for reviews and release the audio from the tech panel at major events (so that there are less calls of fixed, corrupt, consipiracy...) so people can listen to the calls for a specific skater if there were questions on why was so and so not called/called...
5) Any judge caught cheating gets a lifetime ban
6) Judges should never be related to a high ranking official...

These are things just off the top of my head

[Nate]

The quad is just another jump. It just is worth a ton. The only emphasis is on how many points you leave on the table by doing a triple loop as your last triple instead of a quad toe early on and leaving the loop out.

No commentator has enough time to explain the scoring system, period. You cannot explain components scoring and even if they did reputation judging etc. would still leave the spectator scratching his or her heads.

Sent from my Galaxy Note 3 using Tapatalk

[Nate]

While there is no perfect judging system, I do think falls should be penalized differently than they are.

At my level (single and double jumps), falling on a jump essentially means you have negative points for that element.  Between lost points in GOE and the 1 point deduction, you'd be better off not trying the jump at all.  A very severe penalty - I'd rather see the jump be worth 0.

However, at the elite level, a 1 point deduction per fall, when scores are over 100 points, seems like it's not severe enough. 

The REAL incentive for the quad, IMO, is the HUGE increase in base value that it gives you.  Because competitors are limited to repeating only 2 triple jumps in the program, adding a quad, even one that you fall on, raises that BV significantly.  If a program without a quad repeats the axel and lutz, that means you can only do 1 flip, loop, salchow, and toeloop.  Then you are filling in the rest of your jumping passes with double axels, which has a BV of 3.3 points.

A quad toe loop has a value of 10.3 points.  Fall on it, loose 3 points in GOE plus 1 point for the fall, you are still at at 6.3 points.  A bit less than a 3A (8.5), but more then any other triple.  It's a no brainer.

I'd like fall deductions to be proportional to the value of the jump, I'm at a loss of how much though.  Or, maybe, there should be an automatic deduction in PCS due to falls instead?  They do really disrupt the flow of a program.

GOE is factored based on the value of the element. It's why hitting levels is important, it also affects the GOE factoring for the element. You cannot get negative points for an element you fall on. You can only get zero if it's an invalid element.

Your base value explanation is spot on however.

If you judge by the criteria and leave reputation out then I don't find much to complain about the ladies results which are likely the catalyst for this thread.

I think the unspoken system of paying your dues does more harm than the IJS.

Sent from my Galaxy Note 3 using Tapatalk

[JSM]

GOE is factored based on the value of the element. It's why hitting levels is important, it also affects the GOE factoring for the element. You cannot get negative points for an element you fall on. You can only get zero if it's an invalid element.

I know.  But, after losing points in GOE AND having the additional 1 point deduction for a fall, it's like getting negative points for the element.  Example (from a competition some time ago), I fully rotated but fell on a 2T.  BV is 1.3.  Lost .55 points in GOE, plus had the 1 point deduction for a fall.  Therefore, though I got .75 points for the element, with the fall deduction it was -.25 points.  Should have done a single!! 

What I meant was that I'd rather have the fall deduction be lower on elements that are not worth a lot of points.  We're not elite skaters at my level, so 1 point is a LOT!

[PhysicsOnIce]

Well, if we make the fall be a percentage of the jump, that could help, but still would penalize more at the lower levels than the elite.  Would it be better to just have the missed jump count as 0 (since, heck, they didn't actually "do" it anyway),  Thereby lowering the base value of the program?  PLUS there can be some dinging for the actual fall in the artistic section?  Or would that count as double-jeopardy?  I like the idea of considering the under-rotated value.  How would that work? If you underrotate a 3A, is it then counted as a 2A?  How would that be scored (assuming no fall)?

In terms of what PhysicsOnIce said about throwing out highs and lows......would you recommend then just taking a basic average of the scores?  Do we know how often the highs and lows are greater than one sigma value away?  My stats are rusty, but sigma is standard deviation, right?

Yes a Sigma is a standard deviation. In Science we normally use a 3 sigma clipping to exclude results which have a a high probability of being outliers.  Ie in this case your distribution would be the range of scores and would be described by the mean of the scores and the standard deviation.

My biggest problem with the Trimmed Mean value is that it produces bias results if the distribution is not symmetric. lets say for  example an element you have (8.9,9.0,9.2,9.2,9.2,9.2) that is clearly an not symmetric distribution of scores. The trimmed mean as it is described in the IJS simply lowest and highest goes one value of the 9.2 and the 8.9 would disappear. so your distribution is 9.0,9.2,9.2, 9.2 , which is heavily biased toward 9.2 so your mean value would be 9.15.  If we for example take the whole range and calculate the standard deviation and do a 1 sigma (which would be 0.13) clipping anything between 8.98 - 9.24 would be taken  since the mean value would be 9.11  or if we do a 3 sigma anything between    8.7 - 9.4. Anything outside of that range would be thrown out. You have to be a bit careful with how you define the mean and the sigma since you do include all numbers in the definition.

I think it the best way of implimenting it would be something like the TS defines the "mean value" ie the expected Base Value and GoE and all scores inside the 1 sigma  or 3 sigma level are considered valid, where the expected distribution is simply the standard normal distribution around the mean value defined by the TS.

The other thing is considering not the Mean but the Median, after all it is more a robust estimator, or maybe a weighted mean (which would penalize scores by being far away from the expected mean value, by simply not counting them complete).

As for knowing when things are further than a sigma away, it is nearly impossible from a spectarors point of view now days. Back in the 6.0 system you could very quickly tell( I keep referring to the 6.0 system because I missed everything in between), but know that things are quickly added together and presented as one number it is impossible to know. You can find it on the program breakdowns when the are online.

Okay, I know that it would become horribly hard to explain that to spectarors.

[irenar5]

Changes:
1) find a method of judging the PCS mark so that the whole >> than the sum of parts.  Basically, the sum of parts under IJS >> than the whole.  I don't know how to fix this in the current system, this is what 6.0 was good for
2) spread the wealth of the tech panel.  In the ladies event at the most recent Olympics, Alexander Lakernik (RUS) was the TC and Olga Baranova (FIN  ) was the ATS.  Olga Baranova is orignally from RUS (and, according to some stories, was Sotnikova's coach when she was younger).  I think whoever the expected top 5 countries are in an event should have 3/5 of the tech panel from random draw and a fourth the data operator.  This spreads the wealth/responsibility/politiking around and doesn't give power to just one country (where Lakernik and Baranova could outvote Guesmeroli on reviews, even though one was "Finnish").  That puts double pressure on your skaters to do well to get a shot at a tech panel from your country
3) get rid of anonymous judging
4) if we are to maintain this current system, release the video the tech panel saw for reviews and release the audio from the tech panel at major events (so that there are less calls of fixed, corrupt, consipiracy...) so people can listen to the calls for a specific skater if there were questions on why was so and so not called/called...
5) Any judge caught cheating gets a lifetime ban
6) Judges should never be related to a high ranking official...

These are things just off the top of my head

Amen!

[Doubletoe]

I would start with the current system but get rid of judges' anonymity and impose much stricter rules and penalties for conflicts of interest and collusion, including the number of judges who can be from any one country on any panel (and definitely only one person from any given country on the technical panel).
I would also implement a process whereby judges' marks and the technical controller's calls could be challenged by any skater in that competition or any other judge.  In my proposed process, an ISU panel would be asked to review that judge's GOE marks and PCS marks for certain skaters and the judge would have to explain them.  If the GOE marks did not follow the ISU guidelines (which are pretty clear), the judge's scores would be thrown out for that competition and he/she would be given a warning.  3 warnings and the judge would be suspended.  Same for the technical controller.

[AgnesNitt]

Given the insane 'passion' some skating fan have, I'm not so sure about non-anonymous judging. People get death threats for Tweets someone disagrees with. Imagine some 'fan' who sees their country 'disrespected' at the olympics stalking and killing a judge.
No, anonymity is not that important to me.

[Doubletoe]

Given the insane 'passion' some skating fan have, I'm not so sure about non-anonymous judging. People get death threats for Tweets someone disagrees with. Imagine some 'fan' who sees their country 'disrespected' at the olympics stalking and killing a judge.
No, anonymity is not that important to me.

People can already see who is on the judging panel and can pretty much guess which judges would favor skaters from which countries (usually their own).  Other sports do not have anonymous judging and figure skating didn't used to.  How many judges were ever actually killed before anonymous judging?  How about in sports that are not anonymously judged?  Judges should have to be accountable for the scores they give.

[AgnesNitt]

People can already see who is on the judging panel and can pretty much guess which judges would favor skaters from which countries (usually their own).  Other sports do not have anonymous judging and figure skating didn't used to.  How many judges were ever actually killed before anonymous judging?  How about in sports that are not anonymously judged?  Judges should have to be accountable for the scores they give.

Non-anonymous judging started in 2004, right?
Twitter started 2006. Reddit 2006. 4chan 2003.
Sample Twitter death threats.

I can find examples death threat examples on Reddit, 4chan, other places. This isn't the world it was in the 6.0 days. It's not even the world it was in 2002. I think from a risk mitigation perspective anonymity  for judges is not a bad idea.

As far as anonymity goes I think most people have forgotten that the reason for the anonymity was that a judge at the 2002 SLC Olympics got a death threat - I believe it was a Canadian judge.  As a judge I can see where one might want to be anonymous if that is what was happening!

The problem with ISU is it has been run for speed skating by speed skaters since 1980. It historically has not dealt with investigations of figure skating malfeasance with any dispatch-- or at all. Ottavio Cinqatta doesn't know about figure skating, and given his lack of response personally, or the ISU's inability to respond to accusations in a speedy manner,  he doesn't care to bother. ISU uses figure skating to fund speed skating.