Interesting Article re: IJS

[fsk8r]

Unfortunately, most of the commentators are from the 6.0 era and still don't really have a grasp of the IJS rules.  I would personally love to have a feed of the technical specialist calling each element for the judges, but then I'm a nerd like that. 

I think this is where British Eurosport has the advantage. Chris Howarth spends most of the year coaching so has to know the IJS rules.
He might not be a technical specialist but as a coach needs to know how to maximise points.
But it's hard for them to be knowledgeable about all the disciplines.

[Doubletoe]

I think this is where British Eurosport has the advantage. Chris Howarth spends most of the year coaching so has to know the IJS rules.
He might not be a technical specialist but as a coach needs to know how to maximise points.
But it's hard for them to be knowledgeable about all the disciplines.

I agree that British Eurosport commentating is less annoying than most.  Maybe that's why!  Having said that, it's amazing how many coaches fail to keep up with the IJS rules!  Neither my current coach nor my previous coach have managed to keep up with the rules, and I'm sure would have been clueless as to why Patrick Chan only got a level 1 on the final spin in his Skate Canada freeskate (answer: He did not get 2 revolutions in sit position and the skater needs to have 2 revolutions in all 3 basic positions--camel, sit and upright--in order to get a level 2 or higher on a change-foot combination spin).

[fsk8r]

Why aren't coaches keeping up with the rules?
All the British coaches are now having to swot up as the majority of competitions are now IJS from the first test level up. The only level on the old system is Beginners.
While the ones with skaters at Novice and higher are completely on top of the rules, the rest need to understand enough to increase spin levels and improve GOE on jumps.

[taka]

I'm surprised coaches aren't keeping up with IJS rules! Surely most competitions in most places are run under IJS now? Almost all competitions are IJS (beginner level is still RJS) in the UK so coaches who coach anything above SkateUK need to know at least the basics appropriate for their levels of skaters. I presumed the same would hold true in other parts of the world too!

Even our test system is in the process of being updated to IJS too. The 1st step is moving to an IJS style GOE marking (+3 to -3) on test elements along with scoring program components too (Poor/Weak/Acceptable/Good/Excellent). Apparently at some point down the line there will be a move to passing NISA level tests in competition for all levels of skater, not just the competitive test passes for Advanced novice / Junior / Senior equivalent levels we have currently.

[FigureSpins]

Why would you assume that "everyone" is using IJS throughout their structure?  Absolutely untrue in the US.  Here, the standard track free skate events from Juvenile.  The Adult Gold/Adult Jr-Sr free skate events are supposed to be IJS as well.  However, the Basic Skills/No Test/Pre-Prel through Pre-Juv/Adult Pre-Bronze through Silver events are judged under 6.0 using ordinals.  All the freestyle-related events (jump, spin, compulsories, artistic, showcase, etc.) are under 6.0.  (Pairs is structured similarly; I'm not sure when IJS is used for Ice Dance; I assume the higher levels as well.)

US non-qualifying Club competitions have the option of using IJS for all levels, but very few make that choice, afaik.  I know that several clubs on the other side of the country do so; Sherwood Invitational is one.

The full-blown US IJS system is expensive to use for competitions because of the technology and additional trained volunteers needed.  The paper-based "Modified IJS" would be good to use in lieu of 6.0 scoring, imo.  Ordinals are meaningless and often swing widely from competition to competition.    I'd rather the skaters get a solid numeric score so they can set goals and strive for "personal bests" before the next event.  It would be wonderful to get an element-by-element plus overall score, like the old ISI results, but I suspect it's too time-consuming. 

The USFSA Continuing Education Requirements (CER) are required for any coach presenting students for tests or competitions.  There are two tiers: CER-B for those whose skaters are not competing in qualifying events and CER-A for those with qualifying-event skaters.  The difference is simply that the qualifying coaches have to take an IJS course.  There are a number to choose from, including "Current Season Singles Skating," which outlines things like spins with 3 revs in position, difficult entries, step sequence vs. chore sequence, etc. 

I don't have any skaters at the qualifying levels, but I comply with CER-A each year to pick up additional knowledge for the future, even though it costs an extra $25.  I bring that to my lower-level skaters since many judges have indicated that they use the IJS standards to judge tests and 6.0 events. 

One note: just because a skater doesn't do three revs in each spin position doesn't indicate the coach was ignorant or the choregrapher blew it - sometimes, it's the skater's fault.  You have to know that the elite skaters at the ISU Senior Grand Prix know they're supposed to do something.  There are any number of reasons why they give away points.

[sarahspins]

IJS is eventually coming to ALL levels, right down to snowplow 1.

[FigureSpins]

When?

[Query]

Just go to www.isu.org and find the event (there will be a link on the main page if it's a major event), click on that, then click on the link for "Starting Orders/Detailed Results" and then "Judges' Scores". 

Thanks! That's a lot of what I wanted to see.

I am surprised to find that judges typically vary in their GOE scores for the same move by most of the available range. [I'm ignoring component factors because I don't understand them at all.] I thought the whole point of judge's training was to get the judges to the point of scoring people almost identically, especially by the time judges are working at the highest levels. Apparently such a goal is not met.

Of course, given that the score differential between judges is so large, what I wanted to see - a step by step explanation of how the scores for each skating performance move are arrived at - is impractical. You can't explain what the score should be, point by point, if the score assigned varies judge-by-judge by most of the available range.

And the fact that they aren't giving each score by named judge, but are reported in random order, means you can't see the biases of each judge. One of the major goals of IJS was to eliminate the perception that judges from one country might be tempted to favor their own skaters. What the random reporting order does instead is to hide any existing bias. Are each judge's scores reported by judge name anywhere?

I get that some aspects of figure skating have to be subjective, but I am more convinced than ever that figure skating at the upper levels should have professional judges, subject to strong peer review, with severe penalties for excess score deviation or signs of bias.

Edit: Obviously that would be a very painful standard to hold amateur judges to who have to pay for their own training, but it is a standard to which professional referees are held to in many sports.
[Obviously the technical specialist is subject to individual scrutiny.]

[fsk8r]

I was timekeeper at an IJS adult competition and sat in the middle of the panel. The judges were split either side of me. At one point when the referee was signing off the score sheets, the judge sitting next to her commented on how the GOE varied from one side of the panel to the other and how she'd not realised the angle could make such a difference.

While we'd like the scores to be goals to improve upon, there's still an element of human judgement so GOE will vary from one judge to another. As long as the judges are consistent across the field that variation will come out in the wash.

I do however agree that judges shouldn't be anonymous. There's more accountability when people are identified.
But the judges do have review meetings after competitions where they need to justify if they deviate to far from the norm.

It is interesting that the US still hasn't switched to IJS. It is a lot more expensive for clubs to run the competitions and you do need more officials, but most club competitions in Europe seem to be IJS and that has driven the UK to switch. And we're always under the constant impression the whole world has switched and we're slow.

[Doubletoe]

I am surprised to find that judges typically vary in their GOE scores for the same move by most of the available range. [I'm ignoring component factors because I don't understand them at all.] I thought the whole point of judge's training was to get the judges to the point of scoring people almost identically, especially by the time judges are working at the highest levels. Apparently such a goal is not met.

Of course, given that the score differential between judges is so large, what I wanted to see - a step by step explanation of how the scores for each skating performance move are arrived at - is impractical. You can't explain what the score should be, point by point, if the score assigned varies judge-by-judge by most of the available range.

And the fact that they aren't giving each score by named judge, but are reported in random order, means you can't see the biases of each judge. One of the major goals of IJS was to eliminate the perception that judges from one country might be tempted to favor their own skaters. What the random reporting order does instead is to hide any existing bias. Are each judge's scores reported by judge name anywhere?

I was timekeeper at an IJS adult competition and sat in the middle of the panel. The judges were split either side of me. At one point when the referee was signing off the score sheets, the judge sitting next to her commented on how the GOE varied from one side of the panel to the other and how she'd not realised the angle could make such a difference.

While we'd like the scores to be goals to improve upon, there's still an element of human judgement so GOE will vary from one judge to another. As long as the judges are consistent across the field that variation will come out in the wash.

I do however agree that judges shouldn't be anonymous. There's more accountability when people are identified.
But the judges do have review meetings after competitions where they need to justify if they deviate to far from the norm.

It is interesting that the US still hasn't switched to IJS. It is a lot more expensive for clubs to run the competitions and you do need more officials, but most club competitions in Europe seem to be IJS and that has driven the UK to switch. And we're always under the constant impression the whole world has switched and we're slow.

It's absolutely true that different judges have different viewing angles and some might catch under-rotations or edge changes while others won't.  Here are a few other things to keep in mind before crying bias:
 
(1) Different judges value different things; some value jump height and distance, others value clean landings, others value clean takeoffs, others value air position, etc.  When you have 9 judges, those different emphases tend to balance out, which is exactly why you have 9 judges.
(2) The highest and lowest scores are dropped, so being the one judge who gives a -3 or +3 on an element won't get you anywhere.  The same is true for PCS.
(3) You can't just look at the scores for one skater and decide that the judge who gave the lowest marks or highest marks for that skater was biased.  In most cases, judges are very consistent with the way they score skaters and either score everyone leniently or everyone strictly.  You have to look at that judge's scores across the entire flight of skaters.

Judges are not anonymous at U.S.F.S.A. competitions.  The main scoresheet has every judge listed (with the number) and they are also announced by name and number during the 6-minute warmup at the competition.  There's one judge in my area whose name I don't need to look for on my scoresheet.  If I see a string of negative GOE's down one of the judging columns on the scoresheets, I know what number judge he was, just because he is notoriously strict and always gives the lowest scores.  Sometimes I wonder if it bothers him that his scores are always the ones that get dropped, LOL!

In U.S. adult competitions, Adult Gold and above (i.e., skaters who are required to land a single axel on the test) are all judged on IJS.  In addition to cost, the other reason to keep the lower levels on 6.0 is that so many skaters are under-rotating single jumps, not getting enough revolutions in position on spins, etc., that it makes it tough to judge (and tough on the skaters, too).

[fsk8r]

In U.S. adult competitions, Adult Gold and above (i.e., skaters who are required to land a single axel on the test) are all judged on IJS.  In addition to cost, the other reason to keep the lower levels on 6.0 is that so many skaters are under-rotating single jumps, not getting enough revolutions in position on spins, etc., that it makes it tough to judge (and tough on the skaters, too).

We had this argument in the UK as well about the under-rotation and the spin positions. The first year at the British Adults the only things being called were step sequences so it was all down to PCS to sort out placement. Now adults are getting low enough on sit spins and hold the legs high enough on camel spins. And under-rotation is being reduced as well. It helps that the Technical Specialists are getting more practice calling lower level elements, but it also helps that we're all aiming to get the calls so are working hard to fix things.
There's no incentive to fix your jump rotation unless you can see if having a clear impact on your score.

[Doubletoe]

We had this argument in the UK as well about the under-rotation and the spin positions. The first year at the British Adults the only things being called were step sequences so it was all down to PCS to sort out placement. Now adults are getting low enough on sit spins and hold the legs high enough on camel spins. And under-rotation is being reduced as well. It helps that the Technical Specialists are getting more practice calling lower level elements, but it also helps that we're all aiming to get the calls so are working hard to fix things.
There's no incentive to fix your jump rotation unless you can see if having a clear impact on your score.

You have a good argument.

[Query]

1. I think it is good that USFSA judges are not anonymous.

But an obvious source of bias is Nationalism - though judges who know individual skaters might also be biased. So it is the ISU more than the USFSA that really needs to be open.

Perhaps the ISU decision makers felt that anonymous judges would be less biased, because the national skating associations couldn't penalize the judges for being unbiased. I seem to recall that a Soviet judge made that accusation against the Russian Skating Federation just before the ISU went to IJS, and that it caused the scandal that drove the ISU to change to IJS. (I wonder if it is possible for the national skating organizations to use cameras or binoculars to capture the judges' score sheets.)

But the assumption that openness causes more corruption than secrecy isn't necessarily correct. That certainly isn't usually the assumption in politics. A more obvious solution would have been to remove the judges' affiliations with National associations. Something that a separate professional corp organized by the ISU could perhaps do.

2. As to the idea that the spread in scores is unimportant if the individual judges are consistent - that can't be evaluated if you can't figure out which scores belong to the same judge.

3. I buy the idea that viewing angle makes a big difference. But by the Grand Prix competition level, don't the judges use video replays?

4. Oh well. I guess there is no more potential for unfair bias than in reality TV shows where the public votes.

[Doubletoe]

2. As to the idea that the spread in scores is unimportant if the individual judges are consistent - that can't be evaluated if you can't figure out which scores belong to the same judge.

You CAN figure out which scores belong to the same judge, since each judge has a number that is used consistently throughout the same competition.  The Judge #3 whose scores show up under the Judge #3 column on the scoresheet for skater #1 is the same Judge #3 on the scoresheet for skater #12.

[Query]

Oh. 

[Sk8tmum]

Judges are also evaluated regularly for consistency. There is a "corridor" I.e. A range of +/- GOE and a judge's scores are checked to see if they're within the corridor. If they're out of kilter it is addressed ... the goal is consistency always. Frankly, I have seen improvements with judging consistency and more importantly with the tech specialist calling of the elements as we've moved further into IJS. An inaccurate tech specialist is a bigger factor than an individual judge.

Tech specialists use video replays. Judges don't.

As for consistency... I remember very clearly the inconsistency in OBO scores in comps...

[aussieskater]

You CAN figure out which scores belong to the same judge, since each judge has a number that is used consistently throughout the same competition.  The Judge #3 whose scores show up under the Judge #3 column on the scoresheet for skater #1 is the same Judge #3 on the scoresheet for skater #12.

Domestically yes, but I'm told they absolutely mix the judges' columns between skaters for internationals.  Certainly for the major internationals they do (source = my coach who is regularly on international panels).

While I love the IJS for the feedback it gives skaters on the technical side ("did I rotate that jump?"), its major flaw in my view is judge anonymity.  There are no good reasons of probity to have it, and so many good reasons against it.

The other major flaw of the IJS is corridor judging in the PCS marks.  It's ridiculous and unrealistic that nearly every skater will score all their individual PCS's within a band of less than one mark; judges need to be able to give, for example, 3 for transitions and 6 for skating skills, where that is warranted.

[Query]

As for consistency... I remember very clearly the inconsistency in OBO scores in comps...

OBO?? Huh?

P.S. Interesting that Johnny Weir, a U.S. sportscaster who skated under IJS, doesn't discuss it at length. Maybe he has it right. After all, most sports fans don't know all the rules of their favorite sports.

[aussieskater]

OBO?? Huh?

One By One.  Also known as 6.0 and ordinals.  A somewhat convoluted system, where the winner was chosen by place (highest number of judges giving highest place wins), rather than by score (highest score wins).  The skaters got very little feedback from the resuts generated by the OBO system.  A skater could have two judges placing him/her 1st, two 4th placings, a 7th, and come 2nd, depending on how the other skaters placed.

[FigureSpins]

One By One.  Also known as 6.0 and ordinals.  A somewhat convoluted system, where the winner was chosen by place (highest number of judges giving highest place wins), rather than by score (highest score wins).  The skaters got very little feedback from the resuts generated by the OBO system.  A skater could have two judges placing him/her 1st, two 4th placings, a 7th, and come 2nd, depending on how the other skaters placed.

You just described the USFSA 6.0/Ordinals system perfectly. 

The ISI used to judge differently, which was better than 6.0/Ordinals and more like IJS.   I believe the ISI software now just lists the skaters placements; I'm not even sure if the total score is still listed on the sheet.  (All of the rinks in my state have switched to Basic Skills.)


Each judge on the panel would be asked to score specific elements from that level's required skills, along with some more-subjective items like music interpretation/rhythm, posture, duration, edges/flow, and general overall.  There was a specific point range for each element/category.  When the event finished, the accountant would add up all the judge's scores for each element/category and whoever had the highest score won.  The published result sheet, which was stuck on the wall, showed each of the required elements and "other" items across the top of the page.  The skater could see that their Flip jump was the lowest-scored one of the entire flight and compare their score with the same element score from prior competitions.

Xanboni did a good job of explaining the ISI judging on a blog post, but the illustration is not real; it shows all three judges using the same sheet.  Just focus on one judge at a time and you'll see the division of labor: http://xan-boni.blogspot.com/2011/09/sample-isi-judging-sheet-explained.html  This judging system was one of the reasons the ISI asks coaches to judge; you have to know the correct way to do a Flip jump in order to judge it and most well-meaning spectators don't have that knowledge.

I would really like to see the USFSA Basic Skills program implement (and the ISI, return to) a scoring result sheet like that; IJS is fine for the higher levels with the +/- GOE on each element, but at the learn-the-skate levels, focusing on the skills is more important. Simplifying the score result sheet would make that more clear.  The transparency is also good, imo. 

[Doubletoe]

OBO?? Huh?

P.S. Interesting that Johnny Weir, a U.S. sportscaster who skated under IJS, doesn't discuss it at length. Maybe he has it right. After all, most sports fans don't know all the rules of their favorite sports.

Johnny was never one of those skaters who studied the judging system and used it strategically to his advantage.  He was known for not doing jump combinations just because he didn't want to, even though tacking on a double toeloop could mean the difference between a podium spot or not.  So if Johnny isn't very specific about IJS scoring, it may be because he's never learned the rules very thoroughly himself.

[icedancer]

Johnny was never one of those skaters who studied the judging system and used it strategically to his advantage.  He was known for not doing jump combinations just because he didn't want to, even though tacking on a double toeloop could mean the difference between a podium spot or not.  So if Johnny isn't very specific about IJS scoring, it may be because he's never learned the rules very thoroughly himself.

Agreed!!

His programs were basically big, empty '90's programs... beautiful but not so intricate...

I do like his commentary though.  He is very kind to the skaters.