Latest updates

Check the Important info page for latest updates! (8 May 2018)
Twitter

Saturday, July 28, 2018

Uncertainties about FIFA's new calculation method

Example: let's assume that in an elo-system team A has 1200 points as 'the value before a match' and team B has 1500 points as 'the value before a match' when they play a friendly match (not in a FIFA International Match Calendar window) against each other with A as the home team. A wins 1:0.




The points to exchange P for this match are calculated as K * (W - We)
where
K (or I in FIFA's terminology) is the weight of the match = 5
W is the actual result = 1 (a win for home team)
We is the expected result = 1/(1 + 10^(- (1200 - 1500) / 600)) = 0,2402531
P = 5 * (1 - 0,2402531) = 3,79873
So rating points for team A after this match will be 1200 + 3,79873 = 1203,79873 and rating points for team B after this match will be 1500 - 3,79873 = 1496,20127. The rating points for team A remain unchanged until they play their next match when their 'value before a match' will still be 1203,79873.

Now FIFA can for instance decide to keep the 'value before a match' constant for a team during all matches played by the team between one published ranking and the next published ranking. The provided information is unclear about such details. For now I think they will follow the classic elo-approach as much as possible, so that means an update of  the 'value before a match' after each match.

The crux of this system is the starting value to use when a team hasn't played a match calculated with this elo-system yet. In classic elo when a new team is introduced in the ranking it gets an arbitrary starting value assigned, a sort of scientific guess about it's strength compared to the other teams in the ranking. That guessed rating will, according to the formulas above, be automatically adapted with each match the new team plays and after some 30 matches the rating of the new team will be converged to a sort of steady state: 'the real strength' of the new team at that moment. During those 30 matches 'the real strength' of the new team is therefore indeterminate.

Now with FIFA introducing this new elo-like calculation method, they will have to choose starting values for all teams. About the starting values to use FIFA says in it's introductionary document only this:
"One of the main advantages of SUM is that it allows for a smooth transition from the current ranking formula to the new one without displacement of teams in the existing ranking table. The current FIFA / Coca-Cola World Ranking will be replaced seamlessly by the new SUM formula without gain or loss of member association rank positions".
I assume, based on this information, that the starting values will be the June ranking points, but it could be anything really. 

These starting values will be applied to the first match each teams plays from June 4th onwards, because June 3rd was the deadline for the published June ranking. All matches played on June 3rd or earlier were incorporated in the June ranking. Matches from June 4th onwards will be calculated with the new method. So that explains partly why I use the June ranking points as starting values.

Other, some smaller, details are also unsure yet:
- FIFA published in June two sets of match type weights K (or I) within two weeks. The second set was a factor 10 smaller than the first set. What are the correct weights for each match type ?
- what is the actual result (W) of a knock-out match that goes into extra time for each team in that match ?
- what is the actual result (W) of a knock-out match that's decided by a penalty shoot-out for each team in that match ?
- what to do with the points of teams that lose a knock-out match in extra time or after penalties? Subtract or not ?

So the resulting August ranking could be anybody's guess really ! We will just have to wait and see what FIFA publishes in August as their first ranking calculated with the new method and just try to deduct what assumptions they decided to apply about all these issues.
Or maybe FIFA publishes a document in the meantime that's 'a bit less vague' about the used method. I don't hold my breath for that to happen, because that would mean real transparancy from FIFA :)

About me:

Software engineer, happily unmarried and non-religious. You won't find me on Twitter or other so called social media. Dutchman, joined the blog in March 2018.

9 comments:

  1. The rankings converge after about 30 games only assuming the following:
    1. The K value is high enough (consider for example a K value of 0.01 - obviously the rankings won't converge)
    2. The strength of the teams doesn't change too much during those 30 games - 30 matches for national teams means at least 4 years, so I don't think this assumptions hold for international football.

    Therefore any rankings for international football would always be at least one of the following :
    1. The rankings reflect past strength, and not the current strength of the teams.
    2. The rankings are very volatile and based on luck.
    3. The rankings take more than just the actual results of the national teams into consideration.

    ReplyDelete
  2. I think most who have followed along with the new formula during the World Cup agree that the K values are too small. Of course that's based on the assumptions that you (and most of us) have made about the starting point values. I don't know how seriously FIFA or the company who created the formula tested it with past results, but they clearly failed to try the "extremes" like the #1 team going out in the group stage or the #20 team making the final.

    There's still the negative points problem too with the teams on and near 0 points in the June 2018 rankings.

    ReplyDelete
  3. I think the ranking system start after every world cup, I mean after 4 years. Example- France win world cup, so France Ranking No. is 1 and Croatia Ranking No. is 2 and Beljium Ranking No. is 3.

    ReplyDelete
  4. The K-values are too low, so I agree with Cody here. If you double the K-factor you would get a much more realistic ranking, which sees Brazil on top, Belgium no.2 and France no.3, calculated with the June ranking points. The top 3 would be separated by just a few points, so for example if France would have won the match against Denmark instead of the obligatory draw, they would have gone on top.
    So doubling the K-factor would solve a lot of the problems, but not all and doubling is arbitrary.

    And the matter of the points after a knockout-game, that's something FIFA has to be transparent about. If they choose an approach: fine. If they choose another approach: fine also. But FIFA needs to be consistent and transparent about this. Now it's a bit of a black box.

    In my calculations for the KO-games, I kept all the losing teams on the same points they had before that game. And a win after PSO, I considered to be a full win (so W=1). Why? Because if you consider a win after PSO to be a draw (W=0.5), a stronger team that wins a PSO could still lose points. Take Russia vs Croatia for example. The win expectancy was much, much higher for Croatia. They won on PSO and when you consider that a draw, they would have lost around 22 pts. which would go to Russia. So in this case, the losing team would gain points, while the winning team would lose points or (at maximum) would stay on the points they had.

    Indeed, it's anyone's guess really

    ReplyDelete
  5. If FIFA starts with unchanged rankings they would still have the freedom to start with any set of ranking points that keeps teams in the same order.

    If I were running the changeover I would backtest the proposed ELO system and work out the 5 year average ELO points for #1, #2, #3 and so on and then assign that number of ranking points to the #1, #2 and #3 teams per the existing world ranking per the previous system.

    Given the imperfection in the previous system I expect the best outcome is for FIFA to artificially bunch initial ranking points for most teams fairly tightly around a median point and then it is much more likely that there will be some rapid early fluctuations in any new system before this decreases over time.

    That is much better than the worst case of rankings hardly changing for several years due to the scenarios others have assumed in their previous analysis on this site and elsewhere on the internet.

    ReplyDelete
  6. FIFA has issued a new bulletin about the calculation method for the new ranking. See: https://resources.fifa.com/image/upload/fifa-world-ranking-technical-explanation-revision.pdf?cloudid=p9nvs47hyrmxtlthw9cj

    The formula itself and the k- (or i-)factor remain the same, the only thing that is different from the old bulletin is the explanation of the PSO-rules and a clear statement about negative points in knockout-games.

    If I read the bulletin correctly and apply it on the results, it seems Germany will be the leader of the August rankings.

    Seems odd to me that FIFA needed a whole month to figure out they don't want to change anything to their new calculation method.

    ReplyDelete
    Replies
    1. After reading the publication a bit better, I've noticed the K-factor HAS changed. It has been scaled down a factor 10. I firmly believe thes means FIFA will have to adjust the ranking points beacause if they use the current points, nothing will change anymore.

      I'm getting more curious by the minute to see what the new ranking will look like.

      Delete
  7. It seems I guess the 0,75 points for a win in penalty shoot-out, but not after extra-time that will be 1 as a win in regular times

    ReplyDelete
  8. "Note: In some instances, games end with a winner after regular or extended time, but still include
    a PSO to determine the team that eventually will get to the next round. These games are treated
    as normal wins and defeats". No more events like Jordan-Kyrgyzstan.
    And another important statement: "Teams that earn negative points in the knock-out round
    of a final competition (e.g. as a result of losing, or even winning after PSO against a weaker team)
    do not lose any points".
    Again, still no details about starting points but I am conviced too that they will not the actual points

    ReplyDelete