Welcome to the ICM Forum. If you have an account but have trouble logging in, or have other questions, see THIS THREAD.
Podcast: Talking Images (Episode 4 released May 26th)
Polls: 1950s (Results), 1966 awards (May 28th), 1935 (May 29th)
Challenges: Comedy, Western, Iberian Peninsula
Film of the Week: Unter den Brücken, June nominations (May 29th)
World Cup S4: Round 2 schedule, Match 2A: Poland vs Mexico (Jun 4th), Match 2B: Tajikistan vs Italy (Jun 14th)

Doubling the Canon 2021 - Improvement suggestions

500<400, Favourite 1001 movies, Doubling the Canon, Film World Cup and many other votes
User avatar
Angel Glez
Posts: 2146
Joined: Apr 02, 2012
Location: Spain
Contact:

Doubling the Canon 2021 - Improvement suggestions

#1

Post by Angel Glez » May 14th, 2020, 10:45 am

As you know there are some changes planned for next update. It will also be a right moment to introduce improvements in our game.

Name: Doubling the Canon
Host: Angel Glez
Canon: TSPDT GF top 1000 + TSPDT 21st Century top 1000 + TSPDT GF top 1001-2000
Doubling list: 1100 1000 titles
Nominations: 12 films per doubler + TSPDT drop-offs + DtC holdovers
Rating system: 0-6
Adjustment for popularity: For every 10 votes the lowest vote gets dropped (???)
Threshold: 8 votes
Re-voting: After 5 years

EVERYTHING is open to discussion, but keep in mind that what we are mainly looking for is a friendlier game, higher participation and more film discussion and discovery. You've got the floor!

User avatar
sol
Donator
Posts: 8404
Joined: Feb 03, 2017
Location: Perth, WA, Australia
Contact:

#2

Post by sol » May 14th, 2020, 12:16 pm

Name: Doubling the Canon
Host: Sir Angel Glez, Esq.
Canon: TSPDT GF top 1000 + TSPDT 21st Century top 1000 + TSPDT GF top 1001-2000
Doubling list: 1100 1200 titles
Nominations: 12 15 films per doubler + TSPDT drop-offs + DtC holdovers
Rating system: 0-6
Adjustment for popularity: For every 10 votes the lowest vote gets dropped (not needed if [see below])
Threshold: 8 votes - 20% of the total number of participants*
Re-voting: After 5 years

* This would create a truer consensus list as opposed to a niche list of obscure favourites, which I believe is more in line with Lee Price's original vision.

I also think that it is problematic that a 3-vote is meant to be "good enough for the bottom half of the canon" and yet 3.90 is the cut-off - which is why I think a bigger list is better. Again, in line with Lee's original vision as I understood it.

Just my two cents; feel free to ignore...
Former IMDb message boards user // iCM | IMDb | Letterboxd | My top 750 films // Long live the new flesh!
Image Image Image

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#3

Post by mjf314 » May 14th, 2020, 12:47 pm

One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.

The films near the top will be films that a lot of people like. Films with few votes can still make the list, but they need a higher rating. The list above contains a few films rated 6.0 with 3 votes, but if you want to get rid of them you can increase the penalty.

I'm not sure about the list size, but the list size can be decided after the voting is finished.

I don't really like the idea of a constant minimum. For example, suppose there are 2 films:
Film A is rated 5.5 with 7 votes.
Film B is rated 3.9 with 8 votes.
Surely film A deserves to be on the list more than film B.

@sol: 20% of total participants would have been 18 this year. That seems to me like too much.

edward5
Posts: 297
Joined: Aug 03, 2013
Contact:

#4

Post by edward5 » May 14th, 2020, 1:08 pm

Some voters have a high average rating. And some voters have a low average rating.
it is because of different evaluation criterion.
So I think everyone's rating need to adjust.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#5

Post by mjf314 » May 14th, 2020, 1:15 pm

edward5 wrote:
May 14th, 2020, 1:08 pm
Some voters have a high average rating. And some voters have a low average rating.
it is because of different evaluation criterion.
So I think everyone's rating need to adjust.
Do you have a suggestion of how to do this?

User avatar
zzzorf
Posts: 753
Joined: Apr 14, 2018
Location: Kempsey, NSW, Australia
Contact:

#6

Post by zzzorf » May 14th, 2020, 1:16 pm

Most of everything seems fine to me, I just like to participate as I managed to find a couple of great movies in the nominations this year. My only thought would be the name. With the Canon having grown larger we aren't really Doubling it anymore. Maybe changing it to something like "Extending the Canon" may better suit the new landscape?

AdamH
Site Admin
Posts: 12313
Joined: May 05, 2011
Contact:

#7

Post by AdamH » May 14th, 2020, 1:26 pm

Is there a post that very clearly lays out exactly how Doubling the Canon works? Voting, re-voting, holdovers, nominations etc.? I'd personally find that very useful as the seemingly complicated system has always put me off.

User avatar
Melvelet
Posts: 2000
Joined: Mar 29, 2013
Contact:

#8

Post by Melvelet » May 14th, 2020, 1:26 pm

mjf314 wrote:
May 14th, 2020, 12:47 pm
One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.

The films near the top will be films that a lot of people like. Films with few votes can still make the list, but they need a higher rating. The list above contains a few films rated 6.0 with 3 votes, but if you want to get rid of them you can increase the penalty.

I'm not sure about the list size, but the list size can be decided after the voting is finished.

I don't really like the idea of a constant minimum. For example, suppose there are 2 films:
Film A is rated 5.5 with 7 votes.
Film B is rated 3.9 with 8 votes.
Surely film A deserves to be on the list more than film B.

@sol: 20% of total participants would have been 18 this year. That seems to me like too much.
I really like this. And if we define a sensible way to determine the holdovers (a list that penalizes few votes (way) less and determining a good length for the holdover list), we'd keeping having an interesting spotlight for the niche films.

User avatar
brokenface
Donator
Posts: 13436
Joined: Dec 29, 2011
Contact:

#9

Post by brokenface » May 14th, 2020, 1:47 pm

This is just throwing something out, but would there be a possibility of making the vote a 2-stage thing, so rather than having holdovers for the following year, you first whittle down the big list of 1300+ nominees to say 200 films using some combination of lower threshold of rating/lower # of votes and then you have a smaller ballot where people might have time to watch some of the ones on the borderline that same year, rather than them essentially starting from scratch the following year. Also the 2nd ballot would be a bit less overwhelming to go through so you might get more casual participants joining at that point.

I'm counting myself as casual as I haven't participated for a few years just through not getting round to going through the ballot of 1200+ films. I can go through with IMDB open and try to translate my ratings, but I do find it a bit of a cumbersome process to get through & I'm never sure I'm being fair to films as it's hard to calibrate that many votes.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#10

Post by mjf314 » May 14th, 2020, 1:54 pm

Melvelet wrote:
May 14th, 2020, 1:26 pm
I really like this. And if we define a sensible way to determine the holdovers (a list that penalizes few votes (way) less and determining a good length for the holdover list), we'd keeping having an interesting spotlight for the niche films.
If we add 4 2-votes to each film:
The holdovers can be every film with fewer than 8 votes AND a bayesian rating of 3.27 or higher.

If we go with a larger penalty and add 5 2-votes to each film:
The holdovers can be every film with fewer than 8 votes AND a bayesian rating of 3.14 or higher.

Either way, the holdover cutoffs would be approximately the same as they are now.

User avatar
sol
Donator
Posts: 8404
Joined: Feb 03, 2017
Location: Perth, WA, Australia
Contact:

#11

Post by sol » May 14th, 2020, 1:58 pm

mjf314 wrote:
May 14th, 2020, 12:47 pm
@sol: 20% of total participants would have been 18 this year. That seems to me like too much.
It depends what you see the purpose of the list as being.

When lee-109 first proposed the project circa 2005, I recall talk about creating a consensus canon of films that we collectively thought were worthy enough to be in the official TSP canon.

20% could be lowered to 15% - but then do we really have a consensus canon when the list is full of films that 85% of participants have never seen?

I really like this though, which would seem to resolve some of my reservations about how the game has evolved since its inception: :thumbsup:
mjf314 wrote:
May 14th, 2020, 12:47 pm
One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.

The films near the top will be films that a lot of people like. Films with few votes can still make the list, but they need a higher rating.
Former IMDb message boards user // iCM | IMDb | Letterboxd | My top 750 films // Long live the new flesh!
Image Image Image

User avatar
OldAle1
Donator
Posts: 4344
Joined: Feb 09, 2017
Location: Dairyland, USA
Contact:

#12

Post by OldAle1 » May 14th, 2020, 2:08 pm

sol wrote:
May 14th, 2020, 1:58 pm

When lee-109 first proposed the project circa 2005, I recall talk about creating a consensus canon of films that we collectively thought were worthy enough to be in the official TSP canon.

The thing is though, in 2005 the "canon" was 1000 films, and now it's 3000. BIG difference. How many films out there are there that could be in any was considered "consensus" choices beyond those 3000? While I tend to agree with you overall, and do find too much obscurity on the list now, I'm not sure how to avoid that and highlight films that are really loved, especially with all the obscurity-chasers and mainstream-haters that make up at least half of the voters.

User avatar
St. Gloede
Moderator
Posts: 10718
Joined: May 06, 2011
Contact:

#13

Post by St. Gloede » May 14th, 2020, 2:17 pm

I would love to see DtC drop-off revotes as well.

The popularity scoring seems pretty good to be honest, maybe tweak slightly further, i.e. for every 5 votes over X threshold - unless that gets too complicated.

User avatar
72aicm
Donator
Posts: 3234
Joined: Nov 13, 2016
Contact:

#14

Post by 72aicm » May 14th, 2020, 2:43 pm

Exlude TSPDT 1001-2000. No need to change it other than that. I love that list and the quirky, and sometimes unfair voting system. :lol:

User avatar
joachimt
Donator
Posts: 31484
Joined: Feb 16, 2012
Location: Netherlands
Contact:

#15

Post by joachimt » May 14th, 2020, 3:16 pm

Set the list length to 1000. I never understood how the length has been determined.
ICM-profile
Fergenaprido: "I find your OCD to be adorable, J"

AdamH
Site Admin
Posts: 12313
Joined: May 05, 2011
Contact:

#16

Post by AdamH » May 14th, 2020, 3:28 pm

joachimt wrote:
May 14th, 2020, 3:16 pm
Set the list length to 1000. I never understood how the length has been determined.
Definitely agree with this. Anything over 1000 is too long (especially as I think it was up to 1300+ at one point). Even 1000 is a huge list.

User avatar
ChrisReynolds
Donator
Posts: 2558
Joined: Dec 29, 2011
Location: Berlin, Germany
Contact:

#17

Post by ChrisReynolds » May 14th, 2020, 3:58 pm

mjf314 wrote:
May 14th, 2020, 12:47 pm
One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.
Could you share the formula you used?

I think moving to this method would produce a better list, especially as the current balancing method (minimum number of votes, lowest vote gets dropped for every ten votes) seems crude and creates weird situations where a film could have got on the list by one extra person voting a 1.

Other than that I would support limiting the list to 1000 films and keeping the films people are allowed to nominate at 12.

User avatar
sol
Donator
Posts: 8404
Joined: Feb 03, 2017
Location: Perth, WA, Australia
Contact:

#18

Post by sol » May 14th, 2020, 4:05 pm

I must be the only user who finds longer lists more interesting. The longer the list, the more fun (i.e. challenging) it is to complete. Also, with more films, there is a greater likelihood of coming across films that have not already been placed on my radar thanks to other lists. :shrug:
Former IMDb message boards user // iCM | IMDb | Letterboxd | My top 750 films // Long live the new flesh!
Image Image Image

User avatar
Angel Glez
Posts: 2146
Joined: Apr 02, 2012
Location: Spain
Contact:

#19

Post by Angel Glez » May 14th, 2020, 4:19 pm

AdamH wrote:
May 14th, 2020, 1:26 pm
Is there a post that very clearly lays out exactly how Doubling the Canon works? Voting, re-voting, holdovers, nominations etc.? I'd personally find that very useful as the seemingly complicated system has always put me off.
Actually what we ask the doubler is not complicated at all. In Phase I we only need 12 films that he thinks are great but are not on a given list (the Canon). In Phase II he must rate the films he has seen from the roster of nominees. Everything else is a intrinsic process that he cannot control.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#20

Post by mjf314 » May 14th, 2020, 4:30 pm

ChrisReynolds wrote:
May 14th, 2020, 3:58 pm
mjf314 wrote:
May 14th, 2020, 12:47 pm
One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.
Could you share the formula you used?

I think moving to this method would produce a better list, especially as the current balancing method (minimum number of votes, lowest vote gets dropped for every ten votes) seems crude and creates weird situations where a film could have got on the list by one extra person voting a 1.

Other than that I would support limiting the list to 1000 films and keeping the films people are allowed to nominate at 12.
(rating * num_ratings + 2 * 4) / (num_ratings + 4)

It's the formula that the IMDb top 250 uses, but with different numbers. The 2 is the "average rating" (which I chose arbitrarily), and the 4 is how many average ratings I added to each film. Both numbers can be tweaked to make the list more or less obscure.

I used the adjusted ratings because I didn't have the unadjusted ratings, but it's probably better to use the unadjusted ratings.

blocho
Donator
Posts: 3071
Joined: Jul 20, 2014
Contact:

#21

Post by blocho » May 14th, 2020, 4:39 pm

I suggest a higher threshold because the rules as they are encourage obscurity. Nominators at this point generally know that they have a much better chance if they nominate avoid anything that will get more than 25 votes.

I also suggest making an adjustment to the rating system. 0-6 is a seven-point system, which is already kind of tough. Furthermore, the descriptions for each point in that system are hard to parse. How should I know a movie would be between my 600th and 1200th favorite movie? It forces me to have every movie I've ever seen ranked. Also, a movie you would put as a 5 means very different for things for someone who has seen 3000 movies versus 8000 movies. I would suggest using instead a 1-5 scale. That's a five-point scale, which matches well with the imdb 10-point scale that many people use (to adjust, you just divide your imdb rating by 2).

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#22

Post by PeacefulAnarchy » May 14th, 2020, 5:20 pm

mjf314 wrote:
May 14th, 2020, 4:30 pm
ChrisReynolds wrote:
May 14th, 2020, 3:58 pm
mjf314 wrote:
May 14th, 2020, 12:47 pm
One possible change is replace "adjustment for popularity" with the bayesian estimate formula, to penalize films with fewer votes. I posted a sample list here:
viewtopic.php?p=640215#p640215
The numbers can be tweaked if you want a bigger or smaller penalty.
Could you share the formula you used?

I think moving to this method would produce a better list, especially as the current balancing method (minimum number of votes, lowest vote gets dropped for every ten votes) seems crude and creates weird situations where a film could have got on the list by one extra person voting a 1.

Other than that I would support limiting the list to 1000 films and keeping the films people are allowed to nominate at 12.
(rating * num_ratings + 2 * 4) / (num_ratings + 4)

It's the formula that the IMDb top 250 uses, but with different numbers. The 2 is the "average rating" (which I chose arbitrarily), and the 4 is how many average ratings I added to each film. Both numbers can be tweaked to make the list more or less obscure.

I used the adjusted ratings because I didn't have the unadjusted ratings, but it's probably better to use the unadjusted ratings.
Changing the "average rating" makes absolutely no difference at all in the rankings, it just changes what the displayed "average" is. Only how many makes a difference.

User avatar
Fergenaprido
Donator
Posts: 3894
Joined: Jun 03, 2014
Location: Malaysia
Contact:

#23

Post by Fergenaprido » May 14th, 2020, 5:27 pm

My thoughts.
  • Name: Doubling the Canon :thumbsup: yes, it's no longer a strict doubling, but that doesn't really bother me - if we want to keep DTC it could be changed to "Deepening the Canon" or something similar
  • Host: Angel Glez :thumbsup:
  • Canon: TSPDT GF top 1000 + TSPDT 21st Century top 1000 + TSPDT GF top 1001-2000 :thumbsup:
  • Doubling list: 1100 1000 titles :thumbsup: :thumbsup: I definitely think that it should go back down to 1000 films; makes it more manageable, less intimidating, and now is the perfect time since we're planning to exclude the top 1001-2000 films from the list
  • Nominations: 12 films per doubler + TSPDT drop-offs + DtC holdovers :thumbsup:
  • Rating system: 0-6 :thumbsup: I like this rating system, but perhaps it needs to be adjust so that 6 is Top 500, 5 is 501-1000, etc.
  • Adjustment for popularity: For every 10 votes the lowest vote gets dropped :think: Maybe instead of every 10 votes, it's every X votes, where X is the threshold
  • Threshold: 8 votes :think: How many voters were there this year? I generally like the idea of a 10% threshold, based on the number of participants from the year before
  • Re-voting: After 5 years :thumbsup:
I'm not sure how I feel about the Bayesian proposal. I know the participants don't have to understand the mechanics of how it all works behind the scenes in order to participate, but it should be relatively clear/transparent to ensure buy-in and legitimacy. I wouldn't use arbitrary numbers, though, I would use actual averages from the process, perhaps find the average (unadjusted) rating for all films and the average number of votes for all films, and then use a rolling average of the past 5 cycles to allow for more consistency. I think that would be fairer and clearer to people than arbitrarily picking 2 and 4 as in your example.

Also, I just did a cross-reference and there are 240 films on both DTC and TSPDT 1001-2000 right now. So restricting DTC to a Top 1000 will still likely result in a lower cutoff from this year's 3.90 (I'm guessing around 3.70-3.75), which is a good thing to me, since films in the high 3s almost 4s are still ones that an average of the voters think should be on the list.
blocho wrote:
May 14th, 2020, 4:39 pm
I also suggest making an adjustment to the rating system. 0-6 is a seven-point system, which is already kind of tough. Furthermore, the descriptions for each point in that system are hard to parse. How should I know a movie would be between my 600th and 1200th favorite movie? It forces me to have every movie I've ever seen ranked. Also, a movie you would put as a 5 means very different for things for someone who has seen 3000 movies versus 8000 movies. I would suggest using instead a 1-5 scale. That's a five-point scale, which matches well with the imdb 10-point scale that many people use (to adjust, you just divide your imdb rating by 2).
I don't think the system is too hard to parse, especially among a community consisting largely of chronic listmakers :lol:
I think it's okay if a 5 for a 3000-film person is different from an 8000-film person. Imdb 10-point scale is also used differently by people, so there's no consistency there if that's what you're looking for. At least the current 7-point scale clearly defines what each number means.

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#24

Post by PeacefulAnarchy » May 14th, 2020, 5:30 pm

blocho wrote:
May 14th, 2020, 4:39 pm
I also suggest making an adjustment to the rating system. 0-6 is a seven-point system, which is already kind of tough. Furthermore, the descriptions for each point in that system are hard to parse. How should I know a movie would be between my 600th and 1200th favorite movie? It forces me to have every movie I've ever seen ranked. Also, a movie you would put as a 5 means very different for things for someone who has seen 3000 movies versus 8000 movies. I would suggest using instead a 1-5 scale. That's a five-point scale, which matches well with the imdb 10-point scale that many people use (to adjust, you just divide your imdb rating by 2).
To the first point, you don't need everything ranked, just to have some rough idea of where things fall. If you have 100 10s, 500 9s and 1000 8s, then your 8s might be in one category or the other but you can just ask "is this a really strong 8 or low 8" to parse the difference.
As for the different number of movies watched, that's a judgement that voters should be making, though I don't know if they do. When I had seen much fewer films I took that into account, so I considered my top 300 at the time to be roughly what my top 600 would be if I filled all my blind spots, now my adjustment is more 500=600 and 1000=1200 since I've seen more of the established canon. So while it's personal in the sense of it being my opinion and my ratings, the numbers for the tiers have always been about my "ideal self" who would have watched all relevant films.
I think the fact that conversion is not just imdb rating over 2 is actually something very positive about this since it means that even if you're just converting imdb ratings you have to put a little thought into what they mean. Imdb ratings can mean very different things, some people give 7 to a film they love, others to a film they like and others to a film they feel indifferent to. Converting your personal scale to DtC should require consideration of how your personal scale works.

jeff_v
Posts: 997
Joined: May 09, 2011
Location: Another Place
Contact:

#25

Post by jeff_v » May 14th, 2020, 6:16 pm

Or you can just rate everything you've seen on a 0-100 scale in Criticker, which makes stuff like this easy.

edward5
Posts: 297
Joined: Aug 03, 2013
Contact:

#26

Post by edward5 » May 14th, 2020, 6:28 pm

V0 = numbers of film rating 0
V1 = numbers of film rating 1
V2 = numbers of film rating 2
V3 = numbers of film rating 3
V4 = numbers of film rating 4
V5 = numbers of film rating 5
V6 = numbers of film rating 6

T = Total votes
T = V0+V1+V2+V3+V4+V5+V6

M0 = Median of V0
M0 = (1+V0)/2
M1 = (1+V1)/2
M2 = (1+V2)/2
M3 = (1+V3)/2
M4 = (1+V4)/2
M5 = (1+V5)/2
M6 = (1+V6)/2
Mn = (1+Vn)/2

Cn = from 0 to Mn
C0 = M0 = (1+V0)/2
C1 = V0+M1 = V0+(1+V1)/2
C2 = V0+V1+M2 = V0+V1+(1+V2)/2
C3 = V0+V1+V2+M3 = V0+V1+V2+(1+V3)/2
C4 = V0+V1+V2+V3+M4 = V0+V1+V2+V3+(1+V4)/2
C5 = V0+V1+V2+V3+V4+M5 = V0+V1+V2+V3+V4+(1+V5)/2
C6 = V0+V1+V2+V3+V4+V5+M6 = V0+V1+V2+V3+V4+V5+(1+V6)/2
Cn = V0+V1+...+V(n-1)+Mn = V0+V1+...+V(n-1)+(1+Vn)/2

average rating overall of 2020 = 3.19
G = perfect film rating = 6.38


New Score[adjust] = N
N0 = C0*G/T = C0*6.38/T
N1 = C1*G/T = C1*6.38/T
N2 = C2*G/T = C2*6.38/T
N3 = C3*G/T = C3*6.38/T
N4 = C4*G/T = C4*6.38/T
N5 = C5*G/T = C5*6.38/T
N6 = C6*G/T = C6*6.38/T



i.e.

xianjiro

6 0 0%
5 7 1%
4 37 6%
3 190 31%
2 280 45%
1 96 15%
0 10 2%


V0 = 10
V1 = 96
V2 = 280
V3 = 190
V4 = 37
V5 = 7
V6 = 0

T = Total votes
T = V0+V1+V2+V3+V4+V5+V6 = 620

median
M0 = (1+V0)/2 = 5.5
M1 = (1+V1)/2 = 48.5
M2 = (1+V2)/2 = 140.5
M3 = (1+V3)/2 = 95.5
M4 = (1+V4)/2 = 19
M5 = (1+V5)/2 = 4
M6 = (1+V6)/2 = 0.5

from 0 to median
C0 = M0 = 5.5
C1 = V0+M1 = 58.5
C2 = V0+V1+M2 = 246.5
C3 = V0+V1+V2+M3 = 481.5
C4 = V0+V1+V2+V3+M4 = 595
C5 = V0+V1+V2+V3+V4+M5 = 617
C6 = V0+V1+V2+V3+V4+V5+M6 = 620.5


average rating overall of 2020 = 3.19
G = perfect film rating = 6.38

N = new score
N0 = C0*G/T = 5.5*6.38/620 = 0.06
N1 = C1*G/T = 58.5*6.38/620 = 0.60
N2 = C2*G/T = 246.5*6.38/620 = 2.54
N3 = C3*G/T = 481.5*6.38/620 = 4.95
N4 = C4*G/T = 595*6.38/620 = 6.12
N5 = C5*G/T = 617*6.38/620 = 6.35
N6 = C6*G/T = 620.5*6.38/620 = 6.39

score Vote adjust score
6 0 6.39
5 7 6.35
4 37 6.12
3 190 4.95
2 280 2.54
1 96 0.60
0 10 0.06

name 0 1 2 3 4 5 6 all scores T=all votes ave. score
xianjiro 10 96 280 190 37 7 0 1409 620 2.272580645
peacefulanarchy 53 28 61 52 82 43 41 1095 360 3.041666667
mjf314 4 6 18 73 72 36 36 945 245 3.857142857
matthewscott8 15 45 37 13 20 28 43 636 201 3.164179104
oldale1 15 38 102 143 113 40 36 1539 487 3.160164271
frbrown 2 14 45 52 62 34 28 846 237 3.569620253
ebbywebby 21 70 103 106 71 40 11 1144 422 2.710900474
lineuphere 6 11 28 55 49 49 20 793 218 3.637614679
sol 15 28 83 108 89 59 48 1457 430 3.388372093
lonewolf2003 8 46 42 53 104 46 11 1001 310 3.229032258
jal90 3 7 17 39 32 42 30 676 170 3.976470588
opio 0 8 20 19 32 25 18 466 122 3.819672131
fergenaprido 31 28 41 17 40 53 34 790 244 3.237704918
hurluberlu 17 36 71 139 95 52 8 1283 418 3.06937799
sortile9io 6 37 66 86 58 63 30 1154 346 3.335260116
test 1 100 100 100 100 100 100 100 2100 700 3
test 2 200 0 200 0 200 0 200 2400 800 3
test 3 0 0 0 100 200 200 300 3900 800 4.875
test 4 300 100 50 25 12 6 3 371 496 0.747983871


adjust 0 1 2 3 4 5 6 all scores ave. score
xianjiro 0.06 0.60 2.54 4.95 6.12 6.35 6.39 1980.99 3.195145161
peacefulanarchy 0.48 1.20 1.98 2.99 4.17 5.28 6.03 1151.59 3.198861111
mjf314 0.07 0.20 0.51 1.69 3.58 4.99 5.92 784.74 3.203020408
matthewscott8 0.25 1.21 2.51 3.30 3.82 4.59 5.71 644.38 3.205870647
oldale1 0.10 0.45 1.37 2.97 4.65 5.65 6.15 1556.72 3.196550308
frbrown 0.04 0.26 1.05 2.36 3.89 5.18 6.02 759.22 3.203459916
ebbywebby 0.17 0.85 2.16 3.74 5.08 5.92 6.30 1349.37 3.197559242
lineuphere 0.10 0.35 0.92 2.14 3.66 5.09 6.10 698.61 3.204633028
sol 0.12 0.44 1.26 2.68 4.14 5.24 6.03 1374.89 3.197418605
lonewolf2003 0.09 0.65 1.55 2.53 4.15 5.69 6.28 992.09 3.200290323
jal90 0.08 0.26 0.71 1.76 3.10 4.48 5.84 545.49 3.208764706
opio 0.03 0.24 0.97 1.99 3.32 4.81 5.94 392.37 3.216147541
fergenaprido 0.42 1.19 2.09 2.85 3.60 4.81 5.95 781.55 3.20307377
hurluberlu 0.14 0.54 1.36 2.96 4.75 5.87 6.33 1336.61 3.197631579
sortile9io 0.06 0.46 1.41 2.81 4.14 5.26 6.11 1106.93 3.199219653
test 1 0.46 1.37 2.28 3.19 4.11 5.02 5.93 2236.19 3.194557143
test 2 0.80 1.60 2.40 3.19 3.99 4.79 5.59 2555.19 3.1939875
test 3 0.00 0.00 0.00 0.40 1.60 3.19 5.19 2555.19 3.1939875
test 4 1.94 4.51 5.47 5.96 6.19 6.31 6.37 1585.43 3.196431452

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#27

Post by PeacefulAnarchy » May 14th, 2020, 6:39 pm

Something about that feels very gameable, but I'm still not quite sure what's going on. (I understand the calculations and the way you're normalizing, I'm just not sure I understand the underlying dynamics of how changing one vote or adding a vote ripples through).
One negative is that everything as it is now, despite people regularly being confused about the process, is pretty straight forward and that most definitely is not.
On the plus side no more ties.

User avatar
Fergenaprido
Donator
Posts: 3894
Joined: Jun 03, 2014
Location: Malaysia
Contact:

#28

Post by Fergenaprido » May 14th, 2020, 6:43 pm

PeacefulAnarchy wrote:
May 14th, 2020, 5:30 pm
blocho wrote:
May 14th, 2020, 4:39 pm
I also suggest making an adjustment to the rating system. 0-6 is a seven-point system, which is already kind of tough. Furthermore, the descriptions for each point in that system are hard to parse. How should I know a movie would be between my 600th and 1200th favorite movie? It forces me to have every movie I've ever seen ranked. Also, a movie you would put as a 5 means very different for things for someone who has seen 3000 movies versus 8000 movies. I would suggest using instead a 1-5 scale. That's a five-point scale, which matches well with the imdb 10-point scale that many people use (to adjust, you just divide your imdb rating by 2).
To the first point, you don't need everything ranked, just to have some rough idea of where things fall. If you have 100 10s, 500 9s and 1000 8s, then your 8s might be in one category or the other but you can just ask "is this a really strong 8 or low 8" to parse the difference.
As for the different number of movies watched, that's a judgement that voters should be making, though I don't know if they do. When I had seen much fewer films I took that into account, so I considered my top 300 at the time to be roughly what my top 600 would be if I filled all my blind spots, now my adjustment is more 500=600 and 1000=1200 since I've seen more of the established canon. So while it's personal in the sense of it being my opinion and my ratings, the numbers for the tiers have always been about my "ideal self" who would have watched all relevant films.
I think the fact that conversion is not just imdb rating over 2 is actually something very positive about this since it means that even if you're just converting imdb ratings you have to put a little thought into what they mean. Imdb ratings can mean very different things, some people give 7 to a film they love, others to a film they like and others to a film they feel indifferent to. Converting your personal scale to DtC should require consideration of how your personal scale works.
Once again, PA found a way to say what I wanted to say more clearly :D :worship:

PeacefulAnarchy wrote:
May 14th, 2020, 6:39 pm
Something about that feels very gameable, but I'm still not quite sure what's going on. (I understand the calculations and the way you're normalizing, I'm just not sure I understand the underlying dynamics of how changing one vote or adding a vote ripples through).
One negative is that everything as it is now, despite people regularly being confused about the process, is pretty straight forward and that most definitely is not.
On the plus side no more ties.
Yeah, I'm also not sure I fully understand what your showcasing edward. Could you add a few lines of text to explain it?

edward5
Posts: 297
Joined: Aug 03, 2013
Contact:

#29

Post by edward5 » May 14th, 2020, 6:52 pm

average rating overall of 2020 = 3.19
so the perfect film rating = 6.38 ( this number can be changed to any number)
if we think of all the films from hate to favorite score from 0 to 6.38.
so the first film get 0 , the second film get 0.01 , the third film get 0.02 .... all the way to the last film get 6.38

use median to represent a bunch of the films get the same score
if one rate 620 films and 10 films get zore score . 96 films get one score. 280 films get two score. 190 films get three score. 37 films get four score. 7 films get five score. no film gets six score.


10 films get zore score
median of 10 is (1+10)/2 = 5.5, so all 10 films use the 5.5's film score to be the new adjustment score
the 620's film get the perfect film rating = 6.38
the 5.5's film get the rating = 5.5*6.38/620 = 0.06
so all the first ten films' adjustment score = 0.06

96 films get one score
median of 96 is (1+96)/2 = 48.5, so use 48.5's film score to be the new adjustment score
and not forget we have 10 films already adjust from score 0.
10+48.5 = 58.5 so we use 58.5's film score to be the new adjustment score for film rated 1
the 58.5's film get the rating = 58.5*6.38/620 = 0.60
so all 96 films rated 1 now have a 0.60 adjustment score.

280 films get two score
median of 280 is (1+280)/2 = 140.5
and films got zore score and one score already have adjusted. it is 10+96 = 106 films
140.5+106 = 246.5
so we use 246.5's film score to be the new adjustment score for film rated 2
the 620's film get the perfect film rating = 6.38
the 246.5's film get the rating = 246.5*6.38/620 = 2.54
so all 280 films rated 2 now have a 2.54 adjustment score.

190 films get three score
median of 190 is (1+190)/2 = 95.5
and films got zore score, one score and two score already have adjusted. it is 10+96+280 = 386 films
95.5+386=481.5
so we use 481.5's film score to be the new adjustment score for film rated 3
the 620's film get the perfect film rating = 6.38
the 481.5's film get the rating = 481.5*6.38/620 = 4.95
so all 190 films rated 3 now have a 4.95 adjustment score.

37 films get four score
median of 37 is (1+37)/2 = 19
and films got zore score, one score, two score and three score already have adjusted. it is 10+96+280+190 = 576 films
19+576=595
so we use 595's film score to be the new adjustment score for film rated 4
the 620's film get the perfect film rating = 6.38
the 595's film get the rating = 595*6.38/620 = 6.12
so all 37 films rated 4 now have a 6.12 adjustment score.

7 films get five score
median of 7 is (1+7)/2 = 4
and films got zore score, one score, two score, three score and four score already have adjusted. it is 10+96+280+190+37 = 613 films
4+613=617
so we use 617's film score to be the new adjustment score for film rated 5
the 620's film get the perfect film rating = 6.38
the 617's film get the rating = 617*6.38/620 = 6.35
so all 7 films rated 5 now have a 6.35 adjustment score.


old score Vote adjust score
6 0 6.39
5 7 6.35
4 37 6.12
3 190 4.95
2 280 2.54
1 96 0.60
0 10 0.06

old average score = 2.27
new average score = 3.19


use this method we can adjust everyone's average score to around 3.19
Last edited by edward5 on May 14th, 2020, 7:10 pm, edited 1 time in total.

User avatar
Angel Glez
Posts: 2146
Joined: Apr 02, 2012
Location: Spain
Contact:

#30

Post by Angel Glez » May 14th, 2020, 7:04 pm

https://docs.google.com/spreadsheets/d/ ... sp=sharing

Here's a editable spreadsheet with full results (before and after the current adjustment). Maybe you could show us what happens by applying your formulas.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#31

Post by mjf314 » May 14th, 2020, 7:05 pm

PeacefulAnarchy wrote:
May 14th, 2020, 5:20 pm
Changing the "average rating" makes absolutely no difference at all in the rankings, it just changes what the displayed "average" is. Only how many makes a difference.
Changing the average does affect the rankings. Here's an example.

Average rating = 2
Number of average ratings added = 4
5.5 rating / 10 votes = 4.5
4.5 rating / 40 votes = 4.27

Average rating = 0
Number of average ratings added = 4
5.5 rating / 10 votes = 3.93
4.5 rating / 40 votes = 4.09

When the average is 2, the first film is ranked higher. When the average is 0, the second film is ranked higher.

edward5
Posts: 297
Joined: Aug 03, 2013
Contact:

#32

Post by edward5 » May 14th, 2020, 7:27 pm

Angel Glez wrote:
May 14th, 2020, 7:04 pm
https://docs.google.com/spreadsheets/d/ ... sp=sharing

Here's a editable spreadsheet with full results (before and after the current adjustment). Maybe you could show us what happens by applying your formulas.
xianjiro's score vote xianjiro's adjustment score
6 0 6.39
5 7 6.35
4 37 6.12
3 190 4.95
2 280 2.54
1 96 0.6
0 10 0.06


It needs to change every individual's score. It's a lot of work. if you sent me all the votes, so i can do it in one day.

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#33

Post by PeacefulAnarchy » May 14th, 2020, 7:44 pm

mjf314 wrote:
May 14th, 2020, 7:05 pm
PeacefulAnarchy wrote:
May 14th, 2020, 5:20 pm
Changing the "average rating" makes absolutely no difference at all in the rankings, it just changes what the displayed "average" is. Only how many makes a difference.
Changing the average does affect the rankings. Here's an example.

Average rating = 2
Number of average ratings added = 4
5.5 rating / 10 votes = 4.5
4.5 rating / 40 votes = 4.27

Average rating = 0
Number of average ratings added = 4
5.5 rating / 10 votes = 3.93
4.5 rating / 40 votes = 4.09

When the average is 2, the first film is ranked higher. When the average is 0, the second film is ranked higher.
Well obviously the number of ratings would have to change if you changed the "average" to make things match. My (incorrect) point was that you have two variables but you could achieve the same thing with only one variable. But now that I do some calculations it turns out I was still wrong. I'm not sure why I thought the two variables were tied. You can get close approximations with just the one, though.
Like Ave=0
and # 2.22222
is very close but not equivalent, to your first example.

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#34

Post by PeacefulAnarchy » May 14th, 2020, 8:04 pm

edward5 wrote:
May 14th, 2020, 6:52 pm
Long description of normalization
So as I think about this more my original objection was wrong, it's not very gamable. Any rating/voting system is to some extent, but relatively speaking it's rather robust and hard to manipulate.

The idea is to "standardize" people's ratings. So someone with a high average is very generous and so their "6" is not worth as much as a "6" from someone with a low average. It all balances out, roughly, because the person with a high average has more high ratings so their 6 which is worth a bit less is given to more films and vice versa.

I'm not sure how I feel about this kind of system. It assumes that the differences in average are due to how people perceive the rating system rather than due to the quality of the films and genuine opinion. There's certainly some of that, but I'm not sure it's really the biggest cause of the differences.

I will say that, no matter what, a 0 should be a 0. I realize this would break the tiering mathematically and adjustments to make it work would be less "correct" but I think they'd be more reflective of voter intent. A 6 should probably universally be a 6 too (breaking the cleanness of the math even more), though that's maybe a bit more debatable conceptually.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#35

Post by mjf314 » May 14th, 2020, 8:11 pm

PeacefulAnarchy wrote:
May 14th, 2020, 7:44 pm
Well obviously the number of ratings would have to change if you changed the "average" to make things match. My (incorrect) point was that you have two variables but you could achieve the same thing with only one variable. But now that I do some calculations it turns out I was still wrong. I'm not sure why I thought the two variables were tied. You can get close approximations with just the one, though.
Like Ave=0
and # 2.22222
is very close but not equivalent, to your first example.
It depends on the ratings. If the ratings are lower than it's not as close.

Average rating = 2
Number of average ratings added = 4
3 rating / 10 votes = 2.71
2.7 rating / 40 votes = 2.64

Average rating = 0
Number of average ratings added = 2.22222
3 rating / 10 votes = 2.45
2.7 rating / 40 votes = 2.56

User avatar
Angel Glez
Posts: 2146
Joined: Apr 02, 2012
Location: Spain
Contact:

#36

Post by Angel Glez » May 14th, 2020, 8:15 pm

edward5 wrote:
May 14th, 2020, 7:27 pm
It needs to change every individual's score. It's a lot of work. if you sent me all the votes, so i can do it in one day.
Let me know and the file is yours, but there has to be an easier way to do this.

The bayesian formula could work, but I need to look at this more carefully (films with only 3 votes making the list? :huh: )

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#37

Post by mjf314 » May 14th, 2020, 8:21 pm

PeacefulAnarchy wrote:
May 14th, 2020, 8:04 pm
I'm not sure how I feel about this kind of system. It assumes that the differences in average are due to how people perceive the rating system rather than due to the quality of the films and genuine opinion. There's certainly some of that, but I'm not sure it's really the biggest cause of the differences.
I was going to say something similar. edward5 is assuming that each person's average enjoyment is about the same, but ratings differ due to different scales. I don't think this assumption is correct.

Suppose there are 2 voters:
Person A watches 100 nominees, but he's very careful to only watch films that he thinks he'll enjoy, and he loves most of them.
Person B watches 100 random nominees, including many from his least favorite genres, and he thinks most of them are meh.

Person A will give a lot of high ratings, and person B will give a lot of low ratings.

edward5's formula normalizes them, so both people end up with the same average rating. That doesn't seem right to me.

mjf314
Moderator
Posts: 11256
Joined: May 08, 2011
Contact:

#38

Post by mjf314 » May 14th, 2020, 8:23 pm

Angel Glez wrote:
May 14th, 2020, 8:15 pm
edward5 wrote:
May 14th, 2020, 7:27 pm
It needs to change every individual's score. It's a lot of work. if you sent me all the votes, so i can do it in one day.
Let me know and the file is yours, but there has to be an easier way to do this.

The bayesian formula could work, but I need to look at this more carefully (films with only 3 votes making the list? :huh: )
If you think 3 is too few, it can be tweaked. If a film has a rating of 6.0, how many ratings should need to have, in order to make the list?

User avatar
PeacefulAnarchy
Moderator
Posts: 24705
Joined: May 08, 2011
Contact:

#39

Post by PeacefulAnarchy » May 14th, 2020, 8:29 pm

mjf314 wrote:
May 14th, 2020, 8:11 pm
PeacefulAnarchy wrote:
May 14th, 2020, 7:44 pm
Well obviously the number of ratings would have to change if you changed the "average" to make things match. My (incorrect) point was that you have two variables but you could achieve the same thing with only one variable. But now that I do some calculations it turns out I was still wrong. I'm not sure why I thought the two variables were tied. You can get close approximations with just the one, though.
Like Ave=0
and # 2.22222
is very close but not equivalent, to your first example.
It depends on the ratings. If the ratings are lower than it's not as close.
Yeah, like I said, I messed up somewhere in my mental analysis yesterday and carried that forward. There is no way to make them scale equivalently.
Angel Glez wrote:
May 14th, 2020, 8:15 pm
edward5 wrote:
May 14th, 2020, 7:27 pm
It needs to change every individual's score. It's a lot of work. if you sent me all the votes, so i can do it in one day.
Let me know and the file is yours, but there has to be an easier way to do this.

The bayesian formula could work, but need to look at this more carefully (films with only 3 votes making the list? :huh: )
No, a film with only three votes wouldn't make the list, but a film with 5 might with mjf's numbers. It all depends on which numbers you pick. If you used a real average like 3 you'd need the number of missing votes to be a lot more than mjf's proposed 4 (something like 8-15 depending on how much you want to hamper low voted films) for the formula to be meaningfully useful.

User avatar
tobias
Posts: 419
Joined: Feb 04, 2017
Contact:

#40

Post by tobias » May 14th, 2020, 8:37 pm

Name: Doubling the Canon
Host: Angel Glez
Canon: TSPDT GF top 1000 + TSPDT 21st Century top 500 + TSPDT GF top 1001-2000
Doubling list: 1000 titles
Nominations: 12 films per doubler + TSPDT drop-offs + DtC holdovers
Rating system: 0-6
Adjustment for popularity: I like mjf's suggestion of the bayesian estimate. I think it should be quite lenient but more penalizing to obscure films than the current system is.
Threshold: No threshold (with the bayesian estimate it's unescesarry)
Re-voting: After 5 years

This would also make holdovers as they are now obsolete if I understand it correctly but holdovers could instead just be the 50 films that were the closest to getting in the last time around.

Post Reply