Welcome to the ICM Forum. If you have an account but have trouble logging in, or have other questions, see THIS THREAD.
NOTE: Board emails should be working again. Information on forum upgrade and style issues.
Podcast: Talking Images (Episode 22 released November 17th * EXCLUSIVE * We Are Mentioned in a Book!!! Interview with Mary Guillermin on Rapture, JG & More)
Polls: Directors (Results), 1929 (Results), Directorial Debut Features (Mar 12th), DtC - Nominations (Mar 20th), Favourite Movies (Mar 28th)
Challenges: UK/Ireland, Directed by Women, Waves from around the World
Film of the Week: Lean on Pete, April nominations (Apr 1st)

icm is down

Post Reply
User avatar
3eyes
Donator
Posts: 7172
Joined: May 17th, 2011, 6:00 am
Location: Philadelphia
Contact:

ICM down?

#1

Post by 3eyes »

I get this message: 502 Bad Gateway
anybody know what that means?
:run: STILL the Gaffer!
User avatar
jeroeno
Posts: 3766
Joined: June 22nd, 2011, 6:00 am
Location: Valkenswaard, The Netherlands
Contact:

#2

Post by jeroeno »

Don't know what it means but I get the same thing :huh: :blink: :angry:
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#3

Post by Cippenham »

Its working ok for me now.
Turning over a new leaf :ICM:
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#4

Post by Cippenham »

Now its running slow and I got that error searching, but the site is still running to some extent
Last edited by Cippenham on May 6th, 2013, 2:45 pm, edited 1 time in total.
Turning over a new leaf :ICM:
esiersdale
Posts: 498
Joined: April 1st, 2012, 6:00 am
Contact:

#5

Post by esiersdale »

I'm getting the 502 message now, but wasn't 15 minutes ago
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#6

Post by Cippenham »

working ok now for me
Turning over a new leaf :ICM:
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#7

Post by Cippenham »

site down now completely
Turning over a new leaf :ICM:
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#8

Post by Cippenham »

working again
Turning over a new leaf :ICM:
User avatar
Torgo
Posts: 2134
Joined: June 30th, 2011, 6:00 am
Location: Germany
Contact:

#9

Post by Torgo »

Worked perfect for me all day long. Weird.
User avatar
3eyes
Donator
Posts: 7172
Joined: May 17th, 2011, 6:00 am
Location: Philadelphia
Contact:

#10

Post by 3eyes »

thass all I know.
:run: STILL the Gaffer!
Kasparius
Posts: 24613
Joined: September 14th, 2011, 6:00 am
Contact:

#11

Post by Kasparius »

It's probably only for an hour or so, hopefully they are trying to implement the easy adoption process.
esiersdale
Posts: 498
Joined: April 1st, 2012, 6:00 am
Contact:

#12

Post by esiersdale »

or kick off some cheaters
User avatar
bal3x
Donator
Posts: 13073
Joined: May 26th, 2011, 6:00 am
Contact:

#13

Post by bal3x »

iCM was down yesterday which according to the twitter post was due to databaseserver problems. It would appear those are back..
User avatar
mimurph1
Posts: 139
Joined: March 23rd, 2012, 6:00 am
Location: Pittsburgh, PA
Contact:

#14

Post by mimurph1 »

major problems
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#15

Post by Cippenham »

mimurph1 on Sep 18 2013, 11:58:50 AM wrote:major problems
well its back now
Turning over a new leaf :ICM:
User avatar
jeroeno
Posts: 3766
Joined: June 22nd, 2011, 6:00 am
Location: Valkenswaard, The Netherlands
Contact:

#16

Post by jeroeno »

Oh good, it's working for more people :woot:
Last edited by jeroeno on September 18th, 2013, 6:39 pm, edited 1 time in total.
User avatar
Armoreska
Posts: 12850
Joined: November 1st, 2012, 6:00 am
Location: Ukraine, former Free Territory
Contact:

#17

Post by Armoreska »

down again or "temporarily inaccessible"
Last edited by Armoreska on September 19th, 2013, 9:32 pm, edited 1 time in total.
he or A. or Armo or any

Image
currently working towards a vegan/free world + thru such film lists: GODARD,
Spoiler
ANARCHISTS, 2010s bests, Yasujiro Ozu, Visual Effects nominees, kid-related stuff, great animes (mini-serie or feature), very 80s movies, 17+ sci-fi lists on watchlist, ENVIRO, remarkable Silent Films and Pre-Code (exploring 1925 atm) and every shorts and docu list I'm aware of and
/forum.icmforum.com/viewtopic.php?f=1&t=1434
and "Gordon" Liu Chia-Hui/Liu Chia-Liang and Yuen Woo-ping and "Sammo" Hung Kam-bo
imaginary awards | youtube channels | complaint lounge | explain how big a fan of slavery you are here, ..viewtopic.php?f=12&t=1535 and here: ..viewtopic.php?f=12&t=4484
Cippenham
Donator
Posts: 13434
Joined: May 9th, 2011, 6:00 am
Location: Dorset England
Contact:

#18

Post by Cippenham »

Now it just says Icheckmovies is down
Turning over a new leaf :ICM:
Marijn
Posts: 140
Joined: June 24th, 2011, 6:00 am
Contact:

#19

Post by Marijn »

We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.
User avatar
Gershwin
Donator
Posts: 7149
Joined: May 17th, 2011, 6:00 am
Location: Leiden, NL
Contact:

#20

Post by Gershwin »

I guess that means you need an official forum for days like these. :shifty:
RokP 250

Profiles: Untappd - Last.fm - iCM
Kasparius
Posts: 24613
Joined: September 14th, 2011, 6:00 am
Contact:

#21

Post by Kasparius »

Marijn on Sep 19 2013, 03:58:21 PM wrote:We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.
Thanks for the update, you guys are doing a terrific job!
mjf314
Moderator
Posts: 11669
Joined: May 8th, 2011, 6:00 am
Contact:

#22

Post by mjf314 »

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Last edited by mjf314 on September 19th, 2013, 10:11 pm, edited 1 time in total.
Local Hero -- aka MestnyiGeroi
Posts: 11744
Joined: May 29th, 2011, 6:00 am
Contact:

#23

Post by Local Hero -- aka MestnyiGeroi »

mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
User avatar
St. Gloede
Moderator
Posts: 12071
Joined: May 6th, 2011, 6:00 am
Contact:

#24

Post by St. Gloede »

Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
biglol
User avatar
mightysparks
Site Admin
Posts: 31325
Joined: May 5th, 2011, 6:00 am
Location: Perth, WA, Australia
Contact:

#25

Post by mightysparks »

Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
tehe
"I do not always know what I want, but I do know what I don't want." - Stanley Kubrick

iCM | IMDb | LastFM | TSZDT

Image
Kasparius
Posts: 24613
Joined: September 14th, 2011, 6:00 am
Contact:

#26

Post by Kasparius »

mightysparks on Sep 19 2013, 04:17:43 PM wrote:
Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
tehe
:worship:
User avatar
Gershwin
Donator
Posts: 7149
Joined: May 17th, 2011, 6:00 am
Location: Leiden, NL
Contact:

#27

Post by Gershwin »

You guys should remove this Delluc profile, that should make a big difference. ^_^
RokP 250

Profiles: Untappd - Last.fm - iCM
Kasparius
Posts: 24613
Joined: September 14th, 2011, 6:00 am
Contact:

#28

Post by Kasparius »

Gershwin on Sep 19 2013, 04:28:22 PM wrote:You guys should remove this Delluc profile, that should make a big difference. ^_^
Hey eezee does it!
Marijn
Posts: 140
Joined: June 24th, 2011, 6:00 am
Contact:

#29

Post by Marijn »

mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
Last edited by Marijn on September 23rd, 2013, 3:57 pm, edited 1 time in total.
User avatar
3eyes
Donator
Posts: 7172
Joined: May 17th, 2011, 6:00 am
Location: Philadelphia
Contact:

#30

Post by 3eyes »

I've never understood the neighbor compatibility thing. I have yet to find anyone whose compatibility with me is anything but low (except for my buddy Cinephage, who is VERY low).
:run: STILL the Gaffer!
User avatar
bal3x
Donator
Posts: 13073
Joined: May 26th, 2011, 6:00 am
Contact:

#31

Post by bal3x »

Marijn on Sep 23 2013, 09:55:47 AM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.

Just my 2¢ :)
Last edited by bal3x on September 23rd, 2013, 4:07 pm, edited 1 time in total.
User avatar
Gershwin
Donator
Posts: 7149
Joined: May 17th, 2011, 6:00 am
Location: Leiden, NL
Contact:

#32

Post by Gershwin »

So are you. Thanks for all the time you spend on our hobby. :)
RokP 250

Profiles: Untappd - Last.fm - iCM
User avatar
PeacefulAnarchy
Moderator
Posts: 25938
Joined: May 8th, 2011, 6:00 am
Contact:

#33

Post by PeacefulAnarchy »

bal3x on Sep 23 2013, 10:06:20 AM wrote:But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.
None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.
Last edited by PeacefulAnarchy on September 23rd, 2013, 4:21 pm, edited 1 time in total.
Kasparius
Posts: 24613
Joined: September 14th, 2011, 6:00 am
Contact:

#34

Post by Kasparius »

Thanks for all your work!
User avatar
bal3x
Donator
Posts: 13073
Joined: May 26th, 2011, 6:00 am
Contact:

#35

Post by bal3x »

PeacefulAnarchy on Sep 23 2013, 10:20:44 AM wrote:
bal3x on Sep 23 2013, 10:06:20 AM wrote:But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.
None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.
Yes, I know that, real-time was a wrong term I used, but the idea is to have less frequent updates until new technology is implemented. Just a temporary solution. If less frequent updates means faster site and less down-times I'm certainly willing to accept that compromise. Anyways, thanks for all the hard work, Guys, I'm sure you will arrive at the best solution! :)
allisoncm
Posts: 16746
Joined: May 11th, 2011, 6:00 am
Contact:

#36

Post by allisoncm »

Marijn on Sep 23 2013, 09:55:47 AM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
Maybe the awards could be sent out only once a week (temporarily?). I know this may not be a popular solution. The once a day calculation of checks is good, though. Any less would be a bit frustrating.

Maybe update the neighbors once a month? I'm sure people hardly ever look at that.
User avatar
frbrown
Posts: 6573
Joined: November 1st, 2011, 6:00 am
Contact:

#37

Post by frbrown »

allisoncm on Sep 23 2013, 12:32:28 PM wrote:Maybe update the neighbors once a month? I'm sure people hardly ever look at that.
Or eliminate them completely. I don't think I'll even notice if they're gone.
Image
allisoncm
Posts: 16746
Joined: May 11th, 2011, 6:00 am
Contact:

#38

Post by allisoncm »

frbrown on Sep 23 2013, 01:03:28 PM wrote:
allisoncm on Sep 23 2013, 12:32:28 PM wrote:Maybe update the neighbors once a month? I'm sure people hardly ever look at that.
Or eliminate them completely. I don't think I'll even notice if they're gone.
I agree. Some users like this features, but I rarely use it.
mjf314
Moderator
Posts: 11669
Joined: May 8th, 2011, 6:00 am
Contact:

#39

Post by mjf314 »

Marijn on Sep 23 2013, 09:55:47 AM wrote:1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation
1. I posted a suggestion regarding this issue on Trello several months ago:
https://trello.com/c/OaxSqtVP/567-membe ... ts-anymore

I might be wrong because I'm not very knowledgeable about this kind of stuff, but I think changing 140,000 boolean variables (and then displaying a notification that links to a single page the next time the user visits the site) would be much less work than sending 140,000 messages. I know you mentioned a notification system, so maybe you were already planning to do something similar to this.

4. How do you calculate the top users? I guess you have to sort the users, but which algorithm do you use? The number of items to be sorted is 137759, and the range of values is much smaller (0-6796), so probably something like counting sort would be much more efficient than comparison-based algorithms.

Edit: Actually forget what I just said. Sorting 137759 numbers will get done quickly regardless of algorithm, so that's the easy part. I guess the hard part is calculating how many official checks each user has? Unfortunately I don't have any suggestions for how to improve that part.
Last edited by mjf314 on September 24th, 2013, 3:55 am, edited 1 time in total.
User avatar
tirefeet
Posts: 1679
Joined: November 28th, 2012, 7:00 am
Contact:

#40

Post by tirefeet »

Anyone else having difficulties as of right now?
Post Reply