Welcome to the ICM Forum. If you have an account but have trouble logging in, or have other questions, see THIS THREAD.
500<400 (RESULTS)
Polls: 1994 (Oct 20th), 1997 awards (Oct 24th), 1940s (Oct 26th), Shorts (Nov 16th), Knockout competition (Round 2)
Challenges: Horror, TSPDT, Latin America + Caribbean, 2020 schedule
Film of the Week: Finis terrae, October nominations (Sep 27th)

icm is down

Post Reply
User avatar
3eyes
Donator
Posts: 6829
Joined: May 17, 2011
Location: Philadelphia
Contact:

icm is down

#1

Post by 3eyes » September 18th, 2013, 5:39 pm

thass all I know.
:run: STILL the Gaffer!

Kasparius
Posts: 24612
Joined: Sep 14, 2011
Contact:

#2

Post by Kasparius » September 18th, 2013, 5:40 pm

It's probably only for an hour or so, hopefully they are trying to implement the easy adoption process.

esiersdale
Posts: 498
Joined: Apr 01, 2012
Contact:

#3

Post by esiersdale » September 18th, 2013, 5:49 pm

or kick off some cheaters

User avatar
bal3x
Donator
Posts: 13025
Joined: May 26, 2011
Contact:

#4

Post by bal3x » September 18th, 2013, 5:54 pm

iCM was down yesterday which according to the twitter post was due to databaseserver problems. It would appear those are back..


Cippenham
Donator
Posts: 12506
Joined: May 09, 2011
Location: Dorset England
Contact:

#6

Post by Cippenham » September 18th, 2013, 6:38 pm

mimurph1 on Sep 18 2013, 11:58:50 AM wrote:major problems
well its back now
Turning over a new leaf :ICM:

User avatar
jeroeno
Posts: 2882
Joined: Jun 22, 2011
Location: Valkenswaard, The Netherlands
Contact:

#7

Post by jeroeno » September 18th, 2013, 6:38 pm

Oh good, it's working for more people :woot:
Last edited by jeroeno on September 18th, 2013, 6:39 pm, edited 1 time in total.

User avatar
Armoreska
Posts: 11267
Joined: Nov 01, 2012
Location: Ukraine
Contact:

#8

Post by Armoreska » September 19th, 2013, 9:32 pm

down again or "temporarily inaccessible"
Last edited by Armoreska on September 19th, 2013, 9:32 pm, edited 1 time in total.
Image
currently working towards a vegan/low waste world + thru such film lists (besides TV): 2010s bests, RW Fassbinder, Luis Bunuel, Yasujiro Ozu, Eric Rohmer, Visual Effects nominees, kid-related stuff, great animes (mini-serie or feature), very 80s movies, 17+ sci-fi lists on watchlist, ENVIRO, remarkable Silent Films and Pre-Code (exploring 1925 atm) and every shorts and docu list I'm aware of and
viewtopic.php?f=1&t=1434
and "Gordon" Liu Chia-Hui/Liu Chia-Liang and Yuen Woo-ping and "Sammo" Hung Kam-bo

Cippenham
Donator
Posts: 12506
Joined: May 09, 2011
Location: Dorset England
Contact:

#9

Post by Cippenham » September 19th, 2013, 9:51 pm

Now it just says Icheckmovies is down
Turning over a new leaf :ICM:

Marijn
Posts: 138
Joined: Jun 24, 2011
Contact:

#10

Post by Marijn » September 19th, 2013, 9:58 pm

We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.

User avatar
Gershwin
Donator
Posts: 7069
Joined: May 17, 2011
Location: Leiden, NL
Contact:

#11

Post by Gershwin » September 19th, 2013, 10:07 pm

I guess that means you need an official forum for days like these. :shifty:
RokP 250

Profiles: Untappd - Last.fm - iCM

Kasparius
Posts: 24612
Joined: Sep 14, 2011
Contact:

#12

Post by Kasparius » September 19th, 2013, 10:07 pm

Marijn on Sep 19 2013, 03:58:21 PM wrote:We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.
Thanks for the update, you guys are doing a terrific job!

mjf314
Moderator
Posts: 10807
Joined: May 08, 2011
Contact:

#13

Post by mjf314 » September 19th, 2013, 10:07 pm

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Last edited by mjf314 on September 19th, 2013, 10:11 pm, edited 1 time in total.

Local Hero -- aka MestnyiGeroi
Posts: 11696
Joined: May 29, 2011
Contact:

#14

Post by Local Hero -- aka MestnyiGeroi » September 19th, 2013, 10:15 pm

mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:

User avatar
St. Gloede
Moderator
Posts: 10002
Joined: May 06, 2011
Contact:

#15

Post by St. Gloede » September 19th, 2013, 10:16 pm

Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
biglol

User avatar
mightysparks
Site Admin
Posts: 29591
Joined: May 05, 2011
Location: Perth, WA, Australia
Contact:

#16

Post by mightysparks » September 19th, 2013, 10:17 pm

Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
tehe
"I do not always know what I want, but I do know what I don't want." - Stanley Kubrick

iCM | IMDb | LastFM | TSZDT

Image

Kasparius
Posts: 24612
Joined: Sep 14, 2011
Contact:

#17

Post by Kasparius » September 19th, 2013, 10:20 pm

mightysparks on Sep 19 2013, 04:17:43 PM wrote:
Local Hero -- aka MestnyiGeroi on Sep 19 2013, 04:15:35 PM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

:whistling:
tehe
:worship:

User avatar
Gershwin
Donator
Posts: 7069
Joined: May 17, 2011
Location: Leiden, NL
Contact:

#18

Post by Gershwin » September 19th, 2013, 10:28 pm

You guys should remove this Delluc profile, that should make a big difference. ^_^
RokP 250

Profiles: Untappd - Last.fm - iCM

Kasparius
Posts: 24612
Joined: Sep 14, 2011
Contact:

#19

Post by Kasparius » September 19th, 2013, 10:31 pm

Gershwin on Sep 19 2013, 04:28:22 PM wrote:You guys should remove this Delluc profile, that should make a big difference. ^_^
Hey eezee does it!

Marijn
Posts: 138
Joined: Jun 24, 2011
Contact:

#20

Post by Marijn » September 23rd, 2013, 3:55 pm

mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
Last edited by Marijn on September 23rd, 2013, 3:57 pm, edited 1 time in total.

User avatar
3eyes
Donator
Posts: 6829
Joined: May 17, 2011
Location: Philadelphia
Contact:

#21

Post by 3eyes » September 23rd, 2013, 4:05 pm

I've never understood the neighbor compatibility thing. I have yet to find anyone whose compatibility with me is anything but low (except for my buddy Cinephage, who is VERY low).
:run: STILL the Gaffer!

User avatar
bal3x
Donator
Posts: 13025
Joined: May 26, 2011
Contact:

#22

Post by bal3x » September 23rd, 2013, 4:06 pm

Marijn on Sep 23 2013, 09:55:47 AM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.

Just my 2¢ :)
Last edited by bal3x on September 23rd, 2013, 4:07 pm, edited 1 time in total.

User avatar
Gershwin
Donator
Posts: 7069
Joined: May 17, 2011
Location: Leiden, NL
Contact:

#23

Post by Gershwin » September 23rd, 2013, 4:07 pm

So are you. Thanks for all the time you spend on our hobby. :)
RokP 250

Profiles: Untappd - Last.fm - iCM

User avatar
PeacefulAnarchy
Moderator
Posts: 23315
Joined: May 08, 2011
Contact:

#24

Post by PeacefulAnarchy » September 23rd, 2013, 4:20 pm

bal3x on Sep 23 2013, 10:06:20 AM wrote:But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.
None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.
Last edited by PeacefulAnarchy on September 23rd, 2013, 4:21 pm, edited 1 time in total.

Kasparius
Posts: 24612
Joined: Sep 14, 2011
Contact:

#25

Post by Kasparius » September 23rd, 2013, 4:26 pm

Thanks for all your work!

User avatar
bal3x
Donator
Posts: 13025
Joined: May 26, 2011
Contact:

#26

Post by bal3x » September 23rd, 2013, 4:31 pm

PeacefulAnarchy on Sep 23 2013, 10:20:44 AM wrote:
bal3x on Sep 23 2013, 10:06:20 AM wrote:But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.
None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.
Yes, I know that, real-time was a wrong term I used, but the idea is to have less frequent updates until new technology is implemented. Just a temporary solution. If less frequent updates means faster site and less down-times I'm certainly willing to accept that compromise. Anyways, thanks for all the hard work, Guys, I'm sure you will arrive at the best solution! :)

allisoncm
Posts: 16128
Joined: May 11, 2011
Contact:

#27

Post by allisoncm » September 23rd, 2013, 6:32 pm

Marijn on Sep 23 2013, 09:55:47 AM wrote:
mjf314 on Sep 19 2013, 04:07:47 PM wrote:Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.
No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!
Maybe the awards could be sent out only once a week (temporarily?). I know this may not be a popular solution. The once a day calculation of checks is good, though. Any less would be a bit frustrating.

Maybe update the neighbors once a month? I'm sure people hardly ever look at that.

User avatar
frbrown
Posts: 5932
Joined: Nov 01, 2011
Contact:

#28

Post by frbrown » September 23rd, 2013, 7:03 pm

allisoncm on Sep 23 2013, 12:32:28 PM wrote:Maybe update the neighbors once a month? I'm sure people hardly ever look at that.
Or eliminate them completely. I don't think I'll even notice if they're gone.
Image

allisoncm
Posts: 16128
Joined: May 11, 2011
Contact:

#29

Post by allisoncm » September 23rd, 2013, 7:39 pm

frbrown on Sep 23 2013, 01:03:28 PM wrote:
allisoncm on Sep 23 2013, 12:32:28 PM wrote:Maybe update the neighbors once a month? I'm sure people hardly ever look at that.
Or eliminate them completely. I don't think I'll even notice if they're gone.
I agree. Some users like this features, but I rarely use it.

mjf314
Moderator
Posts: 10807
Joined: May 08, 2011
Contact:

#30

Post by mjf314 » September 24th, 2013, 12:18 am

Marijn on Sep 23 2013, 09:55:47 AM wrote:1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation
1. I posted a suggestion regarding this issue on Trello several months ago:
https://trello.com/c/OaxSqtVP/567-membe ... ts-anymore

I might be wrong because I'm not very knowledgeable about this kind of stuff, but I think changing 140,000 boolean variables (and then displaying a notification that links to a single page the next time the user visits the site) would be much less work than sending 140,000 messages. I know you mentioned a notification system, so maybe you were already planning to do something similar to this.

4. How do you calculate the top users? I guess you have to sort the users, but which algorithm do you use? The number of items to be sorted is 137759, and the range of values is much smaller (0-6796), so probably something like counting sort would be much more efficient than comparison-based algorithms.

Edit: Actually forget what I just said. Sorting 137759 numbers will get done quickly regardless of algorithm, so that's the easy part. I guess the hard part is calculating how many official checks each user has? Unfortunately I don't have any suggestions for how to improve that part.
Last edited by mjf314 on September 24th, 2013, 3:55 am, edited 1 time in total.

tirefeet
Posts: 1575
Joined: Nov 28, 2012
Location: Warsaw, Poland
Contact:

#31

Post by tirefeet » October 2nd, 2013, 12:48 am

Anyone else having difficulties as of right now?

User avatar
funkybusiness
Donator
Posts: 10522
Joined: Jan 22, 2013
Contact:

#32

Post by funkybusiness » October 2nd, 2013, 12:58 am

"This page is temporarily unavailable"

User avatar
Gershwin
Donator
Posts: 7069
Joined: May 17, 2011
Location: Leiden, NL
Contact:

#33

Post by Gershwin » October 2nd, 2013, 12:58 am

"Temporary unavailable". Well, it's bed-time anyway.
RokP 250

Profiles: Untappd - Last.fm - iCM

User avatar
joachimt
Donator
Posts: 29542
Joined: Feb 16, 2012
Location: Netherlands
Contact:

#34

Post by joachimt » October 2nd, 2013, 5:50 am

Gershwin on Oct 1 2013, 06:58:25 PM wrote:"Temporary unavailable". Well, it's bed-time anyway.
:lol:
Great motivation to go to bed!
ICM-profile
Fergenaprido: "I find your OCD to be adorable, J"

allisoncm
Posts: 16128
Joined: May 11, 2011
Contact:

#35

Post by allisoncm » October 2nd, 2013, 6:07 am

It seems to be working right now?

ErikSchierboom
Posts: 315
Joined: May 06, 2011
Contact:

#36

Post by ErikSchierboom » October 2nd, 2013, 7:39 am

We are having some server problems lately. A future blog post will explain how we will be dealing with it.

User avatar
Cocoa
Donator
Posts: 1547
Joined: Jul 17, 2013
Location: Chicago, USA
Contact:

#37

Post by Cocoa » October 2nd, 2013, 11:33 am

It's been down/temporary unavailable for me since September 30th while trying to access the site from my computer. Strange how I was able to access it on my iPod touch and my phone yesterday for a few minutes, but then Temporary unavailable shows up shortly after I log in and I can never get back on it from the devices I used.

ozymandias
Posts: 1
Joined: Oct 02, 2013
Location: Where he lone and level sands stretch far away
Contact:

#38

Post by ozymandias » October 2nd, 2013, 1:00 pm

Cocoa on Oct 2 2013, 05:33:14 AM wrote:It's been down/temporary unavailable for me since September 30th while trying to access the site from my computer. Strange how I was able to access it on my iPod touch and my phone yesterday for a few minutes, but then Temporary unavailable shows up shortly after I log in and I can never get back on it from the devices I used.
I can access it by using Internet Explorer, but not on Chrome. And who wants to use IE? :D

Edit: Now also not working on IE, seems I'll be ICM-less for some time :(
Last edited by ozymandias on October 2nd, 2013, 1:34 pm, edited 1 time in total.

User avatar
3eyes
Donator
Posts: 6829
Joined: May 17, 2011
Location: Philadelphia
Contact:

#39

Post by 3eyes » October 2nd, 2013, 1:57 pm

I can get it on Firefox now.

(I'm no computer geek, but what about clearing the cache, Cocoa?)
Last edited by 3eyes on October 2nd, 2013, 1:58 pm, edited 1 time in total.
:run: STILL the Gaffer!

User avatar
Cocoa
Donator
Posts: 1547
Joined: Jul 17, 2013
Location: Chicago, USA
Contact:

#40

Post by Cocoa » October 2nd, 2013, 5:31 pm

3eyes on Oct 2 2013, 07:57:18 AM wrote:I can get it on Firefox now.

(I'm no computer geek, but what about clearing the cache, Cocoa?)
Thank you. I can now access it on my iPod again after doing that. I'll try doing that on my laptop when I get home.

ETA: It now works on my laptop after clearing it.
Last edited by Cocoa on October 2nd, 2013, 7:20 pm, edited 1 time in total.

Post Reply