icm is down

[3eyes]

I get this message: 502 Bad Gateway
anybody know what that means?

[jeroeno]

Don't know what it means but I get the same thing

[Cippenham]

Its working ok for me now.

[Cippenham]

Now its running slow and I got that error searching, but the site is still running to some extent

[esiersdale]

I'm getting the 502 message now, but wasn't 15 minutes ago

[Cippenham]

working ok now for me

[Cippenham]

site down now completely

[Cippenham]

working again

[Torgo]

Worked perfect for me all day long. Weird.

[3eyes]

thass all I know.

[Kasparius]

It's probably only for an hour or so, hopefully they are trying to implement the easy adoption process.

[esiersdale]

or kick off some cheaters

[bal3x]

iCM was down yesterday which according to the twitter post was due to databaseserver problems. It would appear those are back..

[mimurph1]

major problems

[Cippenham]

major problems

well its back now

[jeroeno]

Oh good, it's working for more people

[Armoreska]

down again or "temporarily inaccessible"

[Cippenham]

Now it just says Icheckmovies is down

[Marijn]

We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.

[Gershwin]

I guess that means you need an official forum for days like these.

[Kasparius]

We're up again... for now. Our database server can't cope with the growth of the site. We're planning an upgrade soon. Sorry about this.

Thanks for the update, you guys are doing a terrific job!

[mjf314]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

[Local Hero -- aka MestnyiGeroi]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

[St. Gloede]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

biglol

[mightysparks]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

[Kasparius]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

Maybe if you're good friends with the iCM server you can just have a chat and find out what's wrong?

[Gershwin]

You guys should remove this Delluc profile, that should make a big difference.

[Kasparius]

You guys should remove this Delluc profile, that should make a big difference.

Hey eezee does it!

[Marijn]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!

[3eyes]

I've never understood the neighbor compatibility thing. I have yet to find anyone whose compatibility with me is anything but low (except for my buddy Cinephage, who is VERY low).

[bal3x]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!

But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.

Just my 2¢

[Gershwin]

So are you. Thanks for all the time you spend on our hobby.

[PeacefulAnarchy]

But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.

None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.

[Kasparius]

Thanks for all your work!

[bal3x]

But couldn't all these activities you mention be made less frequent in the meantime (until you move to new technology)?

I for one really don't care about real-time updates to any of those things (in fact I wouldn't miss if the neighbors thing is completely removed), I'm pretty sure others can wait a bit as well if this is causing such strain on the servers.

None of those things happen in real time right now. I wouldn't want the award and top user calculations to be less frequent. I could stand to not have the biweekly update messages if there was a permanent place on the site where that info could be easily found, same with award updates. I like the messages but they're not strictly necessary. I don't use the neighbors at all.

Yes, I know that, real-time was a wrong term I used, but the idea is to have less frequent updates until new technology is implemented. Just a temporary solution. If less frequent updates means faster site and less down-times I'm certainly willing to accept that compromise. Anyways, thanks for all the hard work, Guys, I'm sure you will arrive at the best solution!

[allisoncm]

Which part of iCM puts the most strain on the servers? Is it neighbor compatibility, or something else? Maybe you could reduce how often neighbor compatibility gets calculated, or maybe only calculate it for active users?

Or, if you're in the mood for a more radical change, maybe just get rid of the neighbor system and replace it with something better - a recommendation database. I'm not sure if that would put more or less strain on the servers, but I'm guessing less because you wouldn't have to recalculate so many scores on a regular basis.

No, it's not only the neighbors compatibility (we already calculate neighbors once a week only for active users). It's just the overall size of the database combined with our database design. MySQL just isn't made for the things we do with it. We need to move certain data to other database systems (such as Redis and Neo4j) and use queuing (RabbitMQ) to update everything in the background when a user interacts with the site.

The biggest problems at the moment are with:

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

We've already built a proof of concept with queuing so we know it's working, but it's just a lot of work to implement it everywhere. We're working as fast as we can, but as some of you know, we have to work on the site in our spare time. We all have full time jobs.

Anyways, thanks for using the site. You're all awesome!

Maybe the awards could be sent out only once a week (temporarily?). I know this may not be a popular solution. The once a day calculation of checks is good, though. Any less would be a bit frustrating.

Maybe update the neighbors once a month? I'm sure people hardly ever look at that.

[frbrown]

Maybe update the neighbors once a month? I'm sure people hardly ever look at that.

Or eliminate them completely. I don't think I'll even notice if they're gone.

[allisoncm]

Maybe update the neighbors once a month? I'm sure people hardly ever look at that.

Or eliminate them completely. I don't think I'll even notice if they're gone.

I agree. Some users like this features, but I rarely use it.

[mjf314]

1. Messages (after toplist update, awards update or blog post)
2. Daily update of the table used for award and neigbor calculation (basically a truncate and reinsert)
3. Neighbor calculation
4. Top user calculation

1. I posted a suggestion regarding this issue on Trello several months ago:
https://trello.com/c/OaxSqtVP/567-membe ... ts-anymore

I might be wrong because I'm not very knowledgeable about this kind of stuff, but I think changing 140,000 boolean variables (and then displaying a notification that links to a single page the next time the user visits the site) would be much less work than sending 140,000 messages. I know you mentioned a notification system, so maybe you were already planning to do something similar to this.

4. How do you calculate the top users? I guess you have to sort the users, but which algorithm do you use? The number of items to be sorted is 137759, and the range of values is much smaller (0-6796), so probably something like counting sort would be much more efficient than comparison-based algorithms.

Edit: Actually forget what I just said. Sorting 137759 numbers will get done quickly regardless of algorithm, so that's the easy part. I guess the hard part is calculating how many official checks each user has? Unfortunately I don't have any suggestions for how to improve that part.

[tirefeet]

Anyone else having difficulties as of right now?