tag:support.appharbor.com,2010-11-23:/discussions/problems/54463-site-has-gone-down-twice-today-whyAppHarbor: Discussion 2016-08-11T00:45:13Ztag:support.appharbor.com,2010-11-23:Comment/323628192014-04-01T17:17:07Z2014-04-01T17:17:07ZSite has gone down twice today, why?<div><p>Hi,</p>
<p>What application do you mean? I can see that there are two
applications associated with your account.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-01T17:17:59Z2014-04-01T17:17:59ZSite has gone down twice today, why?<div><p>Also, can you describe more about how the application has gone
down - does it stop responding entirely, do you see a timeout or is
it just really slow at responded, for instance?</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-01T17:18:32Z2014-04-01T17:18:32ZSite has gone down twice today, why?<div><p>Sorry, wrestlestat.</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-01T17:21:22Z2014-04-01T17:21:22ZSite has gone down twice today, why?<div><p>It can up with a tcp error page that looks like was created by
nginx<br>
server(?). Got my New Relic alert saying it was down (not
responding to any<br>
requests), so I went their and it spun for more than 10 seconds, I
checked<br>
the New Relic email for more info, and when I came back it was
showing that<br>
tcp error.</p>
<p>It was down for around 2 minutes...</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-01T17:43:53Z2014-04-01T17:43:53ZSite has gone down twice today, why?<div><p>Ok thanks! I can see that the load balancer actually was acting
up until the time you wrote - from around 10:05 AM - 10:09 AM (both
PST). I'm sorry about the downtime you experienced and we're
investigating what went wrong with the load balancer during that
timeframe.</p>
<p>I can see that this timeframe also matches the one in the New
Relic alert you forwarded. When was the other (second) downtime? It
may be related to the other problem, but could also indicate a
separate issue as we only have registered one with the load
balancer in question.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T01:14:19Z2014-04-02T01:15:40ZSite has gone down twice today, why?<div><p>Hi again,</p>
<p>Just a quick follow-up on this: We've reached out to AWS as
there appears to have been a sudden spike in I/O usage just around
the time you experienced downtime, and the issue doesn't appear to
be related to any AppHarbor components. Usually this is not a
problem and I wouldn't expect any more downtime because of this
particular issue - however, if you do happen to experience it again
please let me know.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T02:07:04Z2014-04-02T02:07:04ZSite has gone down twice today, why?<div><p>It's down again right now...</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T02:07:54Z2014-04-02T02:07:54ZSite has gone down twice today, why?<div><p>Hi,</p>
<p>Yes we're on it - looking into the issue right now.</p>
<p>Best,<br>
Rue</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T02:10:23Z2014-04-02T02:10:23ZSite has gone down twice today, why?<div><p>Lol, thanks. Sorry to be a pain. Just thought you might want to
know,<br>
obviously you guys are tracking this more than I am...I'll leave
you alone.</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T02:14:06Z2014-04-02T02:14:06ZSite has gone down twice today, why?<div><p>No worries - actually it's great that you let us know. Usually
these issues are not instance-wide and affecting many applications,
and in those cases we still want to attend to the issue
quickly.</p>
<p>Your application should be back up and running now. I'm really
sorry this happened again. The issue appears to be related to the
underlying instance/hardware and we'll migrate to new instance ASAP
and continue to investigate the underlying issue.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T02:32:02Z2014-04-02T02:32:02ZSite has gone down twice today, why?<div><p>You should probably know, I turned on TODAY the "Page Speed
Optimization"<br>
feature. Could it have anything to do with that?</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T06:57:14Z2014-04-02T06:57:14ZSite has gone down twice today, why?<div><p>Actually that shouldn't have any effect when you're using the
shared load balancers since it was temporarily disabled a while
back. Incidentally this was because it appeared to be connected to
an issue similar to this, but since the setting isn't applied it
can't be the cause - but a good guess however, since it has caused
it in the past :-)</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T12:28:38Z2014-04-02T12:28:38ZSite has gone down twice today, why?<div><p>So the Page Speed Optimization is not enabled right now? When is
that<br>
supposed to get turned back on?</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T17:55:02Z2014-04-02T17:55:02ZSite has gone down twice today, why?<div><p>Down again at 12:34 Central time again today.</p>
<p>Is there work being done causing these outages?</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T18:09:56Z2014-04-02T18:09:56ZSite has gone down twice today, why?<div><p>Hi,</p>
<p>Yes we also got alerted about that, but no work was done on the
server while it happened. We're still trying to get to the bottom
of this issue, but it's proving more difficult than expected. Rest
assured that this is our highest priority though. We tried
migrating to new hardware yesterday, but that seemingly has not
resolved the issue.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-02T18:10:30Z2014-04-02T18:10:30ZSite has gone down twice today, why?<div><p>Another at 12:57...</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-04T12:33:55Z2014-04-04T12:33:55ZSite has gone down twice today, why?<div><p>Did you see my question about the Page Speed Optimizations? Are
those in<br>
use right now?</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-05T00:36:30Z2014-04-05T00:36:30ZSite has gone down twice today, why?<div><p>Sorry forgot to answer that part - the Page Speed Optimizations
have been disabled on the shared load balancers for the time being.
We can enable it for you on a dedicated load balancer.</p>
<p>We experienced quite a few issues with the feature on the shared
servers so we're waiting for a confirmation that those are
resolved. Please note however that it's a "labs" feature so it may
be discontinued at any time.</p>
<p>Google also <a href=
"https://developers.google.com/speed/pagespeed/service">PageSpeed
optimizations as a service</a> - you should be able to set that up
with your AppHarbor application to get a similar result.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/323628192014-04-10T12:15:30Z2014-04-10T12:15:30ZSite has gone down twice today, why?<div><p>Any update on the outage issues? I'm still getting them as of
last night.</p>
<p>Thanks</p></div>andegretag:support.appharbor.com,2010-11-23:Comment/323628192014-04-10T21:56:56Z2014-04-10T21:56:56ZSite has gone down twice today, why?<div><p>Hi,</p>
<p>We didn't receive any alerts about downtimes last night - and
we're monitoring multiple endpoints on the same load balancer,
which were also affected on the previous incidents. The last alert
we've seen was 8 days ago at 04/02/2014 10:39:13AM PST.</p>
<p>This issue appears to be related to the application itself. I
took a look at your New Relic graphs (attached) around the time,
which reveals a high CPU usage and time spent in CLR. This can
sometimes indicate a worker process restart - which is also the
case here. Your application is recycled roughly every 24 hours when
you're on the free plan.</p>
<p>The solution is to upgrade to the Catamaran plan, or make sure
the application is capable of starting up quickly. Applications on
a paid plan are not recycled on a daily basis.</p>
<p>Actually attaching the New Relic profiler causes 20-25 seconds
extra startup time for a 1x web worker, so removing that would be
one way to decrease the startup time (however you won't get the
alerts and insight of course).</p>
<p>Best,<br>
Rune</p></div>rune