Home → Support →

Totally lost contact with our application

• Edit

chrispag

26 Feb, 2019 04:51 PM

Is appHarbor current down (or partly down) one way or another ?
None of our applications are answering :(

1. Support Staff 1 Posted by rune on 26 Feb, 2019 04:57 PM

   

   Hi,

   We are unfortunately experiencing an issue affecting multiple servers in the region, and are currently working to restore connectivity as soon as possible -- stay tuned for an update!

   Best,
   Rune

   • Edit

2. 2 Posted by chrispag on 26 Feb, 2019 05:04 PM

   

   Thanks Rune, looking forward to hear from you.

   Regards,
   Christophe

   • Edit

3. 3 Posted by chrispag on 26 Feb, 2019 06:24 PM

   

   Any idea when this will be sorted out? Just need to give some timeline to our customers, Thanks.

   Regards,
   Christophe

   • Edit

4. Support Staff 4 Posted by rune on 26 Feb, 2019 08:52 PM

   

   Hi Christophe,

   I didn't have a timeline for this when you wrote unfortunately, but as you've probably noticed already we were fortunately able to restore connectivity to the applications affected by this issue approximately a bit more than an hour ago. This solution appears to be stable, but is being closely monitored in the event it should crop up again.

   I'm very sorry you experienced such extended downtime today. The root cause is still being investigated so we can get to the bottom of this, but it does appear to have been an issue related to AWS' "classic" EC2 service some of the elastic load balancer IPs in the EU region. The replacement and backup instances that were brought up would intermittently function only to soon have the exact same issue crop up again. The solution implemented so far hasn't shown any signs of being affected by the problem even intermittently, so there's a high likelihood we've found a stop-gap measure that's adequate even if the root cause is not known yet.

   We'll be working with AWS to narrow down the reasons for this problem, and I'll make sure you get a more thorough explanation as soon as the root cause is fully understood, and steps we'll take to avoid it in the future are ready. The solution so far involved migrating part of our infrastructure to leverage AWS' VPC stack, which offer more control over certain networking aspects that may have been related to the problem we experienced to day.

   We may migrate more of our infrastructure to the VPC stack relatively soon if it turns out to have been caused by an issue specifically affecting the "classic" EC2 service (which is mostly maintained by AWS for legacy reasons). I'll know more about any configuration changes such a migration will require, and include more information on that when today's incident has been thoroughly investigated and appropriate next steps assessed.

   In the meantime, please don't hesitate to reach out if you have any questions, or experience any availability issues.

   Thanks,
   Rune

   • Edit