Home → Support →

Suddenly getting "The service is unavailable." out of the blue

• Edit

fictfact

15 Mar, 2016 01:31 AM

Are you all having issues? Have been working fine when all of a sudden "The service is unavailable." started popping up for our app.

1. Support Staff 1 Posted by rune on 15 Mar, 2016 01:35 AM

   

   Hi,

   No there are no general issues at the moment. What application are you seeing this on? Looks like the app associated with your account is working fine now?

   Best,
   Rune

   • Edit

2. 2 Posted by fictfact on 15 Mar, 2016 01:38 AM

   

   We’re are seeing it on fictfact.com. We pulled www.fictfact.com over to AWS Elastic Beanstalk until we figure out what’s going on. If you go to http://ffweb.apphb.com/ you get either “The service is unavailable” or a 504 Gateway timeout.

   -- 
   Brian Kirsten
   Sent with Airmail

   • Edit

3. Support Staff 3 Posted by rune on 15 Mar, 2016 02:00 AM

   

   Ah ok, got it. Took a look and the worker was exceeding the maximum number of simultaneous connections leading to the HTTP 503. It's a bit interesting why that would happen as the request throughput doesn't seem much higher than usual.

   That being said the issue might've been caused by the MongoDB database which the application couldn't connect to for a while there (based on the New Relic data for the application). Perhaps the application is holding on to the incoming connections when it's unable to connect to the database? That could cause issue if there's a high timeout as it's cause connections to build up.

   It looks like the issue has been mitigated for the time being after your most recent deployment. Two things I'd recommend if it happens again:

   • Install the Librato add-on, which among other metrics provides you with the current number of active (concurrent) connections. This should help identify the issue and I'm happy to take a closer look when some data has been collected.
   • Either scale up or out to allow for more active connections -- if the app holds on to connections when the database server is unavailable it'll likely be worth looking into that, but scaling out or up would increase the limit for connections (it's currently 500 per worker unit).

   Best,
   Rune

   • Edit