tag:support.appharbor.com,2010-11-23:/discussions/problems/62397-problem-accessing-external-apiAppHarbor: Discussion 2016-08-11T00:43:08Ztag:support.appharbor.com,2010-11-23:Comment/341606162014-08-16T02:35:34Z2014-08-16T02:35:34ZProblem accessing external API<div><p>Hi,</p>
<p>You're right, the reason the application was running on two
instances was that the application was being migrated to a new
server. The period of time may have been longer than usual, but the
application should be designed to run "idempotently" on multiple
instances primarily for this reason. It also doesn't sound like
that was the issue, unless the Stubhub API somehow automatically
restrict API requests to a single instance/IP - which would be
pretty unusual.</p>
<p>The most likely reason for the problem is if Stubhub requires
you to whitelist IP addresses of the instances that will make API
requests. When migrating to the new server the worker also got a
new IP address and that's a fairly common cause of this type of
issue. The new IP addresses are all within the <a href=
"support.appharbor.com/kb/tips-and-tricks/application-server-ips-ec2-configuration">
ranges described here</a>.</p>
<p>By the way, the problem with weird random requests being
generated by your browser appears to have cropped up again. Did you
recently (within the last few days) use the "old" browser or have
just not used the dashboard in the meanwhile?</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-16T05:40:18Z2014-08-16T05:40:18ZProblem accessing external API<div><p>Regarding the StubHub issue, I agree that it appears likely that
the rejection of the API calls has something to do with the IP
address of the new server. However, I checked the StubHub API
settings and they are set to allow calls from all domains (see
attached screenshot). Would the IP address of the new server be in
a different range than the old server? If so, is it possible to
move the app to a VM with an IP address in the same range as the
old server? BTW, I tried to access the link in your response, but
got a page not found message.</p>
<p>I did not understand your question regarding the "weird random
requests being generated by my browser." If we have discussed that
before, I have totally forgotten about it. Can you elaborate? In
any case I am using the same browser that I always use, latest
version of Chrome.</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-16T05:45:24Z2014-08-16T05:45:24ZProblem accessing external API<div><p>Some further information to add to my previous response. I have
successfully accessed the StubHub API from four other machines
without having to whitelist those machine's IP addresses and for
the many months that I was able to successfully access StubHub from
AppHarbor there was no whitelisting of the AppHarbor server's IP
address. It definitely appears that for some reason stubHub has a
problem with the new IP address.</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-17T12:44:11Z2014-08-17T12:44:11ZProblem accessing external API<div><p>Hi,</p>
<p>Sorry, here's the <a href=
"http://support.appharbor.com/kb/tips-and-tricks/application-server-ips-ec2-configuration">
correct link to the IP ranges</a>.</p>
<p>The new servers all run on new IPs, but the application should
be configured to work with any IP in the ranges listed on that page
for the highest availability - otherwise it will fail when we
upgrade server, failover, maintenance, recovery or similar (like
what seems to be the issue here).</p>
<p>If you think this might be caused by a single IP being
blacklisted you might want to try and scale down the number of
background workers to 0 and back up to 1. That'll deploy the worker
to a randomly chosen server, and likely a different one that your
current.</p>
<p>Otherwise I think it'd make sense to wait and hear what Stubhub
has to say about the issue - they're issuing the 403 and should be
able to tell why, if that information is not already contained in
an error message or similar.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-18T23:31:22Z2014-08-18T23:31:22ZProblem accessing external API<div><p>Rune,</p>
<p>I tried scaling the background workers down to 0 and back up to
one, but that did not resolve the problem.</p>
<p>I agree that the ball seems to be in Stubhub's court.
Unfortunately, I am having no luck with getting them to respond to
my support requests. Hopefully, I will hear something soon. As soon
as I hear from them, I will update you.</p>
<p>Mike</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-20T23:00:36Z2014-08-20T23:00:36ZProblem accessing external API<div><p>Hi Mike,</p>
<p>Did you hear back from Stubhub yet? We can try and move your app
around a bit to see if it helps from a few different IPs. I really
think it's the only plausible explanation for this problem - they
also might have explicitly blacklisted some AWS IP addresses if
they've been used by someone else in the past.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-21T13:47:30Z2014-08-21T13:47:30ZProblem accessing external API<div><p>Rune,</p>
<p>I still have not heard from Stubhub. I was traveling all day
yesterday, so<br>
I will ping them again today in hopes of getting some response.</p>
<p>In the meantime, if we could try moving the app around a bit,
that would be<br>
great. If you move it, let me know, and I will check to determine
if<br>
anything has changed.</p>
<p>Thanks,</p>
<p>Mike</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-21T21:53:27Z2014-08-21T21:53:27ZProblem accessing external API<div><p>Mike, I've tried moving it to another instance with a new IP
address. Seems like some jobs are being scheduled, but couldn't
determine if it has changed anything.</p>
<p>By the way, how do you know that StubHub is returning a 403?</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-22T00:56:13Z2014-08-22T00:56:13ZProblem accessing external API<div><p>Rune,</p>
<p>No luck after the move.</p>
<p>I log (to LogEntries) the results of any HTTP call that does not
return a "200" status code. See the attached screen shot of the
failing HTTP call that was made about ten minutes ago (refer to
highlighted lines).</p>
<p>Still haven't heard from Stubhub. This situation is so weird
because calls made from AppHarbor succeeded for about a year and a
half. Like you suggested maybe they had black listed a range of AWS
IP addresses. I just wish they would respond to me and tell me why
they are rejecting the calls. Like I stated earlier I can still
make calls from any other machine and they work fine. So it doesn't
have anything to do with a credentials issue.</p>
<p>Regards,</p>
<p>Mike</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-22T09:23:34Z2014-08-22T09:23:34ZProblem accessing external API<div><p>Ok got it. I tried accessing the same endpoint from a bunch of
servers, and finally found an IP address within our ranges that
does not appear to be blacklisted. I've moved your background
worker to this server for the time being, and I'd assume that this
means your app also works again.</p>
<p>However even if it works this is by no means a permanent
solution - the application may be moved again in the future, and
you need to figure out a solution with StubHub. That might either
be whitelisting our IP ranges, or verifying credentials rather than
IP addresses.</p>
<p>For your reference I've attached a screenshot of the result
returned by StubHub from a blacklisted server.</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-22T21:40:58Z2014-08-22T21:40:58ZProblem accessing external API<div><p>Rune,</p>
<p>Moving the background worker did resolve the problem. I have a
companion web app, named FiddleFlights, that also makes calls to
StubHub. Would it be possible to move that app to the same server
as the background worker temporarily while I continue to attempt to
get some response from StubHub?</p>
<p>Yesterday I opened another support request, number three, with
StubHub but stiil have not gotten any response. I will continue to
hound them as I realize that the current situation at AppHarbor is
not a permanent fix.</p>
<p>The thing about this that is weird is that StubHub does require
and validate credentials when accessing their API. All calls
require a token to be submitted with the call for
authentication.</p>
<p>Thank you for all of your assistance with this issue. I fully
realize that this is not an AppHarbor problem and I appreciate your
going "above and beyond." Your support has always been outstanding
and I appreciate it.</p>
<p>Regards,</p>
<p>Mike</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-23T10:29:38Z2014-08-23T10:29:38ZProblem accessing external API<div><p>Mike,</p>
<p>I've moved the FiddleFlights app to the same server, hope that
helps for now!</p>
<p>It's not entirely uncommon that services completely block IP
ranges in cases of severe abusive behavior. There might have been a
DDoS attack or similar in the past which could've caused
blacklisting of some IP addresses. The "access denied" page seems
like it takes precedence over the credentials being sent to them,
and ideally they'd change this behavior at least for our
ranges.</p>
<p>You might want to consider spinning up a small instance
somewhere (like EC2) and run a simple proxy server that you can use
for your API calls. It should be relatively straightforward to set
it up, and you'll be more independent from both AppHarbor's IP
ranges and StubHub's support.</p>
<p>Thanks for the kind words about the service! Always happy to
help :-)</p>
<p>Best,<br>
Rune</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162014-08-23T13:31:04Z2014-08-23T13:31:04ZProblem accessing external API<div><p>Rune,</p>
<p>Thanks. Everything is now working properly. I will update you
when/if I<br>
hear from StubHub.</p>
<p>Mike</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162014-08-23T14:46:19Z2014-08-23T14:46:19ZProblem accessing external API<div><p>Great!</p></div>runetag:support.appharbor.com,2010-11-23:Comment/341606162015-03-15T20:23:29Z2015-03-15T20:23:29ZProblem accessing external API<div><p>Just to bring this issue to a close. I never heard from StubHub
when we were discussing this issue last summer. Since I only use
the functionality in question during baseball season I had dropped
the issue. I did some testing a couple of weeks ago and once again
the requests to StubHub from AppHarbor were being blocked. So I
reached out to StubHub again and finally got the following
response:</p>
<pre>
<code>Due to some recent changes in our security, traffic coming from AWS servers is
being blocked. My suggestion would be to use an IP address (US based) where
your application traffic could go through without being blocked, One way to do this it
to take all the traffic coming through AWS and route it to a proxy server (that is US
based as well) so that all incoming and outgoing traffic is recognizable as yours and
not from AWS, We apologize for the inconvenience.
Thanks,
API Support Team</code>
</pre>
<p>So, following their suggestion (that you had suggested earlier
as well), I setup a proxy server on DigitalOcean. Well, StubHub was
blocking requests from Digital Ocean as well. When I reported this
to StubHub they agreed to white list the IP address of the proxy
server. Which solved the problem. I asked StubHub if they would
white list the IP address ranges that AppHarbor uses, but they will
not do that.</p>
<p>So, it appears that the issue is resolved for the time being.
They sure make using their public API difficult!</p></div>mjmooretag:support.appharbor.com,2010-11-23:Comment/341606162015-03-16T20:15:52Z2015-03-16T20:15:52ZProblem accessing external API<div><p>Hi Mike,</p>
<p>Good to hear that they were able/willing to whitelist your
proxy's IP address, might also be valuable for other customers to
know that this is an option.</p>
<p>Thanks for the heads up on this issue!</p>
<p>Best,<br>
Rune</p></div>rune