SOLVED: Cannot access dashboard | 503 and 504 errors

[SBK]

@outages-io-visuel
@kmichailg

Please let me know if you are still experiencing the same issues now

[outages.io.visuel]

@SBK
dashboard at app.outagesio.com is ok now!

www.outagesio.com gives a timeout now.

[outages.io.visuel]

@SBK
Can not see dashboard at https://app.outagesio.com/agent/dashboardUnified/id/128915 now
Gateway Timeout.....

[SBK]

@outages-io-visuel

So it is definitely something related to DNS replications at this point
Let us check a bit more why this is affecting some regions only

[OutagesIO_Support]

Actually, no DNS changes had to be made for this move other than the dnspoints slave server which is only used for members DDNS.

On this end of the network, other than a database issue that lasted around an hour yesterday, the only issues we still have is a slowness between servers due to heavy traffic that is being worked on.

Nothing else should affect the service other than in short burps so this problem is very perplexing.

[kmichailg]

@OutagesIO_Support

There seemed to be a problem with cookies and local browser data. I am accessing the dashboard using a windows PC and a macOS PC, on both Chrome and Firefox. Accessing it via Incognito/Private Browsing allows me to log back in. Little bit of inconvenience but for almost 2 weeks I thought the dashboard was down. Probably something with the DNS issue you guys have raised.

P.S.
Yes it is still ongoing for me. I haven't cleared my local cache and I am willing to send it to you guys for debugging.

Update:
Tried to login to Incognito on Windows PC and got hit with another 504 error.

Update #2:
Tried connecting via VPN located at Singapore and London. Still can't login. Gateway timeout still.

[kmichailg]

Here's some more info and screenshots:

I can access the login page using Private Browsing/Incognito. But once I submit my login credentials I encounter 504 Gateway Error.

[mikeshand]

Unable to login to the dashboard for at least 5 hours this morning using various browsers. I got in once on Sunday, but mostly got the same error.

Gateway Timeout

The gateway did not receive a timely response from the upstream server or application.

[OutagesIO_Support]

Hi,

That's so strange. Not doubting you at all, just not sure how to diagnose this. We have a similar problem with another person trying to get their agent to work but it cannot reach the receiving servers.

We know there is a gateway issue which is something we've been working on for days but it's not consistent, it's random, and it doesn't take the entire site down, just now and then.

We've been monitoring everything internally non stop and do see some latency problems and know when there are complete disconnections between app/db servers and so on but we reset those as quickly as we learn about them.

There must be something members are seeing that we simply haven't caught which is odd considering we're using the service from remote too so would experience the same things.

Still digging and definitely appreciate your ongoing input on this.

[mikeshand]

Still get the same error when attempting to login to my own account, BUT strangely I CAN login to the demo account! So is it some problem specific to my account?

I tried to do a password reset both using my email address and my username and in both cases it said it had sent a reset to my correct email address (so it knew the correspondence between the two) but nothing has arrived after 10 minutes, and yes I have checked my spam.

[OutagesIO_Support]

Hi,

We can see in the email logs that the emails are making it into the site, which appears to be Google.
The email is not bouncing back so maybe the email service is quietly blocking outagesio.com?

Have you tried sending a reply to us and whitelisting it from the same mail service? Just send an email to noreply@outagesio.com. Even if it bounces, it'll whitelist the service.

/var/log/smtp.log:Apr  5 10:53:12 mx postfix/smtp[29689]: DAFD9AE798: to=<xxxxx@xxxxx.org.uk>, relay=aspmx.l.google.com[172.217.194.27]:25, delay=3.1, delays=0.01/0/1.8/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK  1649181191 f13-202a792ed00159ba2i4079lg.326 - gsmtp)

[SBK]

@mikeshand

Hi Mike,
Since I have no way to access to your dashboard may I ask you your permission to force-change your password so i can test it?

[mikeshand]

Well that's interesting. I've managed to get the password reset email and reset it to the same password. It then automatically logged me in and I could see my dashboard.

So I logged out and tried to login again and I am back to getting the error!

[mikeshand]

Sorry, messages crossed. Yes go ahead and force change the password.

[SBK]

@mikeshand
I will send it to your email address so we can both test and double check

[mikeshand]

The new password gives the same gateway timeout error for me.

[SBK]

@mikeshand
Mail sent

I would like also to suggest one more thing to test meanwhile...

If you open a new tab and copy the following link:
https://app.outagesio.com/agent/manage

Are you able to login going directly to that link?
Or not?

If yes, are you receiving the gateway error only when you try to access the dashboard?
If yes, can you try to access the following link?
https://app.outagesio.com/agent/updateNotifications/id/127083

[mikeshand]

The first link takes me to a login page where I can type in my credentials and I get to my dashboard just fine.

The second link takes me to the notifications page and from there I can move back to my agent with no problems.

If I then logout I can now login again from the usual place.

I'm totally perplexed. Something to do with browser caching?

If I now try from DuckDuckGo it works fine from there too.

So something seems to have sorted the problem.

[SBK]

@mikeshand

That was my bet but could not be sure.
Try for a while switching back and forth using different browsers, eventually clearing cache and forms data from the very beginning (not only the last 48 hours) and lets see if that explains a bit the problem.

Of course test logout and login again few times before stating all is solved

[mikeshand]

Thanks. I'll report back after a few days.