Hi Mark,
Unfortunately, I don't think we've had a 32bit version for many years because Microsoft no longer put out the updates we needed to support it.
Any old versions we might have backed up somewhere would no longer function with all of the changes we've made to the service since then.
I'm one of those people that is trying to hang on to Windows 7 for as long as I can but had to go 64bit since most developers moved on as well.
@markl said in Agent Disconnected:
Hi, I have 2 agents that got disconnected this morning. I tried restarting the Echo, restarting the PC, but still it is disconnected. Can you help me please?
ID: 128664, 128694
128664
This one in Wxxx, Canada seems to have always been connected directly to the Internet and not on your LAN side. I see the PC seems to have always been connected directly to the Internet with a public IP which is dangerous.
That said, you obviously know better than I do what your intended use is so I just felt it important to mention.
128694
This one looks normal in terms of it's connection. I can see it's got private IP so is running on a PC inside your LAN/network. This one is in Vxxx, Canada.
Both seem to have stopped running today, the 27th but both look normal in every other way, sending the correct information, communicating etc then they stop.
Some thoughts I have;
Usually, when an agent stops communicating out of the blue, it means something has decided to block it.
I see the providers are different so no need to suspect if something there might have changed.
Are you using something on both PCs that might have decided to block communications between the agent and OutagesIO? Firewall, virus scanner, something along those lines is the number one thing we see that would stop the service.
While your Windows version might be a bit different, can you check to make sure something didn't block the Echo Networks service.
I tried restarting the Echo
What happened when you restarted the services? No complaints from the system or anything?
If you simply cannot find a reason for this and confirm there is nothing in your PC firewall or LAN/provider firewall that suddenly started blocking, then I would suggest trying a re-install.
Go to the dashboard of the agent, pick Manage then pick Re-install and follow the prompts in the same way as you first installed. Make sure that you uninstall any previous version just to help ensure the installer works perfectly. Make sure you have full administrator rights when you re-install the software.
Do this on both after you've checked the above and we can go from there.
Yes you are right, 128664 is connected directly to the internet with >a public IP, as I am checking whether the outage is caused by our >ISP, or by our MSP. So I agent run within our LAN, and 128664 >runs directly to the internet.
I suspected something along these lines but prefer to point them out just in case someone is not aware.
What you are trying to diagnose is exactly why OutagesIO came to life. When we offered MSP services, customers would point the finger at our managed services before thinking it might be their Internet connection. We needed a way to remotely know where and when problems were happening not only for the customer but to get help from the ISP.
I already reinstalled now, and it is working. Hopefully it will not >disconnect again.
An agent shows Inactive on average around 30 seconds of not being able to communicate. If it remains Inactive for up to 30 minutes, it will then show Disconnected.
In either of these statuses, it's best to ping something like tpw.outages.io and app.outagesio.com from the command line on the same PC before doing anything else.
This helps to see if there is a path problem or if the agent is no longer running. Restarting the service if there is an actual outage in progress will lose all of the data that the agent is keeping until it can send it to your dashboard.
We have seen plenty of cases where a firewall or virus scanner or even a physical firewall will block the agent after a while but not initially making things a bit more confusing.
We've also seen software conflicts so I hope you aren't experiencing any of those things as they are hard to diagnose.
Keep us in the loop please.
Hi,
I just checked and now the status is back.
Yes, this is because we sent it another reboot command to see if would do the same thing again and it didn't.
What you are seeing should be addressed in the next agent version release.
Initially, we thought that the multi-threading function of the agent code which handles a number of simultaneous functions might be not sending some data now and then.
However, situations like what happened here help to confirm that what may actually be happening is that if the agent is not able to send something for what ever reason, it could give up so the data could eventually be lost if it doesn't re-try.
It is supposed to always re-try but there is something in the libraries or method we are using that somehow prevents this re-try now and then. Could even be CPU overload causing the re-try thread to be lost.
The new version should be ready to test internally this week some time. How long that testing takes depends on what we see as problems if they come up as this is a rather heavy re-write.
Hi,
Seems you found something we have to look into.
Report upgrades are done on the application site (app.outagesio.com) but hardware purchases are done on the www.outagesio.com site.
On the www, we use a paypal software package that allows us to automatically calculate the shipping but that code doesn't work and is not available on the app software that we're using.
Thanks for pointing this out. I'll bring it up as something we need to add when a hardware agent is prepared for shipping.
Thank you for using OutagesIO and hope it helps you find the problems you might be experiencing.
Just an update.
From the investigations, it seems that the outages notification email may have accidentally been removed quite a long time ago, maybe as far back as 2021.
It's odd there have not been more complaints.
All other notifications appear to be there.
Of course, if that is the case, that will be re-added.
Thanks for the update. Very odd problem. We'll look deeper. Please let us know when you do start getting those as we work some more on it.
I see an outage that is more recent. Did you set and get the notification/s for outages?
That's the outage I saw also but it might have happened just before the problem was found.
I guess you've not looked at the extra things you have in Extended :).
If you look at the Historical menu, you'll find all of the events in there. You can even download some of the data then format it and combine it in any way that suits your needs using excel or something else.
Indeed, there could be something that is slowly failing, causing many short outages or disconnections.
Could be a cable, power supply, an interface, but keep in mind, it's always possible that it's just a tiny configuration issue too.
For example, sometimes a firewall with an upstream DHCP might show that it knows what its gateway is but might lose it at times causing short outages. Configuring a static GW might help.
I know this is not your situation so just sharing how tiny things can be hard to find and not so obvious sometimes.
Hi,
I see that you did get an outage and have your notifications on. However, I don't know if that happened before or after we fixed the mail problem.
Let us know once you see something.
Ok, just wanted to check since the gateway is what seems to be constantly going down on you for some reason.
BTW, I thank you very much for sticking with the issue of the outages notifications.
We could see that outages were no longer being entered into the database as of 2023-03-21 13:19:34. That was the last entry.
After many hours spent digging, we found that problem and fixed it which caused a massive number of outages to come in from agents that had them backed up, waiting to re-send.
We also saw an equal amount of emails going out and that caused some mail servers to defer receiving so they will make it over time but may end up in trash/spam folders due to the excessive number of emails that were sent instantly.
I don't yet know if this was the only problem so if you can let us know how it goes from this point on, that would be great. I noticed your notifications are currently disabled.
Yes, disconnected comes after around 29 minutes of being inactive. When members turn of PCs or servers running the agent, it becomes MIA to the network so gets a status change.
Still looking into the outages emails point and have found a problem. They either aren't being sent or aren't being sent to all members. It will take a while to figure out why considering all of the other emails are being sent.
In the meantime, you could use Inactive notifications but note that an inactive status occurs after an agent has not reported in for around under 30 seconds.
Based on your reports, you have some kind of internal issue causing all of the inactive statuses. If your agent is on the LAN, behind the device that it complains about, you could try connecting that PC or a PC running the agent directly to the router/modem which might eliminate those so you don't get too many inactive emails.
In terms of why we stop the emails after 3 is for two main reasons.
1: If we don't, people that leave the service typically flag the emails as spam rather than coming back to their dashboard to disable them.
We ended up being blacklisted too many times and getting blacklisted has the nasty effect of blocking wanted emails from other users of the same email service. We cannot take a chance at being blacklisted.
2: A lighter reason... we hope that people don't rely on emails along but keep an eye on their dashboards as much as possible in business situations at least.
It is better to let members know there is a problem to look into than send constant emails about the same thing. If there is a problem with inactives for example, an email or two tells the member so and the dashboard tells the rest.
I'll have to dig deeper on this email issue to find out what's going on. It's going to take a while however.
The problem is that we aren't seeing any issues with outgoing emails. We monitor mail services to make sure they are always up and running.
It's difficult to know why an email would not have gone out or did but was never received.
This is all I know at this point.
The first email out was when the agent was created on the 18th.
Three emails went out on the 16th and the one at 2023-03-16 10:21:47 was the reports upgrade notification.
The rest on the 14th would be what ever other notifications you had enabled at the time.
The thing is, all of those went out and they made it to your email provider. The logs confirm they were received.
In almost all our communications, we ask members not to flag us as spam otherwise it can affect others using the same email service.
It's possible that an email provider blocks an email but for all intents and purposes, for the sending server, it made it.
I really cannot say why you missed an email but maybe keep an eye on it from this point on. If something happens, let us know immediately and we'll check the various logs to see what happened.
The ratio is much higher for software because it's an instant download.
Hardware is mostly used by businesses while software agents are mostly used by consumers and especially free members.
Because most people are becoming used to ordering online that waiting a day to a few days for most might not be a problem.
The Windows software is much more of a challenge to maintain across so many versions and so many possible combinations of packages installed on different PCs.
That said we do continue to try and find the elusive problem we seem to have with the Windows agent.