Track Internet disconnections, provider outages with historical data, and automated speed testing.
For Windows, Linux, ARM64, ARMa7. Learn more by visiting www.outagesio.com
Notice: If you created an account on app.outagesio.com, simply use the same credentials to log in here.
Agents is running but not reporting
-
@SBK I uninstalled and then installed agent 129004 again. It was working after the installation then I shutdown the computer. After starting the computer again, the agent was reported disconnected from the portal.
I had to restart the agent from windows services console in order to see it online from the portal.
-
@SBK Same problem with agent 128994, it stop pinging at 9:24am despite the computer being turned on and the internet connection working properly. I had to restart the agent to see it again online on the portal.
Maybe the agent has some issues with Windows Server 2016 ?
-
I am checking into this but I seem to recall that old operating systems like this can no longer be supported because MS doesn't maintain some of the .NET stuff that the agent needs to run correctly.
That or check if the servers where the agents are not working right are fully updated or not. That could be another reason.
-
Oh ok that may be a reason. Thank you for checking.
I have a routine to install daily all the Windows updates so most of the servers are up to date. I check the server of agent 128994 and it's indeed at the latest level of patches.
-
I looked at that agent and it does look fine. The last time it communicated was 2022-06-18 12:03:51 and it seems to be correctly installed since it was sending everything and getting what it needed.
It just suddenly stopped communicating. Either the service is not running or something has blocked it.
Can you check to make sure the server still has the allowed service in the firewall, to make sure some other security software on the server itself didn't block it.
Also, is it possible something upstream might block it after a while? If the service is running and it's showing Disconnected, it means it is not able to reach the OutagesIO network.
-
@OutagesIO_Support said in Agents is running but not reporting:
Either the service is not running or something has blocked it.
You seem convinced there is a communication problem between the agent and your servers. Can you elaborate why and how can I troubleshoot this further ? Can't it be a bug in the agent preventing it to communicate with your server ? Can I check this with Wireshark or access any log to confirm this ? (e.g. if I don't see the agent trying to communicate with your servers it means it's not a network problem).
My Windows Servers configuration are very simple and there is no security software apart from the embedded Windows Defender. I don't change the firewall configuration so outbound ICMP is allowed. And if you check the 2nd post of this thread, I was able to ping your servers from a computer that was reported disconnected despite the agent running.
-
I would not say I'm convinced of anything yet, just trying to work through possible things to narrow down what's going on. Since you know the Internet is up but you see the agent Disconnected, it means it is unable to reach the OutagesIO servers.
Which server were you pinging and getting something back from? The app and several other things don't allow ICMP. You should be able to ping tpw.outagesio.com however which is a test ping reply server.
A bug is always possible but in this case, it doesn't seem to be an agent problem because the agent did show that it was communicating for a while.
The firewall ports basically just need port 80, 443 and ICMP out/reply for some of the tests. There isn't any log and wireshark will only show connections to the OutagesIO network when the agent is able to communicate.
The only thing I can think of right now are the two things I mentioned.
I truly suspect the Windows versions as being too old because we update the agent now and then when MS makes changes that force us to.
Since you are having to use that version, maybe we can find an older version of the agent that would function on your servers but with that would come some missing functionality that we've added over the years.
Depending on how critical your needs are, our hardware agent is inexpensive, runs 24/7, is self updating so you'll never have any of these issues and it will never miss a thing.
All of the above said, I'll get together with SBK tomorrow, the other person helping you and see if there is anything obvious we can spot.
-
I investigated this further today and noticed OtmWinClient.exe is not running on the servers that are reported disconnected by the portal :
If I restart the service or launch OtmWinClient.exe manually, then the portal sees the agent connected.
Can you explain what is OtmWinClient.exe and do you know why it is sometimes not running ?
-
Hi,
So, the first thing we need in order to help each other is to pick a question and stay there. Right now, we have a mix of things in two questions which is making it very hard to follow everything.
Can we stick to this one please?
As for the agent, I'm not sure what you are asking when you say what is the exe? That is the agent itself, that's the service. Maybe I'm not understanding the question so if you can clarify, that would be great.
As for why the agent might not be running, as mentioned earlier, it might be because these are old operating systems and might not have all of the more modern updates that MS has on the newer OS's. That is the main thing I suspect at least.
Also, you were asked if you could install a regular Win 7/8/10/11 at the same location where one of your agents is not working so that we can compare side by side what is happening. It can be a vm or a BMS, doesn't matter.
Now, one thing I do see that looks very wrong is that the installation notes specifically that you must remove any existing agents installed on the server before installing a new one. From what I can see, you have multiple instances running on this server which could be why it's not able to run correctly or runs eratically, one instance messing up the other.
Can you try this please.
Pick a server that doesn't seem to be working right but DID run correctly at one point, meaning it sent hops, pings, etc. This just proves that ICMP isn't being blocked or wasn't at one point at least.
Head to that agent dashboard on app.outagesio.com.
Click on manage, then pick the re-install option.Before installing anything, remove ALL of the instances you have on this server. Check the services and the software list to make sure there are no echo networks packages or otm services left.
At that point, continue the installation as usual. Click yes to agree, download the agent, save it where you know you can find it, confirm the download, then click on the file to start the installation, enter the codes and complete the process.
Then show us what the above screen looks like and see if it's working.
-
Hi,
The agent developer says you seem to have installed the agent multiple times using different users.
Just uninstall all of those as mentioned then install just one agent using the highest level user you have on the system and it should work again.
And of course, you need to install only on 64bit systems.
-
- I can't install a regular Windows 7/8/10/11 because we need Windows server
- There was no previous installation and I have never installed the agent multiple times. I always install the agent only once and under the Adminstrator user. The reason you see muttiple "OtmServiceApplication.exe" is because it seems to be launched every time a user logs in on the server. If you check the screenshot, you will see that the user is different for each instance (PC1, PC7, Administrator...). I assume this is the tray icon so I am not surprised it is launched for every user. There can be up to 15 to 20 users logged in at the same time on one server.
- Yesterday I checked on a few servers that had this problem and I confirm I can ping tpw.outages.io. I already tested it in my 2nd post on a server that was before working correctly and sent you a screenshot so please recheck.
- I have already tried the reinstall procedure for some servers yesterday in the last couple of weeks and it doesn't make a difference. The agent works fine after the installation but it will stop working at some point.
I think I have covered all your points.
The problem now became obvious to me. OtmWinClient.exe is sometimes not started on the servers which is why I see the agent disconnected. The reason remains unknown, this problem is random and seem to happen on every installation. The only solution I have so far is to restart the service manually. "OtmService.exe and "OtmServiceApplication.exe" are always running, the problem is only with "OtmWinClient.exe". Can you please explain the difference between these 3 programs ?
-
Yes, you covered all of the points and I appreciate that.
Thank you for responding to all that so that we can be in sync. Now we better understand where we are at with everything.
In this case, this needs to be looked at by the developer. He mentioned one thing when saying that the agent appeared to have been installed multiple times;
That’s an interesting screenshot. It looks like multiple different users (column 4) have installed otm. So I need to look into that and make sure the app is always installed as “all users”. Interesting that and something I may have overlooked.I have to wait for a reply to better answer why there are three programs installed as I've only ever seen two but have never run it on 'server' versions.
One more thing that was mentioned is that the server must support .NET 4.7.2 I think but the installer should download it automatically if it's not on the OS already.
Have you tried the new version we made available today? It's possible that the newest version might pull down something different that the OS needs to get this to run right.
Just make sure there aren't any other agents installed which I know you know now :).
And please hang in there. I'm sure we'll figure this out. It just takes a little time to get in sync, gather more info then fix the problem.
-
I have installed agent V1.69.2106 on all the servers so far.
I tried to install the agent V1.72.2202 today and the installation fails when trying to download .Net framework :
I have pasted the logs here : https://zerobin.net/?ba17bcb3567c4c9c#YWUMi/KRCnDBQMbaTB+hOC133xxGI5NBPFvoxbCIvNQ=
I checked on the server and here are the .Net versions currently installed :
-
I don't know why the installer has a certificate error when trying to download the .Net installer. I downloaded the installer from Chrome using the same download link as in the logs without any problem.
After installing .NET 4.7.2, the agent was installed successfully.
-
@OutagesIO_Support For now the agent is running fine but it's too early to tell, the problem usually comes after few days.
-
That might help.
Just a suggestion but before you do too much work on many servers though, maybe stick to one until we have this figured out. Once we do, then do the rest.
The dev noticed he left his test cert on the build. Sometimes, it's hard to fully test new builds. We might not see problems but as members install and report them, we fix them.
The problem is, most people don't report them so it's always nice when someone reports or works with us so we can figure things out, just like in this situation.
We appreciate your patience. We'll get a new build out to you asap.
