Hi Mark,

Unfortunately, I don't think we've had a 32bit version for many years because Microsoft no longer put out the updates we needed to support it.

Any old versions we might have backed up somewhere would no longer function with all of the changes we've made to the service since then.

I'm one of those people that is trying to hang on to Windows 7 for as long as I can but had to go 64bit since most developers moved on as well.

@markl said in Agent Disconnected:

Hi, I have 2 agents that got disconnected this morning. I tried restarting the Echo, restarting the PC, but still it is disconnected. Can you help me please?

ID: 128664, 128694

128664
This one in Wxxx, Canada seems to have always been connected directly to the Internet and not on your LAN side. I see the PC seems to have always been connected directly to the Internet with a public IP which is dangerous.
That said, you obviously know better than I do what your intended use is so I just felt it important to mention.

128694
This one looks normal in terms of it's connection. I can see it's got private IP so is running on a PC inside your LAN/network. This one is in Vxxx, Canada.

Both seem to have stopped running today, the 27th but both look normal in every other way, sending the correct information, communicating etc then they stop.

Some thoughts I have;

Usually, when an agent stops communicating out of the blue, it means something has decided to block it.

I see the providers are different so no need to suspect if something there might have changed.

Are you using something on both PCs that might have decided to block communications between the agent and OutagesIO? Firewall, virus scanner, something along those lines is the number one thing we see that would stop the service.

While your Windows version might be a bit different, can you check to make sure something didn't block the Echo Networks service.

Check firewall

I tried restarting the Echo

What happened when you restarted the services? No complaints from the system or anything?

If you simply cannot find a reason for this and confirm there is nothing in your PC firewall or LAN/provider firewall that suddenly started blocking, then I would suggest trying a re-install.

Go to the dashboard of the agent, pick Manage then pick Re-install and follow the prompts in the same way as you first installed. Make sure that you uninstall any previous version just to help ensure the installer works perfectly. Make sure you have full administrator rights when you re-install the software.

Do this on both after you've checked the above and we can go from there.

Yes you are right, 128664 is connected directly to the internet with >a public IP, as I am checking whether the outage is caused by our >ISP, or by our MSP. So I agent run within our LAN, and 128664 >runs directly to the internet.

I suspected something along these lines but prefer to point them out just in case someone is not aware.

What you are trying to diagnose is exactly why OutagesIO came to life. When we offered MSP services, customers would point the finger at our managed services before thinking it might be their Internet connection. We needed a way to remotely know where and when problems were happening not only for the customer but to get help from the ISP.

I already reinstalled now, and it is working. Hopefully it will not >disconnect again.

An agent shows Inactive on average around 30 seconds of not being able to communicate. If it remains Inactive for up to 30 minutes, it will then show Disconnected.

In either of these statuses, it's best to ping something like tpw.outages.io and app.outagesio.com from the command line on the same PC before doing anything else.

This helps to see if there is a path problem or if the agent is no longer running. Restarting the service if there is an actual outage in progress will lose all of the data that the agent is keeping until it can send it to your dashboard.

We have seen plenty of cases where a firewall or virus scanner or even a physical firewall will block the agent after a while but not initially making things a bit more confusing.

We've also seen software conflicts so I hope you aren't experiencing any of those things as they are hard to diagnose.

Keep us in the loop please.

Hi,

I just checked and now the status is back.

Yes, this is because we sent it another reboot command to see if would do the same thing again and it didn't.

What you are seeing should be addressed in the next agent version release.

Initially, we thought that the multi-threading function of the agent code which handles a number of simultaneous functions might be not sending some data now and then.

However, situations like what happened here help to confirm that what may actually be happening is that if the agent is not able to send something for what ever reason, it could give up so the data could eventually be lost if it doesn't re-try.

It is supposed to always re-try but there is something in the libraries or method we are using that somehow prevents this re-try now and then. Could even be CPU overload causing the re-try thread to be lost.

The new version should be ready to test internally this week some time. How long that testing takes depends on what we see as problems if they come up as this is a rather heavy re-write.

Hi,

Seems you found something we have to look into.
Report upgrades are done on the application site (app.outagesio.com) but hardware purchases are done on the www.outagesio.com site.

On the www, we use a paypal software package that allows us to automatically calculate the shipping but that code doesn't work and is not available on the app software that we're using.

Thanks for pointing this out. I'll bring it up as something we need to add when a hardware agent is prepared for shipping.

Thank you for using OutagesIO and hope it helps you find the problems you might be experiencing.

Hi, have you had the chance to install the new agent?

We'll try to see if we can spot anything useful using those agents next week. In the meantime, if you can keep an eye on the app logs and see if you can find us any logging, that's what we really need.

There is one more thing. we could give you a special version of the agent that you could install on some of the machines you've confirmed are doing this.

The agent logs to a file so any problems would be logged even if it's not in the Windows app logs for some reason.

Hi,

Yes, it's the application log. If you're not seeing anything being logged about the agent, maybe that's a lead to something else. We have never seen it not log.

As for random, not trying to get out of this as we'd love to solve it but it's not enough information to know what's going on.

Random, no logs, really not sure how to help.

I asked the Windows dev and he says it's not possible that the agent is not logging what it's doing. Perhaps you're not looking in the right place?

Also, you mention that most of the machines are offline but you said you guess it's happening to all of them.

The problem with that is that it makes it much harder for us to know where to search for problems :).

We absolutely want to see this working for you and right now, you've told us the new version may not be working after we've spent many months testing so this comment makes us quite nervous.

Is there any way you can confirm how many might be having problems? There should be something in the logs if you could take a deeper look. I think it would be in the application log.

I'll pass this on to our windows dev. It's sad to see this since we spent months trying to find the problem so this is a bit confusing.

Since you don't see anything in the logs, then we need to look at our end to see when the agent(s) stops communicating to see if there is a pattern. Can you provide all the agent IDs involved so we can do that.

The agent is supposed to auto restart every 24hrs so it's possible something is preventing it from doing that and maybe it ends up crashing in memory.

Maybe the solution is to auto restart the service daily. Is that something you could try?

We will look at our end to see if it's going offline around the time it's supposed to restart.

Question
We see 30 agents installed but most of those are always disconnected status.

You mentioned these are all Windows 2016 servers so are some of those turned off at night or something or are they supposed to be running 24/7?

I ask because looking into this, we cannot know which servers are supposed to be (or are) left on 24/7.

Happy that you stuck with us to this point.

Can you let us know how things go once you feel the agents are working as expected. As SBK mentioned, feedback is highly important to us and helps to improve where ever we can.

@lp4y said in Agents is running but not reporting:

@SBK Ok I will test it

If you can, re-install completely on maybe a handful of servers making sure they are all running the latest and we can go from there.

Hi again,

If you get the chance to try the latest version, it seems to be working properly now.

We've tested for weeks on most Windows versions including yours and the agent never stopped.

Hopefully, this will be your experience too.

If you try it, be sure to fully uninstall the old version and use the installer to re-install an agent.

So our Windows dev did some digging and this is what we were told.

I've found the reason for the crash this person experienced when installing. 

If the service fails to start because the Listen() call throws an exception then the exception handler calls Close() and Close() fails to check if the handle pointer is a null reference before using it. 
What I don't know is why we are not able to start a named pipe on that persons machine.
I'll add the missing check to the library code but that won't cure the actual problem they had, so I'll investigate further on what might prevent use opening a named pipe on Windows 10 Pro. 

I've never seen that fail before. So I don't think we have a general issue here, more likely a restriction configured on that machine or possibly the name we use for the pipe is already being used.

Thanks for sharing the ID.

I can see that it does communicate, it sends it's first payload but then it stops.
To me, something on the OS is stopping the service or conflicting with it which is either causing the service to stop or get blocked by something.

The reason you cannot get to the dashboard is because it's not fully running yet so there is no actual dashboard ready to view. I'm looking at the raw data as an admin which is why I can see it tries to communicate then stops.

What you'll see is '...Click to complete' probably, since it's not fully communicating yet so it cannot confirm to the site that it's ready to offer the dashboard.