Track Internet disconnections, provider outages with historical data, and automated speed testing.
For Windows, Linux, ARM64, ARMa7. Learn more by visiting www.outagesio.com
Notice: If you created an account on app.outagesio.com, simply use the same credentials to log in here.
2.5Gbps Hardware Agent Repeatedly "Rebooting", Missing Data
-
Thanks for looking into this so quickly!
Not sure if it makes any difference in terms of what the monitoring and outage reporting looks like, but this is currently installed on a network where the ISP is WeLink, which ISP uses CG-NAT and also may be more subject to signal problems than a fiber or cable connection would be. This WeLink rooftop antenna based wireless high speed (2gbps) internet may be generating more events / different events than typical installations would see. -
The setup is unrelated to the problems you've reported. Once we've resolved those issues, we can try to help with what you're seeing.
-
Just so you know, we are still monitoring this. Since the problem is not clear, we're trying tests which take a while to confirm.
For example, we've updated your agent to the newest version that is not released yet to see if that might help and other things.
We do see that the agent gets a fair number of disconnections which are not IP outages.
Still on this.
-
Update.
Day two of looking into this. We spent a large portion of the day trying to understand why this is happening to your agent only.
There is barely a pattern but we can tell that the main program stops communicating after a while and eventually, just comes back. In the meantime, another program keeps communicating but that one does not monitor the Internet.
Your agent is running the same firmware as all the others are and the same agent version yet yours has this odd problem of the main program not communicating. That said, while it has the same version, it could still be a slightly bit different one that somehow was flashed on yours by mistake.
One thing that came up. Are you using the power supply that came with the agent or another one?
Second, is there any chance you can give us remote ssh access to this agent. You can lock it down to one of our IPs of course but being able to look on the agent itself might show us something we simply cannot see from remote.
If ssh is possible, Ed would get into a chat with you tomorrow to get the into and would then take a look.
-
I am using the power supply that came with the 2.5Gbps agent.
I still think there is something odd going on with the ISP (WeLink). Other devices on the LAN exhibit problems too when using WeLink, which I think are an artifact of WeLink service.
So, I have switched connection back to AT&T fiber for now, starting about 1715 Central 13 Jan.
I have also left a yellow hardware agent (129878) and a software agent (131236) running on the same LAN in case that helps.
I would need specific step-by-step instructions to enable ssh access to the 2.5Gbps hardware agent. -
For ssh access, you'd have to look up what ever firewall you are using and create a rule that allows a remote IP to a LAN device (port forwarding).
Your other option would be to send it back so we can look at it.
The problem is that your ISP could be having certain problems but it would not explain the behavior we're seeing from your agent. From all we can tell, the programs are running correctly but something keeps blocking the monitoring program for long periods of time.
Since both programs communicate with the same remote networks, then both should stop communicating if it was a communications issue but only one does.
It's quite weird which is why gaining access to the device might give us a better understanding of what is going on.
-
BTW, I assume you noticed that the missing notifications was solved right after you brought it up so again, thanks for bringing that up.
At the moment, it seems to be something on your end but what, we cannot tell from here. Quite challenging. By gaining ssh access into it, where the problem is, we would have a better chance to find out what's going on.
No need to send it back, there is nothing wrong with the agent. You might notice it seems to be behaving a bit better already.
We continue testing small changes to see if those affect your agent and the odd behavior.You might notice it rebooting now and then and if you do, that's us testing incremental changes because we don't want to affect all of the other working agents.
We do not see these things with other agents so what ever is happening, it is specific to your location or better said, something on your network.
We are still investigating and making some progress which we'll share.
-
O OutagesIO_Support marked this topic as a question on
-
Hi,
I am trying to compare different behaviors within the same LAN and to do that I am asking if it would be possible to have all 3 agents 131236 (wash geek), 131232 (wash 2.5) and 129878 (wash) up and running.
For the moment I can say that both 131236 and 131232 are behaving in a similar way: i.e. they become inactive but NO outage.
Usually this means the problem has to be located within the LAN (your network, firewall or switching system) and not on the WAN (the ISP provider) but it is not always so crystal clear.In short some data stops from being collected by the agent when it doesn't have access to our servers, but there is no evident "internet outage" recorded: this can happen for some of the reason that were commented at the beginning of this thread (cable, signal, etc.).
Let me know if 129878 can be powered on.
-
The 3 agents are recording a similar situation on Jan 13 at around 13:33 Chicago time but at the same time are not recording any type of outage.
Two different technologies (Windows, Openwrt) and three different versions (MT300 and MT3000 even if they both are openwrt are different in binaries) but they all:
- cannot identify an outage
- are monitoring inactives
- they disagree in some minor timing, which can be related to the way the three agents are connected to the LAN
So next question is: is it possible, without any specific detail, understand if all three agents are connected the same way within the LAN (different VLANs, directly connected to the router or thru a switch, different rules in the firewall)
A simple hand drawn picture is more than enough, as I said no company detail is needed, jsut trying to see where the problem is originated and why they behave in such a way.
-
Retracted. Accidentally replies to Ed :)
-
I see @OutagesIO_Support seeming to quote/reply to @SBK and not sure what portion is directed to @SBK versus the OP (me).
The entire purpose of my getting new 2.5Gbps hardware agent was to better monitor outages from new ISP (WeLink) and compare to old ISP (AT&T Fiber). Unfortunately, there may be something else weird going on with the local network which may or may not be an artifact of something the new ISP does (e.g. CGNAT, RF at unknown reliability vs Fiber, maybe some filtering they did not tell me about, etc).Once the bug was quickly fixed about certain but not all 2.5Gbps hardware agents not reporting, I am not sure what's up, but I do know I did NOT keep the LAN and WAN/ISP setup in a steady state suitable for any in-depth monitoring throughout that time, until late evening Chicago time 14 Jan. Work was underway to ensure internet service back online to users. But that no doubt made any troubleshooting of agents difficult.
Over those couple days I had to swap back and forth between the old and new ISP a couple times (eventually switching back the original - had before 2.5GBPS agent installation - ISP) and added a new software agent and put the old yellow hardware agent back online. A few, maybe several, of the times when agent(s) ceased communication that was likely: when ISP tech came to attempt to adjust their RF (did not solve), ISP tech took connection offline, I had to swap back and forth between ISPs by physically moving cable in 2 different floors and 2 separate buildings, I moved HW agent 129878 to a different floor different switch, routers rebooted, etc.
The current setup is since late evening Chicago time 14 Jan is this:
ISP ONT/Router BGW320-500 (in passthrough mode) 3GbE port -> 2.5GbE TPLink Deco BE25 2.5GbE -> 2.5GBe switch:
-> HW 2.5GB Agent 131232
-> [other devices]
-> Win11 PC running SW Agent 131236 (and in live use by a user, so potential PC reboots or disconnects)
-> ~30-40m cat5e cable -> another 2.5GbE switch:
-> HW Agent 129878
-> [other devices]That setup should now remain stable for a month or two until WeLink ISP can persuade me they have solved whatever technical issues they were having and want to attempt again to supplant current ISP (AT&T fiber), after WeLink failed to perform adequately this month.
-
Yes sorry, I replied to Ed's post in error. I was tired. You can disregard that. Sorry for the confusion.
By the way, only our dedicated hardware agents are rebooted. On Windows, the service restarts itself nightly only. This is because PCs can also be in use by people and we would not want them to lose any work they were doing.
-
I see @OutagesIO_Support seeming to quote/reply to @SBK and not sure what portion is directed to @SBK versus the OP (me).
The entire purpose of my getting new 2.5Gbps hardware agent was to better monitor outages from new ISP (WeLink) and compare to old ISP (AT&T Fiber). Unfortunately, there may be something else weird going on with the local network which may or may not be an artifact of something the new ISP does (e.g. CGNAT, RF at unknown reliability vs Fiber, maybe some filtering they did not tell me about, etc).Once the bug was quickly fixed about certain but not all 2.5Gbps hardware agents not reporting, I am not sure what's up, but I do know I did NOT keep the LAN and WAN/ISP setup in a steady state suitable for any in-depth monitoring throughout that time, until late evening Chicago time 14 Jan. Work was underway to ensure internet service back online to users. But that no doubt made any troubleshooting of agents difficult.
Over those couple days I had to swap back and forth between the old and new ISP a couple times (eventually switching back the original - had before 2.5GBPS agent installation - ISP) and added a new software agent and put the old yellow hardware agent back online. A few, maybe several, of the times when agent(s) ceased communication that was likely: when ISP tech came to attempt to adjust their RF (did not solve), ISP tech took connection offline, I had to swap back and forth between ISPs by physically moving cable in 2 different floors and 2 separate buildings, I moved HW agent 129878 to a different floor different switch, routers rebooted, etc.
The current setup is since late evening Chicago time 14 Jan is this:
ISP ONT/Router BGW320-500 (in passthrough mode) 3GbE port -> 2.5GbE TPLink Deco BE25 2.5GbE -> 2.5GBe switch:
-> HW 2.5GB Agent 131232
-> [other devices]
-> Win11 PC running SW Agent 131236 (and in live use by a user, so potential PC reboots or disconnects)
-> ~30-40m cat5e cable -> another 2.5GbE switch:
-> HW Agent 129878
-> [other devices]That setup should now remain stable for a month or two until WeLink ISP can persuade me they have solved whatever technical issues they were having and want to attempt again to supplant current ISP (AT&T fiber), after WeLink failed to perform adequately this month.
-
I see @OutagesIO_Support seeming to quote/reply to @SBK and not sure what portion is directed to @SBK versus the OP (me).
The entire purpose of my getting new 2.5Gbps hardware agent was to better monitor outages from new ISP (WeLink) and compare to old ISP (AT&T Fiber). Unfortunately, there may be something else weird going on with the local network which may or may not be an artifact of something the new ISP does (e.g. CGNAT, RF at unknown reliability vs Fiber, maybe some filtering they did not tell me about, etc).Once the bug was quickly fixed about certain but not all 2.5Gbps hardware agents not reporting, I am not sure what's up, but I do know I did NOT keep the LAN and WAN/ISP setup in a steady state suitable for any in-depth monitoring throughout that time, until late evening Chicago time 14 Jan. Work was underway to ensure internet service back online to users. But that no doubt made any troubleshooting of agents difficult.
Over those couple days I had to swap back and forth between the old and new ISP a couple times (eventually switching back the original - had before 2.5GBPS agent installation - ISP) and added a new software agent and put the old yellow hardware agent back online. A few, maybe several, of the times when agent(s) ceased communication that was likely: when ISP tech came to attempt to adjust their RF (did not solve), ISP tech took connection offline, I had to swap back and forth between ISPs by physically moving cable in 2 different floors and 2 separate buildings, I moved HW agent 129878 to a different floor different switch, routers rebooted, etc.
The current setup is since late evening Chicago time 14 Jan is this:
ISP ONT/Router BGW320-500 (in passthrough mode) 3GbE port -> 2.5GbE TPLink Deco BE25 2.5GbE -> 2.5GBe switch:
-> HW 2.5GB Agent 131232
-> [other devices]
-> Win11 PC running SW Agent 131236 (and in live use by a user, so potential PC reboots or disconnects)
-> ~30-40m cat5e cable -> another 2.5GbE switch:
-> HW Agent 129878
-> [other devices]That setup should now remain stable for a month or two until WeLink ISP can persuade me they have solved whatever technical issues they were having and want to attempt again to supplant current ISP (AT&T fiber), after WeLink failed to perform adequately this month.
-
The yellow HW agent works great. And when I swap it for the 2.5GBPS HW agent, 2.5GBS HW agent has problems.
It does seem like the perhaps ISP (WeLink) is doing something weird that may be frustrating the 2.5GBps hardware agent. After leaving this agent on one LAN then on the other, I noticed if I switch this 2.5GBPS agent to the LAN with WeLink as the ISP it gets stuck not communicating (for hours, until I give up). And if I put it on the LAN with AT&T as the ISP it instantly communicates properly.
Power cycle of HW Agent between network swapping does not seem to change this.
Restarting the (WeLink) router also does not seem to change this.OTOH - Attaching yellow HW Agent (130727) instead to that same WeLink ISP LAN on same port on the same WeLink router where 2.5GBPS agent fails, the yellow HW Agent works fine, immediately.
The yellow HW agent works seamlessly and immediately whichever LAN I put it on. The 2.5GBPS HW agent works only on the AT&T LAN. When Yellow HW agent then 2.5GBPS agent are tried on the same ports on the same routers, the yellow HW agent works in all cases, but the 2.5GBPS agent seems to always have trouble on the WeLink LAN. The SW agent (131236) seems to work just as well on either LAN.
-
I am definitely interested in troubleshooting this as you came up with the conclusion that the MT3000 is somehow "WeLink intolerant" (joking of course) since there must be something which is triggering a different behavior.
Just for the sake of info the MT300 can only reach a 100Mbit connection while the MT3000 is able to go beyond that value till the nominal 2.5GB
-
I am definitely interested in troubleshooting this as you came up with the conclusion that the MT3000 is somehow "WeLink intolerant" (joking of course) since there must be something which is triggering a different behavior.
Just for the sake of info the MT300 can only reach a 100Mbit connection while the MT3000 is able to go beyond that value till the nominal 2.5GB
So, for now, I will leave 2.5GBPS HW agent (131232) on the WeLink ISP LAN, but it seems it may never connect. If it would be helpful, I can move it to the other LAN to receive updates. It may be best to wait until WeLink claims (again) to have made some sort of fix or adjustment, and then see if 2.5GBPS HW Agent remains frustrated by whatever odd thing WeLink may be doing.
In current installation, WeLink is providing symmetrical 2GBPS service, and tests very close to that speed with the gateway router's built-in speed test. However, the gateway router WeLink provides (Eero Pro 6E with 2x Eero 6+) has only one 2.5Gbe and one 1GBe port. So, in this setup, WeLink's antenna internet source (DHCP and they use CGNAT) is connected to Eero router's 2.5GBe port, and 2.5GBPS HW agent is connected to the Eero router's 1GBe port. No change when connected to the 1GBe port of one the Eero 6+ (connected to gateway Eero Pro 6E via wireless backhaul). Eero router claims to have given the HW agent an IP address via DHCP and claims to be communicating with the HW agent.
FWIW, when I swapped Eero for TP-Link Deco BE5000 (i.e. 3x Deco BE25), using Deco as the gateway for WeLink, the Deco Router also seems to have problems, going offline after less than 1 minute and coming back only occasionally (or not until after my patience is lost). All the gear (HW agents, Deco Routers, switches) seem to work well when connected to AT&T Fiber gateway.
