Track Internet disconnections, provider outages with historical data, and automated speed testing.
For Windows, Linux, ARM64, ARMa7. Learn more by visiting www.outagesio.com
Notice: If you created an account on app.outagesio.com, simply use the same credentials to log in here.
2.5Gbps Hardware Agent Repeatedly "Rebooting", Missing Data
-
I am definitely interested in troubleshooting this as you came up with the conclusion that the MT3000 is somehow "WeLink intolerant" (joking of course) since there must be something which is triggering a different behavior.
Just for the sake of info the MT300 can only reach a 100Mbit connection while the MT3000 is able to go beyond that value till the nominal 2.5GB
-
I am definitely interested in troubleshooting this as you came up with the conclusion that the MT3000 is somehow "WeLink intolerant" (joking of course) since there must be something which is triggering a different behavior.
Just for the sake of info the MT300 can only reach a 100Mbit connection while the MT3000 is able to go beyond that value till the nominal 2.5GB
So, for now, I will leave 2.5GBPS HW agent (131232) on the WeLink ISP LAN, but it seems it may never connect. If it would be helpful, I can move it to the other LAN to receive updates. It may be best to wait until WeLink claims (again) to have made some sort of fix or adjustment, and then see if 2.5GBPS HW Agent remains frustrated by whatever odd thing WeLink may be doing.
In current installation, WeLink is providing symmetrical 2GBPS service, and tests very close to that speed with the gateway router's built-in speed test. However, the gateway router WeLink provides (Eero Pro 6E with 2x Eero 6+) has only one 2.5Gbe and one 1GBe port. So, in this setup, WeLink's antenna internet source (DHCP and they use CGNAT) is connected to Eero router's 2.5GBe port, and 2.5GBPS HW agent is connected to the Eero router's 1GBe port. No change when connected to the 1GBe port of one the Eero 6+ (connected to gateway Eero Pro 6E via wireless backhaul). Eero router claims to have given the HW agent an IP address via DHCP and claims to be communicating with the HW agent.
FWIW, when I swapped Eero for TP-Link Deco BE5000 (i.e. 3x Deco BE25), using Deco as the gateway for WeLink, the Deco Router also seems to have problems, going offline after less than 1 minute and coming back only occasionally (or not until after my patience is lost). All the gear (HW agents, Deco Routers, switches) seem to work well when connected to AT&T Fiber gateway.
-
Definitely interesting.
I will be trying between tomorrow and Friday to get in touch with you directly using the chat to see if I can test few things I have in mind.
If this week is not possible then I have to ask you to postpone to the week of February 10, since I will be traveling next week -
Is the behavior similar to this post?
-
Is the behavior similar to this post?
@OutagesIO_Support Some issues do seem similar to that post. To ensure the 2.5G agent (131232) receives updates and can restart properly once, I've moved it to the LAN that it DOES work on (AT&T Fiber), to get any overnight updates. Then tomorrow I'll put it back on the (troublesome) WeLink LAN again.
I also will leave Yellow HW agent (130727) onto the WeLink LAN (where yellow HW agent already works fine).
Note that WeLink uses CGNAT, so public IPs are 50.20.112.0/20, but the router sees its own WAN address as 100.64.0.0/10. Not sure if that matters to HW agent. -
The hardware agent would not care what the LAN/WAN addresses are, it will just monitor its own LAN network settings (IP, GW, MASK, DNS) and outgoing paths.
The interesting aspect is that because it does not care about those things, it will always pick up the routes being used so over time, you can get a sense of the routes being used, even if they are CGNAT or SD-WAN for example. Just keep an eye on your hops to see those. Then you can compare with problem hops no matter what the route takes.I hope I'm explaining this correctly.
-
So far, with the 2.5g agent and an older hw yellow agent both connected to the same LAN, the 2.5g hw agent always remains disconnected, despite the router claiming to have given it an IP address. The yellow 100M hardware agent on the other hand stays connected just fine. So I'm back to wondering what's the difference between the two hardware agents that makes one perpetually rebooting or disconnected and the other work normally.
-
Ed will be back soon and will chime in again.
That's something we keep talking about also while monitoring this.
While the hardware is different, the software running on both is the same.The yellow device is much less powerful than the 2.5g but the problem doesn't seem to be related to performance issues with the devices.
This should not cause any unique behavior but something is obviously happening that eludes us so far.
I'm not sure if this was asked before but is there any chance we could get ssh access into both? Details could be shared in private chat of course.
By running some command line tests, maybe we can find some difference between the two.
-
If I can figure out how to set up ssh then sure.
For the next 10 days or so, I cannot mess with the production LAN (AT&T Fiber 1000Mbps), but can do whatever with the WeLink 2000Mbps without impacting any users.Keep in mind that the 2.5GBPS hardware agent seems to work just fine on the AT&T Fiber ISP (1000M) LAN, so there does seem to be something weird related to WeLink:
- Yellow HW agent - works with Welink and AT&T Fiber LAN equally well
- 2.5Gbps HW agent - works on AT&T Fiber service, but problematic on WeLink
- Deco BE5000 mesh router - works on AT&T Fiber service but problematic on WeLink
-
For now, we're testing a new firmware build so as long as you keep it online, we can remotely reboot it to flash the new firmware.
-
If I can figure out how to set up ssh then sure.
For the next 10 days or so, I cannot mess with the production LAN (AT&T Fiber 1000Mbps), but can do whatever with the WeLink 2000Mbps without impacting any users.Keep in mind that the 2.5GBPS hardware agent seems to work just fine on the AT&T Fiber ISP (1000M) LAN, so there does seem to be something weird related to WeLink:
- Yellow HW agent - works with Welink and AT&T Fiber LAN equally well
- 2.5Gbps HW agent - works on AT&T Fiber service, but problematic on WeLink
- Deco BE5000 mesh router - works on AT&T Fiber service but problematic on WeLink
A couple reboots of the (WeLink-provided Eero router and suddenly 2.Gbps HW agent is communicating again.
So I will leave it there a while. WeLink did not tell me they fixed anything, but they may have done, since last time they wanted to troubleshoot it with me I told they I did not have any more time to devote to that until a few weeks from now.I tried disabling IPV6 on their router, which forced a reboot. Then I turned IPV6 back on in their router, which caused another reboot. Then I checked 2.5Gbps HW agent and saw it is communicating again (after reporting being "disconnected" for a couple days).....
unfortunately, it went back to "disconnected" again within about 30 minutes :-(I any event, this does really seem like more of a WeLink issue rather than a 2.5GBps HW agent issue.
-
It does seem like something with WeLink but why. We're perplexed by this since the agent runs fine on one connection but not another. What could be causing such a weird behavior.
When something is mysterious like this, we become even more interested because we never know if it's something that can help us to improve the code and to prevent problems with other members too.
For ssh, we just need a port forward from your router to the LAN IP of the agent. We can lock it down to an incoming IP too if you prefer.
-
Can you power cycle your agent again please. Then let us know if it's working now.
-
No, it's fine, it will reboot at 3am UTC time and does every day.
Just didn't know if it was online or not. -
@recalc
I just need you to confirm that the agent is connected to the WeLink network since as far as I can see , now the agent is behaving as expected.@SBK Yes. Connected to WeLink and working well!
It also came back to working well state after enabling IPV6 in Eero router to check that (07 Feb 0814 CST, which caused router reboot).
So, it seems like you have found a way to overcome whatever WeLink was doing differently than other ISPs (at least at this installation), or perhaps WeLink corrected something on their end (they did not tell me about any change, but they usually don't) -
O OutagesIO_Support has marked this topic as solved on
