2.5Gbps Hardware Agent Repeatedly "Rebooting", Missing Data

[SBK]

Definitely interesting.
I will be trying between tomorrow and Friday to get in touch with you directly using the chat to see if I can test few things I have in mind.
If this week is not possible then I have to ask you to postpone to the week of February 10, since I will be traveling next week

[OutagesIO_Support]

Is the behavior similar to this post?

https://support.outagesio.com/topic/302/hw-agent-works-fine-on-one-lan-fails-to-communicate-on-another-why

[recalc]

@OutagesIO_Support Some issues do seem similar to that post. To ensure the 2.5G agent (131232) receives updates and can restart properly once, I've moved it to the LAN that it DOES work on (AT&T Fiber), to get any overnight updates. Then tomorrow I'll put it back on the (troublesome) WeLink LAN again.
I also will leave Yellow HW agent (130727) onto the WeLink LAN (where yellow HW agent already works fine).
Note that WeLink uses CGNAT, so public IPs are 50.20.112.0/20, but the router sees its own WAN address as 100.64.0.0/10. Not sure if that matters to HW agent.

[OutagesIO_Support]

The hardware agent would not care what the LAN/WAN addresses are, it will just monitor its own LAN network settings (IP, GW, MASK, DNS) and outgoing paths.
The interesting aspect is that because it does not care about those things, it will always pick up the routes being used so over time, you can get a sense of the routes being used, even if they are CGNAT or SD-WAN for example. Just keep an eye on your hops to see those. Then you can compare with problem hops no matter what the route takes.

I hope I'm explaining this correctly.

[recalc]

So far, with the 2.5g agent and an older hw yellow agent both connected to the same LAN, the 2.5g hw agent always remains disconnected, despite the router claiming to have given it an IP address. The yellow 100M hardware agent on the other hand stays connected just fine. So I'm back to wondering what's the difference between the two hardware agents that makes one perpetually rebooting or disconnected and the other work normally.

[OutagesIO_Support]

Ed will be back soon and will chime in again.

That's something we keep talking about also while monitoring this.
While the hardware is different, the software running on both is the same.

The yellow device is much less powerful than the 2.5g but the problem doesn't seem to be related to performance issues with the devices.

This should not cause any unique behavior but something is obviously happening that eludes us so far.

I'm not sure if this was asked before but is there any chance we could get ssh access into both? Details could be shared in private chat of course.

By running some command line tests, maybe we can find some difference between the two.

[recalc]

If I can figure out how to set up ssh then sure.
For the next 10 days or so, I cannot mess with the production LAN (AT&T Fiber 1000Mbps), but can do whatever with the WeLink 2000Mbps without impacting any users.

Keep in mind that the 2.5GBPS hardware agent seems to work just fine on the AT&T Fiber ISP (1000M) LAN, so there does seem to be something weird related to WeLink:

• Yellow HW agent - works with Welink and AT&T Fiber LAN equally well
• 2.5Gbps HW agent - works on AT&T Fiber service, but problematic on WeLink
• Deco BE5000 mesh router - works on AT&T Fiber service but problematic on WeLink

[OutagesIO_Support]

For now, we're testing a new firmware build so as long as you keep it online, we can remotely reboot it to flash the new firmware.

[recalc]

A couple reboots of the (WeLink-provided Eero router and suddenly 2.Gbps HW agent is communicating again.
So I will leave it there a while. WeLink did not tell me they fixed anything, but they may have done, since last time they wanted to troubleshoot it with me I told they I did not have any more time to devote to that until a few weeks from now.

I tried disabling IPV6 on their router, which forced a reboot. Then I turned IPV6 back on in their router, which caused another reboot. Then I checked 2.5Gbps HW agent and saw it is communicating again (after reporting being "disconnected" for a couple days).....
unfortunately, it went back to "disconnected" again within about 30 minutes :-(

I any event, this does really seem like more of a WeLink issue rather than a 2.5GBps HW agent issue.

[OutagesIO_Support]

It does seem like something with WeLink but why. We're perplexed by this since the agent runs fine on one connection but not another. What could be causing such a weird behavior.

When something is mysterious like this, we become even more interested because we never know if it's something that can help us to improve the code and to prevent problems with other members too.

For ssh, we just need a port forward from your router to the LAN IP of the agent. We can lock it down to an incoming IP too if you prefer.

[OutagesIO_Support]

Can you power cycle your agent again please. Then let us know if it's working now.

[recalc]

Looks like it is working well right now.
Still want me to power cycle it?

[OutagesIO_Support]

No, it's fine, it will reboot at 3am UTC time and does every day.
Just didn't know if it was online or not.

[SBK]

@recalc
I just need you to confirm that the agent is connected to the WeLink network since as far as I can see , now the agent is behaving as expected.

[recalc]

@SBK Yes. Connected to WeLink and working well!
It also came back to working well state after enabling IPV6 in Eero router to check that (07 Feb 0814 CST, which caused router reboot).
So, it seems like you have found a way to overcome whatever WeLink was doing differently than other ISPs (at least at this installation), or perhaps WeLink corrected something on their end (they did not tell me about any change, but they usually don't)