Agent inactive / disconnected

[fbernitt]

Hey,

my agent running on my raspberry pi stays inactive or disconnected. Restarting and even resetting the agent does unfortunately not help to get the agent back to running. I am using the agent at URL: https://downloads.echonets.com/otm/arm64_otm_1_81_2401.

The /tmp/otm.log is empty.

What might be related: I can't ping 'tpw.outagesio.com' from neither my home network, nor from other servers in Germany (hosted at hetzner):

PING tpw.outagesio.com (66.85.130.3) 56(84) bytes of data.
From 66.85.130.3 (66.85.130.3) icmp_seq=1 Destination Host Unreachable
From 66.85.130.3 (66.85.130.3) icmp_seq=2 Destination Host Unreachable

I can ping other machines (e.g., www.google.com) without problems. Last week I observed that the ping times collected by the agent increased significantly, somewhere around 140ms on average.

Any suggestions on how to get the agent back to work?

Best

[SBK]

Let me investigate first since that host must be available always

[OutagesIO_Support]

Hi, are you able to ping the tpw servers now?
Target hosts must always be up.
As for ping times, not sure yet, might have been traffic being rerouted.

There was in fact a problem with that which we are also investigating.

[fbernitt]

Hey, thanks a lot!

Yes, I can ping tpw.outagesio.com again:

PING tpw.outagesio.com (66.85.130.3) 56(84) bytes of data.
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=1 ttl=47 time=150 ms
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=2 ttl=47 time=148 ms
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=3 ttl=47 time=146 ms
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=4 ttl=47 time=146 ms
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=5 ttl=47 time=150 ms
64 bytes from 66.85.130.3 (66.85.130.3): icmp_seq=6 ttl=47 time=148 ms
...

The slow ping times could be related to some kind of peering problem, at least a steep increase happens within the twelve99 networks (measured with mtr tpw.outagesio.com:

 6. ffm-b5-link.ip.twelve99.net                                                                                                                                                                   0.0%   207    7.9   8.9   6.7  23.3   1.6
 7. ffm-bb1-link.ip.twelve99.net                                                                                                                                                                  0.0%   207    6.9   8.9   6.9  11.2   1.2
 8. prs-bb1-link.ip.twelve99.net                                                                                                                                                                 59.4%   207   18.5  17.5  15.2  19.8   1.2
 9. rest-bb1-link.ip.twelve99.net                                                                                                                                                                 6.3%   207   93.3  93.3  91.1  96.7   1.2
10. atl-bb1-link.ip.twelve99.net                                                                                                                                                                  0.0%   207  111.2 111.9 109.7 115.1   1.2
11. nash-bb1-link.ip.twelve99.net                                                                                                                                                                80.6%   207  116.3 116.4 114.0 123.5   1.7
12. dls-bb1-link.ip.twelve99.net                                                                                                                                                                 88.3%   207  128.2 127.9 125.8 129.5   1.1
13. dls-b23-link.ip.twelve99.net                                                                                                                                                                  0.0%   207  149.7 149.6 147.2 165.7   2.1
14. dls-bb2-link.ip.twelve99.net                                                                                                                                                                  0.5%   206  127.1 127.6 125.3 130.4   1.1
15. phx-b6-link.ip.twelve99.net                                                                                                                                                                   0.0%   206  147.9 149.6 147.1 165.5   1.9

And it seems to also have fixed my inactive agent problem - after restart, it is now online again.

Thanks again!

[OutagesIO_Support]

Yes, ping times are affected by which hops they flow through. We always wanted a more 'real world' result for pings so wanted those to go through many switches getting to our network. We just needed a reliable destination. It didn't matter where or what, so long as the agents algorithm was able to build up averages from which it would recognize latency changes.

We've tested using closer (to the agent) destinations but it's never 'real world' like. We keep testing this to see how we can improve it.

Thanks very much for bringing this up. We're looking into the other things too.