Install infinite loop

PJPorch

@OutagesIO_Support I have the exact same problem with an Ubuntu 20.04.3 LTS server install of the agent. It never actually downloads the otm_linux binary.

ICMP does not seem to be the issue. I can ping and get replys from sites.
It does not seem to be the user/password. If I edit the /etc/agentid file to a bad value I get an error "curl: (3) URL using bad/illegal format or missing URL". When I correct the file it tries to download but cannot and gets stuck in this loop.
I have copied the script version 1.0.3 and changed only the "start" variable to the install directory.
I even tried to download the otm_linux binary from another PC onto this server, but nothing seems to work.

Thanks in advance for any tips to get the agent running.

OutagesIO_Support

Can you share the script as you have it now.

PJPorch

@OutagesIO_Support Below is the script and output from the otminfo.log file.

The only change to the script is the start variable at the top
In the folder "/home/josh/agent" I have this script and the otminfo.log file.
I can give you the agent ID if needed.

OTMinfo.log

HTTP/2 200
date: Wed, 09 Mar 2022 01:12:16 GMT
server: Apache
strict-transport-security: max-age=63072000; includeSubDomains
last-modified: Thu, 11 Feb 2021 12:34:03 GMT
etag: "28888-5bb0ebc1544c0"
accept-ranges: bytes
content-length: 166024
vary: User-Agent
content-security-policy: frame-ancestors 'none';
content-secure-policy: default-src 'self';
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
referrer-policy: same-origin
permissions-policy: fullscreen=(self 'https://app.outagesio.com'),geolocation=*, camera=()

Script

#!/bin/bash

#Version 1.0.3

# Enter the full path where you will be placing the starter_linux_otm.sh script into.
# Our example shows '/agent', change this to what ever your path is.
start="/home/josh/agent"

command -v curl >/dev/null 2>&1 || { echo >&2 "The curl package is needed but not installed. Please install it and try again. Aborting."; exit 1; }

echo ""
echo "Make sure to create /etc/agentid with credentials before starting this script"
echo "Make sure your starter file points to the same directory that you have it in - Edit the 'start' option above accordingly"
echo "You could add this to /etc/rc.local to start it or build a service for it. Please search Google for more."
echo ""
sleep 5

### Define Global Variables
OTMIN="otm"
IN="$start/updater_linux_ocp.sh"
DATE=`date +%d-%m-%y`
OTM_COMMAND="otm_linux"
USERNAME=`awk 'NR==1' /etc/agentid`
PASSWORD=`awk 'NR==2' /etc/agentid`

### OTM section
# Get rid of any old version
rm -f $start/otm_linux
OTM_HTTP_STATUS=''
OTM_REMOTE_LENGTH=''

### Loop
until [[ ${OTM_HTTP_STATUS} -eq 200 && ! -z ${OTM_REMOTE_LENGTH} ]]
do
sleep 5
##
echo "Receiving binary location"
BINARY_LOCATION=$(curl -m 20 -s -u $USERNAME:$PASSWORD --connect-timeout 10 -X POST https://www.foxymon.com/receiver/receiver2.php -F function=receive_binary_location)
echo "Getting the OTM Package"
curl -I -m 30 --connect-timeout 15 "$BINARY_LOCATION" -o $start/otminfo.log
OTM_HTTP_STATUS=$(cat $start/otminfo.log | awk '/HTTP/ {print $2}')
OTM_REMOTE_LENGTH=$( cat $start/otminfo.log | awk '/Content-Length/ {print $2}' | sed -e 's/[\r\n]//g' )
done

###
if [[ ${OTM_HTTP_STATUS} -eq 200 && ! -z ${OTM_REMOTE_LENGTH} ]]; then
echo "Downloading OTM"
curl -m 30 --connect-timeout 15 "$BINARY_LOCATION" -o $start/otm_linux
sleep 2
OTM_LOCAL_LENGTH=$( ls -l $start/otm_linux | awk '{print $5}' | sed -e 's/[\r\n]//g' )
if [ "${OTM_LOCAL_LENGTH}" == "${OTM_REMOTE_LENGTH}" ]; then
echo "OTM downloaded, starting program then exiting this script"
cd $start
chmod +x $start/otm_linux
## exec $start/otm_linux &
#$start/otm_linux &

ps cax | grep otm_linux > /dev/null
if [ $? -eq 0 ]; then
echo ""
echo "Congratulations, your agent is now monitoring your Internet connectivity."
echo "Type 'ps cax | grep otm_linux' to see the process number"
else
echo "Something didn't work, your agent is not running."
fi

fi
fi

OutagesIO_Support

Quick look seems to show this script is for the Linux version, not the Pi.
Can you make sure you are using the Pi version. Just create a new agent then copy that script.

If that doesn't work, then please also share the systemd service.

PJPorch

@OutagesIO_Support This is installed on an Ubuntu server, so I am using the linux script. The reason I posted here is I am seeing the exact same issue where it keeps trying to download the otm_linux binary and can never successfully do it. I can repost to the Linux section if required. I was just thinking the symptoms are identical, so the solution might be the same.

I have also tried to disable the service, adjust the script as below, and then run in manually (screenshot). I get the same result where it loops trying to download the file.

Script when service is enabled:

## exec $start/otm_linux &
#$start/otm_linux &

Script when service is disabled and the script is run manually:

# exec $start/otm_linux &
$start/otm_linux &

Here is my service file. Located at: /etc/systemd/system/otm.service

[Unit]
Description=Startup of the OutagesIO monitoring agent

[Install]
WantedBy=multi-user.target

After=network.target

[Service]
Type=simple
ExecStartPre=/home/josh/agent/starter_linux_ocp.sh
ExecStart=/home/josh/agent/otm_linux
WorkingDirectory=/home/josh/agent
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n

Results when running the script directly.

OutagesIO_Support

Yes, it doesn't seem to be finding the binary. Can you try this manually.
Go into the same directory and run both;

curl http://downloads.outagesio.com/otm/otm_1.67.2104_linux -o otm-linux

See if that downloaded the binary. Delete it and try https.

curl https://downloads.outagesio.com/otm/otm_1.67.2104_linux -o otm-linux

If it's there, make it executable then run the version.

$ chmod +x otm-linux
$ ./otm-linux -v

Show me the output please.

PJPorch

@OutagesIO_Support Both the HTTP & HTTPS links successfully download the file. I am able to get the version following your commands as well.

PJPorch

@PJPorch In looking at the script I notice the URL you gave to directly download the file is from outagesio.com, but the script goes to foxymon.com. Could there be an issue with this foxymon.com URL?

BINARY_LOCATION=$(curl -m 20 -s -u $USERNAME:$PASSWORD --connect-timeout 10 -X POST https://www.foxymon.com/receiver/receiver2.php -F function=receive_binary_location)

OutagesIO_Support

Yes, it's correct. Foxymon.com is a domain we started with years ago and never got around to changing it since it would require a ton of programming so we just use it in the background.

The script checks a function which tells it what the download URL is based on the credentials you enter into /etc/agentid.

I'm not sure if you shared this before but can you share the agent ID for this install as well please.

I see three which seem to be yours. 128600 which is active, 128588 which is also active and 128593 which is deleted.

PJPorch

@OutagesIO_Support
Agent 128588 is working without any issues.
This post is regarding Agent 128600.
The 3rd must have been one I created during testing then deleted.

OutagesIO_Support

@PJPorch said in Install infintie loop:

This post is regarding Agent 128600.

This agent is currently running, it's in Active status.

PJPorch

@OutagesIO_Support

Nothing is being populated in the ping field or recent events.
The public IP is not updated to the correct address.
The service on the Ubuntu machine shows it is running, but I cannot tell if it was able to download the agent or not. Manually downloading might have helped this get started.

OutagesIO_Support

Wait now, one thing at a time. We need to be in sync.
You said that you were not able to run the agent at all, that it kept downloading but that's all.

The script shows you can use this to see if the agent is running.

ps cax | grep otm_linux

Can you check so we can see if there are one or more instances running.

Since we see it running, kill it so we have a starting point.

The first thing is to download the agent on it's own, without using the script, make it executable then start it then run the above command to confirm it is running and only one is.

If you see it running, then kill the process and we'll do this next.

Next, try this.

curl -m 20 -s -u xxxx:xxxxxxxxxxxxxxxxxx --connect-timeout 10 -X POST https://www.foxymon.com/receiver/receiver2.php -F function=receive_binary_location

The above is from the script. You replace the xxxx:xxxxxxxxxxxxxxxxxx with the credentials you have /etc/agent. Note the semicolon.

What this does is to query your account for the new agent you've created so it can give you the latest version of the agent for that particular OS. Show me the output of this please.

We'll deal with the rest one thing at a time to prevent confusing each other.

PJPorch

@OutagesIO_Support
Sorry...I guess I was jumping ahead when you said you showed the agent as active.

I have stopped the service from running.

sudo service otm status

If I search for the running process with your script I get nothing.

ps cax | grep otm_linux

If I search for the name of the script I do get some results, so I manually have killed all processes running the script,

ps -aux | grep 'starter_linux_ocp'

When I run the curl command with my credentials I get a response of
"https://downloads.outagesio.com/otm/otm_1.67.2104_linux" which looks like it exactly matches the URL I was able to manually download from, so that part of the script seems to be working correctly.

OutagesIO_Support

I think the first thing would be to kill everything that's been left over from your testing so you are starting from scratch. make sure the service is stopped, all 'starter' scripts processes are killed.

Then check your dashboard until the agent is Inactive. Once you know there is no agent or anything else related running, start your agent manually, not using systemd and check if the agent becomes active.

Once sure there is nothing else running and the agent goes active, then kill that process and start the agent using your systemd start command. I don't know for sure but it looks like that part was working, only there were other instances running too which could have caused conflicts maybe. Let's get to the point of the agent running and we'll look into the other stuff after.

PJPorch

@OutagesIO_Support

Disabled the service
Killed all running processes
Agent shows as "Inactive"
Edited the script to run manually.

# exec $start/otm_linux &
$start/otm_linux &

Run the script manually and I'm right back at the beginning where it won't download the file. There are no apparent errors with credentials, permission to download the file, or anything indicating why it can't download the file.

OutagesIO_Support

You seem to be doing all the right things. It's a little perplexing to say the least. It must be something small and so obvious that it's not obvious.

We will dig deeper into this tomorrow because it really isn't making sense and must be something on our end.

OutagesIO_Support

Ok, some good news for you. The problem should be resolved now.
Just create a new agent then download the updated script, version 1.0.5. You can let that new agent get deleted automatically by the system in 24hrs. Use that script and as before, make sure to upgrade the path's of course.

It should work now.

Also made a small change in the systemd if you're using it. Still have to solve why it's echo'ing out the text from the script but it works.

[Unit]
Description=Startup of the OutagesIO Internet monitor agent
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
ExecStartPre=/otm/starter.sh
ExecStart=/otm/otm_binary
WorkingDirectory=/otm
Restart=always
RestartSec=5
StandardError=syslog
SyslogIdentifier=%n

Give it a try, let us know how it goes.

PJPorch

@EchoMin
Whatever changes you made to the script seems to have fixed it. I deleted everything, reinstalled with the new scripts, and everything seems to be downloading and running properly.

Thanks for all the help.
Josh

OutagesIO_Support

Nice to hear.

Which one is your most important agent ID? One that you'll keep going without deleting? We'll keep an eye on that for a bit too.

Information and Support

Install infinite loop