How to monitor your internet connection with virtual custom sensors

Client tools or tools that can connect with Domoticz. Tools for Windows, iOS, Android, Linux etc.
User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

How to monitor your internet connection with virtual custom sensors

Postby safi78 » Saturday 22 October 2016 17:38

Hi all,

I was having some troubles with my internet connection and couldn't figure out where they were coming from. I really like the graphing functionality of domoticz and I have implemented a script that notifies me with Telegram, so I wanted to monitor my download speed with domoticz, thought I'd share:

1. Install speedtest

Code: Select all

sudo apt-get install speedtest-cli


2. Create 3 Virtual Sensors of the type 'Custom Sensors' (for ping, download, upload-values) and remember the idx's (you need that later on)

3. Create a script (I called it st2domo.sh) with:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
pingidx=xxx
downloadidx=xxx
uploadidx=xxx

# no need to edit
speedtest-cli --simple > output.txt
ping=$(cat output.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')
download=$(cat output.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat output.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')

#output if you run it manually
echo "ping = $ping ms"
echo "download = $download Mbps"
echo "upload =  $upload Mbps"

curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$pingidx&svalue=$ping"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$downloadidx&svalue=$download"
curl -s -i -H "Accept: application/json" "http://$username:$password@$host:$port/json.htm?type=command&param=udevice&idx=$uploadidx&svalue=$upload"


and fill in all the details after #setup:
host = your domoticz server (e.g. localhost or 192.168.x.x)
port = your domoticz port (e.g. 8080)
username = your domoticz username
password = your domoticz password
pingidx = the idx of the virtual sensor for accepting the ping-value
downloadidx = the idx of the virtual sensor for accepting the download-value
uploadidx = the idx of the virtual sensor for accepting the upload-value

Don't forget to make it executable (I always do):

Code: Select all

chmod +x st2domo.sh


4. edit your crontab (I scheduled this @hourly):

Code: Select all

crontab -e


And put in the command:

Code: Select all

@hourly /path/to/scrip/st2domo.sh


If setup correctly, domoticz will receive your results from speedtest every hour:

Screen Shot 2016-10-22 at 19.57.29.png
Screen Shot 2016-10-22 at 19.57.29.png (124.73 KiB) Viewed 2821 times


And you can do 'useful' stuff with it ;)

Enjoy!
Last edited by safi78 on Wednesday 02 November 2016 18:26, edited 17 times in total.
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

woody4165
Posts: 391
Joined: Monday 14 March 2016 14:55
Target OS: Linux
Domoticz version: beta
Location: Rome, Italy
Contact:

Re: monitor internet download speed with virtual sensor

Postby woody4165 » Saturday 22 October 2016 18:20

Hi and thanks for sharing your script.

I have not found speedtest as package, but only speedtest-cli and probably the usage is different.
Should I add some source to download this package?

Thanks

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet download speed with virtual sensor

Postby safi78 » Saturday 22 October 2016 18:22

my bad, you install 'speedtest-cli' but the command you use is 'speedtest' (so the script is ok). I'll update my post accordingly.

Thanks for the heads up!
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

manjh
Posts: 142
Joined: Saturday 27 February 2016 13:49
Target OS: Windows
Domoticz version:
Contact:

Re: monitor internet download speed with virtual sensor

Postby manjh » Saturday 22 October 2016 18:36

This is great for the R-Pi, but is there an alternative for a Windows based server?
Hans

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet download speed with virtual sensor

Postby safi78 » Saturday 22 October 2016 18:37

manjh wrote:This is great for the R-Pi, but is there an alternative for a Windows based server?


Sorry manjh, I do not use Windows ever.
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

woody4165
Posts: 391
Joined: Monday 14 March 2016 14:55
Target OS: Linux
Domoticz version: beta
Location: Rome, Italy
Contact:

Re: monitor internet download speed with virtual sensor

Postby woody4165 » Saturday 22 October 2016 19:10

@safi78
it's working like a charm, just replaced

Code: Select all

speedtest
with

Code: Select all

speedtest-cli
and it's ok.
I've also added --server xxxx where xxx is the server number near to myself so that test is more accurate.
You can find your test server with

Code: Select all

speedtest-cli --list

and you will find the number associated to the server name.

So my test is:

Code: Select all

speedtest-cli --server xxxx | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')

Do you know how can I get also ping value?

the return is like this:
......[15.27 km]: 12.356 ms

Thanks!

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet download speed with virtual sensor

Postby safi78 » Saturday 22 October 2016 19:32

changing the speedtest command to speedtest --simple will just produce this output:

Code: Select all

Ping: 27.302 ms
Download: 36.26 Mbits/s
Upload: 5.03 Mbits/s


Then you can use:

Code: Select all

sed -ne 's/^Ping: \([0-9]*.[0-9]*\).*/\1/p'


results to:

Code: Select all

27.302
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

woody4165
Posts: 391
Joined: Monday 14 March 2016 14:55
Target OS: Linux
Domoticz version: beta
Location: Rome, Italy
Contact:

Re: monitor internet download speed with virtual sensor

Postby woody4165 » Saturday 22 October 2016 19:36

thanks again!!

what i would like to understand is if I have to run three times the speedtest to get DL,UP and Ping, or if there is a way to get all the three numbers with one speedtest.

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet download speed with virtual sensor

Postby safi78 » Saturday 22 October 2016 19:38

Sure, this was just a quick hack ;)

just run speedtest --simple > output.txt

then cat output.txt | sed (etc) ?

edit: I'll change the script to that, more efficient (I was only chasing after the Download value ;) )
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

woody4165
Posts: 391
Joined: Monday 14 March 2016 14:55
Target OS: Linux
Domoticz version: beta
Location: Rome, Italy
Contact:

Re: monitor internet download speed with virtual sensor

Postby woody4165 » Saturday 22 October 2016 19:49

Fantastic

so now my script is like this:

Code: Select all

#!/bin/bash

#setup
host=xxx.xxx.xxx.xxx
port=xxxx
username=xxx
password=xxx
#idx for download, upload and ping
idxdl=xxx
idxul=xxx
idxpng=xxx

# speedtest server number
serverst=xxxx


# no need to edit
speedtest-cli --simple --server $serverst > outst.txt

download=$(cat outst.txt | sed -ne 's/^Download: \([0-9]*\.[0-9]*\).*/\1/p')
upload=$(cat outst.txt | sed -ne 's/^Upload: \([0-9]*\.[0-9]*\).*/\1/p')
png=$(cat outst.txt | sed -ne 's/^Ping: \([0-9]*\.[0-9]*\).*/\1/p')


#output if you run it manually
echo "download = $download Mbps"
echo "upload =  $upload Mbps"
echo "ping =  $png ms"


# Updating download, upload and ping ..
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxdl&svalue=$download" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxul&svalue=$upload" >/dev/null 2>&1
wget -q "http://$host:$port/json.htm?type=command&param=udevice&idx=$idxpng&svalue=$png" >/dev/null 2>&1


working and updating.

Perfect!

Thanks again
Last edited by woody4165 on Saturday 22 October 2016 20:14, edited 1 time in total.

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet download speed with virtual sensor

Postby safi78 » Saturday 22 October 2016 19:52

you're welcome & thanks for your input as well :)
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

User avatar
jumbotroll
Posts: 471
Joined: Tuesday 13 January 2015 15:36
Target OS: NAS (Synology & others)
Domoticz version: Beta
Location: Oslo,Norway
Contact:

Re: monitor internet download speed with virtual sensor

Postby jumbotroll » Saturday 22 October 2016 20:20

manjh wrote:This is great for the R-Pi, but is there an alternative for a Windows based server?


On windows 10 it's working here.
Install python 3.5.2 from https://www.python.org/downloads/
Open cmd as Administrator and type: pip install speedtest-cli
Then you can type: speedtest
Regards
Jumbotroll
http://www.jadahl.com

simonrg
Posts: 1041
Joined: Tuesday 16 July 2013 22:54
Target OS: Raspberry Pi
Domoticz version:
Location: North East England
Contact:

Re: monitor internet connection with virtual sensors

Postby simonrg » Sunday 23 October 2016 1:01

Nice approach, a couple of comments.

I only discovered that speedtest appears to download 30MB per test, when I used up my 50GB limit in 7 days by testing every 5 minutes :oops: (~4GB per day :lol: ).

I took a slightly different approach based on a mash up of ideas from this forum.

A simple bash script runs speedtest:

Code: Select all

#!/bin/sh
/usr/bin/speed-test --json >/var/tmp/speed-test.json 2>&1

set to run by cron every hour on the hour (min=00). The script could also run via a timer in Domoticz to keep more of the setup inside Domoticz.

A Lua script runs at 3 minutes past the hour to read the file created by speedtest, decode the json, send the value to Domoticz via commandArray and send a notification if download value is below 3.0Mbps.

Code: Select all

-- ~/domoticz/scripts/lua/script_time_speed.lua

commandArray = {}

JSON  = require('JSON')
date = os.date("*t")

function dospeed(filename, devdown, devup, devping, name, index)
-- filename - full path and filename where results from speedtest are stored
-- devdown, deveup, devping - Domoticz virtual sensor indexes
-- name - used in notification to identify isp
-- index - normal 1, increment by 5 for each additional internet connection tested
    file = io.open(filename, "r")
    json_text = file:read()
    file:close()
    speeds = JSON:decode(json_text)
    print(name..": Ping="..speeds.ping.." Download=".. speeds.download.." Upload=".. speeds.upload)
    commandArray[index]={['UpdateDevice']=devdown..'|0|'..speeds.download}
    commandArray[index+1]={['UpdateDevice']=devup..'|0|'..speeds.upload}
    commandArray[index+2]={['UpdateDevice']=devping..'|0|'..speeds.ping}
    if speeds.download <3.0 then
       commandArray[index+3]={['SendNotification']=name..' internet download speed alert#'..name..' download '..speeds.download..'Mb/s'}
    end
    if speeds.upload <1.0 then
       commandArray[index+4]={['SendNotification']=name..' internet upload speed alert#'..name..' upload '..speeds.upload..'Mb/s'}
    end
end

-- Once an hour
if (date.min == 3) then
--    os.execute('wget http://ipinfo.io/ip -qO - > /var/tmp/ip.txt')
-- 1st internet connection
    dospeed("/var/tmp/speed-test.json", 958, 959, 960, "Eclipse", 1)
-- 2nd internet connection
    dospeed("/var/tmp/speed-test-lnc.json", 961, 962, 963, "Boundless", 6)
end

return commandArray
Raspberry Pi 2 B - [email protected] PSU - Raspbian + Domoticz + RFXtrx(89), LightwaveRF House(dimmers, sockets, wireless/mood switches), Owl CM113, 4 LaCross Temp / Humidity Sensors, 4 Siemens PIR, Smappee, Solaredge, ESP8266

woody4165
Posts: 391
Joined: Monday 14 March 2016 14:55
Target OS: Linux
Domoticz version: beta
Location: Rome, Italy
Contact:

Re: monitor internet connection with virtual sensors

Postby woody4165 » Sunday 23 October 2016 7:54

Hi simonrg

one question is why you should test the line every five mins...

In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test

Thanks

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet connection with virtual sensors

Postby safi78 » Sunday 23 October 2016 10:23

Nice example with JSON, like it.

But same here, the --json doesn't seem to be included in the debian package? What kind of speedtest are you using?

Regards,

Sander
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

simonrg
Posts: 1041
Joined: Tuesday 16 July 2013 22:54
Target OS: Raspberry Pi
Domoticz version:
Location: North East England
Contact:

Re: monitor internet connection with virtual sensors

Postby simonrg » Sunday 23 October 2016 12:46

woody4165 wrote:one question is why you should test the line every five mins...
In any case, nice approach.
with speedtest-cli there is no --json option
where can I install from speed-test command? I tried sudo apt-get install speed-test

Why every 5 minutes? Why not? Ok because it uses up my bandwith, but a realtime monitor would allow me to spot issues early and sort them out.

I have both a fibre and an over the air long range wifi-connection. While speed at one time can appear to be fast, I still to see stuttering on streaming services etc., so I was trying to see if there was anything systematic - farmer moving tractor into line of sight, wind blowing trees into line of sight etc..

So I am not sure once an hour is really useful to track long term trends, but not really useful to me for intermittant faults, as fault could already have happened 55minutes ago, and more annoylingly I could leave home after fault has occured without realising, so could have fixed it if I had known. Basically speed test is not the not the way to do it, I really need access to ISP network infrastructure as this will have the information I need on quality of connection with no unnecessary bandwidth.

For example, this morning
161023Speed.jpeg
161023Speed.jpeg (39.31 KiB) Viewed 2680 times
my speed has dropped from 10 to <2Mbps, I have no idea why, no wind no rain no obvious obstructions, but lets see what ISP says.

I am running on Raspberry Pi, I got speed-test from https://www.npmjs.com/package/speed-test.

I had forgotten my code is largely copied from https://www.domoticz.com/forum/viewtopic.php?t=11970#p86162 which has a lot more discussion of measuring network speed.
Raspberry Pi 2 B - [email protected] PSU - Raspbian + Domoticz + RFXtrx(89), LightwaveRF House(dimmers, sockets, wireless/mood switches), Owl CM113, 4 LaCross Temp / Humidity Sensors, 4 Siemens PIR, Smappee, Solaredge, ESP8266

User avatar
safi78
Posts: 374
Joined: Thursday 05 September 2013 15:39
Target OS: Linux
Domoticz version: LastBeta
Location: Alkmaar, NL
Contact:

Re: monitor internet connection with virtual sensors

Postby safi78 » Sunday 23 October 2016 15:38

simonrg wrote:Why every 5 minutes? Why not?


Because it will try to suck the shit out of your connection every 5 minutes and that will affect everything else you do? ;)

Even if it is just 30MB every 5 minutes you'll have major lag on everything else right.
I don't know if you noticed or not, but I'm an extremely arrogant man who tends to think all of his plans will work ...

https://safiweb.nl

homebridge, rfxcom, zwave, nest, applamp, hue, debian, apple, mysensors, netatmo, fibaro, synology, foscam, otherz

User avatar
Derik
Posts: 1515
Joined: Friday 18 October 2013 23:33
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Arnhem/Nijmegen Nederland
Contact:

Re: monitor internet connection with virtual sensors

Postby Derik » Sunday 23 October 2016 17:08

@Safi78

Try it:

Code: Select all

[email protected] ~ $ sudo apt-get install speedtest-cli
Pakketlijsten worden ingelezen... Klaar
Boom van vereisten wordt opgebouwd
De status informatie wordt gelezen... Klaar
E: Kan pakket speedtest-cli niet vinden
[email protected] ~ $


Edit:

Did some work:

Code: Select all

 sudo apt-get install python-pip


Then:

Code: Select all

pip install speedtest-cli


Then try the script :
cd pad to the script:
then:
./scriptname.sh

And working... :D


Only what is the time when i use:
@hourley

Edit 2:
I do not like to run this script exactly @ x.00 hour.
So i made the crondtab:

Code: Select all

11 0 * * * /home/pi/domoticz/scripts/speedtest.sh

11 minutes past a hour..
Hope this is going to work, and not kill my internet speed every hour...

Thanks guys.. for this script!!
CB 2: Beta Extreme antenna RFXcomE,WU Fi Ping ip P1 Gen5 PVOutput Harmony HUE SolarmanPv OTG Winddelen Alive ESP Buienradar MySensors WOL Winddelen counting RPi: Beta SMAspot RFlinkTest Domoticz ...Different backups

User avatar
Derik
Posts: 1515
Joined: Friday 18 October 2013 23:33
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Arnhem/Nijmegen Nederland
Contact:

Re: monitor internet connection with virtual sensors

Postby Derik » Sunday 23 October 2016 18:43

mm
Only i stupid question...
What is the best ping high or low..?
CB 2: Beta Extreme antenna RFXcomE,WU Fi Ping ip P1 Gen5 PVOutput Harmony HUE SolarmanPv OTG Winddelen Alive ESP Buienradar MySensors WOL Winddelen counting RPi: Beta SMAspot RFlinkTest Domoticz ...Different backups

User avatar
emme
Posts: 379
Joined: Monday 27 June 2016 11:02
Target OS: Raspberry Pi
Domoticz version: latest
Location: Milano, Italy
Contact:

Re: monitor internet connection with virtual sensors

Postby emme » Sunday 23 October 2016 18:55

low ping means fast response.... so it's good
high ping means slow response... that' bad

Watch out the unit: normally it's ms-->milliseconds
That's one small step for a programmer, one giant leap for me!
- - - - -
Raspberry Pi3 + AeonStick G5 + RFXCOMM433 + iRTrans Server + NodeJS
There's no stress in the mess!!


Return to “Third party and supplementary tools”

Who is online

Users browsing this forum: No registered users and 1 guest