Python plugin: Presence detection from wireless router

Python and python framework
Prutsium
Posts: 70
Joined: Monday 18 June 2018 11:31
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Germany / Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Prutsium » Sunday 28 October 2018 12:54

As my stupid cable provider provided me with a new router (Fritzbox 6490) i had to change my network setup.
Now the Asus is back to dumb AP mode and most functions are off (including the DHCP) so i was affraid i would lose all the presence functions.

But somehow the Asus is still able to find all devices in the network even when it's not acting as DHCP server and just a dumb gateway.
Even the devices that connect to the FritzBox directly seems to be found.

Pitty there is no option (anymore) to get SSH at the Fritzbox but at least this way it seems to work.

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Sunday 28 October 2018 13:05

@dextm80

De plugin uses SSH, not telnet. If that is also unavailable, then the plugin will indeed not work. Do you have another router in your setup? The arp tables might be a bit slower to update, but it might be usable.

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Sunday 28 October 2018 13:16

@Prutsium

The plugin does not use DHCP (arp, brctl etc have nothing to do with DHCP) .Is the plugin using arp or brctl on ANY of your routers (should be visible in the log)? That would explain why it detects devices connected tot other acces points. Arp and brctl info will propagate throughout the network, since switches and routers have to know where network clients are located (by mac address).

DHCP assigns IP addresses to clients... something the plugin doesn't care about.

But is always nice to find out something is still working when you didn't expect it to :D

Prutsium
Posts: 70
Joined: Monday 18 June 2018 11:31
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Germany / Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Prutsium » Sunday 28 October 2018 13:43

Yupp forgot to mention i was using it (also before) on BRCTL due the setup at that time (Routers behind the Asus).

So anyway still happy it works as i depend on the presence detection heavily for my Domotica setup.

dextm80
Posts: 102
Joined: Tuesday 24 October 2017 18:32
Target OS: Linux
Domoticz version: 4.10159
Contact:

Re: Python plugin: Presence detection from wireless router

Post by dextm80 » Sunday 28 October 2018 19:16

EscApe wrote:
Sunday 28 October 2018 13:05
@dextm80

De plugin uses SSH, not telnet. If that is also unavailable, then the plugin will indeed not work. Do you have another router in your setup? The arp tables might be a bit slower to update, but it might be usable.
yes, i ve a linksys and tplink router that is used it for switch
Domoticz on AsRock j3455-ITX 8gb ram - Aeotec ZWave Usb Stick - RFLink 433Mhz
1x Fibaro Wall Plug
1x Fibaro Motion Sensor
2x NeoCoolcam Wall Plug
Netatmo Weather Station - Netatmo Thermostat
Philips Hue Bridge

Freekers
Posts: 17
Joined: Friday 20 October 2017 17:36
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Freekers » Sunday 28 October 2018 19:40

EscApe wrote:
Sunday 28 October 2018 9:54
Glad it's working :D

You might be able to tweak arp table management on the routerboard. Maybe set some time-outs(?) I don't know the platform, but it seems well documented and very tweakable. Refreshing the arp table on every polls seems a bit drastic. The plugin might miss mac addresses if the table isn't refreshed quick enough.

Querying the wireless interfaces will probably work much better than the generic arp command. You could investigate how to get the associated mac addresses from the wireless chipset. That is where the plugin is at is best.

Another question to really support RouterOS in the plugin is how to detect the routerboard as a routerboard with a command that will consistently identify the routerboard and that will consistently fail on other routers/firmwares. That way it can be included in the auto detection mechanism of the plugin.
Eg.
To check if a router has a broadcom chipset the plugin will try to run: 'type wl' on the router. type will return an error code if the wl command is not available. The plugin uses this info (and more checks like that) to identify the router. Since routeros offers no real linux-like shell the 'type'' test will not work.
I agree, refreshing the arp table on every poll is a bit drastic.

I've found a way to query the wireless interface using SSH. By executing the command

Code: Select all

interface wireless registration-table print
you get the following table:
Image

This table is instantly updated when the connection of a device drops or reconnects. If it's possible to incorporate this command instead of the arp table, that would be awesome and indeed the best solution :D !

I've also found a command to detect the routerboard, which will consistently identify it as a routerboard using the command

Code: Select all

system routerboard print
The output of this command is as follows:
Image

Please let me know if you need any more debug output.

Kind regards.

Prutsium
Posts: 70
Joined: Monday 18 June 2018 11:31
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Germany / Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Prutsium » Monday 29 October 2018 9:21

EscApe wrote:
Sunday 28 October 2018 13:16
@Prutsium

The plugin does not use DHCP (arp, brctl etc have nothing to do with DHCP) .Is the plugin using arp or brctl on ANY of your routers (should be visible in the log)? That would explain why it detects devices connected tot other acces points. Arp and brctl info will propagate throughout the network, since switches and routers have to know where network clients are located (by mac address).

DHCP assigns IP addresses to clients... something the plugin doesn't care about.

But is always nice to find out something is still working when you didn't expect it to :D
Seems the new way was not very stable as i see various dropouts that my phone went offline (sleep mode).

Did anyone try this: https://github.com/gasperphoenix/FritzB ... eDetection
Not sure if it works but would be nice addition to iDetect :)
(Will start trying it to see if it even works)

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Monday 29 October 2018 21:17

@Freekers

Great finds!
The latest version (just uploaded) uses the wireless command you supplied.

<routerip>=routeros will force the routeros wireless query
<routerip>=routeros-arp will force arp in the routeros syntax

For now it will remain a custom setting. Using the auto detection script on the routerboard seems like a bad idea in the current approach, because the linux commands will just generate lots of errors and i don't want to break up the script into even more parts (with ssh session overhead for every one).

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Monday 29 October 2018 21:33

@Prutsium

Yeah, thats the downside of the brctl method. I can imagine it will be more susceptible to (sleepmode) dropouts in a multi router environment because the sleeping device will produce less traffic in the network. A longer 'grace period' in the plugin settings might help.

The detection tool you are referring to uses the information from a http source/webpage on the fritzbox. It will probably work better on the fritzbox because the information is very likely based on the same tools my plugin uses over ssh. I have no short term plans to include other methods than ssh in my plugin. It would be nice to include http (also for status pages on mesh setups) or even ping as a fallback for some specific devices.

I did see some info on the web (google) on how to enable/install ssh on a fritzbox but the info looked outdated and might only be usable on much older routers.

Freekers
Posts: 17
Joined: Friday 20 October 2017 17:36
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Freekers » Monday 29 October 2018 22:12

EscApe wrote:
Monday 29 October 2018 21:17
@Freekers

Great finds!
The latest version (just uploaded) uses the wireless command you supplied.

<routerip>=routeros will force the routeros wireless query
<routerip>=routeros-arp will force arp in the routeros syntax

For now it will remain a custom setting. Using the auto detection script on the routerboard seems like a bad idea in the current approach, because the linux commands will just generate lots of errors and i don't want to break up the script into even more parts (with ssh session overhead for every one).
I've just tested the new version; it works perfectly! Thank you very very much for the great support & quick fixes. Awesome!

Freekers
Posts: 17
Joined: Friday 20 October 2017 17:36
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Freekers » Wednesday 07 November 2018 20:55

Have you considered adding your plugin to the Plugin Manager for easy installation & updating :) ?
viewtopic.php?t=22339

Kind regards.

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Thursday 08 November 2018 19:59

Hi Freekers,

Have considered it (briefly) but since the installation can not be 100% automated I didn't add it. The updating part is a good point. I might reconsider ;) Figuring out how to get it added is not a priority for me right now though.

Freekers
Posts: 17
Joined: Friday 20 October 2017 17:36
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Freekers » Monday 12 November 2018 22:42

Would it be possible to make the ssh key location an user configurable option? I'm asking this because the ssh key location on Synology devices is different from the normal location. I have to manually edit the python file to reflect the correct path. This way, updating becomes a bit harder, because I cannot do a git pull since it will overwrite my changes :)
EscApe wrote:
Thursday 08 November 2018 19:59
Hi Freekers,

Have considered it (briefly) but since the installation can not be 100% automated I didn't add it. The updating part is a good point. I might reconsider ;) Figuring out how to get it added is not a priority for me right now though.
Adding the plugin to the plugin manager is quite easy :) You just submit a pull request to the plugin manager's repository with your plugin name & github URL and that's it :)

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Wednesday 14 November 2018 15:36

New in version 0.7.7

If you need to specify a custom key file name for ssh authentication, you can add file name (full path) after the router username in the username field, separated by # (e.g. admin#/home/pi/.ssh/id_rsa)

@Freekers,
I had another look at the plugin manager. Since I am not planning to use the plugin manager myself and do not want to publish something untested I will not add my plugin to the plugin manager myself. Feel free to add it though.

I'm running out of (sensible) options to add custom configuration variables and the plugin needs some cleaning up after adding all these extra's :oops: :lol:

mikeoo
Posts: 156
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Wednesday 14 November 2018 15:43

Using the plug-in now for a while and 95% it is working fine and live it.

Using it with Unify Edgerouter X and ARP option because other command are not working with the EdgeRouter X as fas a i could find and know.

Sometimes when a phone leave the house iDetect keeps thinking it is still at home.

Today a phone come back at 06:30 am (dog walk) and iDetects registered that phone is back. Phone leaves again the house at 09:30 am and came back a 13:50 pm. Last Status in Domoticz is 06:30 am and did not change anymore while phone was gone for a few hourse. Next day all is working fine and working fine for days and the i could happen again.

As a test I run a other script what is also using ARP and when the above problem happens i check the status for this phone that uses the other script and that one is working fine and the phone status is then off while iDetect is On.

So 2 scripts both using ARP are getting difference result sometimes.

Any tips how the find/tweak this because i am lost. Mabe other users use an Edgerouter also and have other experiences and tips.

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Wednesday 14 November 2018 17:16

@mikeoo,

I noticed exactly the same reliability issue you described when using the ARP method on my Asus router (as mentioned in the instructions). But that should not be any different for other scripts... unless the other script uses another method than running the ARP command on the router.

Have you looked at that script and can you share the method it is using (or the entire script)?
Is the Edgerouter the only router you are polling?

mikeoo
Posts: 156
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Wednesday 14 November 2018 17:31

EscApe wrote:
Wednesday 14 November 2018 17:16
@mikeoo,

I noticed exactly the same reliability issue you described when using the ARP method on my Asus router (as mentioned in the instructions). But that should not be any different for other scripts... unless the other script uses another method than running the ARP command on the router.

Have you looked at that script and can you share the method it is using (or the entire script)?
Is the Edgerouter the only router you are polling?
I just find out i could SSH to the Unfi AP point. I was under the imprecision the this was not possible because i use the Unify Controller software.
So the good news is they iDetect now scan all 3 AP and log is telling:

Code: Select all

 2018-11-14 16:18:26.577 Status: (iDetect) Using chipset specific wlanconfig command on router 192.168.0.201 for interfaces ath0 & ath1 & ath3 & ath4 & eth0 (=wlanconfig ath0 ath1 ath3 ath4 eth0)
2018-11-14 16:18:27.222 Status: (iDetect) Using chipset specific wlanconfig command on router 192.168.0.202 for interfaces ath0 & ath1 & ath3 & ath4 & eth0 (=wlanconfig ath0 ath1 ath3 ath4 eth0)
2018-11-14 16:18:27.889 Status: (iDetect) Using chipset specific wlanconfig command on router 192.168.0.203 for interfaces ath0 & ath1 & ath3 & ath4 & eth0 
So it finally use the chipset and not ARP anymore if i get it right :mrgreen:

So hopefully all will work much better now.

Next step is to find out how to create SSH key authentication.
I use a Raspberry Pi for Domoticz and 3 Unify AP but need to find out how and were the place the ssh key files.
The Unify Controller have also an option for SSH keys but don't know if this is the same way. Not my strongest part of networking :D
https://help.ubnt.com/hc/en-us/articles ... Fi-Devices

EscApe
Posts: 223
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Wednesday 14 November 2018 17:52

So it finally use the chipset and not ARP anymore if i get it right :mrgreen:
Thats’s great! It should be much more reliable and quicker to respond.

Since I don’t own any unifi ap’s I can offer no help on the ssh key configuration. Maybe someone else on the (unifi) forum?

mikeoo
Posts: 156
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Wednesday 14 November 2018 18:53

EscApe wrote:
Wednesday 14 November 2018 17:52
So it finally use the chipset and not ARP anymore if i get it right :mrgreen:
Thats’s great! It should be much more reliable and quicker to respond.

Since I don’t own any unifi ap’s I can offer no help on the ssh key configuration. Maybe someone else on the (unifi) forum?
Will try tomorrow and when i have succes i will post it for other if it is easy to do :D
But for your info Unify AP lite and AP long range working with only filling the ip-adresses for all AP in the field for router. Handy for others with same setup.


Tnx for the help and plugin

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests