Python plugin: Presence detection from wireless router

Python and python framework
EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Wednesday 26 September 2018 11:40

@mideoo

Thanks!

I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.

Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).

You could experiment by increasing them a little in line 84 of the plugin:

Code: Select all

def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):
I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.

The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Wednesday 26 September 2018 20:52

Just published version 0.7.2

Enhanced: Found a different way to prevent multiple router configurations from overloading the (single threaded) python plugin framework when timeouts occur.
The timeouts per ssh session are a little more relaxed now and the plugin will just skip a poll if a previous poll cycle could not complete in time.
It will also warn you if a poll cycle takes over 50% of the poll interval time, because that kind of (wait) times could indicate that something is wrong (slowed down by cpu, network, router). On my single router setup it takes about 3% of a 10 second interval and i would expect most configurations to stay below 15%, even with multiple routers. Higher percentages might have a performance impact on Domoticz or at least it’s single threaded python plugin framework. If you just want to check the percentages on your setup you can see them in debug mode.

mikeoo
Posts: 157
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Thursday 27 September 2018 8:18

EscApe wrote:
Wednesday 26 September 2018 11:40
@mideoo

Thanks!

I suspect the timeouts (that are hardcoded into the plugin) are a bit short for your setup. The router might occasionally take too long to create the ssh session. Especially if it has a slow cpu or is running other tasks. A busy Domoticz host or sub optimal netwok would also slow things down. I also get some timeouts while i'm compiling Domoticz on my raspberry pi (the pi is just to busy). Using sshpass instead of key based might add some extra time, but i have never measured the difference, so i'm not sure.

Since you are forcing the arp method the 'detection' never fails because it doesn't need a ssh session to detect the capabilities. Your only concern would be the poll timeout(s).

You could experiment by increasing them a little in line 84 of the plugin:

Code: Select all

def getfromssh(self, host, user, passwd, routerscript, port=22, alltimeout=2, sshtimeout=2):
I kept the timeouts short to prevent the plugin from blocking the plugin framework and/or domoticz during the ssh sessions, which becomes more important if you use the plugin on multiple routers/accespoints. Maybe i'll make it a setting or calculated value (based on number of routers) in a future version, but it would be better to get these response times down in your setup.

The plugin is at its best if it can query wifi chipsets directly. Are you connecting to the EdgeRouter X? If possible you should try to connect to the accespoints instead. That would make the plying more responsive to arriving and leaving. I don't know if that is even possible on the unify's or if they support ssh.
I run it now for 24 hours and it look like it is working fine. I use an other system that also use ARP and running it next to each other the test how the plugin is doing. Until now your plugin is appropriable 30 sec quicker with detecting a phone then the older one so that look great and it respond much better.

Only problem i have with ARP based function that sometimes the phones goes offline/online for a short while.

This is one phone log with the plugin active and the other is the log of the same phone and with the old ARP system. So with your plugin it goes less Offline/Online but still not found a solution for it. Battery saving and Wifi is always active on the Android phone but i think it is a problem with Android to try the save battery and kill things.

So i will test with Grace period if i can set this longer to maybe fix the on/off situation for about 2 min
You mean to adjust alltimeout=2. Is the time set per minute ?

Plugin

Code: Select all

2018-09-27 06:47:21	On
2018-09-27 06:45:29	Off
2018-09-27 06:20:06	On
2018-09-27 06:18:13	Off
Old ARP system

Code: Select all

2018-09-27 07:52:20	Off
2018-09-27 06:47:28	On
2018-09-27 06:47:16	Off
2018-09-27 06:20:08	On
2018-09-27 06:19:56	Off
2018-09-27 05:44:25	On
2018-09-27 05:32:57	Off
2018-09-27 02:05:41	On
2018-09-27 02:04:35	Off
Unifi use a Unify Controller to control the AP and if you don't use that you need to use a App for phone/tablet
You can't login in directly to a AP but it is centralized with a DB running on my Raspberry. The edgerouter X have it's one Gui but there is also an UniFy secure Gateway which is a bit the same as the Edgerouter but that one is allso using the Unify Controller software. So you can change settings for everything with one piece of software.

But need more digging to see if what option there are for this but it is quit complex so the take a while :D

Found Bug:
I can't switch the iDetect - Override to On. When i click on it a pop-up said Switching On but it stays Off. Also nothing in the logfile
For testing I removed everything so plugin was gone in Hardware and noting left in Devices.
Reboot Raspberry
Add the plugin-in again
Still same problem

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Thursday 27 September 2018 17:33

@mikeoo

I published 0.7.2 last night, which has more relaxed timeouts. You should not need to modify them in the plugin.

Unfortunately there are still devices that temporarily drop the wifi connection completely. My iPhone 5 would (occasionally) but later iPhones do keep the wifi connection intact. At least for the chipset based detection methods... It is thinkable that they are dropped from the arp tables if the don't send any data through the router/switch (i don't know the inner workings of arp that well). A longer 'grace period' is indeed the right way to ignore those dropouts. Devices will still be detected quickly if they get home, but it takes longer to detect that they have left.
Using arp on two routers seems a bit redundant at first, but it might indeed speed up the detection if devices start communicating through one of the two routers only.
I can't switch the iDetect - Override to On
Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess ;)

mikeoo
Posts: 157
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Friday 28 September 2018 8:12

EscApe wrote:
Thursday 27 September 2018 17:33
@mikeoo
I can't switch the iDetect - Override to On
Which override mode did you select in the plugin settings? There is a 'do not allow override' option. That would be my fist guess ;)
Fixed indeed when you choose the right options :oops:

Love it :mrgreen:

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Friday 28 September 2018 22:56

Just updated to version 0.7.2
Using a Asus Ac86U and Ac68u in AImesh mode.
Was curious if the updates would work without the #forcegeneric use. This is not the case, without the #forcegeneric iDetect only sees the phones connected to the main router (and not phones connected to the mesh)

The logging without #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using chipset specific wl command on router 192.168.1.1 for interfaces eth5 & eth6 (=wl eth5 eth6)

With #forcegeneric
Initialized version 0.7.2, author 'ESCape'
Status: (iDetect) Using generic brctl command on router 192.168.1.1. Will respond slower and on some routers a little less reliable to absence

Please do not remove this feature, in AImesh it is still needed

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Friday 28 September 2018 23:09

Strange. I didn’t remove that option yet, but if it is somehow ignored anyway you can use the new approach. Add =brctl or =arp after the router address eg, 192.168.0.1=brctl

This should have the same effect as forcegeneric with one difference: you now specify it on a per server basis.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Saturday 29 September 2018 9:39

=brctl seems to work fine

what is the difference between =brctl and =arp ?

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Sunday 30 September 2018 10:33

I am no network expert, so if you want know more then you will have to find a better source. Basically both give you information about your network layout, but on a different level. Arp is about discovering corresponding mac addresses from ip addresses. De bridge has to know what mac addresses are on which (virtual) port.

In my experience with a single router setup the information from the bridge (brctl) is quicker in detecting when a device leaves. On the downside it is also quicker to forget a device that has temporarily dropped the connection (power saving on some phones). Arp is to unreliable on my asus router. In some cases it would keep devices listed even if they had disconnected hours ago.

Bottom line: both are good in detecting when you get home. Brctl is quicker (and more reliable on my router) to detect that you have left.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Thursday 04 October 2018 20:00

Tried again to run without #forcegeneric or =brctl because idetect stopped logging the phones

Before 0,7.2 I did not get these, but this is in the loging:

2018-10-04 19:50:13.079 Error: (iDetect) 'onHeartbeat' failed 'TypeError':'getrouter() got an unexpected keyword argument 'port''.
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 656 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 584 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function onHeartbeat
2018-10-04 19:50:13.079 Error: (iDetect) ----> Line 287 in '/home/hjz/domoticz/plugins/iDetect/plugin.py', function getactivemacs

after the initial errors this message

2018-10-04 19:50:58.069 Status: (iDetect) Warning! Skipping this poll cycle because the previous session has not finished yet (investigate what is slowing things down if you frequently get this message)

Update with =brctl started things back up ok.

I am on AC86U and AC68U in AImesh config.

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Thursday 04 October 2018 20:49

I have quickly corrected the type error directly on github. That should fix the redetection from crashing. It was an oversight on my part.

The fact that tou ran into this error means the router capabilities were not detected on script startup. The plugin will then retry to get the capabilities on the next heartbeat (that’s why i didn’t catch my mistake when testing 0.7.2

The debug log should tell you more about why it fails to get the capabilities.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Thursday 04 October 2018 21:40

Updated to 07.3. All I get now is:

Error: (iDetect) Failed 3 times to get capabilities for 192.168.1.1. Retry interval automatically reduced for this router.

Just also noticed the switches have not updated since 30 Sept .....

Error: (iDetect) Polling 192.168.1.1 has failed 3 times. Poll interval automatically reduced for this router.

Don't see the connection in the router log anymore (Pub key). Strange

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Friday 05 October 2018 7:04

There is not much to tell without a debug log, apart from the plugin not being able to get info from the router. Any changes on the 30/9?

Have you tried ssh from the command line (from the same user account that is running Domoticz.)
New router firmware could lead to the router having a new (host) key.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Friday 05 October 2018 7:09

Thanks for the suggestion, there was a firmware update. Will check the host key.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Saturday 06 October 2018 9:12

I needed to add the router to known hosts again, was the reason for failing.

Just to report. I tested without =brctl again. iDetect then only sees the mac addresses on the main router, not on the mesh router. With =brctl it working again as it was :-)

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Saturday 06 October 2018 9:40

For detection using the wifi chipset the plugin must connect to the router containing that chipset, so in your mesh setup you will have to add both mesh nodes to the plugin.

arp and brctl do not rely on the information from the wifi chipset, but on discovery protocols on the entire network. That's why they will see every device on every router, but are a bit slower to respond to changes.

hjzwiers
Posts: 27
Joined: Friday 12 January 2018 9:26
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by hjzwiers » Saturday 06 October 2018 19:37

With Asus AImesh putting in both IP addresses does not work, still only sees the main router.

=arp fails, =brctl works

Just giving the info, I'll stick to brctl thnx

EscApe
Posts: 225
Joined: Thursday 02 April 2015 8:46
Target OS: Raspberry Pi
Domoticz version: v3+beta
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe » Saturday 06 October 2018 20:48

Thanks for sharing the info. If you ever want to investigate why the chipset approach doesn’t work just let me know (forum or pm). I’m interested to find out how the aimesh works.

AurelioF
Posts: 29
Joined: Wednesday 03 October 2018 18:01
Target OS: Raspberry Pi
Domoticz version:
Location: Spain
Contact:

Re: Python plugin: Presence detection from wireless router

Post by AurelioF » Thursday 11 October 2018 15:10

I confirm that it works with Unifi AP Pro :D
Thanks for sharing

mikeoo
Posts: 157
Joined: Sunday 22 March 2015 8:35
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Holland
Contact:

Re: Python plugin: Presence detection from wireless router

Post by mikeoo » Friday 12 October 2018 7:20

AurelioF wrote:
Thursday 11 October 2018 15:10
I confirm that it works with Unifi AP Pro :D
Thanks for sharing
Do you use ARP with the Unifi. I use also Unifi and interesting how you use it.

Post Reply

Who is online

Users browsing this forum: Ewaldharmsen, TheeDude and 1 guest