Polling strategy for Tradfri plugin

[vwout]

@akbooer

I’m happy to try and advise if there’s a well-posed problem to solve.

I’ll give it a try
The protocol is CoAP, which uses a TLS encrypted UDP session internally. Because it is less trivial to implement this in LUA, I use a custom lua library (luacoap). For some logic this uses async I/O, but for some it is synchronous.

One way to synchronize the Tradfri device representation on the UI is to use polling. This polling is done on a fixed interval that is configurable from the plugin and set to 30 seconds by default. The poll function will iterate over all Tradfri devices and individually queries the status, so this scales linearly with the amount of devices.

The polling is triggered via luup.call_delay. Alternatively I could implement this via a job, but that does not seem to bring any advantages.

On Vera - at least according documentation - each luup.call_delay runs in its own thread.
When I read the openLuup scheduler code it looks like a long running timer callback (or job) blocks the scheduler.
Is the conclusion correct that on this point openLuup behaves different than Luup?
If so, can I run a timed operation in openLuup that does not block the main execution luup?

[akbooer]

Sorry it’s taken a while to answer. There are a lot of tricky questions here!

It looks as though you need a solution that works both for Vera and openLuup, but are willing to code the separate cases??

@vwout

The protocol is CoAP, which uses a TLS encrypted UDP session internally. Because it is less trivial to implement this in LUA,

Are you referring to UDP or TLS, or the combination of both? I should just point out that, in openLuup, creating a UDP callback listener (say, on port 2222) is as easy as:

luup.register_handler ("myUDPcallback", "udp:2222")

with the callback handler having the same parameters as for any other user request handler.

@vwout

On Vera - at least according documentation - each luup.call_delay runs in its own thread.
When I read the openLuup scheduler code it looks like a long running timer callback (or job) blocks the scheduler.
Is the conclusion correct that on this point openLuup behaves different than Luup?

I think you’ll find that Vera has two threads per plugin, and all the delays will run in one thread. It depends what you mean by a long-running timer. The actual delay time is not an issue, but once the callback starts, it’ll block until it’s finished. The only thing that generally can block in these circumstances is I/O, so openLuup tries to make sure that as much of its own I/O is done asynchronously.

For a delay callback using UDP there should be virtually no delay at all, since it is transaction-free. For this reason, all of the communication in my DataYours plugin, and in the built-in openLuup Data Historian connection to external databases is done using UDP.

Am I not understanding what you’re trying to achieve here?

AK

[vwout]

Eventually the plugin should work on Vera as well, but since I rely on a (C) library to handle coap, the primary target is openLuup. Tradfri uses the coaps protocol, which is secure CoAP. CoAP uses UDP and relies on Dtls for encryption. Dtls is a TLS implementation for datagram protocols.

I did not know about luup.register_handler. I could use that, but then I would have to implement a CoAP and a Dtls implementation in plain Lua. That … probably is a lot of work.

Regarding my actual question. What I’m trying to achieve is keeping the state of the luup devices up to date with the Tradfri devices. There are two patterns: observing and polling. Observing is efficient, but is not (yet) reliable nor stable. Polling works well and is implemented in my Tradfri plugin at line 731:

function tradfriPollDevices()
    luup.call_delay("tradfriPollDevices", Config.GW_PollInterval, "")

    for k, d in pairs(Config.GW_Devices) do
      tradfriCommand(GW.METHOD_GET, {GW.ROOT_DEVICES, d.tradfri_id})
    end
end

This functions first sets a timer to call itself after an interval (30 secs. by default). After that it GETs a resource at the Tradfri gateway for each device. This GET request is synchronous and blocking. A response usually takes <100ms.
This works well, but in case - like some users - you have over 40 Tradfri devices, this may take a few seconds. If the gateway immediately responds - which it does not do all the time. When the function above runs in a thread, separate from the mainloop (as expected according to the luup documentation), those few seconds should not be an issue.

Hence my question whether I can do a similar thing in openLuup.
But maybe is should take this to a new thread instead of the plugin topic

[akbooer]

This is an interesting problem, and common enough to merit some careful consideration.

There’s two potential directions, or, rather, two different ways to address this issue (at least):

1. @rigpapa has coded a potential solution, but it may not work with this particular I/O call. I leave it for them to respond further.
2. It’s possible to run multiple copies of openLuup simultaneously, in completely separate threads. I do this when testing. It should be possible to arrange your polling in one, and communicate the result asynchronously to the main one.

This would require a bit of experimentation, but perhaps it could eventually turn into a kind of Unix fork instruction for openLuup plugins…

Hmmmm… interesting!

AK

[rigpapa]

@akbooer

@rigpapa has coded a potential solution, but it may not work with this particular I/O call. I leave it for them to respond further.

I think you’re referring to SockProxy? Currently it doesn’t do UDP, but it would not be a stretch to add it…

SockProxy, like any proxy, sits as a man-in-the-middle between a client and a server/endpoint. The benefit it provides is that, for Luup plugins, it sends an asynchronous notification to the plugin when data is waiting from the endpoint, so the plugin can immediately read the data and act, rather than polling. The plugin reads on the same socket it transmits on as usual (at least for TCP), so plugins can be pretty quickly and easily adapted to its use without a ton of code changes, and still be usable if SockProxy is not installed or not operating.

As I said, I’d be happy to look in to adding UDP support if it sounds like this is a usable piece of plumbing…

[vwout]

The UDP communication is handled in C, not lua.

I was looking into using coroutines as an intermediate alternative, but since the CoAP GET request itself is blocking, coroutines don’t solve the issue - once the code gets scheduled, it will block anyway.

[akbooer]

Yes, coroutines are great, but the Lua 5.1 implementation does not allow coroutines to run across C-code calls, which cuts out many interesting applications. Newer versions have overcome this limitation.