I’m working on the X10 CM11 interface to try to improve the reliability of the RS232 comms but this really a more general question about serial comms on the Vera.
The CM11 sends out a polling byte. The connected device then sends an acknowledge byte and then the CM11 sends its command buffer. It first sends the number of bytes its going to send first rather than using a termination marker so the comms on the Vera is set to “raw” i.e. byte by byte which means multiple invocations of the incoming function to pick it all up before taking action.
To try and pick up the whole command in one pass I changed the incoming code so that when the polling byte is read I call luup.io.intercept(), then send the ack byte as before and then read the port with a timeout. Just to be sure I’ve got it waiting 500ms and trying again 3 times. I call the intercept again before each read attempt.
What happens is that the 3 reads timeout with no data received. According to the log, the incoming function is then invoked by the system and the data starts coming in byte by byte as it would have done normally without the intercept and the extra reads. According to the log the first byte comes almost immediately as my attempts to read finish so it does not seem to be me trying to read the port too quickly. All the extra code does is just delay the data coming in through the incoming route.
If you search the forum for luup intercept you will find a significant number of posts. Some of these date back to the very early days, but are still relevant. This is one such
In my implementation of openLuup, a Vera emulation environment, I have always struggled with this, and never achieved consensus within the community on how it does/should work.
Suffice to say that this is an appallingly badly ‘designed’ and implemented functionality. I would follow the sage advice of @futzle from the above thread
She was a true expert on such matters, and sadly rarely participates in the forum these days… a huge loss.
Abandon Vera’s flawed approach, use the LuaSocket library for low-level I/O, and roll your own state machine, per her advice.
Second that. I found that the whole incoming logic is only useful in a read only mode or simple send some, receive some scenarios. When true handshaking is required go with your own sockets routines.
I’ll add a third confirmation to all of the above. To make matters worse, the <incoming> functionality only returns one byte at a time in raw mode; there’s no way to read all available data as a unit, even though the system may have it buffered. So if the datagram is of any size, the system overhead of reading it goes pear-shaped. In one instance, I had a response to a TCP query that was normally about 30K bytes–it would take the (Plus) system 5-6 seconds just to read that data and pass it to the callback (where I was discarding it for timing purposes). Unacceptable when the same thing can be done in single-digit milliseconds by calling recv() on a socket directly.
Sockets direct. Always sockets direct. But then for serial…
Here are the important parts of the code that receives serial data from the RFXtrx transceiver:
The first section is in the I_RFXtrx.xml file:
<functions>
local pluginLib
function startup(lul_device)
luup.log("RFXtrx: loading library L_RFXtrx ...")
if (package.path:find ("/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua", 1, true) == nil) then
package.path = package.path .. ";/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua"
end
package.loaded.L_RFXtrx = nil
pluginLib = require("L_RFXtrx")
if (package.loaded.L_RFXtrx == nil) then
luup.log("RFXtrx: plugin is not installed correctly. Library L_RFXtrx cannot be loaded.", 1)
luup.task("Plugin not correctly installed", 2, "RFXtrx", -1)
return false
end
luup.log("RFXtrx: library L_RFXtrx loaded")
-- This calls the startup function in the plugin lua file
return pluginLib.startup(lul_device)
end
</functions>
<incoming>
<lua>if (pluginLib ~= nil) then pluginLib.incomingData(lul_device, lul_data) end</lua>
</incoming>
The rest is in the plugin LUA file:
function startup(lul_device)
THIS_DEVICE = lul_device
local IOdevice = getVariable(lul_device, tabVars.VAR_IO_DEVICE)
if ((luup.io.is_connected(lul_device) == false) or (IOdevice == nil))
then
error("Serial port not connected. First choose the seial port and restart the lua engine.")
task("Choose the Serial Port", TASK_ERROR_PERM)
return false
end
log("Serial port is connected")
-- Check serial settings
local baud = getVariable(tonumber(IOdevice), tabVars.VAR_BAUD)
if ((baud == nil) or (baud ~= "38400"))
then
error("Incorrect setup of the serial port. Select 38400 bauds.")
task("Select 38400 bauds for the Serial Port", TASK_ERROR_PERM)
return fals
end
log("Baud is 38400")
return true
-- Function called when data incoming
function incomingData(lul_device, lul_data)
if (buffering == false)
then
return
end
-- Store data in buffer
local data = tostring(lul_data)
local val = string.byte(data, 1)
-- Messages from the RFXtrx transceiver are at least 4 bytes and never more than 40
if (#buffer == 0 and (val < 4 or val > 40))
then
warning("Bad starting message; ignore byte "..string.format("%02X", val))
return
end
buffer = buffer..data
local length = string.byte(buffer, 1)
if (#buffer > length)
then
local message = getStringPart(buffer, 1, length + 1)
buffer = getStringPart(buffer, length + 2, #buffer)
debug("Received message: "..formattohex(message))
setVariable(THIS_DEVICE, tabVars.VAR_LAST_RECEIVED_MSG, formattohex(message))
setVariable(THIS_DEVICE, tabVars.VAR_VERATIME, os.time())
if (getVariable(THIS_DEVICE, tabVars.VAR_COMM_FAILURE) ~= "0")
then
luup.set_failure(false)
end
local success, error = pcall(decodeMessage,message)
if(not success)then
warning("No decode message for message: "..formattohex(message).."Error: "..error)
end
end
end
-- Helper function to get a substring that can handle null chars
-- Code from RFXCOM plugin
local function getStringPart( psString, piStart, piLen )
local lsResult = ""
if ( psString )
then
local liStringLength = 1 + #psString
for teller = piStart, (piStart + piLen - 1)
do
-- if not beyond string length
if ( liStringLength > teller )
then
lsResult = lsResult..string.sub(psString, teller, teller)
end
end
end
return lsResult
end
And I will fifth this, being no expert and having struggled and researched around this topic for my pioneer AVR plugin, I too completely abandoned the use of this function following other’s advices…
Cheers but I don’t think there anything there that I’m not doing. The problem is that the CM11 sends a signal saying that it has data ready which needs a response therefore the code in incoming also has a write to make the data flow. What’s so maddening is that it almost works.
Thanks, I only have to send and a receive a handful of bytes so in principle “raw” could work. It works some of the time but after watching it tick away for way to many hours, what is interesting is that when the system does not get the full X10 command, the CM11 sends its Eeprom data marker. This is telling me something but I don’t know what. I’m going to hook it back up to the PC to try and simulate what is going on. Its more than just dropping bytes perhaps.
What’s really annoying is that the send (Vera > CM11) part of the interface works perfectly and works even when CM11 is part way through forwarding on a received X10 command to the Vera. The send involves several co-ordinated writes and reads each one using io.intercept. Why does it all work here but not in the incoming routine? Is the incoming routine mucking up the timing?
Would it be possible to set up another device, not connect it explicitly to the serial device but read the port directly at a fixed frequency? Would this stop the “incoming” routing from interfering with the traffic?
Is there a good plugin example I can use as a go by for this? By state machine do you just mean code that reads data and changes state for other code to monitor / react but not actually do anything itself?
You can easily write a delay callback routine which polls the socket to see if there is anything to read.
The LuaSocket library select() function is what you need.
I’ll try and find a simple example. I have written more than a few myself, but they may not be the best to follow.
Basically you’re having to cope with the fact that the return response is asynchronous. This either means polling (using select) or using blocking I/O in a separate thread (which is harder, but it has been done.)
Or maybe you don’t mind blocking for a short period of time?
Here is a very general purpose piece of code. It’s actually a stand-alone module which is lifted verbatim from my http_async module which offers asynchronous HTTP on both Vera and openLuup.
It essentially offers two methods: one to watch a socket and call you back when there is something to read (with the socket as an argument), and the other to cancel the watching (when you’ve closed the socket.)
local function scheduler_proxy ()
local NAME = "_Async_Request"
local POLL_RATE = 1
local socket_list = {} -- indexed by socket
local function check_socket ()
local list = {}
for sock in pairs (socket_list) do list[#list+1] = sock end
local recvt = socket.select (list, nil, 0) -- check for any incoming data
for _,sock in ipairs (recvt) do
local callback = socket_list[sock]
if callback then
local ok, err = pcall (callback, sock)
if not ok then luup.log ("async.request ERROR: " .. (err or '?')) end
end
end
luup.call_delay (NAME, POLL_RATE, '')
end
local function socket_watch (sock, callback_handler)
socket_list[sock] = callback_handler
end
local function socket_unwatch (sock) -- immediately stop watching for incoming
socket_list[sock] = nil
end
_G[NAME] = check_socket; check_socket()
return {
socket_watch = socket_watch,
socket_unwatch = socket_unwatch,
}
end -- of scheduler_proxy module
If it’s still not clear, I’ll send an example of its use.
I thought I’d do a bit more investigating first…….
I have the interface sending very reliably now. The problem is the fact that the incoming data from the CM11 does not reliably follow the normal pattern of # bytes, bit mask, address and function codes. However what it does send is fairly repeatable - about 80% of the time it seems to send the poll byte x5A and then send x5B byte is something to do with the eeprom. The CM11 then sends two bytes however the second one was also x5A which I think is not another poll byte. It seems to be sending the pattern instead of the usual string.
I then swapped the CM11 back to by PC + VB test program where I can much more easily play with all the settings but this time I used the new FTDI adaptor I bought to get the Vera working. Lo and behold the PC to CM11 started exhibiting the same behaviour. I swapped the serial adaptor back to one my original PL2303 based ones I developed the program on and the PC<>CM11 started to work correctly.
The problem seems to be the FTDI adaptor (exactly the same make / model as recommended on the Vera site). It could be a faulty device of course but I suspect not. I going to try and set up one of the PL2303 I’ve tried to understand all the various posts on manually setting up an adaptor but I think some of the file names and contents have changed so am struggling. Got the vendor codes etc. but not quite sure which serial config file I need to update.
Richard
Best Home Automation shopping experience. Shop at Ezlo!