Luup.io.intercept() not working as expected

I’m working on the X10 CM11 interface to try to improve the reliability of the RS232 comms but this really a more general question about serial comms on the Vera.

The CM11 sends out a polling byte. The connected device then sends an acknowledge byte and then the CM11 sends its command buffer. It first sends the number of bytes its going to send first rather than using a termination marker so the comms on the Vera is set to “raw” i.e. byte by byte which means multiple invocations of the incoming function to pick it all up before taking action.

To try and pick up the whole command in one pass I changed the incoming code so that when the polling byte is read I call luup.io.intercept(), then send the ack byte as before and then read the port with a timeout. Just to be sure I’ve got it waiting 500ms and trying again 3 times. I call the intercept again before each read attempt.

What happens is that the 3 reads timeout with no data received. According to the log, the incoming function is then invoked by the system and the data starts coming in byte by byte as it would have done normally without the intercept and the extra reads. According to the log the first byte comes almost immediately as my attempts to read finish so it does not seem to be me trying to read the port too quickly. All the extra code does is just delay the data coming in through the incoming route.

What’s going on?

Richard

If you search the forum for luup intercept you will find a significant number of posts. Some of these date back to the very early days, but are still relevant. This is one such

In my implementation of openLuup, a Vera emulation environment, I have always struggled with this, and never achieved consensus within the community on how it does/should work.

Suffice to say that this is an appallingly badly ‘designed’ and implemented functionality. I would follow the sage advice of @futzle from the above thread

She was a true expert on such matters, and sadly rarely participates in the forum these days… a huge loss.

Abandon Vera’s flawed approach, use the LuaSocket library for low-level I/O, and roll your own state machine, per her advice.

2 Likes

Second that. I found that the whole incoming logic is only useful in a read only mode or simple send some, receive some scenarios. When true handshaking is required go with your own sockets routines.

Cheers Rene

1 Like

I’ll add a third confirmation to all of the above. To make matters worse, the <incoming> functionality only returns one byte at a time in raw mode; there’s no way to read all available data as a unit, even though the system may have it buffered. So if the datagram is of any size, the system overhead of reading it goes pear-shaped. In one instance, I had a response to a TCP query that was normally about 30K bytes–it would take the (Plus) system 5-6 seconds just to read that data and pass it to the callback (where I was discarding it for timing purposes). Unacceptable when the same thing can be done in single-digit milliseconds by calling recv() on a socket directly.

Sockets direct. Always sockets direct. But then for serial…

2 Likes

Here are the important parts of the code that receives serial data from the RFXtrx transceiver:

The first section is in the I_RFXtrx.xml file:

	<functions>
		local pluginLib

		function startup(lul_device)
			luup.log("RFXtrx: loading library L_RFXtrx ...")
			if (package.path:find ("/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua", 1, true) == nil) then
				package.path = package.path .. ";/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua"
			end

			package.loaded.L_RFXtrx = nil
			pluginLib = require("L_RFXtrx")
			if (package.loaded.L_RFXtrx == nil)	then
				luup.log("RFXtrx: plugin is not installed correctly. Library L_RFXtrx cannot be loaded.", 1)
				luup.task("Plugin not correctly installed", 2, "RFXtrx", -1)
				return false
			end

			luup.log("RFXtrx: library L_RFXtrx loaded")
			-- This calls the startup function in the plugin lua file
			return pluginLib.startup(lul_device)
		end
	</functions>

	<incoming>
		<lua>if (pluginLib ~= nil) then pluginLib.incomingData(lul_device, lul_data) end</lua>
	</incoming>

The rest is in the plugin LUA file:

function startup(lul_device)
    THIS_DEVICE = lul_device
        local IOdevice = getVariable(lul_device, tabVars.VAR_IO_DEVICE)
        if ((luup.io.is_connected(lul_device) == false) or (IOdevice == nil))
            then
            error("Serial port not connected. First choose the seial port and restart the lua engine.")
            task("Choose the Serial Port", TASK_ERROR_PERM)
            return false
        end
        log("Serial port is connected")
        -- Check serial settings
        local baud = getVariable(tonumber(IOdevice), tabVars.VAR_BAUD)
        if ((baud == nil) or (baud ~= "38400"))
            then
            error("Incorrect setup of the serial port. Select 38400 bauds.")
            task("Select 38400 bauds for the Serial Port", TASK_ERROR_PERM)
            return fals
        end
        log("Baud is 38400") 
        return true
-- Function called when data incoming
 function incomingData(lul_device, lul_data)
 
 	if (buffering == false)
 		then
 		return
 	end
 
 	-- Store data in buffer
 	local data = tostring(lul_data)
 	local val = string.byte(data, 1)
 	-- Messages from the RFXtrx transceiver are at least 4 bytes and never more than 40
 	if (#buffer == 0 and (val < 4 or val > 40))
 		then
 		warning("Bad starting message; ignore byte "..string.format("%02X", val))
 		return
 	end
 	buffer = buffer..data
 
 	local length = string.byte(buffer, 1)
 	if (#buffer > length)
 		then
 		local message = getStringPart(buffer, 1, length + 1)
 		buffer = getStringPart(buffer, length + 2, #buffer)
 		debug("Received message: "..formattohex(message))
 		setVariable(THIS_DEVICE, tabVars.VAR_LAST_RECEIVED_MSG, formattohex(message))
 		setVariable(THIS_DEVICE, tabVars.VAR_VERATIME, os.time())
 		if (getVariable(THIS_DEVICE, tabVars.VAR_COMM_FAILURE) ~= "0")
 			then
 			luup.set_failure(false)
 		end
 		local success, error = pcall(decodeMessage,message)
 		if(not success)then
 			warning("No decode message for message: "..formattohex(message).."Error: "..error)
 		end
 	end
 end
-- Helper function to get a substring that can handle null chars
-- Code from RFXCOM plugin
local function getStringPart( psString, piStart, piLen )
	local lsResult = ""

	if ( psString  )
		then
		local liStringLength = 1 + #psString
		for teller = piStart, (piStart + piLen - 1)
			do
			-- if not beyond string length
			if ( liStringLength > teller )
				then
				lsResult = lsResult..string.sub(psString, teller, teller)
			end
		end
	end
	return lsResult
end
1 Like

And I will fifth this, being no expert and having struggled and researched around this topic for my pioneer AVR plugin, I too completely abandoned the use of this function following other’s advices…

1 Like

Cheers but I don’t think there anything there that I’m not doing. The problem is that the CM11 sends a signal saying that it has data ready which needs a response therefore the code in incoming also has a write to make the data flow. What’s so maddening is that it almost works.

Richard

Thanks, I only have to send and a receive a handful of bytes so in principle “raw” could work. It works some of the time but after watching it tick away for way to many hours, what is interesting is that when the system does not get the full X10 command, the CM11 sends its Eeprom data marker. This is telling me something but I don’t know what. I’m going to hook it back up to the PC to try and simulate what is going on. Its more than just dropping bytes perhaps.

What’s really annoying is that the send (Vera > CM11) part of the interface works perfectly and works even when CM11 is part way through forwarding on a received X10 command to the Vera. The send involves several co-ordinated writes and reads each one using io.intercept. Why does it all work here but not in the incoming routine? Is the incoming routine mucking up the timing?

Would it be possible to set up another device, not connect it explicitly to the serial device but read the port directly at a fixed frequency? Would this stop the “incoming” routing from interfering with the traffic?

Richard

Is there a good plugin example I can use as a go by for this? By state machine do you just mean code that reads data and changes state for other code to monitor / react but not actually do anything itself?

Richard

You can easily write a delay callback routine which polls the socket to see if there is anything to read.

The LuaSocket library select() function is what you need.

http://w3.impa.br/~diego/software/luasocket/socket.html#select

I’ll try and find a simple example. I have written more than a few myself, but they may not be the best to follow.

Basically you’re having to cope with the fact that the return response is asynchronous. This either means polling (using select) or using blocking I/O in a separate thread (which is harder, but it has been done.)

Or maybe you don’t mind blocking for a short period of time?

Thanks, The CM11 buffers up its incoming X10 for a short while so you don’t have to respond to each poll request straight away.

Richard

Here is a very general purpose piece of code. It’s actually a stand-alone module which is lifted verbatim from my http_async module which offers asynchronous HTTP on both Vera and openLuup.

It essentially offers two methods: one to watch a socket and call you back when there is something to read (with the socket as an argument), and the other to cancel the watching (when you’ve closed the socket.)

local function scheduler_proxy ()
  local NAME = "_Async_Request"
  local POLL_RATE = 1
  local socket_list = {}    -- indexed by socket
    
  local function check_socket ()
    local list = {}
    for sock in pairs (socket_list) do list[#list+1] = sock end  
    local recvt = socket.select (list, nil, 0)  -- check for any incoming data
    for _,sock in ipairs (recvt) do
      local callback = socket_list[sock]
      if callback then 
        local ok, err = pcall (callback, sock) 
        if not ok then luup.log ("async.request ERROR: " .. (err or '?')) end
      end
    end
    luup.call_delay (NAME, POLL_RATE, '')
  end

  local function socket_watch (sock, callback_handler) 
    socket_list[sock] = callback_handler
  end

  local function socket_unwatch (sock)       -- immediately stop watching for incoming
    socket_list[sock] = nil
  end

  _G[NAME] = check_socket; check_socket()

  return {
    socket_watch = socket_watch,
    socket_unwatch = socket_unwatch,
    }
end -- of scheduler_proxy module

If it’s still not clear, I’ll send an example of its use.

1 Like

Thanks for that.

I thought I’d do a bit more investigating first…….

I have the interface sending very reliably now. The problem is the fact that the incoming data from the CM11 does not reliably follow the normal pattern of # bytes, bit mask, address and function codes. However what it does send is fairly repeatable - about 80% of the time it seems to send the poll byte x5A and then send x5B byte is something to do with the eeprom. The CM11 then sends two bytes however the second one was also x5A which I think is not another poll byte. It seems to be sending the pattern instead of the usual string.

I then swapped the CM11 back to by PC + VB test program where I can much more easily play with all the settings but this time I used the new FTDI adaptor I bought to get the Vera working. Lo and behold the PC to CM11 started exhibiting the same behaviour. I swapped the serial adaptor back to one my original PL2303 based ones I developed the program on and the PC<>CM11 started to work correctly.

The problem seems to be the FTDI adaptor (exactly the same make / model as recommended on the Vera site). It could be a faulty device of course but I suspect not. I going to try and set up one of the PL2303 I’ve tried to understand all the various posts on manually setting up an adaptor but I think some of the file names and contents have changed so am struggling. Got the vendor codes etc. but not quite sure which serial config file I need to update.

Richard

Best Home Automation shopping experience. Shop at getvera!

© 2020 Vera Control Ltd., All Rights Reserved. Terms of Use | Privacy Policy