Master branch updated from development. Significant changes since last tagged release (v16.8.5):
[ul][li]added Apache 2.0 license[/li]
[li]updated documentation (particularly for new system installs)[/li]
[li]AltAppStore - improved error checking[/li]
[li]VeraBridge - reducing polling overhead for status requests[/li]
[li]graphite_cgi - CGI interface to DataYours now bundled with openLuup system (for Grafana, etc.)[/li]
[li]relocation of logs files to /etc/cmh-ludl/logs/[/li]
[li]change to the default user_data.json checkpoint frequency (to 60 minutes)[/li]
[li]new console UI to view some system internals (scheduler, parameters, backups, logs)[/li]
[li]luup.io - add implementation for io.read() with ācrlfā protocol[/li]
[li]status requests - startup jobs now shown correctly[/li]
[li]scenes - day-of-week and day-of-month timing error fixed, and status updates[/li]
[li]openLuup plugin room allocation preserved across reload[/li]
[li]extended and updated unit tests[/li]
[li]miscellaneous bug fixes and improvements[/li][/ul]
The openLuup plugin now has a HouseMode variable, which may be used as a variable watch trigger to kick off scenes when the house mode is changed (no need for a separate HouseMode plugin.)
[ul][li]HouseMode variable on openLuup device - can trigger scenes (no need for HouseMode plugin)[/li]
[li]server allows POST requests (used now by AltUI for some actions)[/li]
[li]VeraBridge now registers with AltUI as a Data Storage Provider for mirroring variables to Vera[/li]
[li]VeraBridge BridgeScenes variable - enable/disable linking to remote scenes[/li]
[li]new storage schemas for DataYours storage provider (1/5/10/20 min, 1/3/6 hour, 1 day sample rates)[/li]
[li]updated documentation[/li]
[li]misc. bug fixes[/li][/ul]
Master branch roll-up of developments from over the last year (itās been too long!)
The changes are too extensive to document in detail, but recent updates have been to anticipate Vera security āimprovementsā to be rolled in next month and also the recent GitHub changes which deprecate old SSL security access levels. As a result, the normal update approach (simply clicking the update ārefreshā button on the Plugins page) will not work and you will need to use the following procedure to allow the AltAppStore plugin to work and then update openLuup as usual:
The problem arises when more than one plugin decides to modify a library module (eg. string) by adding extra functions with the same name, but possibly different functionality. For example
function string:trim()
return self:match "^%s*(.-)%s*$"
end
Whilst itās relatively straight-forward to ensure that different plugins using this syntax are isolated from each otherā¦
local paddedString = " a string with extra blanks at the ends "
local trimmed = string.trim (paddedString)
ā¦itās actually somewhat harder to handle this syntax, which also needs to workā¦
local trimmed = paddedString: trim()
However, the latest release does handle this.
There is an additional console > Scheduler > Sandboxes page (the scheduler handles both scheduling and context switching) to show which plugins are the offenders:
Here, device #3 is AltUI and #345 is ALTHue. You can see the code fix which @amg0 made to avoid a name conflict with the split function, before this was fixed in openLuup. No such code work-arounds should now be necessary. Both string and table libraries are now sandboxed.
I still believe, however, that it is bad practice to modify system libraries.
add archive_video request (&format=1 for snapshots only)
refactored HTTP and SMTP servers
improved io.server closed socket handling
The big change here is a full implementation POP3 server, along with some permanent mailboxes:
mail@openLuup.local - a normal mailbox from which full messages (with any attachments) may be retrieved
events@openLuup.local - only the subject line of any mail sent here will be retained
Together with the SMTP server, the POP3 server allows a fully internal LAN system to send, store, and retrieve email messages. It can conveniently be accessed by almost any email client. I send trigger messages from my camera to both images@openLuup.local which just stores the attached images in the images/ folder, and also events@openLuup.local which lets me review any trigger times on an old iPod Touch.
Of course, if you want this to work when you are away from your home LAN, then youād need to forward these via VeraAlerts or some such additional plugin.
Your email client can be used to manage the deletion of files from the mailboxes, or, alternatively, a timed scene using the openLuup SendToTrash and EmptyTrash actions will do the job automatically.
Other applications are up to your own imagination (please share!)
VeraBridge āonDashboardā attribute, to allow Vera favourites (thanks @rafale77)
add Trash to console Files menu
link to images from console Images page
fix LastTrip name in I_openLuupCamera1.xml motion sensor (was LastTripped, wrongly)
The major enhancement here is a callback for incoming UDP datagrams from specified ports.
The luup.register_handler() function has been extended to be able to specify the UDP protocol and a port to listen on. The handler is called with a similar parameter list to regular callback functions, but with slightly different semantics:
port - the incoming port as a string, eg. ā2222ā
data - a table with {datagram = received_datagram_string, ip = sender_ip}
protocol - āudpā, in this case
So, for example, in Lua Startup, or within a plugin you could write:
function myUDPcallback (port, data, protocol)
print "Incoming UDP!"
print (port)
print ((json.encode(data)))
print (protocol)
end
luup.register_handler ("myUDPcallback", "udp:2222")
The callback handler is called for each incoming datagram.
I donāt use the testing branch very often, but this is a slightly unusual set of circumstances.
To address some fundamental problems of device loading, I have replaced the XML decoder with a DOM-style implementation. This has meant some significant modifications to the loader module, which reads all the device / service / implementation files. Since itās such a basic change, I thought it wise to test thoroughly. I have done my best to do so, updating the unit tests (now over 200 separate ones are run) and running the loader itself on over 350 UPnP files. In the process I have uncovered some interesting errors and omissions, not only in third-party files, but also in standard Vera/MiOS ones. Iāve let a few other plugin developers know about theirs, but hold out no hope for Vera changing anything.
You may blame @a-lurker for instigating this change:
There is some problem with empty tags or tags that contain no āchildā tags being searched for when there is a single āparentā tag. Just white space, or missing child tag or completely empty is allowed between tags but in this case needs to be ignored.
As an additional carrot to try this version out, Iāve taken the opportunity to fix/add a couple of other things:
RunScene - the openLuup plugin now supports this action (within the openLuup service. Thanks to @rafale77 for the suggestion.) It means that this is now available as a menu-selectable scene action. Use with care and try not to create recursive calls to the same scene, or an infinite loop!
SetHouseMode - the VeraBridge plugin now supports this action (within the VeraBridge service.) It means that changing house mode is now accessible as a menu-selectable scene action
category and subcategory - should now be correctly retained across reloads (thanks @rafale77)
scheduled scene time - should now update correctly for scenes which are cancelled by their Lua code (thanks @rafale77)
Although this is in the testing branch, Iām using it on one of my everyday systems. It seems fine, but I hope I havenāt broken too many things for others, you can always revert to the development branch.
Just enter ātestingā in the Update box and click the update button to try this out.
Iām going to be away from my development system for the next 10 days, and not checking the forum, as Iām still recuperating, so youāre on your own until then.
PS: Something more about the new XML parser:
Features include:
- ignores processing instructions, comments, and CDATA
- expands self-closing tags
- reads and saves attributes
- element relationship links:
- parentNode, firstChild, lastChild,
- previousSibling, nextSibling
- some navigation routines:
- nextNode(), including optional filter function
- getElementsByTagName()
- basic XPath searching:
- xpath(),
- xpathIterator()
- .documentElement field of the model accesses the root XML document element
In a break from the WWW3 standard, text is NOT stored in a child text element,
but in the .nodeValue attribute of the element itself. It seemed much easier this way.
Itās quite easy to navigate this structure and extract information you might need from any XML file.
This development release is a significant enhancement to openLuup functionality. It includes a āData Historianā which stores previous device variable values in both in-memory cache and (optionally) on-disk archives. Taking lessons learned from DataYours and EventWatcher, also from AltUIās Data Storage Providers, it offers an almost configuration-free approach to storing (and retrieving) variable values. In addition, and in contrast to those other options and the dataMine plugin, it provides a natural way of reading historic variable values through the usual Luup interface. Also included is a Grafana Data Source API so that data may be easily visualised if you already have a Grafana installation. A big plus here is that you donāt have to have set up storage for any specific variable, they are (almost) all available.
This development has been long in the making, and indeed the ideas for it come from well before openLuup was even thought ofā¦
I do encourage you to read the (short) Ian Mercer blog page linked to by the above URL.
Some more on the Data Historian:
only numeric variable values are stored
By default, the in-memory cache stores all variable changes since system startup, with a small number of exceptions, retaining the most recent 1000 values (by default)
Variables not cached (by default) include timestamp-like ones, including LastUpdate, Polls, low-level Zwave, and some dates
The on-disk archive is enabled by the single LuaStartup line (path relative to cmh-ludl/ where you want the data to reside.)
By default, only a small subset of cached variables are archived on disk, including security sensors Tripped, temperature, humidity, and generic sensors Current values, energy metering KWH and Watts, battery levels, openLuup system memory and cpu stats, ā¦
The on-disk archives are implemented as Graphite Whisper files (as in DataYours)
different data types have different sample rates and retention policies. For example, temperature data is sampled every 20 minutes, initially, but reduced in a number of stages to once per day after a year, and battery levels are just sampled once per day.
Default total on-disk archive duration is 10 years. Typical file sizes, one per variable (containing multiple resolution data) are about 300Kbyte.
System impact is minimal, with high performance. The additional system load is less than writing to the logs.
Database maintenance effort is zero. The disk archive is of fixed size per variable, and total file space is roughly proportional to the number of archived variables.
the openLuup Console has two pages to view historian activities: openLuup > Historian (for the in-memory cache), and Files > History DB (for the on-disk archives)
the URL for the Grafana Data Source interface is simply your openLuup:3480 port
programmatically, data is retrieved using the luup.variable_get() call
Iāve been running versions of this for a while now. On my āproductionā system which is linked to 3 Veras, Netatmo, Philips Hue (thank you @amg0!), and some MySensors Arduinos, there are over 4000 device variables. About 300 of those are cached in memory, and their on-disk archives take about 90 Mbyte. There are, on average, 12 updates per minute, each one taking about 1.5 mS on an RPi. (I donāt have any security sensors triggering frequently on this system.)
For plugin developers, reading data from the cache/archive is trivial, using an additional {start,end} time parameter to variable_get():
local v0,t0 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33)
print ("normal current value", v0,t0)
local v2,t2 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33, {os.time()-3600, os.time()})
print ("over the last hour", pretty {value = v2, times=t2})
gives
normal current value 31.3 1530026929
over the last hour {
times = {1530023886,1530023926.5466,1530024526.9677,1530025127.1189,1530026327.7046,1530026929.0275,1530027486},
value = {31.1,31,31.2,31.3,31.2,31.3,31.3}
}
If the data over the requested time range is all in cache, then the time resolution will be better than 1mS. Once on disk, it depends on the retention policy of the particular archive (sensible defaults, but all configurable if necessary) never finer than 1 second, but usually 5-10 minutes. If there is no data within the time range, then arrays are empty, but within a non-empty array there are no nil points, the times simply denoting (approximately) when the variable changed.
There are further developments to do before turning this into a master release:
mirroring on-disk archives to remote Graphite or InfluxDB servers
total replacement for DataYours
Also, I plan to implement the shockingly missing Digest authorization for luup.inet.wget().
Edit: this release also includes the luup.openLuup flag to denote that this is openLuup, not Vera.
Iāve just merged the lastest development branch with the master to roll-up all the various changes and bug fixes which have accumulated.
This is not quite the master release I wanted to make, but it does fix a couple of significant problems, and, of course, include the latest things like Data Historian. However, it doesnāt include digest authorization in luup.inet.wget() or a couple of other things I wanted to add. It seemed necessary, though, with recent interest in openLuup, and in particular fixed a long-standing install error.
Iāve been a bit short of quality development time recently.
Thanks to all who have helped to make improvements over the last few months (and before!)
Fixes synchronisation issue between system Mode attribute and openLuup HouseMode variable if luup.attr_set() was used to change mode. Thanks to @DesT for pointing this one out.
enables historian mirroring to external Graphite and InfluxDB databases via UDP
implements missing āactionsā request
In addition to the internal Graphite database used by the Data Historian, all archived variables can be mirrored to an external Graphite and/or InfluxDB database. In order to make this efficient in terms of CPU and network resources, the transaction-free UDP protocol is used, avoiding any possibility of system deadlock.
InfluxDB instructions for configuring a UDP port are here:
DataYours may also be used as a destination for the Graphite historian mirror.
To enable sending to Graphite or InfluxDB, the following openLuup system attributes (for example) with IP and port number should be set in the Lua Startup:
Either of those target databases may then be queried for archived variable data. Grafana queries work well, with auto-population of variable name menus.