ALTUI & DataStorage Providers

ALTUI & DataStorage Providers

DataStorage Providers are an alternative to the thingspeak integration that ALTUI provides. they are optional external module which can be used to receive the notification of a variable change and do whatever you want with it ( storage in another cloud, in a DB etc ). The user can choose which underlying DataStorage provider he wants to chose when he selects the Variable Push option.

You can implement DataStorage Providers as a LUA plugin, a piece of code in openLuup, or even something completely external like a node.js app running on some kind of Ux box ( like a raspberry ) as long as you can call a http GET and have a handler to be called in return in a url via a GET request

The high level summary on how it works is:

  • ALTUI load DataProviders config from its variable
  • or ALTUI UPNP action is called to register a new data provider
  • the User configures a watch on a variable by specifying the provider he wants to use and the parameters required by this provider
  • when the variable changes, ALTUI calls the DataStorage provider with the same parameters as a LUA watch + the DataProvider specific parameters

Lets see the details :

DataStorage Declaration

ALTUI device has a new variable called DataStorageProviders to keep this data provider configuration persistent accross reload/reboot

Thingspeak is allways provided by ALTUI but additional DataStorage can be dynamically declared by a UPNP action “RegisterDataProvider” requiring 3 parameters.

<action> <name>RegisterDataProvider</name> <argumentList> <argument> <name>newName</name> <direction>in</direction> <relatedStateVariable>Name</relatedStateVariable> </argument> <argument> <name>newUrl</name> <direction>in</direction> <relatedStateVariable>Url</relatedStateVariable> </argument> <argument> <name>newJsonParameters</name> <direction>in</direction> <relatedStateVariable>JsonParameters</relatedStateVariable> </argument> </argumentList> </action>

-Name : must be a unique name. ( thingspeak with the built in ALTUI thingspeak integration )
-Url : a URL that will be called as a callback when a variable change ( variable for which the user has selected a data push for this provider )
-JSON parameter: a string representation of a JSON object specifying the user-interface parameters that the user must fill in.

That JSON is an indexed array of parameter objects. each parameter object defines a particular parameter

For thingspeak for instance, in LUA such an object would be like this. note it is a true indexed array. order and indexes matters.

{ [1] = { ["key"]= "channelid", ["label"]="Channel ID", ["type"]="number" }, [2] = { ["key"]= "readkey", ["label"]="Read API Key", ["type"]="text" }, [3] = { ["key"]= "writekey", ["label"]="Write API Key", ["type"]="text" }, [4] = { ["key"]= "fieldnum", ["label"]="Field Number", ["type"]="number", ["default"]=1 }, [5] = { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url" , ["default"]="//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true"} }

in the UPNP action you would use the JSON stringified format of that object so something like:
in JSON

[{"type":"number","key":"channelid","label":"Channel ID"},{"type":"text","key":"readkey","label":"Read API Key"},{"type":"text","key":"writekey","label":"Write API Key"},{"default":1,"type":"number","key":"fieldnum","label":"Field Number"},{"default":"//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true","type":"url","key":"graphicurl","label":"Graphic Url"}]

key is internal , label is what the user sees in the dialog box, type is a HTML 5 INPUT Type, default is a default value

By convention, the parameter named “graphicurl” is reserved to be a url with some {n} placeholders inside which are replaced at the time of displaying a graphic in an iFRAME
whose src attribute is that url where the {n} are replaced by the value of the n-th parameter of the array

When a variable change happens

if a DataStorage Provider was selected for that variable, ALTUI will call the url passed as part of the datastorage declaration with the following parameters:

-http://url...?lul_device=<n>&lul_variable=<n>&old=<n>&new=<n>&lastupdate=<n>&key=value&... for all parameters provided on the DataStorage declaration -http verb = GET

Caveat: calling UPNP action using http as documented by MCV requires to url encode the parameters. example, for a datastorage provider defined as:

[{"type":"text","key":"toto","label":"To To"},{"default":"//www.google.com/{0}","type":"url","key":"graphicurl","label":"Graphic Url"}]

The url to call the UPNP action to register the provider would be:

http://192.168.1.16/port_3480/data_request?id=action&output_format=json&DeviceNum=216&serviceId=urn:upnp-org:serviceId:altui1&action=RegisterDataProvider&newName=toto&newUrl=http%3A%2F%2F192.168.1.16%2Fport_3480%2Fdata_request%3Fid%3Dlr_ALTUI_Handler%26command%3DdataPush&newJsonParameters=%5B%7B%22type%22%3A%22text%22%2C%22key%22%3A%22toto%22%2C%22label%22%3A%22To%20To%22%7D%2C%7B%22default%22%3A%22%2F%2Fwww.google.com%2F%7B0%7D%22%2C%22type%22%3A%22url%22%2C%22key%22%3A%22graphicurl%22%2C%22label%22%3A%22Graphic%20Url%22%7D%5D

In return, when receiving the parameters back from ALTUI when it calls http://url...?lul_device=<n>&lul_variable=<n>&old=<n>&new=<n>&lastupdate=<n>&key=value&...
remember that the parameter values will be urlencoded so you need to urldecode them ( replace + with space etc )
For example ALTUI is calling this hypothetical “toto” data storage provider by

http://url?&toto=This+is+a+text&graphicurl=%2F%2Fwww.google.com%2F%7B0%7D

EDIT : Examples of Data Providers.

[ul][li]DataYours : http://forum.micasaverde.com/index.php/topic,35966.msg266380.html#msg266380[/li][/ul]

This looks great! Can’t wait to give it a go with DataYours, it will be a much better interface than I have at the moment for defining new variables to watch.

Will let you know how it goes! (and probably have a few questions.)

Thanks again.

I added [url=http://emoncms.org/]http://emoncms.org/[/url] integration as a new (built-in) DataProvider in Version V>= 1128

NOTE : new parameter “ifheight” to set the height of the iframe to display a graphic ( if graphicurl field is present )

{
		[1] 	= { ["key"]= "nodeid", ["label"]="Node ID", ["type"]="number" ,["default"]=1},
		[2] 	= { ["key"]= "feedid", ["label"]="Feed ID", ["type"]="number" },
		[3]	= { ["key"]= "inputkey", ["label"]="Input Key name", ["type"]="text" },
		[4] 	= { ["key"]= "readwritekey", ["label"]="Read/Write API Key", ["type"]="text" },
		[5] 	= { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url", ["ifheight"]=460, ["default"]="http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}"}
}

Just about to start trying this out.

[ul][li]What happens if the URL for the storage provider can’t be reached?[/li]
[li]What’s the timeout?[/li]
[li]Surely this will slow things down a lot?[/li][/ul]

Also, I believe you can use the routines located in the socket.url module to do the url encode/decoding without having to write your own:

local url = require "socket.url"

local x = "space percent%ampersand&plus+"
local y = url.escape (x)
local z = url.unescape(y)

print("encoded", y)
print("decoded", z)

gives:

encoded 	space%20percent%25ampersand%26plus%2b
decoded 	space percent%ampersand&plus+

[quote=“akbooer, post:4, topic:190668”]Just about to start trying this out.

[ul][li]What happens if the URL for the storage provider can’t be reached?[/li]
[li]What’s the timeout?[/li]
[li]Surely this will slow things down a lot?[/li][/ul]

Also, I believe you can use the routines located in the socket.url module to do the url encode/decoding without having to write your own:

local url = require "socket.url"

local x = "space percent%ampersand&plus+"
local y = url.escape (x)
local z = url.unescape(y)

print("encoded", y)
print("decoded", z)

gives:

encoded space%20percent%25ampersand%26plus%2b decoded space percent%ampersand&plus+ [/quote]

Cool, I have to try these routines.

regarding call from ALTUI to external url , I prepare the data, then I call it in a delayed routine so I hope and do not believe we should be impacted by the time it takes

luup.call_delay("myhttpget",1,url)

then I used it pretty much all default ( 10s I think )

function myhttpget(url) debug(string.format("myhttpget(%s)",url)) local response_body = {} local response, status, headers = http.request{ method="GET", url=url, -- source = ltn12.source.string(data), sink = ltn12.sink.table(response_body) } debug("https Response=" .. json.encode({res=response,sta=status,hea=headers}) ) return response end

I see the latest build includes a “callback” parameter in the DataStorageProviders structure.

{
  datayours = {
    callback = "",
    parameters = {},
    url = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&format=altui"
  },
  emoncms = {
    callback = "sendValueToStorage_emoncms",
    parameters = {{
        default = 1,
        key = "nodeid",
        label = "Node ID",
        type = "number"
      },{
        key = "feedid",
        label = "Feed ID",
        type = "number"
      },{
        key = "inputkey",
        label = "Input Key name",
        type = "text"
      },{
        key = "readwritekey",
        label = "Read/Write API Key",
        type = "text"
      },{
        default = "http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}",
        ifheight = 460,
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  },
  thingspeak = {
    callback = "sendValueToStorage_thingspeak",
    parameters = {{
        key = "channelid",
        label = "Channel ID",
        type = "number"
      },{
        key = "readkey",
        label = "Read API Key",
        type = "text"
      },{
        key = "writekey",
        label = "Write API Key",
        type = "text"
      },{
        default = 1,
        key = "fieldnum",
        label = "Field Number",
        type = "number"
      },{
        default = "//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  }
}

Is this a local callback routine we can provide instead of constructing a URL for saving the data, or is this only for your built-in data providers?

DataYours as a data storage provider is now working in a development version - it required very few changes, so this is an easy API to use. I’m moving on to the plotting aspect now.

[quote=“akbooer, post:6, topic:190668”]I see the latest build includes a “callback” parameter in the DataStorageProviders structure.

{
  datayours = {
    callback = "",
    parameters = {},
    url = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&format=altui"
  },
  emoncms = {
    callback = "sendValueToStorage_emoncms",
    parameters = {{
        default = 1,
        key = "nodeid",
        label = "Node ID",
        type = "number"
      },{
        key = "feedid",
        label = "Feed ID",
        type = "number"
      },{
        key = "inputkey",
        label = "Input Key name",
        type = "text"
      },{
        key = "readwritekey",
        label = "Read/Write API Key",
        type = "text"
      },{
        default = "http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}",
        ifheight = 460,
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  },
  thingspeak = {
    callback = "sendValueToStorage_thingspeak",
    parameters = {{
        key = "channelid",
        label = "Channel ID",
        type = "number"
      },{
        key = "readkey",
        label = "Read API Key",
        type = "text"
      },{
        key = "writekey",
        label = "Write API Key",
        type = "text"
      },{
        default = 1,
        key = "fieldnum",
        label = "Field Number",
        type = "number"
      },{
        default = "//api.thingspeak.com/channels/{0}/charts/{3}?key={1}&width=450&height=260&results=60&dynamic=true",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }},
    url = ""
  }
}

Is this a local callback routine we can provide instead of constructing a URL for saving the data, or is this only for your built-in data providers?

DataYours as a data storage provider is now working in a development version - it required very few changes, so this is an easy API to use. I’m moving on to the plotting aspect now.[/quote]

yes indeed , this is for “embedded” providers ( thingspeak, emoncms ) as I could not figure out how a function name and function pointer could be used by an external module , nor passed as parameters of a UPNP action.
There is no persistency of these type of providers, meaning they need to re-register at each lua restart as I have to have also another datastructure which holds the mapping between the name and the function itself.
All my attempts with loadstring() and things like that to change context were failures probably due to my incompetency in lua :slight_smile:

local DataProvidersCallbacks={}		-- map names to functions in the local context, only for embedded providers registered within this module in LUA

Alexis

Yes, I’d thought about that problem. It seems to me that it could be overcome if the callback parameter was actually the name of a module you could use with [tt]require()[/tt] with a known function name entry point, say [tt]DataStorageProvider[/tt]. So in the Lua code you’d do something like:

local usermodule = require (userSuppliedName)
usermodule.DataStorageProvider (...)

However, probably not worth the effort since I doubt many people will want to write local storage providers.

For openLuup, it’s not an issue since a local HTTP call never gets to the TCP stack and is converted directly into a call to the request handler anyway. For DataYours this is particularly effective since it then issues a UDP request, possibly to a remote system, which is much lower overhead and doesn’t have a handshake which can timeout. (It also doesn’t guarantee receipt, but over my LAN I’ve logged literally millions of UDP packets for DataYours, none of which have ever been ‘lost’.)

Success! See attached screen capture of DataYours plotting a graph as a DataStorage Provider in AltUI.

You’ll see that the [tt]graphicurl[/tt] is parameter-less, but it would be great if there were some substitutions which were available for deviceNo, serviceId, variableName, etc., and the url could then be built automatically.

DataYours, running DataWatcher, should be installed on the local machine, but can send the data to a remote DataCache. Also, in this case the [tt]graphicurl[/tt] points to the local machine, but there’s no reason this cannot be remote as well.


Edit: added a second version with manual parameters for the device/service/ etc. It’s these for which it would be great to have automatic substitutions.

[quote=“akbooer, post:9, topic:190668”]Success! See attached screen capture of DataYours plotting a graph as a DataStorage Provider in AltUI.

You’ll see that the [tt]graphicurl[/tt] is parameter-less, but it would be great if there were some substitutions which were available for deviceNo, serviceId, variableName, etc., and the url could then be built automatically.

DataYours, running DataWatcher, should be installed on the local machine, but can send the data to a remote DataCache. Also, in this case the [tt]graphicurl[/tt] points to the local machine, but there’s no reason this cannot be remote as well.


Edit: added a second version with manual parameters for the device/service/ etc. It’s these for which it would be great to have automatic substitutions.[/quote]

Great stuff.

one undocumented feature here to control the height of the graphic IFrame. this is for emoncsm. see the graphicurl field definition. it contains a ‘ifheight’ attribute. if it is present, I use this for the IFrame height

{ [1] = { ["key"]= "nodeid", ["label"]="Node ID", ["type"]="number" ,["default"]=1}, [2] = { ["key"]= "feedid", ["label"]="Feed ID", ["type"]="number" }, [3] = { ["key"]= "inputkey", ["label"]="Input Key name", ["type"]="text" }, [4] = { ["key"]= "readwritekey", ["label"]="Read/Write API Key", ["type"]="text" }, [5] = { ["key"]= "graphicurl", ["label"]="Graphic Url", ["type"]="url", ["ifheight"]=460, ["default"]="http://emoncms.org/vis/editrealtime?feedid={1}&embed=1&apikey={3}"} }

For your substitution idea, I ll give some thoughts

Really appreciate the flag for variables using the DataStorage Provider facility in build 1151 - thanks!

and if you hover over it , ( not on mobiles ) it tells you the provider name

Is it possible to modify the default fields which are being sent with the request?
If I’m correct these fields are sent by default:

[ul][li]lul_service[/li]
[li]lastupdate[/li]
[li]lul_variable[/li]
[li]old[/li]
[li]lul_device[/li][/ul]

I’d like to add the name for example.

[quote=“rutger, post:13, topic:190668”]Is it possible to modify the default fields which are being sent with the request?
If I’m correct these fields are sent by default:

[ul][li]lul_service[/li]
[li]lastupdate[/li]
[li]lul_variable[/li]
[li]old[/li]
[li]lul_device[/li][/ul]

I’d like to add the name for example.[/quote]

Currently the parameters passed on the data watch notification url are:
lul_device, lul_service, lul_variable,old, new, lastupdate, provider_params

provider_params being the data provider specific parameters passed at the time of the registration via the UPNP call. UPNPregisterDataProvider()

What name would you like to add ?name of the device owning the variable ?

Isn’t that what the [tt]newJsonParameters[/tt] variable is for?

When registering DataYours (see DataYours - a Data Storage Provider for AltUI) I use the following Lua code:

 local newJsonParameters = {
    {
        default = "unknown",
        key = "target",
        label = "Metric Name",
        type = "text"
      },{
        default = "/data_request?id=lr_render&target={0}&hideLegend=true&height=250&from=-y",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }
    }
  local arguments = {
    newName = "datayours",
    newUrl = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&target={0}",
    newJsonParameters = json.encode (newJsonParameters),
  }

  luup.call_action ("urn:upnp-org:serviceId:altui1", "RegisterDataProvider", arguments, AltUI)

When selecting a variable for storage through the AltUI interface, the “Metric Name” label parameter appears on the menu to be filled in with the desired name. In my case, I use this to indicate the name of the file to be stored and the maximum storage time of the archive. But it could be whatever you like for your own provider code.

[quote=“akbooer, post:15, topic:190668”]Isn’t that what the [tt]newJsonParameters[/tt] variable is for?

When registering DataYours (see DataYours - a Data Storage Provider for AltUI) I use the following Lua code:

 local newJsonParameters = {
    {
        default = "unknown",
        key = "target",
        label = "Metric Name",
        type = "text"
      },{
        default = "/data_request?id=lr_render&target={0}&hideLegend=true&height=250&from=-y",
        key = "graphicurl",
        label = "Graphic Url",
        type = "url"
      }
    }
  local arguments = {
    newName = "datayours",
    newUrl = "http://127.0.0.1:3480/data_request?id=lr_DataWatcherRelay&target={0}",
    newJsonParameters = json.encode (newJsonParameters),
  }

  luup.call_action ("urn:upnp-org:serviceId:altui1", "RegisterDataProvider", arguments, AltUI)

When selecting a variable for storage through the AltUI interface, the “Metric Name” label parameter appears on the menu to be filled in with the desired name. In my case, I use this to indicate the name of the file to be stored and the maximum storage time of the archive. But it could be whatever you like for your own provider code.[/quote]

that is correct. I was wondering if he wanted something like the device’s name to be added automatically to the list of parameters of the callback url, independently of the provider specific parameters that are declared and which the user has to fill in

Hi

Is there any possibility that we can utilize system variables for monitoring basic performance values like CPU, HardDisk, Logins, etc ?

Thank you

[quote=“rmitsos, post:17, topic:190668”]Hi

Is there any possibility that we can utilize system variables for monitoring basic performance values like CPU, HardDisk, Logins, etc ?

Thank you[/quote]

you could install such a plugin ( I think there are a few out there ) then ALTUI will allow you to send these devices variables to your storage

Yeap, I have installed “System Monitor” which is the only I found about that. Unfortunately it provides information only about cpu, memory but not anything else like disk, users, etc.

Thanx for replying

I’ve been trying to setup a local instance of emoncms and I’m scratching my head on setting it up as a valid data provider.

First, I have created a copy of function sendValueToStorage_emoncms to sendValueToStorage_Local_emoncms, changed the url to point to the local instance, and defined it in the startup lua script.

Next, from the examples in this thread, I ran the following code in the “test lua”

[code]local json_loc = require “openLuup.json”
local newJsonParameters = {
callback = “sendValueToStorage_Local_emoncms”,
parameters = {{
default = 1,
key = “nodeid”,
label = “Node ID”,
type = “number”
},{
key = “feedid”,
label = “Feed ID”,
type = “number”
},{
key = “999999999222222222233333334444474”,
label = “Input Key name”,
type = “text”
},{
key = “999999999222222222233333334444476”,
label = “Read/Write API Key”,
type = “text”
},{
default = “http://127.0.0.1/emoncms/vis/editrealtime?feedid={1}&embed=1&apikey={3}”,
ifheight = 460,
key = “graphicurl”,
label = “Graphic Url”,
type = “url”
}},
url = “”
}
local arguments = {
newName = “emoncms_local”,
newUrl = “”,
newJsonParameters = json_loc.encode (newJsonParameters),
}

luup.call_action (“urn:upnp-org:serviceId:altui1”, “RegisterDataProvider”, arguments, AltUI)
[/code]

I was hoping to have a new data provider template added with all the defaults (and the keys). Although I now see the new data provider, I am simply unable to have the api keys “stick” in place…

I have since updated the code above. The keys had to be placed in the “key” elements of the structure; doing this resolved my initial issue. The next thing I found is that ALTUI does not save the callback parameters in the json. I was hoping a “set and forget” where I would run the code and let ALTUI restore the values when the luup engine is restarted, but that didn’t work as expected. The reason seems to be that the callback and callback names are decoupled in a separate lookup which gets constructed only when they are fully registered.

So I forked the code and added some logic to get an actual function added to the callback table if the function is found in the global table. Then I created a new pull request after testing it successfully in my own system.