Tools: openHAB Item file generator

I spent a little time today building an Item file generator, since it seems like a common stumbling block for folks getting started. It was inspired by @Ap15e’s work but does the conversion in a very different manner (no burden on Vera, and a complete list of StateVariables is converted to their Item counterparts)

Anyhow, it’s a two-step process:

a) the first step (“miosLoad.sh”) retrieves an XML version of the MiOS Device MetaData from your Vera Unit.
In some cases, Vera spits out “invalid” XML, so this content may have to be hand-edited prior to feeding it into the next step. The output of this phase is a file called “user_data.xml

b) the second step (“miosTransform.sh”) converts the XML into a Textual openHAB Items file.
If this completes successfully, then you’ll end up with an output file like “house.items

This is an example of what it’s like to run these command line scripts:

me$ ./miosLoad.sh 192.168.1.100 Loading MiOS Unit Metadata from 192.168.1.100... Metadata Loaded into user_data.xml! me$ ./miosTransform.sh house Transforming MiOS Unit Metadata from user_data.xml... Metadata Transformed into house.items! Duplicate Item names requiring manual fixes: String DownstairsDeviceStatus "Downstairs Device Status [%s]" (GDevices) {mios="unit:house,device:385/status"} Number DownstairsId "ID [%d]" (GDevices) {mios="unit:house,device:385/id"} String LivingRoomSonosPIcon "Living Room Sonos (P) Icon [%s]" (GDevices,GRoom2) {mios="unit:house,device:295/service/DeviceProperties/Icon"} String MasterBedroomSonosIcon "Master Bedroom Sonos Icon [%s]" (GDevices,GRoom7) {mios="unit:house,device:331/service/DeviceProperties/Icon"} String SceneControllerConfigured "_Scene Controller Configured [%s]" (GDevices) {mios="unit:house,device:394/service/HaDevice1/Configured"} String SceneControllerDeviceStatus "_Scene Controller Device Status [%s]" (GDevices) {mios="unit:house,device:393/status"} String SceneControllerDeviceStatus "_Scene Controller Device Status [%s]" (GDevices) {mios="unit:house,device:394/status"} Number SceneControllerId "ID [%d]" (GDevices) {mios="unit:house,device:394/id"} Number SceneControllerId "ID [%d]" (GDevices) {mios="unit:house,device:4/id"} String SceneControllerScenes "_Scene Controller Scenes [%s]" (GDevices) {mios="unit:house,device:393/service/SceneController1/Scenes"} String SceneControllerScenes "_Scene Controller Scenes [%s]" (GDevices) {mios="unit:house,device:394/service/SceneController1/Scenes"} String SceneMasterBathroom "Master Bathroom Scene" <sofa> (GScenes) {mios="unit:house,scene:34/status", autoupdate="false"} String SceneMasterBathroom "Master Bathroom Scene" <sofa> (GScenes) {mios="unit:house,scene:35/status", autoupdate="false"} String SceneTestArmed "TestArmed Scene" <sofa> (GScenes) {mios="unit:house,scene:73/status", autoupdate="false"} String SceneTestDisarmed "TestDisarmed Scene" <sofa> (GScenes) {mios="unit:house,scene:76/status", autoupdate="false"} String UpstairsDeviceStatus "Upstairs Device Status [%s]" (GDevices) {mios="unit:house,device:337/status"} Number UpstairsId "ID [%d]" (GDevices) {mios="unit:house,device:337/id"} Contact SceneMasterBathroomActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:34/active"} Contact SceneMasterBathroomActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:35/active"} Contact SceneTestArmedActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:79/active"} Contact SceneTestDisarmedActive "Active [%s]" <sofa> (GScenes) {mios="unit:house,scene:80/active"}

At this point, you’ll have a fairly complete house.items file for use in openHAB. I’ve included almost all of the “by-hand” rules for conversion that I’ve used in the past, so SwitchPower1/Status become Switch, etc, but I’m sure there’s a lot more to add as people try out devices (etc) that I don’t have.

The conversion script relies upon the command-line utility called [tt]xsltproc[/tt], which seems to come with Linux and Mac OSX.

Not everything is neatly parceled, but it will get you a very close starting point. You will want to manually check the output Items file, since I don’t perform any name-collision checks what-so-ever, and Item-names might have gotten duplicated (actually, I know they do in some cases)

Downloads…

[ul][li]Tested 1.8.x version - TAR ball from Box.net
[/li]

[li]Nightly version - MiOS Binding conversion Scripts and MAP files
[/li][/ul]

Note: You can also use this trick to retrieve the MAP files.

Howdy,

Just gave your script a run under ubuntu 14.04. Did have to change the 1st line of both scripts to

#!/bin/bash

Otherwise, some tests won’t work as the /bin/sh standard shell doesn’t recognize them.

The resultant items file looks pretty complete (really complete – prolly need to do some trimming). I notice a lot of instances of “FIXME” in the names. Could you give me a quick idea what the presence of FIXME may typically mean? (i.e. bad character, duplicate identifier, etc, etc).

Very promising boostrap tool!

Gerry

@guessed - awesome work!! I’m sort of glad I had to stumble through it because I understand it better now, but for up and comers, I think this is an awesome tool. And props to Ap15e as well for the work used in the effort. This is going to ramp up MiOS usability of OH for sure!

[quote=“gduprey, post:2, topic:185623”]Just gave your script a run under ubuntu 14.04. Did have to change the 1st line of both scripts to

#!/bin/bash

Otherwise, some tests won’t work as the /bin/sh standard shell doesn’t recognize them.[/quote]

:slight_smile: That’s what I get for using a Mac… I’ve patched the scripts and repushed to Box, thanks for the heads-up!

The resultant items file looks pretty complete (really complete -- prolly need to do some trimming). I notice a lot of instances of "FIXME" in the names. Could you give me a quick idea what the presence of FIXME may typically mean? (i.e. bad character, duplicate identifier, etc, etc).

So the FIXME’s are just in the Label of each Item. For the most part I don’t always have a good Label to apply to stuff, so I glue something together from the bits I have.

In this case, I’m glueing together the Name of the Device (eg. “Kitchen Lights”) and the State Variable name (eg. “HaDevice1/LastUpdate”).

In that example, because this StateVariable maps to an openHAB DateTime, the result is a Label something like:

DateTime KitchenLightsLastUpdate "Kitchen Lights FIXME LastUpdate [%1$ta, %1$tm/%1$te %1$tR]" <calendar> {mios=unit:...HaDevice1/LastUpdate"}

and I figured people would go back over their Labels and fix them up, so they’d need something to search on.

The XSLT is in the file [tt]transform.xslt[/tt] and, for the most part, is readable. I’m sure there’s a lot more rules to put in there, the most obvious being to move to using the UPnP Service Aliases (“SwitchPower/Status”) instead of their fully-qualified counterparts (“urn:upnp-org:serviceId:SwitchPower1/Status”)… for brevity/readability, if nothing else.

Anyhow, we can grow it as people come across other use-cases.

Thanks for the feedback!

There’s something to be said for doing it by hand, esp if you want to understand how to do more with the Binding than it’s OOBox defaults, but we’re starting to see an adoption rate higher than I was expecting (if PM’s are anything to go by) so the Generator is to ease the pain a little for folks wanting a more wholesale bridging of their Vera bits instead of the one-or-two Devices that some people might experiment with…

… and you can always build SiteMaps using the [fat] Designer tools or Chris’s nice web-based HABmin interface. That’s a little harder to automagically build, since so-much stuff is based upon user-layout preference.

And props to Ap15e as well for the work used in the effort.

Totally. His effort [effectively] goaded me to do this.

I used a different approach as I wanted something that was more-like my hand-built Items file, and I knew there was only one way to get it :wink:

Hi guessed,

I’ve tried to excecute your tool . The first step returned the following msgs and the .xml was created.

[size=8pt]root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142…
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied
[/size]
The second step returned the following msgs and the items was created too :

[size=8pt]root@raspberrypi:/opt/tool# sh transform.sh DBHomeVeraEdge
Transforming MiOS Unit Metadata from user_data.xml…
transform.sh: 16: [: 0: unexpected operator
Failed to Transform, Check for bogus XML in user_data.xml.
[/size]

I’ve attached the two files created.

Any idea ?

tnks donato

[quote=“d55m14, post:6, topic:185623”][size=8pt]root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142…
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied
[/size][/quote]

Instead of ‘sh load.sh …’ use ‘bash load.sh …’

There are operators in there that require bash and not the standard shell.

[quote=“gduprey, post:7, topic:185623”][quote=“d55m14, post:6, topic:185623”][size=8pt]root@raspberrypi:/opt/tool# sh load.sh 192.168.55.142
Loading MiOS Unit Metadata from 192.168.55.142…
load.sh: 17: [: 0: unexpected operator
Failed to load, Check IP Address supplied
[/size][/quote]

Instead of ‘sh load.sh …’ use ‘bash load.sh …’

There are operators in there that require bash and not the standard shell.[/quote]

thank you very much…

donato

The newer version of the script, uploaded at 8:40pm PDT (about 90 minutes after the originals) has the #!/bin/bash directive at the top (thanks goto @gduprey for catching that).

You should only need to set the executable bit on the scripts, after untarring, and leave out the “bash” prefix when executing it.

I tested these as working on both MacOSX Yosemite 10.10.1 and on Ubuntu/Linaro 14.04.1 LTS (on my ODroid C1).

[quote=“guessed, post:9, topic:185623”]The newer version of the script, uploaded at 8:40pm PDT (about 90 minutes after the originals) has the #!/bin/bash directive at the top (thanks goto @gduprey for catching that).

You should only need to set the executable bit on the scripts, after untarring, and leave out the “bash” prefix when executing it.

I tested these as working on both MacOSX Yosemite 10.10.1 and on Ubuntu/Linaro 14.04.1 LTS (on my ODroid C1).[/quote]

A new-new version uploaded, changes include:
a) Rename load.shmiosLoad.sh
b) Rename transform.shmiosTransform.sh
c) Support for generating all UPnP ServiceId Aliases defined in the MiOS Binding Wiki page.

It now generates something a lot closer to the hand-built version that I’m using.

Just tweaked the script a little, and re-up’d to Box.net.

Turns out not every MiOS Device has the category_num attribute in the XML. If a particular Plugin wasn’t configured to put this value in, then the attribute would be missing altogether and the XSLT Filter rules would skip over generation for that Device.

In my environment, this was skipping the Device → Item generation for the following Plugins:

[ul][li]ListMasterGarageDoor[/li]
[li]SystemMonitor[/li]
[li]WebPowerSwitch (the root/parent Device)[/li]
[li]Nest (the root/parent Device)[/li]
[li]Weather[/li]
[li]ParadoxAlarm (root/parent Device)[/li][/ul]

Others may have been impacted also, so best to re-run it, and manually “merge” the new results.

New version uploaded, now incorporates removal of most special characters in the Item name normalization process.

eg. _#-!@$%^&*=+~`[]{}\|:;"<>?/.()

There are others that likely should be removed, but these can be generated from my [US] Keyboard.

New version uploaded, this one:

[ul][li]Filters some of the Legacy Sonos ServiceId’s (long since changed)[/li]
[li]Handles duplicate “LastUpdate” (preference to HaDevice1)[/li]
[li]Handles duplicate “Target” (preference to SwitchPower1)[/li]
[li]Handles duplicate “Status” (preference to SwitchPower1)[/li]
[li]Handles duplicate “ZoneName” (preference to UPnP DeviceProperties)[/li]
[li]Suffixes “Heat” on all TemperatureSetpoint1_Heat Items[/li]
[li]Suffixes “Cool” on all TemperatureSetpoint1_Cool Items[/li]
[li]Suffixes “Fan” on all HVAC_FanOperatingMode1 Items[/li]
[li]Suffixes “User” on all HVAC_UserOperatingMode1 Items[/li]
[li]Filters “urn:schemas-micasaverde-com:device:avmisc:1” Items (not a ServiceId, old Sonos code throwback?)[/li]
[li]Filters “urn:schemas-upnp-org:service:RenderingControl:1” Items (not a ServiceId, old Sonos code throwback?)[/li][/ul]

This gets a lot closer to a “duplicate-free” Item declaration file (at least for my configuration). More data will be needed to know how much better it is in the wild.

Runs great and I like the changes. The items file is looking better and better and requiring less manual tweaking. I’ve said it before, but this is a great tool for the new OH/MiOS user to get going (especially if you have a lot of devices).

Not that you’re looking for more work, but a few thoughts:

It might be nice for temperature items (i.e. 'CurrentTemperature") to have a format more like “Temperature… [%.1f ?F]” instead of the %d. Not hard to fix at all, but for OH newcomers, all the hints that can be put in for formats help.

On my thermostats, I’m seeing one of the setpoints items marked as String (others are correct). Again, super easy to fix, but thought you might want to know. Item is:

String   DownstairsThermostatSetpointTarget "Downstairs Thermostat FIXME SetpointTarget [%s]" {mios="unit:nightmare,device:16/service/urn:upnp-org:serviceId:TemperatureSetpoint1/SetpointTarget"}         /** SAMPLE=[67] **/

Not sure how possible this is, but as there is a catalog of room groups in the items file, it might be nice to add the room group to each item that is in a room.

These are all pretty minor changes (the result, not speaking of the work), and the tool is pretty great as is.

Gerry

New version uploaded, this one:

[ul][li]Added Group definition GDevices[/li]
[li]Changed Room-based Group generation to GRoom “”[/li]
[li]Reduced the number of FIXME strings in Device-Item Descriptions[/li]
[li]Changed Device-Item generation to use Group “GDevices” instead of “GDevice”[/li]
[li]Changed Device-Item generation to add Group “GRoom” if the device is in a Room (not for ID and DeviceStatus attr)[/li]
[li]Changed Scene-Item generation to add Group “GRoom” if the scene is in a Room[/li]
[li]Add " Scene" at the end of the Main Scene-Item description[/li]
[li]Use “[%.1f F]” format for various Temperature-related Device-Items[/li]
[li]Fix Typo in Service Alias TemperatureSetpoint1_Cool and TemperatureSetpoint1_Heat[/li]
[li]Dumps a list of Duplicate Item definitions that need to be manually fixed by the user[/li][/ul]

Done for most of the Temperature-related items I could find valid ServiceId’s for. I hardcoded the UoM to “F”, so apology upfront to those in the Metric world :wink:

On my thermostats, I'm seeing one of the setpoints items marked as String (others are correct). Again, super easy to fix, but thought you might want to know. Item is: [code] String DownstairsThermostatSetpointTarget "Downstairs Thermostat FIXME SetpointTarget [%s]" {mios="unit:nightmare,device:16/service/urn:upnp-org:serviceId:TemperatureSetpoint1/SetpointTarget"} /** SAMPLE=[67] **/ [/code]

I’m not sure where that comes from. In Vera, there are _Heat and _Cool variants of the ServiceId as well as “CurrentSetpoint”, but I’ve not seen “SetpointTarget” anywhere.

For now, I’ve left it off the magic fix-it list that I maintain inside the XSLT file.

Thanks for the suggestions, most of them are now inside the latest version that I just published.

Howdy,

Didn’t realize I had so many duplicates! I wrote a gawk script that ‘fixes’ duplicates. Much like your FIXME references, this just finds all duplicate item names and prefixes them with a ‘DUP_x’ where x is the device/scene # (there is logic for duplicates within a single device too – which I had for a device with two different level values). Makes it immediately parsable for mios testing and still easy to find the dups for manual fixup.

I’m not 100% sure this will work on mac. It should work with any modern awk implementation, but I had seen a few old, conflicting reports that certain more recent (i.e. last 15 years) awk features didn’t work under MacOSX. Hopefully that is no longer the case.

I added some logic in the miosTransform to direct duplicate output to a file and only warn if there are actually any dups. Then check to see if they have gawk and run the de-duplicator if so (if they don’t, it pretty much works as before, printing the list of duplicates and noting that installing gawk and re-running will get de-duplication).

Feel free to merge this into your tool or not (I won’t be offended in the slightest if not).

Attached is a .zip (this forum wont let me attach a .tar??) with a modified miosTransform.sh and the awk script.

Gerry

If duplicates, due to Device label-strings, are really common then I can add the DeviceId to ALL Item names that are emitted.

It’ll make them uglier than the current format, but it’ll be a lot closer to unique output from the script… without having to repair afterwards.

This would also break any existing dependencies (Sitemaps, Rules, etc) that are based upon the old naming model. Not sure how much of a problem that really is at the moment, but it might be worth doing before too many users experience the issues.

Let me know what you think about the above change.

Anyhow, I’ve attempted to do any automated fixes directly in the XSLT. My logic is that the XSLT is i18n enabled, and Shell typically isn’t (without great care), so there’s a better shot at creating a valid i18n-compatible Items file if all the logic can be pushed inside.

That may/may-not be the right approach, but it’s the one I’ve taken so far :wink:

there is logic for duplicates within a single device too -- which I had for a device with two different level values
For these cases, I've slowly been cataloging the Dup StateVariable names (that I'm aware of) and putting in a preference as to which one "gets" priority. All others get the "state" Id added into their names to make them more unique.

See the XSLT Variable section “ItemName” to see what I mean.

If you know of specific others that should be added here, then let me know.

PS: gawk isn’t on the Mac by default. It has standard awk, but I don’t know the functional differences between the two, and whether your script has a strong dependency upon stuff in gawk that’s not in regular awk.

Howdy,

First – there was a typo in my uploaded .zip when I went and quoted all the env var references. I’ve corrected that and updated the attachment in my previous post.

My thinking would be to not “penalize” the non-duplicate, non-troublesome item names. Depending on how someone has maintained their Vera setup, all their names may come out fine, where as having all the device IDs means that someone will very likely want to edit every single item they keep.

I have about 320 out of 1300 item definitions with dups, but most are things I don’t think I care about nor will likely leave an item defined for. Most of the things I actually care about are fine.

I’ve checked and while gawk may not ship default on Mac, it is a one-line install command to add it. I’m am depending on it for device/scene ID extraction as doing it without some new features that awk never had is a giant pain. I had to install xsltproc on my stripped down ubuntu box, and for the benefit, doing either is a pretty minor additional step. I would expect gawk (being pretty new and updated regularly) is pretty i18l compliant and I’m not doing anything in there that would typically break strings with i18l chars in them.

As for integrating it all into xslt, that would be great! I am not super up on xslt (I’ve written a few minor transformations and can follow what you did mostly) and wouldn’t quite know how to do duplicate checking and resolution. But I’m all for a one-stop shop style experience and minimal dependencies/files.

My (so far, one) duplicate name within the same device is my power meter. For reasons that make no sense, the device seem to be publishing a LightSensor (it has no light sensor, so this is bogus) and a generic sensor level (which doesn’t seem to match anything reported in the more power-specific items). The items are:

Number   DUP_30_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%d]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:GenericSensor1/CurrentLevel"}             /** SAMPLE=[48] **/
String   DUP_30_1_Panel1MasterCurrentLevel "Panel 1 Master Current Level [%s]" (GDevices,GRoom1) {mios="unit:nightmare,device:30/service/urn:micasaverde-com:serviceId:LightSensor1/CurrentLevel"}             /** SAMPLE=[0] **/

If the de-dupper didn’t add the ‘DUP_30_1’ to the second one, then they’d have duplicate item names.

Gerry

New version uploaded, this one:

[ul][li]Looks for duplicate Device names, when building Item names, and appends DeviceId for duplicates.[/li]
[li]Looks for duplicate UPnP Variable names, when building Item names, and appends StateId for duplicates[/li]
[li]Treats a fixed list of UPnP Variable names as “special” where it’ll never append a StateId (eg. SwitchPower1/Status) but the other duplicates, within the same Device, will have it appended[/li]
[li]Imported some shell script tweaks (quotes around variable expansions from @gduprey)[/li][/ul]

This should eliminate the bulk of duplicate situations. If folks are still seeing them, please PM (privately) me a copy of your user_data.xml file (or the output of the list of Duplicates generated)