openLuup: Data Historian

[powisquare]

Thanks for your patience : ) yes those requests returned the codes you provided and there was no nil error.

Have not been into Lua Test but I did fiddle with the openLuup setup a few days ago by updating openLuup to the latest dev release and updating the VeraBridge plugin. Only other thing I can recall was inserting luup.attr_set (“openLuup.Historian.Directory”,“history/”) into Lua Startup.

Should I reinsert 30107174 into PK_AccessCode?

[akbooer]

@powisquare

Thanks for your patience : )

No, thanks for yours!

Should I reinsert 30107174 into PK_AccessCode?

Well, it obviously is there in the system, which is good. It’s not there in that particular VeraBridge, which is bad. My recollection is that this should be set every Luup reload. You’ve probably done this already, but can you do that again and check the value of PK_AccessPoint amongst your ‘VBOld Forge’ device variables.

I don’t have high hopes for that, since I’m sure you’ve have reloaded several times. Just setting it to the correct value is worth trying, but I do beleive that it will get overwritten on the next reload. That would be thing #2 to try, anyway.

I’ll look at the code but, honestly, it’s working in all my machines and in two of yours, so there is SOMETHING odd going on somewhere. I have a hunch, but it will need a special bit of diagnostic code to try out.

[powisquare]

Upon Reload Luup Engine the variable did not return. After inserting variable and 2 x reloads the variable has persisted! However, I checked on the other two verabridges and VBLodgeMaster has a nil value.

[akbooer]

[powisquare]

lol

[powisquare]

Herding cats.

[akbooer]

Yep. I’m thinking, following up my hunch. The problem is clearly mine.

Is there anything special about the network they’re on? I’m thinking this is a timing problem.

[akbooer]

@powisquare

I checked on the other two verabridges and VBLodgeMaster has a nil value.

…so this, clearly, is nothing to do with the historian, but all to do with VeraBridge. Very odd, because, again, I have never seen this before. Can I have the first part of the main log file again, showing the bridges starting up?

[akbooer]

SOLVED… (I hope.)

Your log shows that you are running a version of VeraBridge (2018.03.02) which does not match the openLuup version that you have.

My guess is that there is a copy of this old L_VeraBridge.lua file in cmh-ludl/ which should simply be deleted (then restart twice.)

Is this, in fact the case?

[powisquare]

Found the file in the openLuup dir. Ok to delete this?

From earlier …

I have a Ubiquity setup. The Old Forge and Cottages connect to the network via wifi/access points. The Lodge is wired but reaches the main network via wireless bridge.

Pinging 192.168.1.170 (Old Forge box) twice reply is 1112ms, 1ms, 4ms, 2ms | 3ms, 2ms, 2ms, 2ms. 192.168.1.171 (Cottages) 112ms, 2ms, 1ms, 2ms | 1ms, 5ms, 1ms, 2ms. 192.168.1.172 (Lodge) 4ms, 3ms, 2ms, 3ms | 2ms, 6ms, 3ms, 6ms

This is from my laptop which is connected wirelessly. From the openLuup box (wired to a switch) they are all under 3ms. (Not sure if any of that is useful!)

I did panic in that the unifi controller was down and only got it working after a firmware upgrade but not certain this would affect the network.

[akbooer]

@powisquare

Found the file in the openLuup dir. Ok to delete this?

No, that’s probably the right one. If you look in that file it should have a much newer version number.

There MUST be another one. Do you have a cmh-lu/ directory, or perhaps in files/?

[Buxton]

I recently updated my unifi controller and it caused enormous havoc on my network. What Unifi doesn’t communicate well is that the version of the controller that you need on your network is highly contingent on the type of equipment you have. In my case, my APs are a couple of years old, and the controller that works with that equipment is several revisions back from the latest and greatest.

You can find out what controller you need by visiting the Ubiquity site and looking up the firmware for your specific devices. There will be a link in the firmware device folder to the version of the latest controller that is compatible with your gear. Backup your settings before a new install because if the controller that you need is older than the controller you’re using, you will be forced to uninstall the existing controller first, potentially wiping out your settings.

[Buxton]

AK, I’m having a difficult time getting Grafana to recognize the Whisper files on my USB drive. This, – on the data source page of the Grafana homepage instance. Any tips on what needs doing as I don’t see any info on the web regarding this particular type of datafile. OpenLuup opens the files fine via historian.

[akbooer]

@Buxton

AK, I’m having a difficult time getting Grafana to recognize the Whisper files on my USB drive.

I wasn’t aware that Grafana could/should be able to access the database files directly?

This, -- on the data source page of the Grafana homepage instance.

…or are you trying to use the Graphite Carbon Metrics dashboard for this?

This relies on database queries of the form:

 /metrics/find?query=carbon.agents.*.updateOperations

which I have not implemented (but could… most of the relevant metrics are gathered internally.)

Any tips on what needs doing as I don't see any info on the web regarding this particular type of datafile.

The datafile format for the original Python implementation is described here:
http://graphite.readthedocs.io/en/latest/whisper.html

However, what you probably haven’t looked at is my documentation for DataYours, which describes the Lua translation:

@DataYours User Guide

The Whisper code, originally written in Python, has been translated to Lua, and then re- factored somewhat. The database code is pure Lua and will run anywhere, but it is not binary-compatible with Graphite Whisper files because I have chosen CSV rather than binary packing. This makes them exactly three times larger than real Whisper ones, but space (outside of Vera) is not a problem.

To make them binary compatible you would need to replace the ‘struct’ implementation with a binary version.

OpenLuup opens the files fine via historian.

Good, … so what it is you are trying to do with Grafana?

[akbooer]

@powisquare

The latest development commit (v18.7.15) should fix your VeraBridge problem, if I have diagnosed things correctly.

I do still believe that this is nothing to do with the Data Historian and it’s a side-effect of having multiple L_VeraBridge.lua files, but I have used this opportunity to rectify a long-standing inconsistency with file search paths, which should address such issues.

Hoping this does fix things… (and breaks nothing else!)

[Buxton]

I was just playing around with the software–trying to understand the capabilities based on your quote:

You don't need DataYours at all, and can link Grafana directly to openLuup.

I misread openLuup to stand as a proxy for the actual data-files. I’ll go over the documentation again.

[powisquare]

Sorry guys - have not been receiving the usual email notifiactions from micasaverde (?!) so thanks for the pm. Have searched the whole pi and only found L_VeraBridge.lua in /etc/cmh-ludl/openLuup/. Have updated to latest development release openLuup and will see what happens. Just having trouble logging into Grafana now ‘This site can’t be reached’. arrgh

Had a new smart meter installed which may have sent the cloud key controller into a tiz. British gas now tell me the meter will provide a reading once a quarter automatically. Not quite what I was hoping for but will get onto energy metering on a another thread soon.

[powisquare]

Ok grafana was not started for some reason.

[akbooer]

[quote=“Buxton, post:36, topic:199464”]I was just playing around with the software–trying to understand the capabilities based on your quote:

You don't need DataYours at all, and can link Grafana directly to openLuup.

[/quote]

Well, as usual, I have not explained things very well. It’s certainly time for an update to the documentation.

DataYours has its own Whisper database, location defined by the contents of DataYours variable LOCAL_DATA_DIR, usually ‘whisper/’. Data Historian has its data wherever you’ve set the system attribute openLuup.Historian.Directory, usually ‘history/’.

Pointing Grafana at openLuupIP:3480/, should allow its metric menus to find both databases, but I’ve just tried this, and it seems not to find the DataYours files. This is perhaps the issue you raised. I will look into it. [edit: it does work, I just didn’t have DataYours running when I tested it!]

However, what DOES work, is to add the following line to Lua Startup

luup. attr_set ("openLuup.Historian.DataYours", "whisper/")               -- overriding DY finder

Assuming that your LOCAL_DATA_DIR points to ‘whisper/’, then after a restart, you should see all your original DataYours file under a metric tree called DataYours…

"DataYours.Vera-88800000.008.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.Vera-88800000.294.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.Vera-88800000.386.urn^micasaverde-com^serviceId^EnergyMetering1.EnergyUsage",
"DataYours.cpu.d",
"DataYours.memory.d",
"DataYours.unknown",
"DataYours.uptime.m",

…along with the other sub-trees generated by the historian (one for openLuup, and one for each bridged Vera.)

HTH, AK

[powisquare]

Brilliant - the nil files are no longer created and PK_AccessPoint values are persistent. Will leave it running and see how it goes. Cheers.