Datayours - Change storage schema after the fact?

I am a newbie to Vera, but I have managed to get Datayours up and running, monitoring a few sensors and storing the data on my NAS (synology). I would like to increase the resolution of the data I’m storing, but I don’t see where I can change the schema once it is set up for a variable. I would prefer not to lose the data I already have, but that wouldn’t be the end of the world.

I can see the existing schema’s in the configuration screen, but I don’t see any way to change them. I am no linux guru, but I can make my way around with vi, but I haven’t been able to locate these in a conf file anywhere with these settings. What am I missing? Thanks!

Well after a bit more reading (I know rtfm), I realized that the schemas are stored in the .wsp whisper database files themselves. So, make sure your schema is set up the way you want it BEFORE you start storing data. Still not sure if there is a way to convert/transfer data from an old whisper database to a new one (more reading in my future).

I deleted the .wsp files from my storage location, then updated the default values in the storage-schemas.conf file, and then re-created the watchers.

[quote=“stymee, post:2, topic:188702”]Well after a bit more reading (I know rtfm), I realized that the schemas are stored in the .wsp whisper database files themselves. So, make sure your schema is set up the way you want it BEFORE you start storing data. Still not sure if there is a way to convert/transfer data from an old whisper database to a new one (more reading in my future).

I deleted the .wsp files from my storage location, then updated the default values in the storage-schemas.conf file, and then re-created the watchers.[/quote]

Hi

I don’ think is possible to convert your old data except for some particular condition in schemas and aggregation but akbooer will be more precise .

Donato

Asked and answered a few times before, it’s tricky. See here for example

http://forum.micasaverde.com/index.php/topic,24669.msg181084.html#msg181084
http://forum.micasaverde.com/index.php/topic,30808.msg224206.html#msg224206

and I’m sure will be asked again until I find a better way.

Hi akbooer,

As you see, I’m asking again … :slight_smile:

I’ve several DY operational wsp files with one and more year of historical data. For application reasons (we have just developed an app to simulate on real energy consumption data the presence of a photovoltaic system with eventually a storage system) I’d like to change the wsp storage schema. I’ve seen the “whisper-resize.py” utility : is it possible to use it in DY ?

In positive case I’ve to change the schema from a less-accurate density of points to a more-accurate density of points. The “whisper-resize.py” does not interpolate points in between the existing ones, so in the more-accurate database, the new points will be null.

Then I can try to write some programming code to fill the null points with the last point’s value.

tnks

donato

Well, it’s another of those things I’ve ‘always been meaning to do.’

It should be a relatively straight-forward rewrite of whisper-resize.py into Lua, especially if you keep interpolation out of it. The Whisper API itself is really very basic. I’ve made a start.

It’s not quite as easy as one might expect because of the variety of possible aggregation functions and different archives. One could end up with inconsistent data between different resolutions.

AK

[quote=“akbooer, post:6, topic:188702”]Well, it’s another of those things I’ve ‘always been meaning to do.’

It should be a relatively straight-forward rewrite of whisper-resize.py into Lua, especially if you keep interpolation out of it. The Whisper API itself is really very basic. I’ve made a start.

It’s not quite as easy as one might expect because of the variety of possible aggregation functions and different archives. One could end up with inconsistent data between different resolutions.

AK[/quote]

Hi akbooer,

I would change (extend) one particular period in the schema :

from retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y
to retentions = 1m:1d,5m:7d,10m:90d,1h:1y,1d:10y

In this case do you think is there the possibility to end up with inconsistent data ?

tnks

donato

What is your aggregation function?

In this case average .

I’ve made a limited translation of whisper-resize into Lua. I’ve run it on a few files, but not tested extensively. Instead of a command-line invocation, it just runs stand-alone and prompts for input. For example:

input path: whisper/uptime.m.wsp
whisper/uptime.m.wsp
%prog path timePerPoint:timeToStore [timePerPoint:timeToStore]*

timePerPoint and timeToStore specify lengths of time, for example:

60:1440      60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8        15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d        1 hour per datapoint, 7 days of retention
12h:2y       12 hours per datapoint, 2 years of retention

old schema: 	20m:30d [0] average
new archive retentions: 1h:7d 3h:30d 1d:1y
1h:7d,3h:30d,1d:1y
aggregation method: 
average
x files factor: 
0
Retrieving all data from the archives
1	Mon Feb 20 16:40:00 2017
Creating new whisper database: whisper/uptime.m.wsp.new
Created: whisper/uptime.m.wsp.new (27984 bytes)
Migrating data without aggregation...
Mon Feb 20 16:40:00 2017
old archive	1	# non-null points:	643
Program completed in 95.47 seconds (pid: 13292).

You give it the whole path to the input Whisper file (including the .wsp) and it will use the SAME name with .new appended to that for the output file. In this example, I’m changing the archive structure from “20m:30d” to “1h:7d 3h:30d 1d:1y”. If you simply press in reply to any prompt, then it will pick up the existing parameter from the input file (as with the aggregation method and x files factor above.)

The source code has the complete python code as comments, so you should be able to follow along. The only obscure part is the internal callback I use when updating the new file to avoid it being opened and closed each time, as it would be if simply using whisper.update()

No guarantees. I suggest you give it a good test! But do come back to me with any problems.

Good luck

AK

Tnks Akbooer,
I’ll test it and give you a feedback as soon as possible

Tnks again

Hi akbooer,

excuse me for stupid question: I’ve copied your module “whisper-resize.lua” in /etc/cmh-ludl but when I try to execute it I get this error msg : … module ‘L_DataWhisper’ not found.

In the same dir there is L_DataWhisper.lua module.

tnks

donato

That should be sufficient… you’re sure you are running in the same directory?

That should be sufficient… you’re sure you are running in the same directory?[/quote]

Sorry…:frowning: I run lua from root not from /etc/cmh-ludl…

tnks

Hi akbooer,

I did some tests. All seems ok excluded one error in the data.

I changed a wsp archive from :

retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y

to :

retentions = 1m:1d,5m:7d,10m:90d,1h:1y,1d:10y

and I’ve noticed an error in the data point relative to GMT +00:00 (for my time zone is + 1 hour) : the data point before and after are correct.

I attached a screenshot of the two files around GMT +00:00.

Any ideas

tnks

OK, so both good news and bad news.

The time issue is very, very hard for developer working in the GMT timezone! I’ll take a look, but I’m glad that the aggregation seems to work OK.

Hi akbooer,

may be useful if I send you a copy of my wsp file to see if at GMT the result will be the same ?

tnks

donato

Yes, fine. I’ll PM the email address.

Best Home Automation shopping experience. Shop at getvera!

© 2020 Vera Control Ltd., All Rights Reserved. Terms of Use | Privacy Policy | Forum Rules