I am a newbie to Vera, but I have managed to get Datayours up and running, monitoring a few sensors and storing the data on my NAS (synology). I would like to increase the resolution of the data I’m storing, but I don’t see where I can change the schema once it is set up for a variable. I would prefer not to lose the data I already have, but that wouldn’t be the end of the world.
I can see the existing schema’s in the configuration screen, but I don’t see any way to change them. I am no linux guru, but I can make my way around with vi, but I haven’t been able to locate these in a conf file anywhere with these settings. What am I missing? Thanks!
Well after a bit more reading (I know rtfm), I realized that the schemas are stored in the .wsp whisper database files themselves. So, make sure your schema is set up the way you want it BEFORE you start storing data. Still not sure if there is a way to convert/transfer data from an old whisper database to a new one (more reading in my future).
I deleted the .wsp files from my storage location, then updated the default values in the storage-schemas.conf file, and then re-created the watchers.
[quote=“stymee, post:2, topic:188702”]Well after a bit more reading (I know rtfm), I realized that the schemas are stored in the .wsp whisper database files themselves. So, make sure your schema is set up the way you want it BEFORE you start storing data. Still not sure if there is a way to convert/transfer data from an old whisper database to a new one (more reading in my future).
I deleted the .wsp files from my storage location, then updated the default values in the storage-schemas.conf file, and then re-created the watchers.[/quote]
Hi
I don’ think is possible to convert your old data except for some particular condition in schemas and aggregation but akbooer will be more precise .
I’ve several DY operational wsp files with one and more year of historical data. For application reasons (we have just developed an app to simulate on real energy consumption data the presence of a photovoltaic system with eventually a storage system) I’d like to change the wsp storage schema. I’ve seen the “whisper-resize.py” utility : is it possible to use it in DY ?
In positive case I’ve to change the schema from a less-accurate density of points to a more-accurate density of points. The “whisper-resize.py” does not interpolate points in between the existing ones, so in the more-accurate database, the new points will be null.
Then I can try to write some programming code to fill the null points with the last point’s value.
Well, it’s another of those things I’ve ‘always been meaning to do.’
It should be a relatively straight-forward rewrite of whisper-resize.py into Lua, especially if you keep interpolation out of it. The Whisper API itself is really very basic. I’ve made a start.
It’s not quite as easy as one might expect because of the variety of possible aggregation functions and different archives. One could end up with inconsistent data between different resolutions.
[quote=“akbooer, post:6, topic:188702”]Well, it’s another of those things I’ve ‘always been meaning to do.’
It should be a relatively straight-forward rewrite of whisper-resize.py into Lua, especially if you keep interpolation out of it. The Whisper API itself is really very basic. I’ve made a start.
It’s not quite as easy as one might expect because of the variety of possible aggregation functions and different archives. One could end up with inconsistent data between different resolutions.
AK[/quote]
Hi akbooer,
I would change (extend) one particular period in the schema :
from retentions = 1m:1d,5m:7d,10m:30d,1h:1y,1d:10y
to retentions = 1m:1d,5m:7d,10m:90d,1h:1y,1d:10y
In this case do you think is there the possibility to end up with inconsistent data ?
I’ve made a limited translation of whisper-resize into Lua. I’ve run it on a few files, but not tested extensively. Instead of a command-line invocation, it just runs stand-alone and prompts for input. For example:
input path: whisper/uptime.m.wsp
whisper/uptime.m.wsp
%prog path timePerPoint:timeToStore [timePerPoint:timeToStore]*
timePerPoint and timeToStore specify lengths of time, for example:
60:1440 60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8 15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d 1 hour per datapoint, 7 days of retention
12h:2y 12 hours per datapoint, 2 years of retention
old schema: 20m:30d [0] average
new archive retentions: 1h:7d 3h:30d 1d:1y
1h:7d,3h:30d,1d:1y
aggregation method:
average
x files factor:
0
Retrieving all data from the archives
1 Mon Feb 20 16:40:00 2017
Creating new whisper database: whisper/uptime.m.wsp.new
Created: whisper/uptime.m.wsp.new (27984 bytes)
Migrating data without aggregation...
Mon Feb 20 16:40:00 2017
old archive 1 # non-null points: 643
Program completed in 95.47 seconds (pid: 13292).
You give it the whole path to the input Whisper file (including the .wsp) and it will use the SAME name with .new appended to that for the output file. In this example, I’m changing the archive structure from “20m:30d” to “1h:7d 3h:30d 1d:1y”. If you simply press in reply to any prompt, then it will pick up the existing parameter from the input file (as with the aggregation method and x files factor above.)
The source code has the complete python code as comments, so you should be able to follow along. The only obscure part is the internal callback I use when updating the new file to avoid it being opened and closed each time, as it would be if simply using whisper.update()
No guarantees. I suggest you give it a good test! But do come back to me with any problems.
excuse me for stupid question: I’ve copied your module “whisper-resize.lua” in /etc/cmh-ludl but when I try to execute it I get this error msg : … module ‘L_DataWhisper’ not found.
In the same dir there is L_DataWhisper.lua module.
I run lua from root not from /etc/cmh-ludl…