@A.Ardon - Thanks for that additional information. After your last post I did some further investigation to see if I could find out what was causing the “handler failed” message. What I discovered is that sometimes a dataMine log file can get corrupt (e.g. by restarting Vera when it is in the middle of logging something). The current code doesn’t handle this well and just returns the unhelpful error. Editing the files directly by hand shouldn’t cause any problem as long as you follow the correct format.
I have coded a fix that simply logs the error in the dataMine debug log, ignores the corrupt line and continues with the rest of the data; this is in version 1.898 of the code which is currently awaiting approval in the App Store. Whilst that’s being approved, download the attached file and upload it into Vera via the Apps/Develop Apps/Luup Files menu ensuring you check the Restart Luup after upload button. Hopefully that fixes your problem.
As for the spurious values created by the device, I have also seen these in my logs from time-to-time and they can normally be handled by the “Enable out-of-limits filter” feature in the Properties pane of the Configuration tab. Set this value to true, enter values for the filter maximum (e.g. 50) and filter minimum (e.g. -20) and click Save. Does your device support negative values? I just wonder if the value 124 was actually supposed to be a negative number but got screwed up by the device/Vera.
attachment removed - please download the latest version from the app store
As mentioned above, version 1.898 will soon be released and it contains the following changes:
Features
[ul][li]Ability to show column graphs in the main dataMine2 window and the graph.html page. Column graphs replace the old Area/Line graph, although if a column graph is selected and there are more than 180 plot points or Aggregate By (Period) is set to None (raw data) then it will revert back to an Area/Line graph.[/li]
[li]The external graphing page graph.html can now be accessed remotely using the url: https://forwarding-server/username/password/vera-gateway-id/data_request?id=lr_dmPage&page=graph.html&ref=graph-reference[/li]
[li]During the mount process any errors output by the mount command are now logged to the debug log to help with resolving any mount issues[/li]
[li]EventsEnabled now defaults to 0 (disabled) rather than 1 (enabled).[/li][/ul]
Bug fixes
[ul][li]Handle corrupt data file entries by logging an error in the debug log and continuing without displaying “handler failed” message[/li]
[li]The graph.html page now correctly reports an error if contact is lost with the Vera unit instead of just showing the spinner[/li][/ul]
Note that you will need the entire app from the app store and not just the file attached above for all of these features/fixes to work.
50 01/16/16 15:16:23.598 luup_log:305: dataMine: Channel: 41 (Id:41), Start: Fri Jan 15 15:16:32 2016, Stop: Sat Jan 16 15:16:23 2016, AggregateBy: 0, AggregationType: 0 <0x32ac6680>
50 01/16/16 15:16:23.619 luup_log:305: dataMine: Ignoring corrupt line in file /nas/database/41/raw/2402.txt :',1' <0x32ac6680>
50 01/16/16 15:16:23.630 luup_log:305: dataMine: 2:Finished reading files - no more data available/nas/database/41/raw/2403.txt <0x32ac6680>
50 01/16/16 15:16:23.630 luup_log:305: dataMine: rawData points 170 <0x32ac6680>
50 01/16/16 15:16:23.662 luup_log:305: dataMine: Done <0x32ac6680>
Below I have a example from my outside temperature. Its hot for a moment… 126 degrees Celcius…
I will take a look later today to set a minimum and a maximum value. I did not know it was possible.
I am shore that it was not an negative value. Most of the time the last number is the correct value e.g. ‘14’ is a ‘4’. ‘126’ is a ‘6’.
At night it gives the right negative value.
I think that my boiler is not handling the opentherm protocol well, it’s a cheap one
I’m pleased to announce that version 1.898 of dataMine2 has now been approved and available in the app store. I’d be interested to hear your feedback on the new features and bug fixes.
Edit: You might need to clear your browser cache to ensure the new web front end loads. You can do this by hitting Ctrl + F5 or Shift + Ctrl + F5 on most browsers. *
Hello @ConstantSphere, I have updated to Datamine 2 version 1.898 on my VeraEdge with the most recent firmware version 1.7.1598. For Datamine 2, I do not see “columns” as a variable configuration option, see attached screen capture.
Also, I have been running Datamine 2 for 4 or 5 days and am not able to get Imperihome to show a graph of the captured data. I always get an error message “not found”, see attached screen capture.
In the mean time you could you try clearing your browser cache by pressing Ctrl + F5 or Ctrl + Shift + F5 (depends on your browser). When the page reloads, take a note of the 2 version numbers that get displayed and report them back to the above forum.
As for Imperihome - make sure you enter the full URL with http:// e.g. “http://192.168.0.1/dm/graph.html?ref=gas” and you are definitely connected to your local network via Wifi. Have you tried another internal URL you know that works, e.g. the Vera portal “http://your-vera-ip”
@Terryleier - I just had another look at your Imperihome screen shot and realised that you were looking at the temperature graph not a user configurable web page. I think you may need to set the Aggregate By (Period) to None (raw data) and the Aggregation Formula to Sample (default) for the CurrentTemperature variable you are looking at in Imperihome. You may also need to have at least 2 weeks of data (but I’m not sure about that).
Thank you, BTW the forum search does not allow “Datamine 2” (“Each word must be two characters long”) and an alternate search for “Datamine 2.0” turns up 31 hits/topics but does not turn up this thread! (Maybe now that I have put Datamine 2.0 into the text of this thread, the search will now locate this thread.)
That did it, thank you! The GUI version changed from 1.890 to 1.898. BTW, I am not sure what the call-out legend above each column is supposed to indicate. For example, I’ve attached a screen capture that shows the partial column for today, January 20, 2016. The column label at the foot of the column reads: “12:00 20.Jan” with a height indicating a run time of less than 1 hour since midnight. On the other hand the call-out legend when hovering on this column states: “17:00:00 Tue 19 Jan 2016: 0.81”, which seems to indicate that the 0.81 hours run-time is from 17:00:00 hours the previous day or something. Not sure what the date/time of the call-out legend is supposed to indicate.
[quote=“ConstantSphere, post:68, topic:190421”]@Terryleier - I just had another look at your Imperihome screen shot and realised that you were looking at the temperature graph not a user configurable web page. I think you may need to set the Aggregate By (Period) to None (raw data) and the Aggregation Formula to Sample (default) for the CurrentTemperature variable you are looking at in Imperihome. You may also need to have at least 2 weeks of data (but I’m not sure about that).
Yes I am using the default Imperihome graph as it works locally and remotely with Datamine on UI5. I changed the aggregations as you indicate, but no luck. This is actually the reason why I am so interested in Datamine 2.0 as I lost the Imperihome graph capability when I “upgraded” from a Vera Lite UI5 to the VeraEdge. I am just trying to get the outside temperature graph back. I will certainly try the other graphs too using the http:requests, but it seems to me that I will need to set up a copy mechanism to copy the data variable to other instances to enable me to set the various aggregation options you now have provided, since the aggregation function attaches to the variable not to the graph. For example, showing furnace operating time by days (1 graph) and by months, another graph. For this pair of graphs, it is my plan to use the Multistring variable container App with either a scene or PLEG to copy the HVAC_OperatingState1:ModeState variable to a Multistring variable every time the ModeState variable changes so I have 2 instances of the ModeState variable to facilitate the 2 graph examples. Similarly copy the outside temperature to get other aggregations such as heating degree days etc. Please let me know if you have any thoughts on other implementation options.
@Terryleier - The columns are all supposed to represent the data from midnight to midnight and be placed at midnight on the time access. Which time zone are you in? And what do you have set in your Vera web interface under Setup > Location tab under Time zone? I didn’t really consider time zones when I wrote the code - I might need to look at coding a fix.
it seems to me that I will need to set up a copy mechanism to copy the data variable to other instances to enable me to set the various aggregation options you now have provided, since the aggregation function attaches to the variable not to the graph. For example, showing furnace operating time by days (1 graph) and by months, another graph. For this pair of graphs, it is my plan to use the Multistring variable container App with either a scene or PLEG to copy the HVAC_OperatingState1:ModeState variable to a Multistring variable every time the ModeState variable changes so I have 2 instances of the ModeState variable to facilitate the 2 graph examples. Similarly copy the outside temperature to get other aggregations such as heating degree days etc. Please let me know if you have any thoughts on other implementation options
@Terryleier - when you save a graph from within dataMine2 using the “Save Graph” button in the Graph Channels pane, it also saves the current aggregation settings. When you display the graph again from the Saved Graphs pane it will use the saved aggregations rather than the ones set in the configuration. It will not allow you to show the same variable aggregated differently on the same graph, but do you really need that?
@Terryleier - I’ve just taken a look at your log file and it’s relatively empty - it’s possible that the logs were rotated shortly before you captured the log. Could you please try accessing your Imperihome graph again and reposting a fresh log file? Thanks.
Here it is @ConstantSphere. Based on your advice, I saved some furnace run-time aggregation graphs using different aggregation periods, days and months and see that the two graphs I wanted are there. From what I can see, these saved graphs seem to be reflected in the log files. Thanks for that clarification. I see there is no need to copy variables at all as I had mistakenly proposed.
As requested, I ran the Imperihome tablet and tried to set up and get the graph widget to work for the temperature variable. The real-time reading comes through, but the graph feature still does not work. I do not see any interaction in the attached logs for that set of requests, so I put an email into Everty support to seek clarification from them. The graph feature does work on a remote Vera 3 UI5. I have not set up Datamine 2.0 on that device yet as I do not have a way to VPN to it from here. I would like to save/salvage the data logged on the USB stick there, which, I believe, requires me to wait until I can access it locally.
You’ve upgraded it to just a super plugin! Thanks very much for your work on the plugin, its documentation update and especially also your time here.
Hi, just wanted to thanks ConstantSphere for resurecting Datamine and share my positive experience.
I’ve just installed it on a Vera Lite (UI5) and a Vera Edge (UI7) without any trouble.
The only issue was the USB bug of the Vera Edge. I had to contact support and they fixed it very quickly.
Actually Im just logging a two temp sensor and a few TRV.
Like Terryleier, my final goal is to make it work with Imperihome.
I will keep you updated how graph works (or not).
But I don’t think I will need to wait 2 weeks. When I’ve tested the Koubachi plant sensor, graph were available the first day (incomplete of course).
Regarding how Imperihome works, AFAIK it directly read the file/data from Datamine1. So normally, if filepath, database, field are the same it shoud do the trick.
@Terryleier and @Thomasss - thank you for your kind words - if you’d like to leave a rating on a the app store for dataMine2 that would be great :).
Looking in the log files I don’t see any reference to the Imperihome request - it should be easy to recognise as it will be for exactly 2 days worth of data as in my log below:
50 01/22/16 18:58:37.756 luup_log:37: dataMine: Channel: 13 (Id:25), Start: Wed Jan 20 18:58:35 2016, Stop: Fri Jan 22 18:58:35 2016, AggregateBy: 1, AggregationType: 0 <0x2ed82680>
50 01/22/16 18:58:37.797 luup_log:37: dataMine: 2:Finished reading files - no more data available/dataMine/database/25/raw/2404.txt <0x2ed82680>
50 01/22/16 18:58:37.797 luup_log:37: dataMine: rawData points 88 <0x2ed82680>
50 01/22/16 18:58:37.819 luup_log:37: dataMine: Done <0x2ed82680>
If you could paste a section of the full Vera log for the period you try to access dataMine2 with Imperihome that may shine a little more light on it.
I’ve also had a go at fixing the time zone issue and whilst in the code moved the weekly aggregation from Thursday thru Wednesday to Monday thru Sunday which makes more sense. All daily, weekly and monthly charts should start at mid-night and represent the data from midnight to midnight. If you’d like to be my beta tester, download the attached file and save it under Luup Files in the Vera UI with restart checked and let me know if it fixes the problem or not? Thanks.
This is now working. I sent an email to Evertygo support as I did not see the Imperihome request in the log file either. The next day it was working!
I've also had a go at fixing the time zone issue and whilst in the code moved the weekly aggregation from Thursday thru Wednesday to Monday thru Sunday which makes more sense. All daily, weekly and monthly charts should start at mid-night and represent the data from midnight to midnight. If you'd like to be my beta tester, download the attached file and save it under Luup Files in the Vera UI with restart checked and let me know if it fixes the problem or not? Thanks.
OK, tried that LUA version 1.899 and can confirm that the dates on the call outs come out as midnight to midnight here.
@ConstantSphere, I should also mention one other thing that came up regarding the graphs, and that is that neither the Graph Type nor the mathematical treatments (Data Offset, Data Multiplier) are saved with the graph. I found it is not possible to use only one temperature dataset to graph various temperature plots, such as current, then average, then heating/cooling degree days. Current temperatures would be drawn as spline or line, while the average would be drawn as a column, and HDD CDD requires subtracting 65[sup]o[/sup]F. Don’t know if it might also be possible to save the Graph Type and mathematical treatments as well along with the aggregation/summing options, but if so, that would be very helpful too.
Just thinking that it would be easy to save the entire configuration data set at the time the graph is saved. Then there would be no need to try to piece together the current configuration data with the saved data, as the saved graph would have a copy of all of the configuration data at the time of the save.
126 degrees Celsius - phew! That’s hot 