I have been working on some improvements to the excellent DataMine app. Note that as this plug-in uses the same web addresses and data storage locations as the original version of datamine it is not possible to run them at the same time (at least I wouldn’t expect them both to work), so it is necessary to uninstall the original version before installing this one. It is a derivative work using the same code base and same data and config files so it should just work with the data you already have. If it doesn’t work, just uninstall and go back to the original version. That said please make a copy of any USB advanced settings in dataMine1 before moving to dataMine2 as these will be lost in the transition.
On top of the original functionality it has
the ability to aggregate data by none (as before), hour, day, week and month
aggregation functions of sample (as before), difference, sum, weighted average, minimum, maximum and on-duration
plot charts as columns (as well as spline, line, area and scatter)
a new API call to provide aggregated data
a new API call to provide data in a format suitable for Google Charts
the ability to access web UI remotely (ALTUI plug-in needed for this to work in UI7)
ability to save and edit aggregation and chart data with a graph (since 1.910)
ability to clone a variable and use it multiple times on the same graph (since 1.911)
ability to customise graphs via themes (coming in 1.912)
Ability to time shift data for direct comparison (new in 1.913)
various performance improvements
improvements to the documentation
I have now published this version 2.0 in the MiOS Marketplace here: MiOS Apps
[s]Whilst the plugin works on my machine, it seems I made an error when adding it to the marketplace. A major part of the functionality is in the web front end which consists of over 500 files packed into a tar.gz bundle in the /www/dm directory. Upon running the app for the first time it unpacks the bundle and deletes the tar.gz. However, it looks like this file is some how corrupt and therefore cannot be found and unpacked, it’s also too large to attach to this post!
As a workaround, most of the files are the same as the original version of dataMine. If you have previously had that installed then use a tool such as WinSCP to replace the existing datamine.min.js file in the Vera directory www/dm/js and dataMine.css, graph.html, graph2.html and index.html files in www/dm. The other files should already be correctly in place from the installation. You will need your Vera root password to do this.[/s]
Attachment removed - please download the latest version from the marketplace store which doesn’t need this workaround *
I am using datamine Graph to show data from a rain sensor. I am currently using graph type Spline but is not fully satisfied with the graph.
High charts has the graph type “column” which is nice for visualizing rain fall data.
This link shows an example of that:
Any plans to add this graph type in the future? Hoping…
I did have a look at adding a column chart a while ago to dataMine but ran into trouble when trying to combine it with other chart types. It might be that I just need to do some more investigation but I’d like to avoid upgrading to a newer version of Highcharts as it would be a big undertaking.
As an alternative I built a new API for Google Charts and a sample page which does allow displaying charts in columns. It’s still a work in progress, so can be a bit tricky to get right. However, if you follow the above install instructions you will now have a new file accessible from /dm/graph2.html?ref=
This page uses Google Charts to display a graph that you have saved using the Save Graph button on the main chart tab of dataMine. The is what you have entered on the Save Graph dialog under Quickview Reference. To make a chart appear as a column type, chose Area/Line in datamine as the Graph Type under the Configuration Properties tab. Area/Line charts will be rendered as column charts on the graph2.html page.
Note that the background colour of this page exactly matches the background colour of the Imperihome mobile app web view.
Also attached is an example of some column charts I created, the first shows the number of hours the heating has been on over the last few days. To create this I used graphed the heating SwitchPower status of my heating controller and set the Aggregate By (Period) to “Day” and the Aggregation Formula to “On Duration (Hours)”. The second shows monthly gas and electricity usage where I graphed the number of KWH from my energy monitor and set the Aggregate By (Period) to “Month” and the Aggregation Formula to “Difference (Max - Min)”.
I’ll look into something a bit easier to use, if I have time.
I installed DataMine 2.0 whilst storing the data on a NAS. It’s logging OK for 28h now.
Is this the reason why some Icons aren’t displaced on the datamine dashboard? I don’t really bother though, since I only use it for setting things up. I view the graphs on the ImperiHome app.
Hi @ConstantSphere, do you have an installation procedure? Because I installed it from Apps → Install Apps from UI7 but I do not know how to use it. I click on the link “datamine grapphing page”, I chosed the UUID USB, and then appears “Please restart Luup (press the reload button in UI5). Wait a moment before then refreshing the GUI.” Then I can’t do anything. Best Regards,
I uninstalled the older 2.0 then reinstalled today with new app in App Store. It seems to install ok, but when I go to the “dataMine graphing page”, I just get the loading icon from the app. Attached screen shot. I tried to mount my USB on my own, then point to /usb/DataMine. FYI. /usb is the name of my mount point, and I have directory in there named DataMine. Any Suggestions on getting this setup? I am running vera lite, UI5. I have reboot the device a few times as well.
@kyle.dawson any chance you could send me a copy of your log (http:///cgi-bin/cmh/log.sh?Device=LuaUPnP) or the output of http:///port_3480/data_request?id=lr_dmCtrl&control=debug (I’d be interested to know if this works)?
Do you get the same result from http:///dm and http:///port_3480/data_request?id=lr_dmPage ?
Can you also confirm that you have the latest version (downloaded after the 4th January 2016)? None of the previous versions work!
@ttopic it sounds like you progressed further than @kyle.dawson but could you also send me a copy of your logs so I can see what is going on. As above, can you also confirm that you have the latest version (downloaded after the 4th January 2016)? In the meantime you may wish to download the DataMine overview from here: http://forum.micasaverde.com/index.php/topic,13057.0.html. Thanks.
[quote=“ConstantSphere, post:8, topic:190421”]Can you also confirm that you have the latest version (downloaded after the 4th January 2016)? None of the previous versions work!
Thanks[/quote]
Yes, I uninstalled and reinstalled from app store just now.
Hi guys, I solved the problem by allowing the VERA to log data in my USB. Just go to the UI7 Settings → Logs and then check “Store logs on USB device”. At the beginning the USB Status said “Not mounted”. So, I plugged my USB (it can be a FAT32, which is the filesystem I used) on port USB1 and turned off and then turned on the VERA. After that, the USB Status said “Enabled” and my Datamine Plugin worked perfectly. I am learning the UI of the plugin. It is working awesome.
@kyle.dawson - the interesting thing to note in the logs for me is the error “luup_require can’t find json-dm”. json-dm is the pre-installed JSON library used by UI5 but it cannot be found which is why it’s failing to start. Which UI version are you running and have you ever had DataMine1 working on your box?
After looking through the forums it seems that UI7 uses a different json decoder - as an experiment, can you try uploading the attached file to your luup files folder under develop apps and restarting your box? It first tries to load the UI7 decoder “dkjson” and then tries the UI5 decoder “json-dm” if that fails.
Actually - I just noticed you are on UI5 from your signature. Can you try uninstalling DataMine2 and install DataMine1 and see if that works? If it does then uninstall DataMine1 and reinstall DataMine2 and retest. I want to see if DataMine1 included a json library in the distribution. If you are able to tell me the names of any files with “json” in the name in the /usr/lib/lua directory that would also be useful.
UI5 required the file “json-dm.lua” to be manually transferred to Vera. in UI7 the file “dkjson.lua” appears to be automatically included as part of Vera and if I remember correctly, is used regularly by @akbooer’s plugins. “json.lua” is required by the OWserver plugin and was also manually transferred to Vera UI5, so you could check for that as well. It would be nice if the log indicated which is being used.
I’m not too sure if they are all interchangeable (hopefully they are for this usage case) but Chris was having some trouble with the various decoders at one point.
thanks @a-lurker that’s really useful. I’ve done a bit of investigation and discovered that the original DataMine app included the json-dm.lua file as part of the distribution whereas I’d assumed it was already installed on the box (I guess it was -dm for DataMine - obvious really!) I’ve now added the file to my distribution and submitted for approval. I’ve actually called it json-dm2.lua as you are not allowed to reuse file names. It’s probably not the cleanest solution but was the one used in the original version and I’m assuming this approach will work on any version of the UI.
[s]Whilst waiting for approval, there are a couple of work-arounds:
install the original DataMine, uninstall it and then install DataMine2
put the attached json-dm.lua file into /usr/lib/lua[/s]
it’s now approved so just install the latest version
I also saw that the Data_Mine.png icon was missing from my distribution but I’ll come back and fix that later.
The latest version of DataMine2 (1.895) has just been approved in the marketplace. This includes a version of the JSON library, so if you install that, it should all work. @ttopic and @kyle.dawson can you confirm that you are both up and running?
Thanks Micasaverde for approving the updates so quickly
Thank you this solved the USB mounting for me. But I cannot chose the USB from the drop down list at http://vera-ip/port_3480/data_request?id=lr_dmPage ;( I can ssh in to my Vera but dunno how to find my USB UUID
You can get the UUID of your stick by typing [tt]blkid [/tt]on the command line, however, you shouldn’t need to do this if everything is mounted properly as DataMine should find it and offer it as an option. It seems there has been a bit of a history of issues in getting USB drives to mount properly in Vera; this post may be of use: http://forum.micasaverde.com/index.php/topic,32771.0.html.
Also, on that page is a warning about not enabling logging to USB whilst using the USB for storing your dataMine files, the simple reason being that Vera has a habit of wiping the USB stick used for storing its logs. I store my dataMine logs on a USB stick but have not enabled USB logging in Vera because of the risk.
I had v1 on a UI5 Vera 3. USB logging ON. I have bene succesfully logging 32 variables for over 20 months…
What I did:
[ul][li]Uninstalled v1
Installed v2 from appstore
After some refresh etc. I could open the extra TAB as usual. Got a pop up with the the text to configure the USB drive (to my surprise). As recommend on the pop up restarted vera again. But same thing. p.s. I get the pull down menu with the option to chose a USB drive.
Stopped Vera logging on USB
Restarted etc. still the same pop up
Thought; ok , lets then start fresh…and selected the available USB drive
Restart etc.
Still keep getting the pop up even tough selected USB and restarted and refreshed etc.
Uninstalled v2
Installed v1
Now getting the same issue as above with v1
[/li]
[li][/li][/ul]
