Energy/Temperature Publication using SmartEnergyGroups.com (SEG)

Other Devices & Integrations Power Management
#5

In fact, excellent.

No need for any Vera power-related app, then. I’ll just throw all power data from DataYours to SEG and cross that one off the list.

#6

For those interested in pushing data to SEG for analysis, here’s a snippet of Lua that can be used in your Vera Startup.

It’s not a plugin, and it’s not resilient (if SEG goes down, the data isn’t accumulated at Vera) but it works reasonably well (I use it for supplementary data like Room & Outdoor temperatures)

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.

[code]local http = require(‘socket.http’)
http.TIMEOUT = 5

local SEG_SITE = “”
local SEG_URL = “http://api.smartenergygroups.com/api_sites/stream”
local segLog = function (text) luup.log('SEG Logger: ’ 
 (text or “empty”)) end

SEG_DEVICES = {
{node=‘vera’, stream=‘t_upstairs’, deviceId=335, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”},
{node=‘vera’, stream=‘t_downstairs’, deviceId=384, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”},
{node=‘vera’, stream=‘t_outside’, deviceId=318, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”}
}

local function findStream(deviceId, serviceId, serviceVar)
– TODO: Change from a simple scan to something that’ll support more entries efficiently.
for k, v in pairs(SEG_DEVICES) do
if (v.deviceId == deviceId and
v.serviceId == serviceId and
v.serviceVar == serviceVar) then
return v.node, v.stream
end
end

return nil, nil
end

local function initWatch()
for k, v in pairs(SEG_DEVICES) do
luup.variable_watch(‘segWatch’, v.serviceId, v.serviceVar, v.deviceId)
end
end

function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)

if (nodeName == nil or streamName == nil) then
segLog(string.format(“Node not found for %s,%s,%s”, deviceId, serviceId, serviceVar))
return
end

– TODO: Encoding
segData = string.format(“(site %s (node %s ? (%s %s)))”, SEG_SITE, nodeName, streamName, newValue)
segLog(segData)

result, status = http.request(SEG_URL, segData)
end

initWatch()

[/code]

You need only substitute in your SEG Site Id, and the “table” (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track, and the script will watch them for changes and push the values across to SEG.

In my case, I have a SEG Device registered with the label “[tt]vera[/tt]”, and beneath that I have SEG Streams for “[tt]t_upstairs[/tt]”, “[tt]t_downstairs[/tt]” and “[tt]t_outside[/tt]”. In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data ([tt]urn:upnp-org:serviceId:TemperatureSensor1[/tt], [tt]CurrentTemperature[/tt])

Since the data is only pushed when a value changes, you may have to wait a little while before you see the data in SEG (after loading the code into Vera’s Startup method, and restarting Vera)

NOTE: When naming your streams for SEG, use prefixes of

[ul][li]t_ - temperature data[/li]
[li]e_ - energy data (Wh, kWh, etc)[/li]
[li]p_ - power data (W, kW, etc)[/li][/ul]

When using SEG’s “Discoveries” feature, it automatically decodes the Stream names, and applies the correct typing information if the streams are named using these prefixes.

This tool is on http://smartenergygroups.com/tools
MyThings → tools → Discoveries

When using the Discovery feature, you may need to leave SEG on that page for some time, as I’m only sending data over as it changes.

#7
And here's what that looks like

Ooh, that’s Google Charts, but what produced it? SEG??

#8

[quote=“akbooer, post:7, topic:186392”]

And here’s what that looks like

Ooh, that’s Google Charts, but what produced it? SEG??[/quote]
Straight out of SEG, they have an option/menu they display on each Graph to Save it. Formats include PNG, JPEG, PDF and SVG.

Saves taking screenshots :wink:

#9

oh, and I enabled @watou’s option on the Nest T-Stats to get finer grained data. It was a bit chunky before I did that


#10

SEG is great stuff, and run by a lad in Sydney. Always happy to promote enterprising companies from home :wink:

I find myself using it for a variety of measurement/graphing/comparison stuff now, esp given how simply it ingests data.

There is another way you could improve your fuel economy, perhaps without changing your driving style ;-) http://www.discovery.com/tv-shows/mythbusters/videos/dimpled-car-minimyth.htm
That's hilarious! I was laughing a ton as I watched, so thankyou!

Those guys are nuts. Every time I see them I’m reminded of this little “mishap” and wish there was video of it 8)
Map Charts Path of MythBusters' Wayward Cannonball | WIRED

On a related note, just noticed that electricity rates have climbed to 33-39c/kWh back home, which is nearly as bad as Hawaii! No wonder Solar PV installs went nuts down there


#11

Guessed, I am trying to use this script, but I’m pretty weak in this area so I could use some clarification.

You need only substitute in your SEG Site Id

So first I create a device on SEG. That gives me a token number and a stream name which I then use as the site ID and teh stream name in the script? And is that site ID with the < > brackets, or without? I am assuming without.

and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track
In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

So in the advanced settings tab for my energy monitor I have the deviceId under id, the ServiceId info is under device_type, and the State Variable would be Watts. I suppose I would set another stream up for KWH.

But my serviceId looks like this:

urn:schemas-micasaverde-com:device:PowerMeter:1

I have a script pushing this data to Plotwatt with this serviceId:

urn:micasaverde-com:serviceId:EnergyMetering1

I tried that too but nothing working so far. I’m not getting an error from the script, just nothing on SEG.

#12

For SEG, the Site Id looks like the attached image. Once logged in, it’s under:

My Things → Sites → → [Edit]

The string doesn’t have <> characters around it, use it as displayed when you select the pane above.

This is the serviceId:
[tt] urn:micasaverde-com:serviceId:EnergyMetering1[/tt]

The other string you had was the device type, which can’t be used. If you want to validate, just “hover” the mouse over the Service Variable name/label in the Device’s Advanced tab. It’ll tooltip the ServiceID for the Service Variable.

Once you set the Vera-end of this “running” with the SEG Site ID, you can goto SEG:

My Things → Tools → Discoveries

and then press b[/b]. Give it a few minutes and it’ll create ALL of the relevant Devices for you, which you can then just go and fill in the blanks (like UoM, range, etc)

#13

The proper site ID did it. Thanks!

#14

Yup, same here. site-id is actually the token key that’s generated. The info here made it really easy to set up and get data into SEG. Now to figure out how to forecast my monthly bills.

Thanks for the 'fo!

#15

Hi Guessed and others

I am adding a veralite to my ISY HA setup, in part because of my frustration in trying to get energy data from ISY to SEG or any other good energy monitoring site. In my case I want track circuits AND also individual smart plugs on individual devices.

[ol][li] Overall are you happy with what you describe above to report to SEG for both GEM and other Vera energy reporting, e.g. smart plugs? [/li]
[li] Is SEG “smart enough” to allow the user to avoid double counting the smart plug loads which are also included in the whole-house-meter loads?[/li]
[li]I am not a scripter (but can follow along and fill in basic blanks, etc.) Do you expect this is an appropriate solution or will I be jumping off a clifF?[/li][/ol]

Thanks for posting this info and your work to make it straightforward for others!!

#16

Yup, completely. I look at the data in SEG daily after I make significant changes in the house.

I report data to SEG from 2 different sources:

[ul][li]Weather, House-Temperature & Vera Performance data from my Vera
These streams use the technique above to push that to my SEG Account.[/li]
[li]Brultech GEM and ECM-1240 Energy Data
These streams are published using btmon.py running on a RaspberryPi.[/li][/ul]

I moved from using Vera as the conduit for the Energy streams in order to reduce the load on Vera (I have a lot of stuff on Vera that needed to be more real-time than the energy monitoring stuff)

SEG has no knowledge of the relationship between the [data] Streams you register with it. It sees Devices (think “Main Panel”, “SubPanel”, “Vera”) and the Streams that come from those Devices (think “Upstairs Temperature”, “Grid Energy”, “Solar Energy”, “Office Energy”, “CPU Usage”)

Once you’ve connected the Devices and established the Streams, you use the SEG UI to build Dashboards to display the UI for those Streams (you pick the UI/Graph styles, as well as what Streams to display in those graphs). It’s a fairly advanced graphical Portal builder for “energy” data.

You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system. I do this to “rollup” certain low level stats into more useful stuff.

eg. TotalEnvironmentalEnergy = MainPanel/AC Energy + SubPanel/Furnace Energy

So this is where, indirectly, relationships are being captured
 by how you build Graphs, and how you build these Computed Streams. All of this is done in a UI, and is easy to do.

The hardest part of my setup is the bymon.py running on Pi. Since you need to setup the Pi, btmon.py, deal with Serial Ports (for attachment to ECM-1240) and Wifi (for dealing with my GEM) and script it all to start on reboot (etc).

Not impossible, but it would take little searching to pull the parts together.

That said, if you’ve got a Vera, and it’s not doing too much else, then you could just run the Brultech Plugin and it’ll do the work of acquiring the data for you (esp if you’ve got the GEM on WiFi and/or Ethernet). You’d then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG. That’s all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id’s)

#17

I find SEG really confusing. It looks very flexible and powerful and I’m sure I could get it dialed in given enough time with it, but I wouldn’t call it an intuitive UI.

#18

[quote=“guessed, post:16, topic:186392”]
You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system. I do this to “rollup” certain low level stats into more useful stuff.


That said, if you’ve got a Vera, and it’s not doing too much else, then you could just run the Brultech Plugin and it’ll do the work of acquiring the data for you (esp if you’ve got the GEM on WiFi and/or Ethernet). You’d then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG. That’s all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id’s)[/quote]

Guessed, thanks! Those were the answers I was hoping to receive


As to calculated streams on SEG: Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to “AV Stream”, (iii) the AV Stream combined data then merges with other data streams and rolls up into “AV and XYZ Stream”. In thinking about this, I suppose one could just to the upper tier rollup manually from each device, vs as a two-tier roleup.

As to Vera vs PI and complexity: I have an ISY with Zwave module and find it to be VERY reliable/stable. However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN). Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data. Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc. I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-). In that context I am hoping these Vera bells-and-whistles won’t cripple it. (PS: exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Finally, you mention GEM wifi or Ethernet. Are these options in the GEM one buys, or ad-ons/hacks to do after the fact. I have not purchased yet. I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

Thanks again!

#19

Both Add and Subtract, of any Stream with the same Units of measure. In my case, I have some streams that are “rolled up” at the source, based upon where I’ve clamped
 sometimes I re-roll them up from the component streams, in SEG, since there might be errors in measurement that I’m interested in seeing.

I’ll post a series of screen-captures shortly to illustrate these bits


Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".

Yep, Calculations are just another Stream, so you can composite new Calculations based upon others. Sometimes this limits the granularity of the result (to 30 minutes) but in general that’s not an issue.

As to Vera vs PI and complexity: I have an ISY with Zwave module and find it to be VERY reliable/stable. However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN). Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data. Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc. I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-). In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS: exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Take a look at the stuff that Teken is doing on the Brultech forums. He has ISY gear. My Vera is a Vera3, so it has more memory than a VeraLite, but the same as the newer VeraEdge.

When running a reasonable load, Vera is good at Lighting/Z-Wave control (etc)
 but when it runs out of steam, you’re hosed. They don’t currently have a “bigger” (and, in all honesty, less-buggy) option for power-users so I paired back my Vera3 to running the critical stuff, and have moved everything else to Pi’s (and now, via openHAB, I’m expanding/migrating to my Mac)

Finally, you mention GEM wifi or Ethernet. Are these options in the GEM one buys, or ad-ons/hacks to do after the fact. I have not purchased yet.

You buy them as part of the GEM when you order it. Mine is a 1st Gen GEM, which I bought with ZigBee, since I already had ECM-1240 units with the ZigBee module (and the related ZB <-> Ethernet Bridge). At the time the ZB Modules were socketed, so I replaced mine with a RovingNetworks RN-XV module (now part of MicroChip).

In the latest models, these are simply options that you buy with the GEM. I like WiFi here, since I don’t have cabling for Ethernet anywhere near my GEM (and I don’t want to either, since my GEM is right next to my SubPanel)

In my setup, I have a WiFi rPi which is USB-Serial attached to a ECM-1240, reading the Main-Panel. This is on one side of the garage and it’s using WiFi to get access to the GEM, which is next to the Sub-Panel on the other side of the Garage. The rPi aggregates the data and phones it up to SEG (as well as PVOutput, for the Solar bits)

I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

You mean the DashBox. It still appears to be somewhat under development. It’s performing the role of what I’m using a Pi + SEG for, albeit using “locally-stored” data, which will be of interest to some folks. I may do this sometime, to replace the Pi and make my solution more consumer-friendly
 handy if you ever want to sell you house with this stuff as “features” :wink:

#20

A few more SEG images, showing the default Stream-list, a default graph for one of the Streams, as well as custom dashboard tabs I’ve built for Energy Overview/Detail and Temperature monitoring.

#21

guessed, very cool and super helpful. Thanks very much!!

PS: It was after reading Taken’s “book” on ISY / GEM integration that prompted me to look at Vera for energy reporting :wink:

#22

Hi again

I apologize in advance for my inexperience. I turned on my Vera today for the first time, added the Nest plugin, got the Sonos plug-in running, got the ISY manual loading plugin running (needed to learn about WinSCP for that) and am now trying to climb the hill described in this thread


In Vera I added a smart plug and Nest T-stats (via the nest plugin) and have created a “Site” and a couple of “Devices” at SEG and see how to get the token numbers. BUT: I am not sure how/where to write the script guessed provided on the prior page. I’m not a programmer and don’t know Vera, so my questions are VERY basic


[ol][li]Where do I write your script (with the edits relating to my SEG/Vera devices)[/li]
[li]Are all the parts of your script used for for a single device,e.g. a smart plug, or just part of the script (which part)[/li]
[li]Would you mind giving a basic step-by-step for setting up a single smart plug, for example, to report to SEG? I will then use that to expand on my own as much as possible[/li][/ol]

THANKS YOU VERY MUCH AND SORRY TO BE SUCH A NEWBIE

#23

@ccclapp,

[ul][li]1) The script belongs in the Startup Lua section of Vera.
In UI5, this is under:

Apps → Develop Apps → Edit Startup Lua

Don’t forget to confirm the changes, and Save/Reload Vera to force this to startup.

[/li]
[li]2) The Script just does a pick-n-pull of any set of “State Variables” you want to send to SEG.
This can include any combination of things really, from any Vera Device, but typically it’s Power, Energy & Temperature related values.

You need only know the correct ServiceId/StateVariable, and Device#, that you want to Monitor. It’s a list, so it can be as many of these as you want. The code monitors for changes in the value, and just sends it over to SEG.

Note that SEG may not want you sending 100’s of values/second, and it will take your “SEG Device” offline if you sent at a high rate. If you’re using the Brultech Power Monitor Plugin, set the GEM to refresh data every 60-90s seconds (the default is 5s, mine is 90s)

[/li]
[li]3) Step by Step

Not totally step-by-step, but the basic process is:

a) Modify the script to include YOUR SiteId, as well as the list of DeviceId/ServiceId/StateVariable combinations
Save this, and Vera will start pushing data to SEG, on your SiteID.

b) Login to SEG in a browser, and navigate to:
My Things > Tools > Discoveries

and set it running to discover your stuff. After a few minutes, it’ll see your feed data and present you with a UI to include it.
At this point, the Feeds will become a Device/Stream combination. You’ll have a Stream, within the [SEG] Device, for each thing you setup in the Vera/SEG startup script.

If you’re not seeing anything here, you can also try:
My Things > Tools > API Trace

and this will let you see what Vera is sending over. After a few minutes, if you still see nothing, then either Vera isn’t sending stuff OR, more likely, you’re using the wrong Token for SiteID.

c) Edit the Streams to add any metadata that you want.
This might include better Title/Description(s), along with Units-of-measure (in cases where SEG wasn’t able to introspect it from the Stream)

At this point you’ll have basic graphs and the data will start accumulating. From there, once you have some data, it’s probably a good point to build a simple one-sensor dashboard.

[/li][/ul]

#24


wonderful. Thanks guessed!!