New EZLO Linux Firmware pre-Alpha starts today

blacey · March 13, 2020, 1:34pm

@reneboer. The power meter reading bug that I reported a couple firmware versions ago for the Zooz power switch 2.0 hasn’t been fixed either. I’m curious, if you look at /var/log/ha-luad.log do you see anything that looks like the following?

Float value updated, node id: 2, channel id: 0, class id: 50
2020-03-13 09:27:29.365992 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/634_257_13.lua
2020-03-13 09:27:29.366988 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/634_257_13.lua
2020-03-13 09:27:29.368042 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
2020-03-13 09:27:29.378235 ERROR: LuaInterpreter: Couldn't run a Lua code: [handler.update_float_value]: Item id not found by item descriptor: item name: electric_meter_watt, node id: 2, channel id: 0, class id: 0x32, tag: nil
stack traceback:
=[C]
=?
=?
(...tail calls...)
2020-03-13 09:27:29.378747 WARN : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been terminated abnormally
Reachable state was updated, node id: 2, new state: true
2020-03-13 09:27:59.340345 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/634_257_13.lua
2020-03-13 09:27:59.341348 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/634_257_13.lua
2020-03-13 09:27:59.342544 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
2020-03-13 09:27:59.352846 INFO : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been executed successfully

melih · March 13, 2020, 1:53pm

Product Management team: pls deal with this asap.

blacey · March 13, 2020, 2:26pm

In my case, it might be that my Zooz Zen15 v2.0 isn’t fully supported yet on the Ezlo Z-Wave stack. I see the Zen15 v1.0 but not Zen15 v2.0.

In my last communication with the Team, they asked for the Zooz firmware version but the new Ezlo Linux firmware doesn’t expose that information to end users through the interface so I asked if there was a way to get it (i.e. command-line, logging, etc.) other than excluding and including on my Vera Plus just to get the Zooz firmware version. Haven’t heard back yet.

All of that said, I’m not super worried that it hasn’t been corrected yet, I was just comparing notes with @reneboer.

Ioana · March 13, 2020, 3:06pm

Hi @reneboer @blacey - we’ve captured your reports and we’ll work on them. I’ll be back with news once I have them.

reneboer · March 13, 2020, 4:49pm

Hi @blacey,

This is a bit of what I see in that log. Some code issues not handling some exceptions it seems.

Reachable state was updated, node id: 49, new state: true
2020-03-13 16:24:26.682063 ERROR: CoreDRemoteInterface::getSingleDevice(5e5f8e9e10d0762c552f3059), exception : fail to fetch a string at such key: "type"
2020-03-13 16:24:26.682966 ERROR: LuaInterpreter: Couldn't run a Lua code: ?:-1: attempt to index a nil value
stack traceback:
        =?
        =?
        =?
        (...tail calls...)
2020-03-13 16:24:26.683449 WARN : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been terminated abnormally
2020-03-13 17:44:29.775238 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/devices_info/generated/0_0_0.lua
2020-03-13 17:44:29.778937 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/0_0_0.lua
2020-03-13 17:44:29.780116 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/0_0_0.lua
2020-03-13 17:44:29.781128 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
2020-03-13 17:44:29.795689 INFO : LuaAddon: a script: HUB:zwave/scripts/set_item_value has been executed successfully
Reachable state was updated, node id: 48, new state: true
2020-03-13 17:44:29.940117 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/devices_info/generated/0_0_0.lua
2020-03-13 17:44:29.943687 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/0_0_0.lua
2020-03-13 17:44:29.944915 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/0_0_0.lua
2020-03-13 17:44:29.945873 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
2020-03-13 17:44:29.956368 INFO : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been executed successfully
Value updated, node id: 48, channel id: 0, class id: 37, value: 255.0
memory_consumption: value updated 1: 18158.0
memory_consumption: value updated 2: 18243.0
2020-03-13 17:44:29.972967 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/devices_info/generated/0_0_0.lua
2020-03-13 17:44:29.976941 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/0_0_0.lua
2020-03-13 17:44:29.977953 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/0_0_0.lua
2020-03-13 17:44:29.978905 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
memory_consumption: value updated 3: 61351.0
memory_consumption: value updated 4: 92088.0
2020-03-13 17:44:29.996812 INFO : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been executed successfully

Cheers Rene

MCakan · March 13, 2020, 5:22pm

Hi @blacey Thank you for your feedback, I checked with the team if there is any other way to get the Zooz FW version withouth the excluding and including it to your Vera plus but there is none. They also thought about sending an API request but it was not possible So is it possible for you to do an exclude and than include it back on your Vera Plus and provide us the values of ManufacturerInfo and VersionInfo fields from the device Advanced Tab. Btw can you please also provide some more details on the bugs you are experiencing (or if you have already done that can you point me to them ) But please do this update first if you have not done already Ezlo Linux pre-Alpha for Vera Edge Update - Official Announcements - Ezlo Community
Thanks a lot

reneboer · March 13, 2020, 5:32pm

Hi @MCakan,

Good news. I excluded (via the EzloApiTool) the device and included it again (worked first time) and now it is working as expected. This is for the Neo Power plug that has a wizard.

Cheers Rene

blacey · March 13, 2020, 5:36pm

@reneboer, thanks for comparing notes. Your log looks similar to what I am seeing but with a different root cause. I believe the firmware maps product attributes to a device-shim script and from your logs, it seems that the firmware doesn’t know about your device or have the mapping. Sounds like the team is on it so time will tell. Maybe your idea of re-including is a good one? (edited, looks like you posted positive results while I was typing)

Did that as soon as it hit the wild and before reporting that the issue still exists.

cat /usr/lib/opkg/info/firmware-common-git_version.control
Package: firmware-common-git_version
Version: 1.0.16
Depends:  libc
Section: eZLO Innovation
Architecture: ramips_24kec
Installed-Size: 4616
Description:  	Just firmware-common-git_version package

Absolutely. The only barrier is that in the most recent Vera Plus legacy firmware update lost a device that I need to recover/resurrect before messing with my network fabric. I’ll try to knock this off in the next day or so.

Sure - here is a region bug that I reported and still exists in 1.0.16.

And here is the one that we are discussing,

blacey · March 13, 2020, 6:58pm

Screenshots from my V+ with the requested info are attached but it is worth noting a few observations during my exclude/include process:

Excluding the Zooz Zen15 on the Alpha firmware was fast.
Adding the Zen15 with the legacy firmware on the V+ is much slower than than the new Linux firmware on the Vera Edge. The legacy firmware has to reload the engine so the user is forced to wait for that to complete. Nice to see that is gone in the new Linux firmware.
Excluding the Zen15 with the legacy firmware on the V+ is slower because the legacy firmware has to reload the engine so the user is forced to wait.
Pairing the Zen15 on the Vera Edge running the Alpha firmware is fast and painless - no more waiting for engine reloads! I don’t know if the speed improvement is the z-wave stack re-write or the number of devices (I have far fewer on the Edge running the new Linux firmware). I’m hoping it is the new z-wave stack.
Power usage status updates are nearly instantaneous on for devices on the new Linux firmware. With the iOS app launched and displaying the Zooz device, I used tail -f /var/log/ha-luad.log to watch the events fire as I plugged and unplugged a power-consuming device. As soon as I plugged the device into the Zen15, the event log updated and the iOS App UI updated - the legacy firmware updates in a similar fashion but subjectively it feels slower.
Power usage reporting includes watts and volts on the new Linux firmware
Excluding and re-including a device on the Vera Edge increases the node number - shouldn’t the node numbers be re-used?
Excluding and re-including the Zoon Zen15 V2 on the Vera Edge enabled the power usage reporting (as @reneboer first reported), however I still see the following warnings in the log:

Float value updated, node id: 4, channel id: 0, class id: 50
2020-03-13 14:57:38.856400 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/634_257_13.lua
2020-03-13 14:57:38.857657 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/generated/634_257_13.lua
2020-03-13 14:57:38.858637 WARN : Plugin script reader: Could not find plugin file /opt/firmware/plugins/zwave/scripts/model/devices/monitoring/.lua
2020-03-13 14:57:38.876120 INFO : LuaAddon: a script: HUB:zwave/scripts/events/events_handling has been executed successfully

Perhaps it is on the roadmap, but the device details that are available on the Legacy firmware and Web UI should be available on the new Linux firmware.

Zooz_Zen15v2_DeviceInfo-1800×1583 103 KB

MCakan · March 13, 2020, 11:34pm

Thank you @blacey for this detailed answer you provided so fast. Have you applied for Atom v2 beta yet? We would like to have you also here Beta enrollment opened for Ezlo Atom v2 - Official Announcements - Ezlo Community

I’ll double check on that!

thanks for confirming that power usage reporting is up also in your case like @reneboer. I will look also into these warnings.

blacey · March 14, 2020, 12:28am

Yes, I applied yesterday

HSD99 · March 14, 2020, 1:18pm

I have successfully upgraded but the unit’s Wi-Fi is on. I would like to turn it off, but none of the command line methods work. Can you provide guidance?

reneboer · March 17, 2020, 10:40am

Hi,

I would like to suggest some essential functions missing from the current Ezlo firmware before it can be more than a hobby level controller (sorry, harsh but true).

ability to set z-wave device associations
ability to set z-wave device parameters

1 is core functionality imo. If I have a switch device (Aotec Mote something, Nodon Octan, etc) or a sensor i need to be able to set a direct association between that and a z-wave actor (switch, alarm, etc). I cannot have the controller as a single point of failure in these scenarios.

2 is needed to build a reliable z-wave network of any scale and to add new devices that may need a tweak. A known problem with several reporting devices is that they send way too many reports in the default config. For example an Aotec HEM will send 3 reports per second out of the box. Use two of those and you are done. Changing some devices bad behavior is a must.

I could make this list much longer, but this are two that are too important not to have. It should be up to par with the Vera capabilities before I would even consider it, but then reliable.

Cheers Rene

Oleh · March 18, 2020, 11:12am

Hi @reneboer.

On the firmware level these functions are available.
Also we already have support for these functions in Lua.
Now we are working on API extension which will make it available in hub API.

blacey · March 19, 2020, 1:06am

I agree and I’de add the ability to display z-wave device details (e.g. firmware, etc.) akin to what is available in the Vera today.

blacey · March 26, 2020, 2:44pm

@Ioana, FYI, I noticed a spread toolkit security warning in the logs a couple firmware versions ago and it is still there. Just raising awareness because it is a security issue that is easy to fix.

Conf_load_conf_file: using file: /etc/spread.conf
Successfully configured Segment 0 [127.0.0.255:4803] with 1 procs:
	           localhost: 127.0.0.1
ENABLING Dangerous Monitor Commands! Make sure Spread network is secured
Set runtime directory to '/var/run/spread'
Set user name to 'root'
Set group name to 'root'
Finished configuration file.
Hash value for this configuration is: 3871598377
Conf_load_conf_file: My name: localhost, id: 127.0.0.1, port: 4803
Spread: SECURITY RISK! running as root, but unix domain socket is not in a root-only writable directory. May risk denial of service or malicious deletion of unexpected file in directory: /tmp
Hangup

The root cause for the “SECURITY RISK!” seems to be a missing spread directive to set the spread runtime directory properly. As a test, I added the following to /etc/spread.conf and it suppresses the security warning.

RuntimeDir = /var/run/spread

And sent a HUP signal to the spread process to reload the config.

Conf_load_conf_file: using file: /etc/spread.conf
Successfully configured Segment 0 [127.0.0.255:4803] with 1 procs:
	           localhost: 127.0.0.1
ENABLING Dangerous Monitor Commands! Make sure Spread network is secured
Set runtime directory to '/var/run/spread'
Set user name to 'root'
Set group name to 'root'
Finished configuration file.
Spread daemon exiting normally!

Anyway, something small and easy to add to the backlog. Because it presents a security vulnerability and easily remedied, it behooves you guys to fix it sooner than later.

Another thought, as the team looks to secure spread, it might be productive to create a spread user and group and run the spread daemon as that user and group - you can configure the daemon user and group in /etc/spread.conf. Running a communications daemon with root privileges is unnecessary and in terms of security, considered bad form.

Food for thought.

blacey · March 26, 2020, 4:32pm

@Ioana, just a followup on running the spread daemon under a non-root user and the unprotected 4803 socket.

As a quick test, I added a spread user and group to /etc/passwd and /etc/group respectively and added DaemonUser = spread and DaemonGroup = spread to /etc/spread.config. I changed the owner of /var/firmware/log/spread.log to spread. For reference, here are the specific changes that I made:

/etc/passwd

root:x:0:0:root:/root:/bin/ash
daemon:*:1:1:daemon:/var:/bin/false
ftp:*:55:55:ftp:/home/ftp:/bin/false
network:*:101:101:network:/var:/bin/false
nobody:*:65534:65534:nobody:/var:/bin/false
spread:*:1111:1111:spread:/var:/bin/false

/etc/group

root:x:0:
daemon:x:1:
adm:x:4:
mail:x:8:
audio:x:29:
www-data:x:33:
ftp:x:55:
users:x:100:
network:x:101:
nogroup:x:65534:
spread:x:1111:

/etc/spread.conf

DaemonUser = spread
DaemonGroup = spread

>>AFTER RESTART<<

Conf_load_conf_file: using file: /etc/spread.conf
Successfully configured Segment 0 [127.0.0.255:4803] with 1 procs:
	           localhost: 127.0.0.1
ENABLING Dangerous Monitor Commands! Make sure Spread network is secured
Set runtime directory to '/var/run/spread'
Set runtime directory to '/var/run/spread'
Set user name to 'spread'
Set group name to 'spread'
Finished configuration file.
Hash value for this configuration is: 3871598377
Conf_load_conf_file: My name: localhost, id: 127.0.0.1, port: 4803
Spread: SECURITY RISK! running as root, but unix domain socket is not in a root-only writable directory. May risk denial of service or malicious deletion of unexpected file in directory: /tmp

ps w | grep spread | grep -v spread
 6786 spread    2888 S    /usr/sbin/spread -c /etc/spread.conf

Voila! Spread is no longer running as root.

BUT!!! The SECURITY RISK message is back!!

I downloaded the spread toolkit source code and reviewed the configure options for a potential root cause for the spread sockets being opened in /tmp regardless of the RuntimeDir setting. I believe that the firmware build system sets the spread configure option --with-unix-socket-dir=/tmp which isn’t the best. It would be better to set it to /var/run/spread

root@8775fe673d9d:/tmp/spread-src-4.4.1# ./configure --help=short
Configuration of Spread 4.4.0:

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-cflags           Specify additional flags to pass to compiler
  --with-cppflags         Specify additional flags to pass to preprocessor
  --with-ldflags          Specify additional flags to pass to linker
  --with-libs             Specify additional libraries to link with
  --with-mantype=man|cat|doc  Set man page type
  --with-pid-dir=PATH     Specify location of spread.pid file
  --with-unix-socket-dir=PATH     Specify location of Unix Domain Socket for client-daemon connections. If you are running Spread as a root user, you should define this to be a root only directory such as /var/run to avoid some security risks.

Finally, the team probably wants to update spread to version 4.1.4 vs. the current Linux firmware version 4.0.4 from May 2014.

Spread Security Recommendations:

Configure spread build options to use /var/run/spread as the backing store for spread sockets
Run spread under spread user and group (i.e. non-root). This entails adding a spread user and group id, modifying /etc/spread.conf to use them, and adjusting any permissions set by scripts like /opt/bin/firmware/spread.sh and /etc/init.d/spread to the spread user.
Optional: upgrade spread 4.0.4 to 4.1.4

Doing these two pretty trivial development tasks should address the spread security issues. I realize this is alpha software but security is paramount for any device connected to the Internet and sometimes is more difficult to fix security issues later in the release cycle.

Ioana · March 27, 2020, 8:14am

Thank you @blacey for reporting this, the team is looking into it and have it planned to work in the current sprint.

Oleh · March 30, 2020, 9:43am

@blacey.
Hi blacey.
Thanks for your comment.
Spread security issue in our backlog.
It wiil be available in next releases.

Thanks.

blacey · April 5, 2020, 11:05am

FYI, I upgraded to 1.0.17 but spread it is still running with root privileges in the current version.