Vera firmware Feedback

You and many others over the years have provided invaluable support, guidance, tools and plugins to keep this community alive. I guess its a good thing I didn’t get started on that ext root project I was planning! :slight_smile: Just when I thought things were getting better… I guess that’s the most disappointing aspect. I think many of us stuck around because after the change to Ezlo it appeared there was new energy and responsiveness in support. Most of us have technical backgrounds and we can understand bugs. Again, what is inexcusable is poor support response, communication in advance about major structural changes and a lack of follow-through. It now appears there are level 2 engineers out there that don’t understand that the Luup engine isn’t supposed to reboot constantly, which means there is a fundamental lack of understanding around the entire platform. Maybe I just picked up someone who started this week? We’ve been here before, but I may follow you away from Vera this time, as have others. For me that involves the tough decision to abandon 5 years of work and start new elsewhere…tough decision, I hope Ezlo is reading. We can’t hold out much longer.

Thanks… I had full logging going for days, i just didn’t upload them here. It just reveals that the engine decides to reload for some reason, looks like a normal reload signal, no error indicated. The support folks have had those logs for 3 days now. I’m no expert but it seems like something internal is causing the engine to decide it needs to restart but whatever it is, it’s not an event that is hitting a log routine anywhere so it’s not visible. The 10 minute timer is almost exact and predictable within a few seconds.

I am on the exact same boat. I too had several years of creativity, optimization and engineering put into the automation setup and did not want to start from scratch. That’s why @akbooer’s openluup is genius!
If you have not yet, you should look into it as the first step:

  1. Move your scenes and plugin to run on a more reliable machine while calling the commands and getting the signals still from the vera as a device hub through its luup API. This will stabilize your setup somewhat by reducing the load on the vera and making it less dependent on its time keeping/scene management/storage and memory.
    This alone may be sufficient for small setups. Unfortunately not so for most of us who have grown our systems over the years.
  2. Step two is then to swap the device bridge altogether to a far more open and stable software platform with z-way. It doesn’t even need to certify or test devices for compatibility… they all are since they are supported at the command class level.

7.30 was supposed to be a breakthrough firmware (and it was for me) but its release was a debacle because of lack of software release discipline and the eagerness to implement untested major changes and even ignore feedback from the community on critical neck breaking faults. 7.31 is another step forward, essentially fixing the major issue of 7.30 and changing the release management so honestly there is progress! There are just quirks in the design which are rooted so deep that still the whole LuaUPnP software is held by a thread and a snap of the finger can make it reload and corrupt itself rather than turning the lights on. (Not a Thanos snap… I see you coming Patrick.). So yes a new firmware platform is sorely overdue but since it is coming with no compatibility bridge altogether, the easiest migration path is openLuup and then (not) vera…

1 Like

Wanted to get everyone an update. Thank you Sorin, Mayker was able to get in touch with me yesterday. Apparently Vera is having SSH issues, so Mayker needed me to help set up what we used to call an “over the shoulder” set up via remote access. Mayker downgraded the firmware successfully, but unfortunately the two symptoms i was seeing continued 1) Luup Engine reloads every 10 minutes and 2) GCal3 firing unexpectedly

This morning I took a closer look at the GCal3 plugin and lo & behold it somehow lost it’s configuration. Part of the plugin had the correct information but the plugin was behaving as if it was new. One of the features included a test config that had the unit firing at the 5am/6am local time (12 GMT). Unable to exactly determine what was wrong, I performed the start up config actions setting the calendar id and json file and voila! GCal3 reset and started working perfectly again!

OK here’s the goofy part: Once GCal3 was configured and reset ok, the 10 minute Luup Engine failures have stopped! Coming up on 1 hour without a reload, all looking good.

I’ll advise Stuart (GCal3 caretaker) and Mayker to publish to the other Vera engineers. I cannot now say for certain the 7.31 firmware is the problem since I have it working on another Vera Plus that does not have GCal3. It’s clear there is some issue with GCal3 that gets hosed in the firmware upgrade, but perhaps it is easily resolved if the plugin is reconfigured from scratch. For now I’m cool back on 7.29 and a stable platform.

1 Like

Glad to hear you are back on track.
To be honest I am not very surprised either as I see two possible root causes to these upgrade failures which really have nothing to do with the firmware itself which IMHO is a major step forward for vera in terms of stability and longevity of the units.

  1. As @rigpapa suggested, oddball plugins, unsupported or not written quite right.
  2. Data corruption during the upgrade either because this firmware changes the kernel and therefore writes a lot of lower level OS files than others or because of hardware weaknesses.

Not sure about the GCal plugin but I get the hunch that you could upgrade now and reconfigure the plugin and be just fine which would indicate a data corruption problem.

Going forward I imagine any system incorporating unmaintained plugins will eventually tank. The 3rd party plugin market is fraught with pitfalls. So many knock together code to suit their own needs and the code is released into the wild poorly documented and ends up as a plugin.

Is/was the Vera device released as an appliance device. It certainly doesn’t appear to be the case. What we now have is some very accomplished coders “hacking” the device, and rightly so, to provide solutions to the shortcomings of the device but at the same time we have users adopting these hacks and jumping up and down when it all goes wrong.

The logical solution is to have a plug’n’play device that users can just power on and select the required plugins without having to worry about what is going on under the hood.

I’ve seen so many home automation systems that rely on the aftermarket plugin channel to generate sales instead of developing their own in house solutions. This business model creates more problems than it solves

1 Like

Well you are right on the money on the fact that the vera certainly is not an appliance and very far from it. We all know that. As for the second sentence, I would argue that I have not seen anybody jump up and down because these hacks fail… People who jump up and down are by far in majority people who never installed them. Pretty sad that almost any open source solution is more appliance like, save for the UI, than the vera.

This I think is ezlo’s eventual vision but I have yet to see a single one succeed. I would argue that the platforms relying on the user’s developed plugin channel remains much more successful. Compare wink and iris, to SmartThings for the cloud base and Homeseer/vera to the plethora of closed platforms which all went belly up. Sure it is easier to control and have a stable platform in a closed environment but you need to be able to execute to out innovate the competition to differentiate. To me it is more a matter of execution than anything. The openLuup/zway combination is the proof that the old vera platform had all the potential and capability to be fixed and support 3rd party plugins.
Long story short: It isn’t so much a problem with the 3rd party plugin market as it is a collection of bad firmware design decisions.

2 Likes

It’s pretty extreme but understandable.
I even feel that my migration to Z-way was less painful than your upgrade even though I run a lot more devices and plugins.

@Sorin
I took a snapshot from this morning of the top contributor list for the past year on this forum:

If you remove the crew, everyone on the list has expressed the desire, the plan or has already migrated to a different platform. We do not feel heard. I don’t know why I am even posting this as I completely lost hope and am as happy as I can ever be with my automation system since I got rid of the vera. You keep speaking about “corner cases”. This unfortunately is what I talk about when I speak of “test to fail”. You should not be designing for the 40-50% majority of cases but for the 0.01% of cases which are overstressing the systems. Given your current quality, every firmware should be tested with a live 200+ physical node network with devices of all kinds, preferentially every single device you have in your compatibility list, with at least every plugin available in the app store, maybe running 100 of them at a time. This would be the bare minimum to pass quality. That’s what we do in my line of business. Testing should involve inclusions/exclusions, configurations and stress testing, actuating all the devices at the same time.
Every build that is going to public release needs to go through beta testing after the above test is passed. No change can be made between beta and release without a new beta unless full risk assessment is made and the change is deemed trivial/inconsequential and they rarely are.

You should be also outputting a lot smaller incremental beta builds much more frequently and the team should have the ability to take out commits which failed instead of adding more code to patch broken/absurd designs and I am polite here, I remember offending someone in the crew calling it “vera style”. I am sorry but it is your old crew’s signature. I can sense a “we wrote it so let’s use it” attitude. Instead, of “It’s not working, let’s take it out for now and reconsider”, it’s “Let’s keep it in there and write more code to fix it” which then leads to fixes of the fixes or the fixes of the fixes none of which is thoroughly tested and all of which breaks more things. That’s how you end up with a luup reload around every corner of your code and why the luup reload takes over 1 minute when it should be taking a handful of seconds. For example, instead of getting to the root cause of why the user data would have some problems and fix that, you setup routines to check it and correct it upon reload, deleting and creating devices, further corrupting user configurations. My observation is that the reloads are often the only one worse thing to do next to doing nothing, wiping out precious data in the RAM and corrupting user data while taking the system down for an extended period of time. Yet guess what the vera does? I rest my case… as a happy camper on the other side of the fence where the grass is lush green looking into the junkyard.
You are spitting on the best thing you got going for yourself: The community developed apps and customizations/openness of the platform which has sustained many of us, allowing us to survive on workarounds and hacks while ignoring your real problems.

17 Likes

As always, rafale, I know more than anyone the huge amount of contribution we’ve got from fellow users such as you, Patrick, Ak, Rene, and others, and god know how thankful are many of our PM’s and management overall, when they find some critical data already posted, at hand. Please don’t take this personally. but no one is spitting on any of your work or thoughts. From day one since Vera has been taken over by Ezlo, you guys have been encouraged to contribute… if you like, want it and feel like doing so. In exchange, the company has been rolling out and patching the best they could the “existing mess of hardware and software”, or that’s how most of the guys you are constantly calling it. And we’re also working on some new and exciting stuff which many of you already contributed to and seem to like it. The fact that NOT all of your requests are being implemented doesn’t mean you are being ignored. There are individuals in this company that put a tremendous amount of work and time in just deciding which features or issues are worth investing manpower, money and time while balancing with innovation and introducing of new. And no, a company cannot survive without innovating and also investing in “new”.

And PLEASE, if you’re under the impression that you’re not being heard you are wrong on this one. You are being heard. One thing I’ve been asked, and I will have to do. Off-topic discussions must be split if they are not related to the main topic. So please, keep content like this on its own thread. It’s invaluable nonetheless, even if it’s not quite pink. We wanna hear it all.

1 Like

Thanks Sorin,

The fact is that we are receiving little to no feedback or acknowledgement for our inputs. The only person who has reached out to me has been @MCakan with a very candid, honest and hungry attitude, trying to understand what we are trying to say. Every other feedback from me or a number of other members since 7.30 beta, 6 months back, has been either taken with silence or better yet, you deleted posts quietly without notification(except for one). I only discovered it because other members PMd me or posted about it. Sorry, this is rude and it doesn’t encourage me to provide inputs. I am also seeing you splitting and merging threads to try to preserve announcements clean from feedbacks, creating threads like this one making them very hard to follow.
I sent my verbose logs twice for example to @edward what 4-5 months ago? All I got was, a month later, a note saying, “ohh it’s an old problem”… and then nothing else. Are you serious? The problem is so fundamental and gross that I wonder how a firmware ever got released with it. How was it assessed when discovered? Do I need to mention that I discovered critical firmware bugs on products for another company though much less impactful, they were fixed in a couple of days, working with me around the clock and they provided me $1000s of equipment for further testing in gratitude for the help provided and so that I can test them as well? What a contrast!
The other behavior I have seen is extreme defensiveness (and pride) on previous decisions (down to even code written) with people in the crew taking things much too personally. I have no idea who makes what decisions, no matter how absurd they are. It’s not personal. We are all customers trying to help you guys build a better platform, relieving your CS from dealing with the insane workload, avoiding sending customers to other platforms etc. Isn’t it your goal too? I sometimes doubt it when I see the amount of bickering between the crew and members on the forum. Do you not realize that you are the seller, the business here and that customers are king? No matter how rude or how unreasonable a customer is, that you, eZLO, should be the one taking the high road? That your customers are the ones who have your success in their hands and are behaving the way are, frustrated or disgruntled because they are suffering from your mistakes? Yes it’s painful for CS to spend hours with a customer trying to fix a problem for a customer but it is the customer who just flooded his yard, overheated or overcooled his house, failed to lock his door or worse yet missed a burglary alarm because the vera got a “luup reload” or froze itself on a “got CAN” or a “wait after NONCE_Get flood”? Do you not see the irony of having a device called the “vera secure” being the most unreliable and insecure home controller on the market?
Instead of asking your members to change their behaviors, maybe you should ask yourself why and what you can do about it to make them change their mind. Please don’t ask me to not have an impression of being ignored. Do something about it. You have been quietly censoring through post/thread deletion or splitting. How am I supposed to feel? Think about it, how much of our time (for all the devs and contributing members) we are spending here helping each other, finding solutions, building integrations and helping eZLO, giving you inputs so that your business can thrive, all work which you don’t have to do and all of this for free. What do we gain out of it? Free MBA intro class… and getting off of my soap box.

11 Likes

For the record, I second everything @rafale77 just outlined and asked.

I do product reviews in my sleep and eat beta testing for breakfast, but never before encountered the kind of environment we’re being asked to engage in here. (No point me going into detail about defensive Replies, or worse yet, relative lack of communication during the process…)

Like everyone else here, I give of my time and expertise freely not out of any allegiance to the company or its employees, but quite selfishly because I wish to protect my investment and – if I’m fortunate – make the platform better for others while doing so.

I’m humbled by the level of brilliance shown by those well-known (and appreciated!) developers who stuck it out all these years. My hope is that corporate recognizes NONE of us could have gotten this far without their continual input. I, for one, would have quit this ecosystem the moment my first Vera died, and never looked back.

It pains me that users of competing products notice the in-fighting on this Forum and cite it as rationale for staying away, yet I honestly cannot blame them. For what are home automation products but the sum of their latest reviews? Customers today are savvy enough to catch on to any disconnect – however slim – between the marketing hype and the reality on the ground.

We are also free agents, whose time is worth something, so us being concerned about things like this company’s product roadmap, or its long-term support of legacy devices, or its professionalism / corporate identity … is perfectly natural. Personally, my litmus test for joining forces with any tech company is to ask myself, “Would I hire these guys?”

Let me not answer that right now.

  • Libra
7 Likes

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.