Something seems to have happened to my Vera and I am seeing frequenty luup restarts (after six plus months of stability). I am seeing the following in Lua log files when searching for Reactor. I have checked all the Reactor created devices and expressions/conditions and nothing seems out of order.
Any hints on best way to debug this issue?
Thanks,
… selected logs from grep Reactor /var/log/cmh/LuaUPnP.log | grep luup
50
08/01/20 22:18:34.509
luup_log:120: Reactor START-UP! <0x76964520>
50
08/01/20 22:18:34.816
luup_log:120: Reactor: Plugin version “3.7-20190” starting on #120 (“Reactor”) <0x76964520>
…
50
08/01/20 22:19:13.531
luup_log:120: Reactor START-UP! <0x76cb2520>
—
—
—
50
08/01/20 22:19:13.766
luup_log:120: Reactor: Plugin version “3.7-20190” starting on #120 (“Reactor”) <0x76cb2520>
…
50
08/01/20 22:19:44.529
luup_log:120: Reactor START-UP! <0x76b42520>
—
—
—
50
08/01/20 22:19:45.038
luup_log:120: Reactor: Plugin version “3.7-20190” starting on #120 (“Reactor”) <0x76b42520>
…
50
08/01/20 22:20:51.705
luup_log:120: Reactor START-UP! <0x76de4520>
—
—
—
50
08/01/20 22:20:52.147
luup_log:120: Reactor: Plugin version “3.7-20190” starting on #120 (“Reactor”) <0x76de4520>
Hmmm… just because you see Reactor starting itself in the logs, doesn’t mean Reactor is causing the restart.
Reactor will disable itself after 10 Luup/system restarts within 15 minutes, unless you have disabled that feature. I will be very surprised if Reactor is causing the restarts, but it will get itself out of the way so you can figure out what is actually doing it.
Thanks - I didn’t realize that. For some reason I thought Reactor was causing it. I had to disable Reactor going into safety lock down mode (with high thresholds). I will keep looking to see what else is causing it.
So interesting - I am in the same shoes. Sure, maybe 6 days ago sounds close enough. My goodnight and morning scenes were causing upwards of 10+ restarts to complete. I am losing my mind trying to identify the issue. I have Reactor driving all conditions and executing all activities - but I also have several plugins that I’d blame first (and have been trying to determine) but no answers yet. I wish there was some level of observation I could trigger the condition and watch to see what’s hitting the wall.
Last night I thought I uncovered a pin storage issue with the Alarm Panel, so I edited the scenes to include the pin (vs retrieved from said failed storage method) and got my “leave” and “home” scenes to actually complete without restart and thought I found it. But apparently I made it worse with other troubleshooting efforts because overnight, it escalated to include multiple full reboots…
If it helps, here are the plugins that I have in case we have the same one(s) and can narrow it down:
Oh right - sorry I mean I wish I could run the activity and watch live log data show what’s unraveling. I can “run activity” and then all I see is the blue notification bar showing the restarts over and over. It’s crazy - like I’ll see only a couple devices change state before another restart, then the next few, restart, etc. Something is killing it
If you have an activity that reliably causes a problem, I would start removing actions from it, one at a time, and seeing if you can get a positive change in behavior. Of the things you listed, my first go-to would be any actions for the Nest WWN plugin, which I banished from my systems two years ago for being… problematic. Since you’ve got a lot of “AlarmManager” messages logged, the EVL3 actions would be a close second (consider first, since my WWN experience is negative but out of date).
Also, if you put comments in your activity with a “*” as the first character, it will emit the remainder of the string to the event log as a message. This is useful for calling out each device action before you run it (progress breadcrumbs), for example.
The event log, though, gets cleared every time Reactor starts (which it would at reload), so I’m wondering what you’re actually seeing there. My guess is you’re furiously refreshing the logic summary page trying to catch something before it reloads/clears, but that’s got a Vegas chance of you hitting anything useful. There is a way to log the events to a file, which will roll and survive the reloads: set LogEventsToFile on your ReactorSensor to 1, then reload Luup to make the setting change take effect. You’ll then find a file in /etc/cmh-ludl called ReactorSensornnn-events.log, where nnn is the device number of your ReactorSensor.
I love how you built in the ability to log through restarts…
So I enabled that, restarted, entered test time to ‘evening’ (for other aspects of the conditions) and triggered the condition. When it finally completed the scene, I changed the time to ‘morning’ and the morning condition was triggered. Those 2 scenes finally completed 21 minutes and 8 restarts later. I looked through ~1500 lines of event log and nothing stands out. I see reactor restarting, picking up where it left off, lots of mention regarding the test time, but nothing showing issue. The Harmony plugin did it’s thing, the alarm plugin did it’s thing, the ecobee plugin did it’s thing. Regarding WWN, I don’t have any activity to those devices, even with switching house modes (arm/disarm). I only have them to trigger a ‘fire’ scene if they ever alarm out. I can certainly remove that plugin since it’s certainly not used ‘often’ but I’d love to have some ability to react if they did.
Like you mentioned, I can start removing things one at a time to try to isolate but this is taking hours I don’t have at the moment. Might be faster to remove all plugin activities and then add them back in one at a time.
Also, only because it seems peculiar - I should mention that when I’m looking at the UI during the ‘morning’ scene I notice one device that is turned on (a kitchen cabinet light) seems to be turned on but wait for acknowledgement at the first of restarts. It’s a very loose measure of order, bc I know there are many devices being set to a different state but multiple times I’ve noticed it during all this.
That sounds like a good approach as well. You can also try inserting delays between device types. Your last comment regarding “wait for acknowledgement” suggests that it may be having trouble communicating with the device, and you are perhaps getting the dreaded “got CAN/dongle is in a bad state” and from there the Z stack is getting confused and eventually deadlocking (which is a frequent cause of restarts). Your LuaUPnP log will hold the clues here, and maybe that’s the next thing to start studying.
I must have a critical conflict in my 10 reactor sensors. I took all the plugin activity out of the mix, disabled all RS. I ran the goodnight scene (only ‘devices’ like lights, windows, etc.) and it ran fine. I ran morning scene same way and all ran fine. I added all plugin activity (media, ecobee, alarm, etc) for both goodnight and morning and they all ran fine again.
Any tips to check conflicts across all 10 sensors?! haaha… The logic summary and status tab is super helpful for each one, but might need to just sit down with a whiteboard lol
You’ll note the first message makes reference to node 67. That’s the ZWave node ID, which is not the Vera device number.
An easy way to find the Vera device is to use a user_data request. Request the following URL in a browser, substituting your Vera’s local IP address where indicated:
Use the in-page search feature of your browser to search for the string altid="67". Because of overlapping use of numbers in this data, there may be multiple hits to the search. The one you are looking for will typically also have on the same line the term id_parent="1". If you read the other attributes on that line, it should be pretty easy to find the device name.
Now, another thing to keep in mind is that this device may be fine, it was the device before it that was being acted upon that started the problem. So be sure to look above this in the log and identify that device as well.
Edit: also, if you go to the Reactor master device, Backup and Restore tab, you’ll find a “Logic Summary” link there, too. If you request the logic summary from the Reactor master device, it will give you summaries for all ReactorSensors together.
You’re like an oracle - you know that? This traced it back to an Inovelli device (all of which have had issues with) so it doesn’t surprise me that it is acting up again. Usually a power cycle (plug in module) fixes it.
Now that we’ve sorted that - any ideas on how to cross compare multiple RS logic? I must have a conflict in there somewhere…
LOL I did not, and of course you thought of this. Seriously - it’s odd how well you build things
And we are all grateful you have provided keys, color coding, even flashing, to help us figure out our mistakes haha!
lol I love it! I went ahead and sent the logic summary - I’m looking through this over the weekend, but based on the formatting requests, you probably have a macro that can scan through it, so figured I’d send it if you were willing to check it out.
Thanks as always for everything. Hopefully I’ll find something sooner than later
I poked through the Logic Summaries, and at first glance nothing really jumps out, but there are a couple of things you may want to investigate.
First, the big scenes like Goodnight… I have a similar function, but I’ve taken a different approach, not by design, but more by accident because that’s how I was thinking about my use at the time that I did it, which is before Reactor existed… what I do is I have a master off scene for each level of my house. I also have an off scene for each room, unless the room has a really small number of devices (e.g. one). Each “level master off” Vera scene does a RunScene (using scene Lua) of the individual room off scenes on that level, and only the single-device exceptions are done directly by the level master scene as needed. I then have an “indoor off” and “outdoor off” scene; the indoor off scene runs the per-level off scenes (again using RunScene in scene Lua). So in execution, there’s a spread of multiple jobs because multiple scenes are being run. I think this has a big effect on Vera’s timing… there’s something undefined but palpable in the way Vera runs scenes. I think having the additional jobs, which are put through Vera’s scheduler, better spreads things out and slows them down, which gives the system pauses where it can breathe, as opposed to slamming it with one giant scene.
Before going down that road, I would try this simple experiment: there’s a switch on the “Run Scene” action in Reactor activities where you can have Reactor run the scene, or hand it off to Vera/Luup to run it. Having Reactor run it is usually faster, but for a large scene like this, the pace of Reactor’s instructions may just be overwhelming for Vera Luup’s scheduling and Z-wave stack. I think it will pace differently if you let Vera run it itself; see if you notice a difference. If letting Vera run the scene is more stable, then I would just go that route — it may come with a little performance penalty, but for an “all off” scene, that shouldn’t be much of a concern.
If that doesn’t smooth out those larger scenes, then I would try splitting them up as I have done, or by some similar approach.
