Pre-ALPHA Ezlo LUA API Documentation

Hi developers,

As promised, we’re releasing the pre-ALPHA LUA API for the Ezlo Platform today:
Pre_ALPHA_Ezlo_Plugins_API.pdf (614.2 KB)

We want to work with you guys to make the Ezlo Platform the best Home Automation Platform, so we’re counting on your feedback, to tell us everything you want to see in the API. Please share in the comments any questions/feedback/requests you have, to make sure we’re incorporating all.

Thank you,
Ioana - Product Manager

4 Likes

Great work guys!

Please provide feedback how to improve/expand the APIs please. thanks.

A couple of things pop up on first look, for me:

  1. There’s little description of plugin structure and the theory of the plugin architecture to guide implementation. This is largely reference documentation for individual APIs, and while that provides some information, it requires us to “read your mind” about how you are thinking plugins are built, packaged, and interact with the system. We don’t know when they are loaded, or how they are initialized, for example.
  2. My gut reaction to having subscribed event notifications dispatched to scripts, which are presumably separate Lua files, rather than functions within a Lua script/file directly–this makes me very uneasy. It seems like a potential mess for systems that will need to respond to a large number of events/event types. A lot of that depends also on…
  3. What is the Lua environment in which the scripts run? Is there one environment for the entire plugin? One per script? One per instance? Is/are they consistent/persistent across invocations of scripts? If one script file declares a Lua global, is it visible to another script file? If you’re imagining “storage” is the way to persist necessary data between calls in a running plugin, that a substantially higher load on the plugin and whatever backs that storage than the current in-memory model of plugin runtime state. I don’t see clearly how you are diffentiating “runtime” state from long-term (across reload/reboot) persistence, and the docs don’t say.
  4. Your network module… are these convenience functions, or the APIs we must use for all socket access? If yours intended to be the full and complete replacement for real sockets, I see many issues now and well into the future. First and foremost, plugin developers will not be able to leverage existing Lua modules for protocols you haven’t implemented (where’s your IMAP support in this module, for example?). There’s a lot that exists above layer 4 that might be used. Second, if you are imagining providing the sole, entirety of network interaction this way, when/how, for example, will you address posting JSON or other formatted payloads in your HTTP POST (very common in RESTful APIs)? Other methods like PUT and DELETE? SSL/TLS control like minimum protocol, or whether peer certification validation is done, and to what certificate and through what CA chain? You will dealing with an endless tide of feature requests, a huge QA load, and a lot of impatient developers and users waiting for firmware updates to support their feature (if, that is, you decide to implement it).
  5. The network.connect() docs don’t describe how to ask for sync or async, it just describes the possibility.
  6. Since the network module seems to have internal buffering, it would be nice to have a call to flush it (dump the buffer contents without having to read it). What’s the size of the buffer? If the plugin has a bug that causes it to stop reading on a connection to a frequently-transmitting endpoint, will the buffer grow until it consumes all system memory?
  7. Are Lua coroutines functional is this environment?
  8. What version of Lua?
  9. Why is it (again) necessary to restart the entire firmware? Can not the initialization script for a plugin just be called on all demand? Why not?
  10. How are UIs defined for plugins?
  11. What device types are defined in the system?
  12. How do plugins define new device types? Can they? Or are plugins expected to accrete instances of predefined types into a functional representation of the new device they are implementing? If that’s the case, how to users see and manipulate specific settings that may apply to the gateway or its devices?
  13. Why are capabilities like lock codes and lock schedules in the Zwave module? Aren’t these generic to all lock devices (may or may not be supported, but are nonetheless possible capabilities) regardless of gateway? The same question applies to RGB values, and thermostat operating states, and any other device capability that may be expressed in other-than-ZWave devices. This looks like you’ve broken encapsulation of the device model.
  14. Can I subscribe to a device or item? How does a plugin respond to changes in its devices or other devices in the system?
  15. There aren’t really any detailed descriptions of the event model or what’s passed to the script for context for each of the different types of events (except a brush for network)–that’s important to judge the usability of an API that relies heavily on event notifications.
  16. I’m a bit concerned about all APIs throwing errors rather than returning error codes. There are passionate debates about things like this in principal, but I think some sensitivity to the language needs to enter into the discussion here. Lua does not have structured exception handling–it has pcall() and that’s it. So it’s pretty much as you’ve shown in the example–you make a call by making another call with the name/functionref of the function you want to call and a list of arguments, and you get back your function’s result, or the executive’s error response and some error return value. So aside from the general ugliness of the way you have to do it in Lua, it introduces the possibility (likelihood) that every subsystem will return myriad different messages (strings), and callers will be trying to match them or parse them for meaning and decision-making rather than looking for classes of errors. The specs and examples don’t really say what the errors are that are returned, but the example suggests they are strings. I’m not a fan. Recommendation/TLDR: all errors thrown should throw tables only, and the table should contain, at a minimum, keys to describe the subsystem that threw the error (e.g. source), a class for the error (e.g. class such as “parameter” for invalid parameters), a unique ID for the error within the class (e.g. id), and a human-readable message (message), along with any additional keys to provide additional context. Whatever they are, it has to be concrete and immutable, because people are going to write code expecting those certain values.
  17. Add to this that the error model seems inconsistent. For example, why does the example for network.connect() show error handling being done as a boolean/nil test of the returned connection handle? If all APIs throw errors, the test of the returned handle should never occur on a failure (because an error is thrown before the conditional can be executed), and on success it’s redundant (the branch will never be taken). Are some errors, then, expressed by return value and others by exception thrown?
  18. Does a non-interval timer “exist” after it fires, or is it necessary to call cancel() on such a timer to destroy its instance? If so, why? And docs be clear about this as a requirement, otherwise you will have plugins leaking memory due to completed but undisposed timers.
  19. Timer IDs should be strings, not numbers. In my experience, timers tend to be task-based, so it’s beneficial to sanity to be able to identify them as “background-refresh” or “dev395query”. If you’re trying to assign numbers, you quickly lose track of what the numbers mean, particularly if you are creating multiple timers for tasks for multiple devices.
  20. The storage module description… I can’t make out what it means.
  21. On the storage module, I’m trying to imagine how and why I will not quickly decide that storing a single table, with all of my plugin’s persistent data within it, is not the most expeditious and supportable thing to do for the long term. Storing individual integers and strings seems like a verbose waste of time, comparatively.
  22. Can plugins communicate with one-another? If so, how?
5 Likes

What @rigpapa said, plus I don’t like the notation with underscores. It’s probably just me, but camelCase or PascalCase seems more like it’s coming from this century.

I can’t really comment any further with not so many details. I applaude plug-ins have their own directory (No more race to get the best looking filename…), but I’d really want more details.

Since we’re at it, I’d really like to have an emulator rubbing on Linux (and so on Windows subsystem for Linux) to develop. Right now I have to use openluup on WSL,

2 Likes

Thank you for your feedback. It’s very useful for us to get it during developing new API.

Using same Lua API and same plugins were main requirements for new plugins engine. We have wide range of smart controllers on the market and a few new controllers will be released soon( You can find our new controllers on CES 2020 reviews ). Ezlo Atom, PlugHub, VeraEdge, Ezlo Plus have the same Lua API. Main idea is integrate device once and deploy on all controllers. Linux based controllers support a bit more Lua modules as atom but Core, Z-Wave, Storage, Timers modules are the same. Base functionality and API architecture are limited by atom hardware possibility. It means for atom: providing the same functionality as Linux controller have but It’s also share all memory and speed improvements with Linux controllers.

As I said in pre-Alpha/Beta topic: we are not ready to share our API with community because we don’t have external development tools and well organized public documentation. But It’s good policy for product to share functionality/API as soon as possible with community and clients and now you could read part of our internal documentation.

@rigpapa

There’s little description of plugin structure and the theory of the plugin architecture to guide implementation. This is largely reference documentation for individual APIs, and while that provides some information, it requires us to “read your mind” about how you are thinking plugins are built, packaged, and interact with the system. We don’t know when they are loaded, or how they are initialized, for example.

Yes. It’s internal documentation. We will work on better way of API representation.

In a few words:

  • all plugin information described in config.json file( you can find examples of this file on your controllers with new firmware: /opt/firmware/plugins );
  • each plugin has own “virtual” space and doesn’t have access to space of other plugins;
  • each script has 3 minute execution time limit;
  • Storage module should be used for saving states between scripts execution;
  • Timers module should be used for “endless” checking;
  • plugins implement all business logic, modules just provide access to specific functionality;
  • UI displays devices from Core module only;

My gut reaction to having subscribed event notifications dispatched to scripts, which are presumably separate Lua files, rather than functions within a Lua script/file directly–this makes me very uneasy. It seems like a potential mess for systems that will need to respond to a large number of events/event types. A lot of that depends also on…

Atom cannot run a few Lua scripts in the same moment of time. Each script should be finished and don’t take a lot of execution time. event/subscription architecture is main possible option for now. But it also gives benefits : less reaction time and more time and RAM memory for main functionality of firmware. Yes, it also forces you to keep clear the structure of the plugin and split it into “modules”.

What is the Lua environment in which the scripts run? Is there one environment for the entire plugin? One per script? One per instance? Is/are they consistent/persistent across invocations of scripts? If one script file declares a Lua global, is it visible to another script file? If you’re imagining “storage” is the way to persist necessary data between calls in a running plugin, that a substantially higher load on the plugin and whatever backs that storage than the current in-memory model of plugin runtime state. I don’t see clearly how you are diffentiating “runtime” state from long-term (across reload/reboot) persistence, and the docs don’t say.

One per script. We only have persist plugins storage. I think, we will add API for temporary/runtime storage in the future.

Your network module…

Yes. Our network module. It’s prototype. We don’t have any plugin which requires this API. Only Berkeley sockets are available. We will support other protocols in the future.

The network.connect() docs don’t describe how to ask for sync or async, it just describes the possibility.

connection will be established just after call but sending/receiving data should be done via “events”.

Since the network module seems to have internal buffering, it would be nice to have a call to flush it (dump the buffer contents without having to read it). What’s the size of the buffer? If the plugin has a bug that causes it to stop reading on a connection to a frequently-transmitting endpoint, will the buffer grow until it consumes all system memory?

Plugin will receive event with received new data. if plugin doesn’t save it : data would be lost.

What version of Lua?

5.3.5

Why is it (again) necessary to restart the entire firmware? Can not the initialization script for a plugin just be called on all demand? Why not?

It’s possible but not for custom plugins which were installed via SSH.

How are UIs defined for plugins?

Part of that is missed now and we are working on it. UI displays devices and items from core module. List of supported items will be described in the future.

What device types are defined in the system?

New plugins use almost the same categories and subcategories of devices as Luup. It will be shared in the future.

How do plugins define new device types? Can they? Or are plugins expected to accrete instances of predefined types into a functional representation of the new device they are implementing? If that’s the case, how to users see and manipulate specific settings that may apply to the gateway or its devices?

Plugin can use custom device type for own devices but if UI doesn’t support it : it can be displayed wrong.

Specific gateway/device preferences will be added in the future.

Why are capabilities like lock codes and lock schedules in the Zwave module?

Core module - protocol independent device/items model. UI uses this abstraction layer for displaying available devices.

Z-Wave module - protocol specific API for working with Z-Wave stack directly.

Each of this layers doesn’t know each other. It’s possible to implement own Z-Wave plugin instead of our. Or work with Z-Wave specific API from your plugin and our plugin will receive all new changes.

Can I subscribe to a device or item? How does a plugin respond to changes in its devices or other devices in the system?

We have one main request for changing value of item from UI: hub.item.value.set. Author of plugin should define own script for this request in config.json file. After that all new values for his items will be sent to this script.

There aren’t really any detailed descriptions of the event model or what’s passed to the script for context for each of the different types of events (except a brush for network)–that’s important to judge the usability of an API that relies heavily on event notifications.

We will improve documentation. Main events were described. Core module does not have Lua events now but It can be easily added.

I’m a bit concerned about all APIs throwing errors rather than returning error codes

Thank you for your recommendation we will think about described approach. Yes, now we use string errors and calculate “Lua call stack” for each error.

Does a non-interval timer “exist” after it fires, or is it necessary to call cancel() on such a timer to destroy its instance? If so, why? And docs be clear about this as a requirement, otherwise you will have plugins leaking memory due to completed but undisposed timers.

You can create timers as much as you want. exist and cancel methods help to build different business logic on timers without saving special data in storage.

Timer IDs should be strings, not numbers. In my experience, timers tend to be task-based, so it’s beneficial to sanity to be able to identify them as “background-refresh” or “dev395query”. If you’re trying to assign numbers, you quickly lose track of what the numbers mean, particularly if you are creating multiple timers for tasks for multiple devices.

Timer id - is string.

The storage module description… I can’t make out what it means.

key / value storage. Each plugin has own storage space.

Can plugins communicate with one-another? If so, how?

No. But we think about it.

@therealdb

Since we’re at it, I’d really like to have an emulator rubbing on Linux (and so on Windows subsystem for Linux) to develop. Right now I have to use openluup on WSL,

Sorry, we don’t have it now.

Hi,

Looks like a new learning curve. Please make sure you will have functional examples once we should start using this. I could not figure out how to build a plugin for the current environment just based on the Luup extensions in the wiki. I put it together with looking at existing plugins mostly.

The Storage module I find it confusing. The description states each plugin has its own ‘volume’, but at the same time warns for key uniqueness between volumes? If it is a must for variable storage between events/sub-modules that can proof a hassle. It also says ‘other ones’ can use it. Is that other modules for the same plugin, or?

The network module looks to be the replacement of the incoming handler and that bit sure needed a big overhaul. So good to see.
I would assume we can still use the standard Lua socket libraries for more complex request handling. If you have that, why replicate it in the network module? A good version of Lua websockets would be welcome too (or get http://wiki.micasaverde.com/index.php/Verawebsockets working).

I like you can have identify timers. Makes scheduling a lot more robust. It would be nice if there will be an option to have the possibility to set a timer at a specific time as is possible today. Would save a developer the need to do that math them selfs.

With a custom plugin there is always a custom bit of GUI required. I guess that part of the picture will come later.

As always Patrick can ask much of the right questions so that is helping all of us. This is sure a good start, but still it would take me months to make my set of plugins available on the new environment. Unless… you pick up the idea again to do most your selfs (with our help as it seemed a year ago), but then I know it will take you years to add the same value all independent developers have added to Vera over the years.

Cheers Rene

2 Likes

But are the standard Lua socket libraries usable, with full functionality (e.g. non-blocking I/O, or even blocking I/O properly integrated)?

@reneboer,
thanks for your feedback.

Looks like a new learning curve. Please make sure you will have functional examples once we should start using this. I could not figure out how to build a plugin for the current environment just based on the Luup extensions in the wiki. I put it together with looking at existing plugins mostly.

We continue to work on documentation and adding new functionality and examples. It will be available in next releases.

The Storage module I find it confusing. The description states each plugin has its own ‘volume’, but at the same time warns for key uniqueness between volumes? If it is a must for variable storage between events/sub-modules that can proof a hassle. It also says ‘other ones’ can use it. Is that other modules for the same plugin, or?

Each plugin has its own storage. Each script of the plugin has access to the same storage but do not have access to the storage of other plugins.

The network module looks to be the replacement of the incoming handler and that bit sure needed a big overhaul. So good to see. I would assume we can still use the standard Lua socket libraries for more complex request handling. If you have that, why replicate it in the network module? A good version of Lua websockets would be welcome too (or get http://wiki.micasaverde.com/index.php/Verawebsockets working).

This API from this link is blocking script execution.
The main idea of our API – non blocking script execution.

I like you can have identify timers. Makes scheduling a lot more robust. It would be nice if there will be an option to have the possibility to set a timer at a specific time as is possible today. Would save a developer the need to do that math them selfs.

With a custom plugin there is always a custom bit of GUI required. I guess that part of the picture will come later.

We are working on it now.

@Sorin, Does this thread belong in the developers section? I would certainly think so.

@Oleh, It does look like work in progress indeed and a significant learning curve for everyone here. I do want to ask the reason why you would not take inspirations from what already exists in terms of architecture and syntax rather that start something from scratch. Making the API non blocking, multithreading is all good idea. Changing what is underneath to make that happen, I understand but could the layer above be more “compatible”?

1 Like

We have only Lua engine without extensions now.

:muscle: :+1: :+1: :+1:

Haven’t had a lot of time to weigh in lately but this nugget is a sound architectural principle. One can create blocking APIs with non-blocking APIs but not the other way around.

That said, even though there are some very talented Devs hanging out here, it would be good for the core engineers to provide higher-level abstractions/frameworks and tooling that reduce complexity for plugin-developers allowing them to implement “business logic” easily without having to write a bunch of repeatable boiler-plate code against low-level, higher-complexity APIs. It might be productive to solicit the common “patterns” from developers to see if they can be generalized and incorporated into a plugin framework that sits above the lower-level APIs. Goal: Community developers produce a larger number of higher-quality plugins while writing less code inspiring others to codify their innovative ideas into a “simple” plugin/extension.

Yes, but no. I’m concerned there’s a bigger architectural issue that Oleh’s comment alludes to: they view these plugins/extensions as scripts rather than as subsystems that run within the environment. That may be a function of language usage, but I’m not seeing evidence of that in the remainder of the API. First evidence: what about this API and the architecture makes a script non-blocking?

And I repeat my concern about plugin structure here in this context. Having events run scripts rather than functions (or closure/lambda) reduces the ability of the plugin author to create truly modular code structure that is readily reusable, and increases complexity of configuration management in deployment. One cannot, in this system, devise a single source file that contains all of the event handling functions required by the subsystem. Each event handler lives in a separate file. This will quickly add to code complexity, with appurtenant problems (difficult debugging, version control issues, deployment consistency, etc.). This is made worse, in fact, by @andryist 's earlier response to me that each script runs in its own Lua environment.

Patterns provided in the underlying API are nice, but not as necessary in my view as an architecture that supports good modular structure. In such an architecture, pattern implementations can be installed modules, shared by plugins/subsystems or, as needed, isolated to one (a plugin could provide its override version of an otherwise shareable pattern). I see no support for this here.

So are you saying here that if I want to monitor an item in another device, I somehow need to override hub.item.value.set on that device? Or, perhaps worse, I can provide an override of hub.item.value.set for the entire system? Is that even possible? If it is, you’ve thrown security and encapsulation out the window. If I can only do it for my plugin’s own devices, then it’s not useful for creating a plugin that monitors the states of another device in the system, and you haven’t described here or in the API how that would be possible. Is it possible?

1 Like

I was referring to what I hope (presume?) is an architectural “non-blocking execution” mantra that transcends beyond plugins. If you look at a lot of the “performance” issues the Vera suffers from today, blocking is often the culprit. Every time @rafale77 justifiably climbs on the soapbox about queue blocks, dropped messages, etc. I think blocking timeouts are the bane of the Vera’s existence… You can’t code around foundational issues and a non-blocking mantra is a step in the right direction - that’s all.

I’m on the same page with all your points but perhaps “script” is just a poor choice of words (?) even though the verawebsockets example clearly uses #!/bin/lua to specify a “script” interpreter in the classical sense. But heck, Swift is one of the most modern languages and can be run through a #! interpreter but I digress. If plugins are relegated to simple scripts, then that will be a problem IFF the base system requires significant “propping up” through community extensions to be useful as is the case today.

It does appear that plugins will run within their own context, sort of sandboxed, with local variables and storage and with entry points defined by registering handlers for events that are invoked whenever the event occurs (kind of like traditional callbacks through subscription). I’m with you that this begs for a more modern approach of blocks/closures/anonymous functions that I believe Lua supports at a rudimentary level which would help avoid performant reentrancy/multithreading issues. Without it, it may be a more pedestrian approach is that the callbacks are invoked in a serialized fashion to a given plugin to reduce internal plugin concurrency complexity, but hopefully not across plugins. In other words, plugin X’s handling of event Z should not delay plugin Y’s handling of event Z. With what the team has had time to publish, it is too early to tell.

Still, I think teasing out the patterns that will drive the plugin API is a great way to go. In fact, there should be well-understood categories of plugins (e.g. foreign device support) and the harder/gnarlier aspects/use cases should prove/disprove the API early. It is a lot easier to fail fast and change/refactor paper than code.

In the meantime, maybe @andryist et. al can shed some light on the Plugin Vision, Mission and Strategy that also devolves down to addressing details like the plugin lifecycle, execution context, fit/function/form in the system, etc.

1 Like

This is one of my big areas of concern. Aside from inconvenient code structure, @andryist also said that each script runs its own Lua environment. Unless he misspoke, that means there is no common runtime storage between the event handlers and other elements of the plugin, you have to jump through hoops to emulate it (e.g. perhaps using the storage module). That stands to contribute considerable overhead.

But I want to repeat a point here. Let’s not confuse asynchronous, and possibly concurrent, with non-blocking. They are related, but not equivalent. Non-blocking is a behavior that, in this structure, the plugin developer will need to implement–it is a rule to abide by. The essence simply is, don’t allow your script to run too long. That’s not non-blocking, that’s just abiding by a best practice. Technically, it is at best voluntary multitasking. There is nothing inherent in this API that makes it non-blocking. I guarantee that I can write a script that never returns. To be non-blocking, I have to make my script return (to keep the system from killing it at the 180 second time limit). It is still up to the script to do as little work as is necessary as quickly as possible when called upon and return/exit, and in that sense, there is absolutely no difference from this architecture and the current architecture of Luup plugins. The timeout is just longer.

And the current Luup architecture, at least for us on the plugin side, is already asynchronous. Because of the short execution time limit before a deadlock and restart, we perform a series of tasks at startup, and that usually culminates in defining action implementations, and registering handlers for variable watches, job watches, HTTP requests, incoming serial/socket data, and delay/timers. Each of those is called on demand, that is, asynchronously, and when called, must (as I said) scramble to get its work done quickly and return. Failure to do so causes a deadlock and reload. The architecture described in this document is just the same. There’s no new mantra. The mantra is the same for this as it has always been for the old.

Where blocking has been an issue for the current Luup architecture is I/O. Functions like Lua socket’s select() that would normally let you wait on a socket for data don’t work in the Luup environment. Luup’s <incoming> handler, the async alternative, is cumbersome and too thin an API to do work efficiently on all but the smallest packets (in raw mode, it passes only one byte at a time–on a Vera Plus, just the system overhead of this approach makes receiving a 30K message take several seconds, without any processing of the received data). They appear to be trying to address this need with the network module, and while that’s a start, it’s also limiting in that it’s a proprietary API that no existing Lua network module will be able to use. IMO, it would be preferable for them to provide their own version of the standard Lua socket library with a functioning select() that is friendly to their multitasking environment (i.e. it yields to their scheduler when called, returns when ready). That’s a pattern they should provide.

Lua has coroutines, but my question as to whether coroutines is supported in this architecture went unanswered, so it stands to reason that the answer is no. IMO, it would be preferable to implement this entire plugin API in coroutines, thereby allowing plugins to manage “multitasking” within themselves in addition to cooperating with the multitasking efforts of the system. This, again, is another pattern. But the importance here is coroutines facilitate a program/script/module being able to interrupt its own path of execution–blocking until some need is met–in a non-blocking way from the system’s perspective, interoperating with the system’s demand for cooperative multitasking.

TL;DR: Don’t confuse asynchronous with non-blocking. All of these scripts are going to be perfectly capable of blocking. The fact that they are called asynchronously doesn’t change that. The only thing that makes it non-blocking is the requirement that the developer write the script well. There is nothing inherent to the API that makes it non-blocking. Indeed, there are many things missing that would contribute to a useful cooperative multitasking environment. The execution rules for plugins are therefore no different than current legacy Luup.

I am carrying my soapbox around lately… :wink:
The blocked process within the zwave queue has indeed been on my mind lately as I have been suspiciously thinking of single threaded process getting hung up in a timeouts. The fact that the vera won’t even change house mode while in it’s “timeout mode” got me thinking of a deeper design problem like this. The serial API, the web server and the logging are running their own process but the command queue and the LuaUPnP run within one massive one.

And I repeat my concern about plugin structure here in this context. Having events run scripts rather than functions (or closure/lambda) reduces the ability of the plugin author to create truly modular code structure that is readily reusable, and increases complexity of configuration management in deployment. One cannot, in this system, devise a single source file that contains all of the event handling functions required by the subsystem. Each event handler lives in a separate file. This will quickly add to code complexity, with appurtenant problems (difficult debugging, version control issues, deployment consistency, etc.). This is made worse, in fact, by @andryist 's earlier response to me that each script runs in its own Lua environment.

I understand your point. It’s right for small simple plugins. It’s easier to write everything in one file than to split all the logic into a few files. But we already have Z-Wave plugin with a lot of complicated logic. It’s hard to imagine all this logic in one file. Modular project structure gives better visibility of project and we have less conflicts between developers. If we put this code to one file… just loading this file to RAM can takes a lot of memory.

Patterns provided in the underlying API are nice, but not as necessary in my view as an architecture that supports good modular structure. In such an architecture, pattern implementations can be installed modules, shared by plugins/subsystems or, as needed, isolated to one (a plugin could provide its override version of an otherwise shareable pattern). I see no support for this here.

Sorry, I didn’t catch what do you mean here. Core, Storage, Timers, Network are part of firmware. Z-Wave is optional module and can be removed/installed. We can easily add other modules.

You can. Can we? Shared modules, I mean, between plugins/gateways/extensions?

@rigpapa

So are you saying here that if I want to monitor an item in another device, I somehow need to override hub.item.value.set on that device? Or, perhaps worse, I can provide an override of hub.item.value.set for the entire system? Is that even possible? If it is, you’ve thrown security and encapsulation out the window. If I can only do it for my plugin’s own devices, then it’s not useful for creating a plugin that monitors the states of another device in the system, and you haven’t described here or in the API how that would be possible. Is it possible?

hub.item.value.set can be overridden only for your items. Core module can have subscription API for any changes with universal devices/items. Core module has this functionality but Lua API is not provided.

This is one of my big areas of concern. Aside from inconvenient code structure, @andryist also said that each script runs its own Lua environment. Unless he misspoke, that means there is no common runtime storage between the event handlers and other elements of the plugin, you have to jump through hoops to emulate it (e.g. perhaps using the storage module). That stands to contribute considerable overhead.

Yes. Each script has own Lua environment. We already have Z-Wave plugin which implemented with this API and architecture. Object oriented approaches help to manage business logic easier. We restore our object model from storage in the begin of the script by recived data, do required changes and serialize object to storage back. Storage uses string as key for any data. If you need to save your specific information about device you can use next keys approach:
“<device_id>/token” : “your internal token”,
“<device_id>/myType” : “my_internal_type”,
“<device_id>/configuration/enabled”: false,

When you receive device id from event you can extract your data structure from storage and do what you want.

But I want to repeat a point here. Let’s not confuse asynchronous, and possibly concurrent, with non-blocking. They are related, but not equivalent. Non-blocking is a behavior that, in this structure, the plugin developer will need to implement–it is a rule to abide by. The essence simply is, don’t allow your script to run too long. That’s not non-blocking, that’s just abiding by a best practice. Technically, it is at best voluntary multitasking. There is nothing inherent in this API that makes it non-blocking. I guarantee that I can write a script that never returns. To be non-blocking, I have to make my script return (to keep the system from killing it at the 180 second time limit). It is still up to the script to do as little work as is necessary as quickly as possible when called upon and return/exit, and in that sense, there is absolutely no difference from this architecture and the current architecture of Luup plugins. The timeout is just longer.

Differences is lack of choice. You cannot write blocking code. Firmware doesn’t give you choice. Plugin has to return resources to controller. Whatever plugin does : firmware main functionality should work.

Almost each Luup module has blocking operations. Some of that can work only by blocking API. As I said before, Atom firmware can run only one script for one moment of time. If the Lua script takes a long time to execute then other part of Atom firmware will be blocked. Atom cannot run old Luup plugins. They are huge for him. But with using a lot of tricks we have the same API for all Atom-based and Linux-based controllers. We can add almost any protocols and modules in Lua API, but main approaches/architecture are difficult to change.

Coroutines are part of base Lua engine functionality. We don’t use it and didn’t specifically disable this functionality. We will verify this and let you know the results.

You can. Can we? Shared modules, I mean, between plugins/gateways/extensions?

Modules are binary extensions. It can be provided only by company. We thought about sharing general Lua code between plugins or provide functionality for extending exist plugins by other plugins. We have a few ideas how it can be done. We don’t support it now, but it’s in our roadmap.

Did I say that? :wink: