openLuup: auto-update

akbooer · March 14, 2016, 5:07pm

…yes, it’s very frustrating. I was revisiting this just yesterday and am totally confounded by the HTTP redirect which GitHub does in order to access the latest release (or any, in fact.)

If anyone can point me in the right direction for successfully downloading a tagged release, then I am all ears. I am loathe to do this with a UNIX command wget or curl, since I want it to work on Windows too. The LuaSocket library seems incapable of handling this and I’ve had no luck stripping out the redirect information from the returned header manually.

CudaNet · March 14, 2016, 7:40pm

Isn’t cURL supported by Windows ?
[edit] Fixed my URL and everything downloaded and the archive tested ok, not sure if this is what you were looking for though…

Windows:

c:\Download\Test>curl -L -o master.zip http://github.com/akbooer/openLuup/archive/master.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   121    0   121    0     0    227      0 --:--:-- --:--:-- --:--:--   298
100  321k  100  321k    0     0   332k      0 --:--:-- --:--:-- --:--:--  332k

Debian:

cuda@Test:~/vera$ curl -L -o master.zip http://github.com/akbooer/openLuup/archive/master.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   121    0   121    0     0    402      0 --:--:-- --:--:-- --:--:-- 30250
100  321k  100  321k    0     0   447k      0 --:--:-- --:--:-- --:--:--  447k

…yes, it’s very frustrating. I was revisiting this just yesterday and am totally confounded by the HTTP redirect which GitHub does in order to access the latest release (or any, in fact.)

If anyone can point me in the right direction for successfully downloading a tagged release, then I am all ears. I am loathe to do this with a UNIX command wget or curl, since I want it to work on Windows too. The LuaSocket library seems incapable of handling this and I’ve had no luck stripping out the redirect information from the returned header manually.[/quote]

reneboer · March 14, 2016, 9:08pm

Hi akbooer,

Wild thought, what about having ALTUI check for an openLuup update just as it does for ALTUI?

Cheers Rene

akbooer · March 14, 2016, 10:56pm

I have no trouble checking for the release, it’s just the downloading. The AltUI upgrade works on both Vera and openLuup without change because it’s actually openLuup which does the download from the MiOS Trac repository when it receives an install_plugin action request. It’s exactly that analogous download (this time, from GitHub) which I’m struggling with for openLuup.

Well, there you go - just shows how little I know about Windows… I still have to unpack the archive, of course. Anyway, great pointer! I’ll let you know how I get on.

martynwendon · March 14, 2016, 11:03pm

[quote=“akbooer, post:1, topic:191608”] I was revisiting this just yesterday and am totally confounded by the HTTP redirect which GitHub does in order to access the latest release (or any, in fact.)

If anyone can point me in the right direction for successfully downloading a tagged release, then I am all ears. I am loathe to do this with a UNIX command wget or curl, since I want it to work on Windows too. The LuaSocket library seems incapable of handling this and I’ve had no luck stripping out the redirect information from the returned header manually.[/quote]

Definitely should be internal to openLuup IMO as opposed to calling something like wget or curl.

Do you mean the 302?

If so, can’t you just use the “result” of the 302?

I don’t know why they do a 302, it’s always redirected to the same url for as long as I can remember.

[code]
root@ahsoxu41:~# wget https://github.com/akbooer/openLuup/archive/v0.7.0.tar.gz
–2016-03-14 22:52:44-- https://github.com/akbooer/openLuup/archive/v0.7.0.tar.gz
Resolving github.com (github.com)… 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://codeload.github.com/akbooer/openLuup/tar.gz/v0.7.0 [following]
–2016-03-14 22:52:45-- https://codeload.github.com/akbooer/openLuup/tar.gz/v0.7.0
Resolving codeload.github.com (codeload.github.com)… 192.30.252.163
Connecting to codeload.github.com (codeload.github.com)|192.30.252.163|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 306836 (300K) [application/x-gzip]
Saving to: ‘v0.7.0.tar.gz’

v0.7.0.tar.gz 100%[===========================================================================================================>] 299.64K 450KB/s in 0.7s

2016-03-14 22:52:46 (450 KB/s) - ‘v0.7.0.tar.gz’ saved [306836/306836]

root@ahsoxu41:~# wget https://codeload.github.com/akbooer/openLuup/tar.gz/v0.7.0
–2016-03-14 22:53:51-- https://codeload.github.com/akbooer/openLuup/tar.gz/v0.7.0
Resolving codeload.github.com (codeload.github.com)… 192.30.252.160
Connecting to codeload.github.com (codeload.github.com)|192.30.252.160|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 306836 (300K) [application/x-gzip]
Saving to: ‘v0.7.0’

v0.7.0 100%[===========================================================================================================>] 299.64K 479KB/s in 0.6s

2016-03-14 22:53:52 (479 KB/s) - ‘v0.7.0’ saved [306836/306836]

root@ahsoxu41:~# wget https://github.com/akbooer/openLuup/archive/master.zip
–2016-03-14 22:55:01-- https://github.com/akbooer/openLuup/archive/master.zip
Resolving github.com (github.com)… 192.30.252.131
Connecting to github.com (github.com)|192.30.252.131|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://codeload.github.com/akbooer/openLuup/zip/master [following]
–2016-03-14 22:55:02-- https://codeload.github.com/akbooer/openLuup/zip/master
Resolving codeload.github.com (codeload.github.com)… 192.30.252.162
Connecting to codeload.github.com (codeload.github.com)|192.30.252.162|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/zip]
Saving to: ‘master.zip’

master.zip [ <=> ] 321.82K 570KB/s in 0.6s

2016-03-14 22:55:03 (570 KB/s) - ‘master.zip’ saved [329539]

root@ahsoxu41:~# wget https://codeload.github.com/akbooer/openLuup/zip/master
–2016-03-14 22:55:20-- https://codeload.github.com/akbooer/openLuup/zip/master
Resolving codeload.github.com (codeload.github.com)… 192.30.252.162
Connecting to codeload.github.com (codeload.github.com)|192.30.252.162|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 329539 (322K) [application/zip]
Saving to: ‘master’

master 100%[===========================================================================================================>] 321.82K 418KB/s in 0.8s

2016-03-14 22:55:21 (418 KB/s) - ‘master’ saved [329539/329539][/code]

akbooer · March 14, 2016, 11:13pm

Yes, exactly.

I don't know why they do a 302, it's always redirected to the same url for as long as I can remember.

I didn’t know that it never changed.

Sadly, it still doesn’t work for me when using the LuaSocket library. I know it is an HTTPS request, but I was rather hoping not to have to set up a certificate and a key in order to do that, but all I get at the moment is “timeout.”

martynwendon · March 15, 2016, 1:00pm

Ah that may be the problem then, the codeload url only responds to https

Not sure you’d need to use certs etc just for a client request?

root@ahscctvserver:~# cat test.lua 
local https = require("ssl.https")

https.request
{
  method = "GET",
  url = "https://codeload.github.com/akbooer/openLuup/zip/master",
  sink = ltn12.sink.file(io.stdout)
}

 lua test.lua > test.zip 

unzip test.zip

Seems to unzip a valid zip file

akbooer · March 15, 2016, 3:39pm

Thanks @martynwendon!

I had tried this before, but without the manually redirected URL. So we are one step closer. Hurrah!

Need to work this into the plugins module now.

martynwendon · March 15, 2016, 4:50pm

Cool, an auto-update option for openLuup would be sweet … might actually encourage me to update it more often

akbooer · March 15, 2016, 4:59pm

I still have to use os.execute() to unzip it

I did look at some pure Lua decompression code, but that seems an even more fraught option.

martynwendon · March 15, 2016, 6:02pm

[quote=“akbooer, post:10, topic:191608”]I still have to use os.execute() to unzip it

I did look at some pure Lua decompression code, but that seems an even more fraught option.[/quote]

ah yeah

How about using the github API instead? I think it’s pretty simple - Repository contents - GitHub Docs

Should be able to get the contents (file list) at a specific commit and then download the files directly one at a time?

Alternatively you could even add a simple “file list” file to the repo itself that lists all the files, download that first and then download the files it contains directly - there’s no redirect on directly downloading the files as far as I can see:

[code][root@ahsrpi1 ~]# wget openLuup/chdev.lua at master · akbooer/openLuup · GitHub
–2016-03-15 17:52:53-- openLuup/chdev.lua at master · akbooer/openLuup · GitHub
Resolving github.com (github.com)… 192.30.252.128
Connecting to github.com (github.com)|192.30.252.128|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Saving to: ‘chdev.lua’

chdev.lua [ <=> ] 116.22K 153KB/s in 0.8s

2016-03-15 17:52:55 (153 KB/s) - ‘chdev.lua’ saved [119012]
[/code]

The API way might be more appealing as you would also have a way to switch between commits, I think AltUI does similar?

akbooer · March 15, 2016, 7:11pm

[quote=“martynwendon, post:11, topic:191608”]How about using the github API instead? I think it’s pretty simple - Repository contents - GitHub Docs

Should be able to get the contents (file list) at a specific commit and then download the files directly one at a time?

Alternatively you could even add a simple “file list” file to the repo itself that lists all the files, download that first and then download the files it contains directly - there’s no redirect on directly downloading the files as far as I can see:[/quote]

Yes, I’ve studied that API quite extensively, and, in fact, the single-file approach is exactly what I use from the MiOS Trac repository to update AltUI through openLuup, reading the directory first in order to determine which files to download. However, it is very slow doing it one file at a time and I was rather hoping that a zipped file transfer would be much faster (and it is.) Having said that, openLuup is a much smaller codebase that AltUI, and GitHub is probably faster than Trac, so it shouldn’t be too bad. Anyway, it is not something to do too often.

The reason I already tag releases in GitHub is so that I can easily reference them for download. I just need to settle on the right approach and do it: the imperative to avoid os.execute() is strong, so I expect it will be single-file in the end. I do appreciate your support!

PS: I split all this off into a separate thread since we had really diverged from CudaNet’s original install topic.

cybrmage · March 15, 2016, 7:49pm

I have a little script that I use to download the latest/master/development/tagged releases…

No zip/no GitHub api/ no external dependancies (although it does use os.execute() to mkdir the destination directories)… and downloads the GitHub repository to “/tmp/openLuup_UPGRADE/”

It’s attached, in case it can be of use…

akbooer · March 15, 2016, 7:54pm

Great, thanks. I’ll take a look.

...although it does use os.execute() to mkdir the destination directories)...

Don't do that! Use lfs.mkdir() instead.

cybrmage · March 15, 2016, 7:59pm

[quote=“akbooer, post:14, topic:191608”]

…although it does use os.execute() to mkdir the destination directories)…

Don’t do that! Use lfs.mkdir() instead.[/quote]

I only use it on my Raspery Pi’s, so… it hasn’t been an issue for me… but, yes, lfs is a more universal option.