Lua - Request works via curl, but not via Lua (Lua socket)?

I’m trying to interact with the Sipgate api (https://developer.sipgate.io/ ) using Lua, and an example they provide for authentication uses Curl, which works. (Example is not my b64 user/pass)

curl \
  --request GET \
  --header "Accept: application/json" \
  --header "Authorization: Basic Sm9objp0b3BzZWNyZXQ=" \
  https://api.sipgate.com/v2/account

But when I try it via Lua, I get the following error.
Does anyone have an idea of what i’m missing

Message/response…

enc64 Sm9objp0b3BzZWNyZXQ=
ok 1
statusCode 301
statusText HTTP/1.1 301 Moved Permanently
headers: table: 0x1e46ed0
data

Here’s the Lua code I used…*


local https = require "ssl.https"
local mime = require "mime"
local http = require "socket.http"
local data = ""
local username = "user"
local password = "pass"

local function collect(chunk)
  if chunk ~= nil then
    data = data .. chunk
    end
  return true
end

local ok, statusCode, headers, statusText = http.request {
  method = "GET",
  url = "https://api.sipgate.com/v2/account",
  headers = {
    ["Accept"] = "application/json",
    ["Authorization"] = "Basic " .. (mime.b64(username ..":" .. password)),
	},
  sink = collect
}

--debug - 
print("enc64", mime.b64(username ..":" .. password))
print("ok\t",         ok);
print("statusCode", statusCode)
print("statusText", statusText)
print("headers:", headers)

print("data", data)

The URL you are accessing responds with a redirect request which the LuaSocket library doesn’t automatically handle. You would have to manually code another (or possibly several) request to the new URL specified in the response header.

Ok, great thanks that helps, so looking at the response headers

headers:     table: 0x1da1cd8       

Breaking that out…

for k,v in pairs(headers) do
	print(k,v)
end 

Returns the following…

location     https://api.sipgate.com:80/v2/account     
connection     close  
content-length     0 

Is it just a straightforward call to that url or do I need to do or provide something else too ?

No, that should be it, although it may return another redirection. So if you write a loop, make sure that it has a limited number of attempts, to avoid circular redirections (happens, sometimes.)

Will do, although my first hurdle is extracting that redirect url from the ‘headers’ table .

This is an area that drives me mad, as I constantly struggle to get the right structure - e.g. none of the following work. I know the key is ‘location’ but I can’t extract the value ?

for k,v in pairs(headers) do
	-- print(k,v)
	-- print(v[1])
	-- print(['location'].v)
	-- print(v.location)
    -- print(v'location')
	-- print(v['location'])
end

Please could someone let me know the right format?

I would think headers.location would hold the address, according to your previous post

Thanks @ElCid, thar looks to be it,

for k,v in pairs(headers) do
	local redirect = headers.location
    print(redirect)
end

However it seems to return 3 instances of the same url, is there a version that returns just one ?

https://api.sipgate.com:80/v2/account     
https://api.sipgate.com:80/v2/account     
https://api.sipgate.com:80/v2/account

Yes, don’t use the loop.

Thanks again @ElCid, ok, so next I need to create something that handles the redirect.

I have the following code so far, which is looking to see if a redirect is returned within the headers. As one does in this case, how best should I handle that, is it best to create another function, and have the first function call that one ?

local https = require "ssl.https"
local mime = require "mime"
local http = require "socket.http"
local ltn12 = require "ltn12"
local username = "user"
local password = "pass"

local function httprequester(url)
	local ok, statusCode, headers, statusText = http.request {
		method = "GET",
		url = url ,
		headers = {
			["Accept"] = "application/json",
			["Authorization"] = "Basic " .. (mime.b64(username ..":" .. username)),
		},
  		sink = ltn12.sink.table(respbody),
		}

	if headers.location ~= nil then
		local firstredirect = tostring(headers.location)
		print ("first redirect :", firstredirect) 
	
		local result,b,c,h = http.request{ 
		url = headers.location, 
		headers = header,method="GET" }
	
		local secondredirect = tostring(c.location)
		print ("second redirect :", secondredirect) 
		--[[for key,val in pairs(c) do 
			print(key,val) ]]--
	end
end

--[[
--debug 
print("enc64", mime.b64(username ..":" .. password))
print("ok\t", ok);
print("statusCode", statusCode)
print("statusText", statusText)
print("headers:", headers)
print("sink", sink)
]]

return redirect
end

print(httprequester("https://api.sipgate.com/v2/account"))
```

I’ve updated the code in the earlier post above, so it handles the redirect returned, but the subsequent redirect is the same url ?

first redirect :     https://api.sipgate.com:80/v2/account     
second redirect :     https://api.sipgate.com:80/v2/account     

I’m conscious that this is the beginning of a loop ?
What normally occurs to change the redirect url ?

Looks like you need to use a secure request. Not just the https url, but with ssl?

Just a guess.

Agreed, and I had originally tried using https.request instead of http.request,

local ok, statusCode, headers, statusText = http.request {

but that just resulted in errors, and nothing being returned e.g.

Runtime error: Line 19: attempt to index local 'headers' (a nil value)

Any ideas ?

Hi,

It could be that the SSL options are the source of the problem. On Vera the library is old and has some outdated options by default. Does this little wrapper help?

-- Need wrapper for Vera to set protocol correctly. Sadly tls1.2 is not supported on the Lite if remote site now requireds that.
local function HttpsWGet(strURL, timeout)
	local result = {}
	local to = timeout or 60
	http.TIMEOUT = to
	local bdy,cde,hdrs,stts = https.request{
			url = strURL, 
			method = "GET",
			protocol = "any",
			options =  {"all", "no_sslv2", "no_sslv3"},
            verify = "none",
			sink=ltn12.sink.table(result)
		}
	-- Mimick luup.inet.get return values	
	if bdy == 1 then bdy = 0 else bdy = 1 end
	return bdy,table.concat(result),cde
end

Cheers Rene

Thanks @reneboer

A quick update to your wrapper, returned a 401 error (via LuaTest) - see below.

Results
No errors
Runtime: 210.5 ms
Code returned: nil

Print output
0          401     

Code
   1   local https = require "ssl.https" 
   2   local http = require "socket.http" 
   3    
   4   local function HttpsWGet(strURL, timeout) 
   5        local result = {} 
   6        local to = timeout or 60 
   7        http.TIMEOUT = to 
   8        local bdy,cde,hdrs,stts = https.request{ 
   9                  url = strURL,  
  10                  method = "GET", 
  11                  protocol = "any", 
  12                  options =  {"all", "no_sslv2", "no_sslv3"}, 
  13               verify = "none", 
  14                  sink=ltn12.sink.table(result) 
  15             } 
  16        -- Mimick luup.inet.get return values      
  17        if bdy == 1 then bdy = 0 else bdy = 1 end 
  18        return bdy,table.concat(result),cde 
  19   end 
  20    
  21   print(HttpsWGet("https://api.sipgate.com/v2/account"))

401 = unauthorised

Sadly it’s the same 401 error if I add mime, my credentials and the authentication header too.

			headers = {
			["Accept"] = "application/json",
			["Authorization"] = "Basic " .. (mime.b64(username ..":" .. username)),
			},

Just a thought , doing it via curl on another machine I was using worked…

curl \
  --request GET \
  --header "Accept: application/json" \
  --header "Authorization: Basic Sm9objp0b3BzZWNyZXQ=" \
  https://api.sipgate.com/v2/account

Checking the version of Curl on Vera

~# curl --version
curl 7.38.0 (mipsel-openwrt-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.2l zlib/1.2.8
Protocols: file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

Trying it via Curl on Vera I get the following…

curl -X GET "https://api.sipgate.com/v2/account" -H "accept: application/json" -H "Authorization: Basic Sm9objp0b3BzZWNyZXQ=="curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Adding the —insecure option to the curl request and it worked…

curl -X GET "https://api.sipgate.com/v2/account" -- insecure -H "accept: application/json" -H "Authorization: Basic Sm9objp0b3BzZWNyZXQ=="

Returned as expected something like this…

{"company":"","mainProductType":"BASIC","logoUrl":"","verified":true,"historyLifeTime":"UNLIMITED"}

I’m wondering if something like this (zerobrane) might help me out ?

http://notebook.kulchenko.com/programming/https-ssl-calls-with-lua-and-luasec

require("socket")
local ssl = require("ssl")

-- TLS/SSL client parameters (omitted)
local params = {
  mode = "client",
  protocol = "tlsv1",
  verify = "none",
  options = "all",
}

local conn = socket.tcp()
conn:connect("www.google.com", 443)

-- TLS/SSL initialization
conn = ssl.wrap(conn, params)
conn:dohandshake()

conn:send("GET / HTTP/1.1\n\n")
local line, err = conn:receive()
print(err or line)
conn:close()

Thoughts ?

Where would I need to put the headers etc ?

Hi @parkerc,

Have you tried with the exact same Authorization header string as for your CURL command? Maybe calculating the basic string in Lua is not correct?

Hi @reneboer,

I have yes, and sadly it was the same result, so I’m at a loss now what else to do.

Other than assume it’s just not possible :frowning: