xmlhttprequest from LabVIEW as client (HTTP and HTTPS)

[loera]

Greetings,

Our group has embedded devices running web services that browsers can use to pull data from the embedded device. We would like to start using LabVIEW to start pulling the data so we can use other LabVIEW VIs to process the data. I'm not to knowledgeable about web services as of yet but have found this snippet of javascript code that seems to be responsible to getting the data from the device to the browser:

AjaxPost.prototype.send = function()
{
  var xhttp = new XMLHttpRequest();
  var callback = this.callback;
  
  var sPostHeader = "PostDest="+ this.sPostDest+
      "&PostTout="+this.sPostTout+
      "&PostName="+this.sPostName+
      "&PostOpt="+this.sPostOpt+
      "&PostAction="+this.sPostAction;
  
  var sPost = sPostHeader;
  if (this.sNameValuePairs)
  {
    sPost += "&" + this.sNameValuePairs;
  }
  logDebug("POST: " + sPost);
  
  if (g_sServerIpAddress)
  {
    xhttp.open("POST", "https://" + g_sServerIpAddress + "/php/post.php", true);
  }
  else
  {
    xhttp.open("POST", "/php/post.php", true);
  }
  xhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
  xhttp.timeout = (this.sPostTout +2) * 1000;
  xhttp.ontimeout = function ()
  {
    logDebug("POST TOUT: " + this.sPostName);
    callback("post.php timeout");
  };
  xhttp.onreadystatechange = function()
  {
    if ((xhttp.readyState == 4) && (xhttp.status == 200))
    {
      var sResponse = xhttp.responseText;
      logDebug("RESPONSE: " + sResponse);
      if (callback)
      {
        logDebug("CALLBACK: " + sResponse);
        callback(sResponse);
      }
    }
  };

  xhttp.send(sPost);
};

I have tried using just the LV HTTP Post VI but that does not seem to return any data. It does return this:

 HTTP/1.1 301 Moved Permanently
X-Powered-By: PHP/5.6.18
Location: site
Content-type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 07 Jun 2000 16:36:24 GMT
Server: lighttpd/1.4.39

What VIs can I use to replicate this "xmlhttprequest"?

Thanks,

-Jose

[MilanR]

Hi Jose!

The LabVIEW HTTP Client Post VI should be able to emulate the behavior of an XMLHttpRequest POST pretty well. However, in order to emulate it fully you have to make sure all parts of the request are what the server is expecting. The parts you most likely need to focus on are: the URL, Body, Headers, and Method.

To make sure you are getting each part of the request right you should use the browser's developer tools to inspect the network request. A few months back I created a post that walks through inspecting requests that are made by the browser and emulating them in LabVIEW in an article titled Scraping Web Pages in LabVIEW. The article also includes a video that walks you through an example web page.

By using the browser dev tools to inspect the network requests you may be able to find the differences that prevents the device from communicating.

[loera]

Milan,

Super helpful to use the browsers's dev tools to see the network request! Thanks for passing that along. 

This turns out to be enough for get it working (after viewing the network requests):

However, we had to disable the "https" feature on our embedded device in order to get this to work.

I would like to be able to connect using the "https" connection method. It looks like I will not be able to use the LV HTTP library to do this. The other suggestions on the forums are to use ActiveX or .NET. I am currently trying to use the .NET invoke and property VIs. No success so far. 

Part of the problem is that it seems like the cert is broken or invalid or whatever it's called from the embedded device. The first time I connect to the device from a browser, say Chrome,  it give a warning that my connection is not private:

I just click on the "Advanced" button and then "Proceed to...." and I get to the web page. 

It appears that I need to be able to do this programmaticly. This is the error I get:

I started off with this Block diagram:

That I found on this thread: https://forums.ni.com/t5/LabVIEW/NET-Http-Https-Get-document/td-p/405426

According to this forum: http://stackoverflow.com/questions/2675133/c-sharp-ignore-certificate-errors    I should just be able to ignore and move on. I'm just not sure how to do this in LV. I am attaching my most current attempt.

Thanks for the help,

-Jose

[loera · Accepted solution]

Greetings,

UPDATE

Recap what I was trying to do:

• Use LV to pull web data from HTTP and HTTPS websites (POST).

Issues:

• There didn't seem to be a way with the LV http VIs to connect to HTTPS websites
• Trying to use just the .net nodes from the Connectivity -> .NET LV pane did not seem to have needed methods available or there was not straight forward. 

Here was my final solution:

1. Create a C# .net DLL with functions I needed (created it in Visual Studio Community - https://www.visualstudio.com/downloads/). See code below. And this youtube tutorial: https://www.youtube.com/watch?v=hE0TVHqZjBI
2. Use the Connectivity -> .NET LV VIs to use the functions in the DLL

While there are still some unanswered questions. This solution is getting me going for now. Thanks to Milan who responded to this thread and also Eric Wang from NI Support for the suggestion of creating a DLL. Of course, it would have been nice if I could have just done it all in LV in the first place 😉

Best,

-Jose

[loera]

Just a BTW.

This website: http://posttestserver.com/post.php was helpful in debugging webPost messages. Please note that in the LV code above I forgot to add the /post.php to the web address.

[MilanR]

Hi Jose!

Yep it is definitely preferable to have SSL enabled for communication with a device. I am not an expert in SSL security so take the following with a grain of salt.

When you use the normal approach of Open Handle -> Get / Put / etc -> Close Handle for making an HTTP request to a secure server (HTTPS), the HTTP Client connects to the server and the server presents a certificate signed by a Certificate Authority (CA). The HTTP client's job is to validate that the signed certificate was signed by a trustworthy source. With SSL there is a Root CA that can sign certificates, and other CAs can take those certificates and use them to sign other certificates, etc making a chain from the Root CA.

Roughly speaking, operating systems / browsers / some apps come with a list of Certificates all on the chain signed by the Root CA. Using these, you can take an existing certificate and validate that someone in the chain signed it. 

My guess at what the device is doing is called a self-signed certificate. This is where, instead of using that chain of trust leading back to the Root CA, you / the device / the device manufacturer signed the certificate / created their own CA certificate outside of the chain and uses it to sign the certificate for the device. This 3rd party blog may explain it better than me.

If it is the case that the devices have self-signed certificates / a self-signed CA there are a couple things you can do with LabVIEW:

1) Add the CA cert that the devices certificates were issued from using the ConfigSSL VI. 

There should be a way to get the self-signed cert either from the device / manufacturer or I think you may be able to get it from Chrome / Firefox when visiting the server in a browser. You can then pass the cert / CA cert to the CA Certificate file input of the ConfigSSL VI. The ConfigSSL VI would be used prior to making the request, ie Open Handle -> Config SSL -> Get / Put / etc -> Close Handle

2) You could also replicate what was done using the .NET HTTP libraries and skip the validation step. You should be able to do this by setting verify server to false on the ConfigSSL VI.

As documented in the help, setting verify server to false "does not verify the server's identity and does not provide optimal security". Finding a way to pass in a valid cert as mentioned in the first option is definitely ideal.

Another longer-term thing to do is mention to the device maker that you are interested in them using Let's Encrypt in their product. Self-signed certs / CAs were the only way to get free certs for a long time which is great when you don't want to spend $100+ to make a cert per device, but it is not ideal to get used to ignoring verification in browsers or having to manually handle certs. Let's Encrypt is a free and open source way to create certificates on the Root CA chain (sponsored by Mozilla, the Firefox makers, to enable SSL security for everyone).

[loera]

The web interface now requires Username and Password... I thought I would add the code necessary to do that:

        public static string post(string url, string postData)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
              
            //Add Username and Password (hardcoded, for now) 
            var credentialCache = new CredentialCache();
            credentialCache.Add(
              new Uri(url), // request url's host
              "Digest",  // authentication type 
              new NetworkCredential("USERNAME", "PASSWORD") // credentials 
            );
            request.Credentials = credentialCache;

            //Request type and other data
            request.Method = "POST";
            request.ContentType = "application/x-www-form-urlencoded";
            byte[] data = Encoding.UTF8.GetBytes(postData);
            request.ContentLength = data.Length;

            Stream requestStream = request.GetRequestStream();
            requestStream.Write(data, 0, data.Length);
            requestStream.Close();

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            Stream stream = response.GetResponseStream();
            StreamReader reader = new StreamReader(stream);

            string result = reader.ReadToEnd();

            stream.Dispose();
            reader.Dispose();

            return result;
        }

-Jose

[BenDur]

Loera, 

Thank you for sharing this code ! 

This is exactly what I was trying to do, it works flawlessly, it is simple yet powerful ! 

Thank you ! 

Kudos to you !