Retrieve full hmtl file from url

[VGans]

Hi everybody!

I'm working on a small project lately where I have to retrieve the weather announcement of the day from the web.

I'm thus using the HTML VIs from the website weather.com.

The idea is that I would like the know the tendency of the day (sunny, cloudy, ...), and I thought I could do it from two informations: either from the text, or from the little icons they always show on the website.

I've thus been using the web inspector from Firefox to find the information I need, and in Firefox, I find it very easily: either it's written in full text, or the icon is loaded in the html, making it very easy to know what the weather will be from the HTML file.

Therefore, I thought I could just using the parsing VI of Labview to find the information I need from the loaded HTML file from the url.

But... I don't know why, the html that Labview gives me is not completely identical to the one Firefox gives me: some lines are missing, and of course, the ones I need, because it wouldn't be fun otherwise!

Anybody has an idea why this would happen?

I'm working with Labview 2014 for Mac.

Thanks in advance!

[nathand]

Can you post both the HTML as received by Firefox, and the HTML as received by your LabVIEW code, so we can see the difference?

Some web servers will return different pages depending on what browser is used to access it. It's also possible that the web page includes other pages inside of a frame; Firefox will download those as well, but the LabVIEW HTTP GET will not. It could also be an issue of data being presented through a script, which Firefox will run but the LabVIEW function will not.

[VGans]

Thanks for your answer 🙂

Here are the screenshots:

[nathand]

You could try adding a User-Agent header that matches the one used by Firefox, and see if you get different results. If that doesn't do it, you'll have to hope for an answer from someone who knows more about HTTP than I do.

[MilanR]

A couple of things:

1. When you look at the Firefox Web Developer Tools Inspector tab you are seeing the in memory Document Object Model (DOM) representation of the HTML. Web browsers tweak, poke, and clean the html to close tags, etc. while building the DOM representation so it may be different than what was initially sent to the browser. To see the original source HTML that was sent to the browser you can right-click the page and choose view source.
   
   
2. It's possible a header, such as User Agent, etc like @nathand mentioned is causing the rendering difference. You can see all the headers sent and received by using the networking tab in the Firefox Web Developer Tools and selecting the html page that was loaded (usually the first item)
   
   
3. Alternatively, instead of scraping the page you might want to use one of the APIs that Weather Channel exposes. The website Weather Underground has a developer API to retieve weather information. Weather Underground and the Weather Channel are both owned by the Weather Company. Using an API they expose makes it easier to avoid having your application break when they decide to change to look of the html pages.

[VGans · Accepted solution]

Hi everybody!!

First of all, thanks a lot for your answers!! 🙂

I finally found the problem, thanks to a HTML king I work with 😉

The problem was that when checking with the inspector of Firefox, Javascript is taken into account, while it's not in Labview. It is then possible to get the correct HTML with Firefox by disabling javascript in about:config. The inspector then shows the same HTML as the one that Labview gets, which allows you to easily find the part that interests you.

Hope it's clear!

Thanks a lot again for your answers everybody! 🙂

Val

[MilanR]

I also gave the weather underground api a shot and it wasn't too bad:

Has a bunch more properties that might be useful

[VGans]

Nice! That might be useful for a more complete application of my project! Thanks again!

[21998]

can you send me that file as i am not able to get api link from website.we have project in which we have to take realtime data from website and also run ni myrio so could help me how to do it?