Count of discussion on LabVIEW community forum

[dvrubins]

Hello everyone!

I would like to know if there is any tool or API to see how many discussions have been opened year by year on forums.ni.com with the label "LabVIEW". I am interested in historical data up until 1998, if possible. I am looking for something like SEDE for StackOverflow (StackExchange - Query Stack Overflow - Stack Exchange Data Explorer).

Do you have any ideas?

[GerdW]

Hi dvrubins,

---

@dvrubins wrote:

I would like to know if there is any tool or API to see how many discussions have been opened year by year on forums.ni.com with the label "LabVIEW". I am interested in historical data up until 1998, if possible.

Do you have any ideas?

---

• Use HTTP functions to load the NI forum website in your VI.
• Use the HTTP address "https://forums.ni.com/t5/LabVIEW/bd-p/170/page/2" and replace the last number with a counter starting with "1" to load ALL possible pages.
• Parse the HTML content to get all threads and their labels.
• Do your counting…

See how far you get!

[tst]

Some thoughts:

1. You can use Gerd's method and you don't have to get all of the pages. You could try jumping around to find the pages which begin the year and you know how many threads are in each page. I think it's safe to assume that the vast majority of threads are closed soon after they begin, so I think you can ignore the fact that the threads are sorted by their last reply and not the first one.
2. You could try using the wayback machine on the main page to see how many posts there were in the LabVIEW board and throw a number converting posts to threads.
3. You could try looking at Khoros' API to see if there is any convenient way to get that data.
4. The posting rate has gone down significantly. See this for instance. Some of it could possibly be explained by people posting elsewhere, but the places I look at don't have a lot of activity either.

[thols]

"historical data up until 1998"

Back then there were no forums I know of. When I started using LabVIEW in 2000, there was a mailing list. Maybe ask Altenbach if he has an archive :).

[dvrubins]

I have a few reservations about this method. One is that it ends up being a scraper. I am afraid that this may violate the terms of service of the NI forum. Although I haven't found anything on the Service Terms - NI page that says this. Other forums consider this type of scraping to be an invasion of privacy or piracy. My intentions are only to get the final amount of discussion at the end of the year.

With your method I could only scrape HTML tags with the 'local-date' class, but as I said, I'm not sure if it doesn't violate anything ^^'.

[dvrubins]

I tried to go page by page, but it didn't work. The suggestion to jump through the pages is not amusing because the discussions, although packed by year, are sorted by """relevenace""".

I will look into the Khoros API to automate the process. Do you have a link to the API for Khoros on NI forum?

Thanks for the link to JB's post. I'll try to ask him about his methodology if I can't solve it via Khoros.

[GerdW]

Hi dvrubins,

---

@dvrubins wrote:

One is that it ends up being a scraper.

---

Yes.

IMHO it doesn't matter if I load the website by clicking a button/link in the browser or by loading the same HTML code using a HTTPRead function.

As long as you don't intend to create a DDoS tool (aka include some small waits) you should be free to load all the forum pages and "read" (aka parse) them…

[tst]

---

@dvrubins wrote:

I tried to go page by page, but it didn't work. The suggestion to jump through the pages is not amusing because the discussions, although packed by year, are sorted by """relevenace""".

---

I don't understand your sentence. The threads are sorted in reverse chronological order of the latest reply. While this means they're not sorted by start date, most threads are usually only active for a short time, so you can treat the thread starting time and the thread ending time as the same time and ignore the outliers. The suggestion to jump between pages was serious, because each year would span multiple pages (and again, you know how many threads are on each page). You can also assume the threads on the LabVIEW board are the LabVIEW discussions. They're not the only ones, and they're not all about LV, but you're not going to get 100% accurate numbers anyway.

You can also change the threads per page to be up to 200, which will make the number of pages smaller, although to maintain the setting, you would to do that in a session where you set that.

---

@dvrubins wrote:

Do you have a link to the API for Khoros on NI forum?

---

I don't, but look at their website. I haven't used the API since Khoros was called Lithium (quite a few years by now), so I don't know what their current API looks like. I'm assuming they still have one.