Design architecture Difficulty

[zerotolerance]

Hello all. I'm having some difficulty comming up with a viable and stable design architecture for the problem listed below.

------------------------------------------

We are downloading 5 files from a server through TCP/IP.

Each file consist of say 20 parts/chunks.

Each part is further divided into chunks of say 15.

So, 15 chunks make 1 part. 20 parts make 1 file.

Each chunk is say 1MB where each part is about 15MB in size.

Now, we are downloading these files through 30 available TCP/IP connections.

Each connection can download 1 chunk at a time.

So, we have 30 chunks being downloaded in parallel through 30 TCP/IP connections.

So, 2 parts of a file (i.e. 30 chunks) are downloading at once.

User may pause the file being downloaded (i.e. file 1) and start downloading a different file (i.e. file 4).

Now, these 30 TCP/IP connections will have to abandon downloading file 1, and be reused to download file 4.

When file 4 is finished, the user decides to start downloading file 1 again. The progress of downloading file 1 should start from where it left off.

The progress may only be deleted when user Cancels the download. 

-------------------------------------------

Basically, the above section describes the jist of it. Any sugestions are appriciated.

Kas

[altenbach]

I assume you are trying to design both sides of the connection (client and server) from scratch. Is this correct? Are you trying to design your own protocol using plain TCP or does the server use a well known standard? What imposed the limit on 30 concurrent connections?

[zerotolerance]

Hi altenbacbach. 

"I assume you are trying to design both sides of the connection (client and server) from scratch. Is this correct?" 

No. The design is only for the client side. No need for the server side. The server side uses the NNTP standard which are easily implemented through TCP/IP protocol through various command structures.

The 30 concurrent connection limit is imposed by server side. I can have 1 to 30 concurrent connections, but no more.

Kas 

[altenbach]

Well, I would start reding the NNTP RFC. I cannot find an obvious way to split a single file transmission into multiple concurrent connections, but I did not look very hard.

How much of the client side NNTP have you implemented so far in LabVIEW. Does everything work with a single connection?

[zerotolerance]

Hi altenbach. 

There is no need to split a file into multiple parts/chunks etc. The file that resides on the server is allready split in parts, and then further split in chunks. Each chunk has a unique identifier, they call this "message ID". So, if I have the message IDs, I can download the chunks and put them together in client side. 

There is no NNTP API's for Labview. But I can easily establish conections, download each chunk, then put them together to make a file. And yes, everything works eith 1 connection, but download speed suffers, since its limited from server side. So, they give 30 connections, and using them in manner described above, users High speed internet connection can be utilised in full.

So far, I have no problem with NNTP communication protocol. As I mentioned, I can connect, download, recombine chunks and parts to form a single healthy file. 

My problem it utilising this in a much bigger scale as mentioned above.

Kas

[zerotolerance]

So far, the attached is what I came up with, but I don't think the code is very scalable. I don't how I can solve the problem depicted in paragraph 3 on the first post. 

If the user pauses a file download and starts another, than resumes the download of the first one again, this example fails. I cannot think of a way on making it behave like described on the 1st post.

Let me know your thoughts. 

Thanks again.

Kas

[Henrik_Volkers]

I have no LV on this PC, however,

How about an array of cluster with chuck name / PostID#,  priority weight and status (waiting for dl, in progress) , whenever a connection is free / a chunk download has finished, the element with the highest weight is downloaded.

Could also be a queue or a AE ....

Commandline controllable NNTP reader/ downloader may already (or should I say may still?)  exist.

If bandwidth is limited and you have multiple downloaders, why not use a torrent like approach?

Still remember the time reading this forum with nn (or was it tin?) on a freebsd system connected via telnet to a DEC VT102  

 (here)

[zerotolerance]

Hi Henrik. 

I follow a similar-ish idea on the example attached above through queues. But that will still not solve the problem on keeping in memory the progress of a download incase the user pauses it and starts to download another file with different "chunks", and then back to original download again. The key here is that when the previous download starts, it should start from where it first left it, unless the user cancels the operation all together. 

As for the cmd NNTP reader, there is no need for it, since the protocol is allready implemented fully in native Labiew. Attached example shows this. Its not particulalrly difficult since it uses TCP/IP protocol. Moreover, I have more controll on how I can rearrange the downloaded files on the background.

As for the torrent-like approach, I'm not really sure how they work. 

Kas

[Henrik_Volkers]

so add a file location and all other data needed to restart the process in the array of cluster.

after each data-block (of whatever size) written to disk also store this array

Even after a sudden crash that would enable a restart

[zerotolerance]

Hi Henrik.

Correct. That is the general way that I was thinking as well. But having to put all these pieces together (i.e. progress report, user functionality, and various data manipulation needed after the download) is what makes up the wholde design architecure of the software. 

While individualy I am able to program and accomplish each task, I am lacking the experience I gues of a way of putting all of it together in a good way that performs all tasks seemlesly, hence the subject of the post. So I was hoping, knowing the requirements, someone may be able to help me plan the design architecture (i.e. placing all these pieces together).

Thanks 

Kas