Fasted way of stripping 18 bit samples from UDP packet

[drak]

Hi All,

I'm hoping someone will be able to point me in the right direction for the best way to dissect large UDP packets that contain 18 bit data samples. I'm new to labview with my experience being more on the electronics/embedded C side.

The UDP packets that I'm reading are 1200 bytes long. I'm able to strip out the Ethernet header, IP header and UDP header using the "Unflatten from String" function without issue, which leaves me with the IP data which is 1158 bytes long.

There is then 78 bytes of hardware status information and the remaining 1080 bytes is made up of three blocks of 360 bytes, each of those blocks consists of 160, 18-bit samples from an 8-channel ADC (160 samples per block, 480 samples per packet). I've gotten as far as breaking the packet into those blocks, but the best way to break it into individual samples is something I could use help with.

The only way I can think of separating the data into 18-bit groups in Labview is to convert it to a boolean array and then parse it, but this seems very clunky.

I'm sure I can brute force something together that works but the VI will need to parse 1000 packets per second, so speed with be an issue. If anyone has any high level recommendations that would be great (don't expect anyone to build this for me - just pointing me in the right direction or making suggestions would be fantastic).

Cheers,

[GerdW]

Hi drak,

you can convert the 360 byte block into an 1D array of booleans. Then you could run a loop getting 18 bit entities and convert to I32 integer. Or reshape the 1D array of booleans into a 18×160 2D array of booleans, then you can convert to I32 in a loop without using ArraySubset to get 18 bits…

(You can also do all this in just one loop without any booleans by "shifting in" new bytes into a larger integer datatype and "shifting out" your 18 bit ADC values: you can use any bit-oriented function (shift, rotate, AND, OR, …) also in integers!)

[altenbach]

Any data is quantized to bytes, so how exactly are the 18bits represented? It seems they are not aligned with byte boundaries, reminding me of the first bit-twiddling challenge long ago (link missing), but it is certainly something that can be done easily.

Can you attach a simple VI containing a string diagram constant containing a typical UDP packet?

Explain exactly how the data is represented.

What is generating these packets and why do they contain packet headers? is this a pcap capture??

[altenbach]

---

@GerdW wrote:

Hi drak,

you can convert the 360 byte block into an 1D array of booleans. Then you could run a loop getting 18 bit entities and convert to I32 integer. Or reshape the 1D array of booleans into a 18×160 2D array of booleans, then you can convert to I32 in a loop without using ArraySubset to get 18 bits…

(You can also do all this in just one loop without any booleans by "shifting in" new bytes into a larger integer datatype and "shifting out" your 18 bit ADC values: you can use any bit-oriented function (shift, rotate, AND, OR, …) also in integers!)

---

I definitely would not use anything green (i.e. boolean arrays, etc.). It will be significantly more efficient with the second option. (8x less data structures!)

[altenbach]

---

@altenbach wrote:

reminding me of the first bit-twiddling challenge long ago (link missing),

---

Some historical perspective:

The bit-twiddling challenge was held in ~2002(?) and was the first in a series of pretty cool LabVIEW challenges. Unfortunately, most of these old pages have permanently disappeared from NI.com 😞

The bit-twiddling challenge dealt with data that was not aligned at byte boundaries as follows:

Input string is encoded as nine bit characters -- eight data bits followed by a parity bit. Output string should contain the eight bit data character if the character parity is even, otherwise output a question mark '?'.For example: Sixteen characters making the string "ABCDEFGHIJKLMNOP" are encoded as in Encoder.vi, adding the parity bits, and resulting in an eighteen byte string of "4184 0C25 52D4 E811 2449 952E 61DA C4C9 5328". Given this eighteen byte string as an input, your VI should produce the original sixteen byte string. If any nine bit input character has odd parity, this indicates an error in transmission, and you should output a '?'."

I still have most of my old code. I am sure it could be adapted to your problem.

[GerdW]

Hi Christian,

---

@altenbach wrote:

---

---

I definitely would not use anything green (i.e. boolean arrays, etc.). It will be significantly more efficient with the second option. (8x less data structures!)

---

I know, that's why I wrote the 2nd paragraph.

But for "someone new to LabVIEW" going the green way might be more intuitive and easier to understand (those younger folks most often did not learn to bit-shuffle on 8/16 bit processors).

(And we still are talking about 360 bytes = 2880 bits…)

[billko]

---

@drak wrote:

Hi All,

I'm hoping someone will be able to point me in the right direction for the best way to dissect large UDP packets that contain 18 bit data samples. I'm new to labview with my experience being more on the electronics/embedded C side.

The UDP packets that I'm reading are 1200 bytes long. I'm able to strip out the Ethernet header, IP header and UDP header using the "Unflatten from String" function without issue, which leaves me with the IP data which is 1158 bytes long.

There is then 78 bytes of hardware status information and the remaining 1080 bytes is made up of three blocks of 360 bytes, each of those blocks consists of 160, 18-bit samples from an 8-channel ADC (160 samples per block, 480 samples per packet). I've gotten as far as breaking the packet into those blocks, but the best way to break it into individual samples is something I could use help with.

The only way I can think of separating the data into 18-bit groups in Labview is to convert it to a boolean array and then parse it, but this seems very clunky.

I'm sure I can brute force something together that works but the VI will need to parse 1000 packets per second, so speed with be an issue. If anyone has any high level recommendations that would be great (don't expect anyone to build this for me - just pointing me in the right direction or making suggestions would be fantastic).

Cheers,

---

I don't recall ever having to strip the header from the packet using UDP.  How are you reading this packet?

[altenbach]

---

@billko wrote:
I don't recall ever having to strip the header from the packet using UDP.  How are you reading this packet?

---

---

@altenbach wrote earlier:

What is generating these packets and why do they contain packet headers? is this a pcap capture??

---

See also

[Ben]

This challenge would be trivial for a KL-10 that featured a 36 bit CPU and variable length word length.

Spoiler

I used to be able to write the machine language code in less time than it would do it in LabVIEW.

😁

Casting the data as an array of U32 and using a shift register to hold a working buffer with a logical shift right then applying a bit-wise mask to pull out each 18-bit value.

each time a 18-bit value is pulled the array of U32 is indexed to pad out the high order bits in the shift register..

Ben

[drak]

I'll try to answer each of your questions as best I can.

1. Packet Structure

Refer to the attached PDFs, each 360 byte block is made up of 20 data samples, each of which contains data for 8 channels. They are not aligned with byte boundaries.

2. What is generating these packets?

The packets are generated by an atmel processor interfaced to three ADCs via SPI within some bespoke hardware. The packet buffer is filled simultaneously by those three SPI peripherals using DMA (all three ADCs share the same clock line to ensure synchronous comms). Once each of the peripherals has filled its allotted 360 bytes this triggers an interrupt and the processor sends out a broadcast UDP packet.

3. Why do these contain packet headers? Is this a pcap capture?

Yes that's correct

I noticed GerdW flagged the most efficient way would be to use shifting operations. Would this "larger integer data type" mentioned just be an integer array? Or can you define a custom data type with a nominated length of bytes?