I can hardly guess what amy be happening without looking at the code. I have applications that correctly manage communications down to 5 ms (more or less the raw ts/rx time) so there may be something that is conusimg time elsewhere in your application.
More than raising thread priority, I would switch from an async timer to a true multithreading setting the communications thread to not sleep with SetSleepPolicy: this is what I'm doing with most critical applications and I find it gives satisfactory results.