Why do you need low latency? you may have to rethink your application architecture.
If you need ultra-low latency you need to implement your application on an FPGA and not an OS-based system.
The driver layer and OS will definitely hinder your latency.