The standard for measuring the resolution of both scanners and digital cameras is set out in ISO-12233 and is based on slanted edge method. The Standard defines test target, illumination conditions and numerical procedure to calculate the MTF. The technique uses image of a sharp edge, slightly slanted relative to the vertical and horizontal axes, which transitions from near full black to near full white. Imaging this edge with a camera produces an Edge Spread Function, a measure of how much the camera resolves or blurs the edge. Spatial differentiation of the ESF results in a Line Spread Function, a measure of how the camera resolves and blurs lines. By orienting the spatial differentiation along the axis closest to the edge rather than the axis across it, the ESF is essentially oversampled. This enables the LSF to be determined at many times the limiting resolution of the digital structure, effectively multiplying the Nyquist limit. Using a Fourier Transform, the Modulation Transfer Function is determined directly from the LSF.
Here
http://www.i3a.org/downloads_iso_tools.html you can find both Matlab and Labview tools to perform ISO-12233 test on RECORDED image. If you need to calculate MTF during live video you will have to write your own program. Labview together with Vision module has all necessary basic functions (extraction, differentiation, FFT) to calculate MTF, although processing of oversampled slanted edge has to be developed in a way similar to how it's done in above mentioned tools.
Imatest
http://www.imatest.com/docs/sharpness.html has commercial program that implements ISO-12233 test technique and has enhanced capabilities. They also describe in more details how actual processing is done.
Please note that this technique gives MTF of complete system (camera lens + detector) and not of camera optics alone.
