Inside LV 8 I found no way of modifying the size of the radio button inserted inside the "radio buttons" cluster. Instead I can modify the size of a single round radio button (found in Classic boolean palette).
Speaking of CVI, I don't think you will be able to modify the size of the single button in a radio group of toolslib controls palette, since it relies on a feature of the tree control (specifically the mark type attribute of the tree items).
You could try to emulate the behaviour of a radio group control using multiple radio buttons (which can be customized individually) adding a callback function that manages the state of individual controls when clicked.