Conservative blogger Bill Whittle was the third panelist to withdraw from the event after recent controversy.