The Governor may, in his judgment, remove any member of the board for neglect of duty or other sufficient cause, after due notice and hearing.