The board may by a majority vote of its members remove the warden at any time.