All permits shall be terminable at the discretion of the Secretary having jurisdiction.