Each time a person is convicted of a violation enumerated in Section 50-9-1120, the number of points assigned to the violation must be charged against the person. For each calendar year that passes after assignment in which the person received no points, the department shall deduct one-half of the accumulated points if the total number of points is greater than three. If a person has three or less points at the end of a calendar year in which no points were received, the department shall reduce his point total to zero; however, a person's record must not be less then zero points.