(d) All laboratory testing shall be conducted or verified by independent testing authorities appropriately certified by the National Association of State Election Directors, the EAC or approved by the commissioners of the State Board.

1. (1) Software and hardware qualification tests. Qualification of voting system software and hardware shall consist of a series of tests, code analyses, and inspection tests performed at the Federal and State levels, to verify that the software and hardware meet design requirements and that characteristics are correctly described in the documentation items. Qualification shall also include a functional configuration audit and a physical configuration audit.

2. (2) Functional configuration audit. A functional configuration audit shall be performed to verify that the software complies with the software specification (as defined in paragraph [f][3] of this section) and applicable laws and regulations. Federal qualification test data may be used in partial fulfillment of this requirement; however, the State Board or its designee shall perform or supervise the performance of additional tests, or order additional laboratory testing, to verify system performance in all operating modes, including but not limited to disability access and alternate language modes and to validate the vendor's test data reports. The functional configuration audit shall be performed in a facility selected by the State Board.

   1. (i) Vendor responsibility. The vendor shall provide a list of all documentation and data required to be included as part of the independent review, and vendor technical personnel shall be available to the State Board during the performance of the functional configuration audit.

   2. (ii) Technical data. The vendor shall provide the following technical data:

      1. (a) copies of all procedures used for module or unit testing, integration testing and system testing;
      2. (b) copies of all test cases generated for each module and integration test and sample ballot formats or other test cases used for system;
      3. (c) records of all tests performed by the procedures listed above, including error correction and retest.
   3. (iii) Audit procedure. The State Board, with the assistance of an independent testing authority, shall subject each voting system to a complete functional test, including but not limited to actual use testing of all components used by voters to enter or review votes. Additionally, the State Board and its independent testing authority shall review the vendor's test procedures and test results. This review shall include an assessment of the adequacy of test cases and input data to exercise all system functions and to detect program logic and data processing errors if such be present. The review shall also include an examination of all test data which is to be used as a basis for qualification.

3. (3) Physical configuration audit. The physical configuration audit is an examination of the software configuration against its technical documentation to establish a configuration baseline for approval. The physical configuration audit shall include an audit of all drawings, specifications, technical data and test data associated with the system hardware and this audit shall establish the system hardware baseline associated with the software baseline. All subsequent changes to the software or hardware shall be subject to re-examination.

   1. (i) Vendor responsibility. The vendor shall provide a list of all documentation and data required to be audited by the State Board. Vendor's technical personnel shall be available to the State Board during the performance of the physical configuration audit.

   2. (ii) Technical data. The vendor shall provide the following technical data:

      1. (a) identification of all items which are to be a part of the software release;
      2. (b) identification of all hardware which interfaces with the software;
      3. (c) configuration baseline data for all hardware included within the system;
      4. (d) copies of all software documentation which is intended for distribution to users, including program listings, specifications, operator manual, user manual and software maintenance manual;
      5. (e) proposed user acceptance test procedure and acceptance criteria;
      6. (f) an identification and explanation of any changes between the physical configuration audit and the configuration submitted for the functional configuration audit.

   3. (iii) Audit procedure. Required data items include draft and formal documentation of the vendor's software development program which are relevant to the design and conduct of qualification tests. The vendor shall identify all documents, or portions of documents, which the vendor asserts contain proprietary information not approved for public release. The State Board or its designee shall agree to use any proprietary information contained therein solely for the purpose of analyzing and testing the software and shall refrain from disclosing proprietary information to any other person or agency without the prior written consent of the vendor or a court order. The State Board or its designee shall review the vendor's source code and documentation to verify that the software conforms to the documentation, and that the documentation is sufficient to enable the user to install, validate, operate and maintain the voting system. The review shall also include an inspection of all records of the baseline version against the vendor's release control system to establish that the configuration, being qualified, conforms to the engineering and test data.

      (e) Functional tests, security tests and simulated voting.

      Prior to certifying a voting system, the State Board shall designate an independent expert to review, all source code made available by the vendor pursuant to this section and certify only those voting systems compliant with this Part. At a minimum, such review shall include a review of security, application vulnerability, application code, wireless security, security policy and processes, security/privacy program management, technology infrastructure and security controls, security organization and governance, and operational effectiveness, as applicable to that voting system.

4. (1) For all systems or equipment, functional tests shall consist of the validation of equipment functional performance, and shall be performed in an open and public venue, in conformity with written procedures adopted by the State Board.
5. (2) All votes entered shall use the identical interfaces as would be used by the actual voters during the actual voting process. By way of explanation, touch-screen votes, or votes cast via alternative accessible devices such as tactile-discernible key pads or pneumatic switches shall be used as the voter would use them rather than casting simulated votes via any of these processes into the voting system using any type of diagnostic input cartridge.
6. (3) Functional tests of voting system software which runs on general purpose data processing equipment shall include all tests similar to those in procedures which are necessary to validate the proper functioning of the software and its ability to control the hardware environment. The tests shall also validate the ability of the software to detect and act correctly upon any error conditions which may result from hardware malfunctions. Detection capability may be contained in the software, the hardware or the operating system. It shall be validated by any convenient means up to and including the introduction of a simulated failure (power off, disconnect a cable, etc.) in any equipment associated with vote processing.
7. (4) Each system shall be submitted for electronic and technical security and integrity analysis by independent certified security experts, who shall be given full unrestricted access to production units of the system, for such analysis. Whenever the vendor is able to provide documentation for the State Board and its testing authority, to establish that the standards of this section of these regulations have been met; then the State Board may, in its discretion, accept such documentation as satisfaction of the tests required by this Part.

8. (5) Functional tests for the following types of equipment shall be required:

   1. (i) Standard commercial, off-the-shelf production models of general purpose data processing equipment (PC's, printers, etc.) shown to be compatible with these requirements and with the voting system.

   2. (ii) Production models of special purpose data processing equipment (scanners, bar code readers, etc.) having successfully performed in elections use and having been shown to be compatible with the voting system.

      (f) Software, hardware, operating and support documentation.

9. (1) Software qualification. The following system software and firmware vendor data items shall be submitted as a precondition of certification of acceptability for elections use.
10. (2) Vendor documentation. Complete product documentation shall be provided to the State Board for voting systems, their components and all auxiliary devices. This documentation shall be sufficient to serve the needs of the voter, the operator, maintenance technicians, and other appropriate county board personnel. It shall be prepared and published in accordance with standard industrial practice for electronic and mechanical equipment such documentation shall include:

11. (3) Software specification. The software specification shall contain and describe the vendor's design standards and conventions, environment and interface specifications, functional specifications, programming architecture specifications, and test and verification specifications. Vendor must also provide document identification, an abstract of the specification, configuration control status and a table of contents. The body of the specification shall contain the following material:

    1. (i) System overview. The vendor shall identify the system hardware and the environment in which the software will operate and the general design and operational considerations and constraints which have influenced the design of the software.
    2. (ii) Program description. The vendor shall provide descriptions of the software system concept, the array of hardware in which it operates, the intended operating environment, the specific software design objectives and development methodology and the logical structure and algorithms used to accomplish the objectives.
    3. (iii) Standards and conventions. The vendor shall provide information which can be used as a partial basis for code analysis and test design. It should include a description and discussion of the standards and conventions used in the preparation of this specification and in the development of the software.
    4. (iv) Specification standards and conventions. The vendor shall identify all published and private standards and conventions used to document software development and testing. Vendor internal procedures shall be provided as attachments to this software specification.
    5. (v) Test and verification standards. The vendor shall identify any standards or other documents which are applicable to the determination of program correctness and acceptance criteria.
    6. (vi) Quality assurance standards. The vendor shall describe all standards or other documents which are applicable to the examination and testing of the software, including standards for flowcharts, program documentation, test planning and test data acquisition and reporting.
    7. (vii) Operating environment. The vendor shall provide a description of the system and subsystem interfaces at which inputs, outputs and data transformations occur. It shall contain or make reference to all operating environment factors which influence the software design.

    8. (viii) Hardware constraints. The vendor shall identify and describe the hardware characteristics which influence the design of the software, such as:

       1. (a) the logic and arithmetic capability of the processor;
       2. (b) memory read/write characteristics;
       3. (c) external memory device characteristics;
       4. (d) peripheral device interface hardware data I/O device protocols; and
       5. (e) operator controls, indicators and displays.
    9. (ix) Software environment. The vendor shall identify all compilers, assemblers, or other software tools to be used for the generation of executable code and a description of the operating system or system monitor. This section shall also contain an overview of the compile-time interaction of the voting system software with library calls and linking.
    10. (x) Interface characteristics. The vendor shall describe the interfaces between executable code and system input-output and control hardware.
    11. (xi) Software functional specification. The vendor shall provide a description of the overall functions which the software performs in the context of its mode or modes of operation. The vendor shall also describe the capabilities and methods for detecting and handling exceptional conditions, system failure, data input/output errors, error logging and audit record generation and security monitoring and control.
    12. (x) Configurations and operating modes. The vendor shall describe the various software configurations and operating modes of the system; such as preparation for opening of the polling place, vote recording and/or vote processing, closing of the polling place and report generation. For each software function or operating mode, a definition of the inputs (characteristics, tolerances or acceptable ranges) to the function or mode, how the inputs are processed and what outputs are produced (characteristics, tolerances or acceptable ranges) shall be provided.
    13. (xiii) External files. In the event that external files are used for data input or output, the definition of information context and record formats shall be provided. The vendor shall also describe the procedures for file maintenance, access privileges and security.
    14. (xiv) Security. Security requirements and security provisions of the system's software shall be identified for each system function and operating mode. The voting system must be secure against attempts to interfere with correct system operation. The vendor shall identify each potential point of attack. For each potential point of attack, the vendor shall identify the technical safeguards embodied in the voting system to defend against attack, and the procedural safeguards that the vendor has recommended be followed by the election administrators to further defend against that attack. Each defense shall be classified as preventative, if it prevents the attack in the first place; detective if it allows detection of an attack; or corrective if it allows correction of the damage done by an attack. Security requirements and provisions shall include the ability of the system to detect, prevent, log and recover from the broad range of security risks identified. These procedures shall also examine system capabilities and safeguards claimed by the vendor to prevent interference with correct system operations. The State Board, with the assistance of its ITA, shall conduct tests to confirm that the security requirements of this Part have been completely addressed. Notwithstanding any other provisions of this Part, the State Board shall determine whether all or a portion of such security requirements and security provisions shall be available for public inspection, but shall exclude any information which compromises the security of the voting system.
    15. (xv) Programming specifications. The vendor shall provide an overview of the software design, structure and implementation algorithms. Whereas the functional specification of the preceding section provides a description of what functions the software performs and the various modes in which it operates, this section should be prepared so as to facilitate understanding of the internal functioning of the individual software modules. Implementation of functions shall be described in terms of software architecture, algorithms and data structures and all procedures or procedure interfaces which are vulnerable to degradation in data quality or security penetration shall be identified.
    16. (xvi) Test and verification specifications. The vendor shall provide a description of the procedures used during software development to verify logical correctness, data quality and security. This description shall include existing standard test procedures, special purpose test procedures, test criteria and experimental design and validation criteria. In the event that this documentation is not available, the qualification test agency shall design test cases and procedures equivalent to those ordinarily used as a basis for verification (see below).
    17. (xvii) Qualification test specification. The vendor shall provide a description of the specification for verification and validation of overall software performance, including acceptance criteria for control and data input/output, processing accuracy, data quality assessment and maintenance, exceptional handling and security. The specification shall identify specific procedures by means of which the general suitability of the software for elections use can be assessed and demonstrated. The vendor's specification and procedure shall be used to establish the detailed requirements of the tests described in "Laboratory Environmental Test Procedures for Hardware and Software" of this standard.
    18. (xviii) Acceptance test specification. The vendor shall provide a description of the specification for installation, acceptance and readiness verification. This specification shall identify specific procedures by means of which the capability of the software to accommodate actual ballot formats and format logic, and pre-election logic, accuracy and security test requirements of using jurisdictions may be assessed and demonstrated. The vendor's specification shall be used to establish the detailed requirements of the tests described in "Laboratory Environmental Test Procedures for Hardware and Software" of this standard performed to evaluate the adequacy of the vendor's procedures and it shall be suitable for inclusion in the regulations and procedures of user counties when preparing for the conduct of actual elections.

    19. (xix) Appendices. The vendor shall provide descriptive material and data supplementing the various sections of the body of the software specification. The content and arrangement of appendices shall be at the discretion of the vendor. Topics recommended for amplification and treatment in appendix form include:

        1. (a) Glossary. Provide a listing and brief definition of all software module names and variable names with reference to their locations in the software structure. Include abbreviations, acronyms and terms which are either not commonly used in data processing and software development or which are used in an uncommon semantic context.
        2. (b) References. Provide a list of references to all related vendor documents, data, standards and technical sources used in software development and testing.
        3. (c) Program analysis. Provide the results of software configuration analysis, algorithm analysis and selection, timing studies and hardware interface studies reflected in the final software design and coding.
        4. (d) Security analysis. Provide a detailed description of the penetration analysis performed to preclude intrusion by unauthorized persons and fraudulent manipulation of elections data. Identify security policies and measures and selection criteria for audit log data categories.
12. (4) Operator information. This documentation shall include a physical description of the equipment sufficient to identify all features, controls and displays. It shall include a complete procedure for energizing the equipment, for testing and verifying operational status and for identifying all abnormal equipment states. It shall include a complete operating procedure for inserting ballots to be tabulated, for controlling the tabulation process, for monitoring the status of the equipment, for recovering from error conditions and for preparing output reports. It shall also include troubleshooting instructions. The documentation shall also include a description of the relationship of the sensitive area, voting target, and ballot position. For paper-based systems, this description shall include a description of the nature of the marks the system will and will not count as votes, for example, the types of marks made with each of a variety of pens and pencils that should be counted and that should not be counted. For DRE voting systems, this description shall include a description of the nature of the voter action required to cast a vote in the sensitive area, for example, the force and duration of contact required.

13. (5) Maintenance information.

    1. (i) This documentation shall contain a complete physical and functional description of the equipment and a theory of operation which fully describes the electrical and mechanical function of the equipment, how the processes of ballot handling and reading are performed, how data are handled in the processor and memory sections, how data output is initiated and controlled, how power is converted or conditioned and how test and diagnostic information is acquired and used.
    2. (ii) A complete parts and materials list shall be provided which contains sufficient descriptive information to identify all parts by type, size, value or range and manufacturer's designation.
    3. (iii) Technical illustrations and schematic representations of electronic circuits shall be provided with indications of all test and adjustment points and the nominal value and tolerance or waveform to be measured. Fault detection, isolation and correction procedures or logic diagrams shall be prepared for all operational abnormalities identified by design analysis and operating experiences.
14. (6) Logistics, facilities and training. The vendor shall identify all operating and support requirements of the system or component. These requirements include material, facilities and personnel, including furnishings, fixtures, and utilities which will be required to support system operation, maintenance and storage.

15. (7) Maintenance training and supply.

    1. (i) The vendor shall identify all corrective and preventive maintenance tasks, including the calibration of the system, as appropriate, and the level at which they shall be performed. Levels of maintenance shall include operator tasks, maintenance personnel tasks and factory repair.
    2. (ii) Operator tasks shall be limited to the activation of controls to identify irrecoverable error conditions and to the replenishment of consumables such as printer ribbons, paper and the like.
    3. (iii) Maintenance personnel tasks shall include all field maintenance actions which require access to internal portions of the equipment. They shall include the conduct of tests to localize the source of a malfunction; the adjustment, repair or replacement of malfunctioning circuits or components and the conduct of tests to verify restoration to service.
    4. (iv) Factory repair tasks shall be minimized, and repairs shall be made on site whenever reasonably possible. Factory repairs shall only include complex and infrequent maintenance functions which require access to proprietary or to specialized facilities and equipment which cannot be obtained by the county board.
    5. (v) The vendor shall identify by function all personnel required to operate and support the system. For each functional category, the number of personnel and their skills and skill levels shall be specified.
    6. (vi) The vendor shall specify requirements for the training of each category of operating and support personnel, including but not limited to voters, poll workers, and elections staff. The vendor shall prepare all materials required in the training activity and shall provide or otherwise arrange for the provision of as many qualified instructors as are necessary to properly and fully train said personnel in each category.
    7. (vii) The vendor shall recommend a standard complement of supplies, spares and repair parts which will be required to support system operation. This list shall include the identification of these materials and their individual quantities and sources from which they may be obtained. The vendor shall supply, at vendor's expense, any special tools required to repair or maintain the equipment.
    8. (viii) The vendor shall provide complete instructions for all methods of voting which voters may use to cast their vote, including instructions on entering and changing votes, write-in voting, verifying votes and accepting the cast votes. Written and audio instructions shall be provided in each language in which voting shall occur within the State.
16. (8) Usability test. Vendors shall make available to the State Board, in a quantity to be determined by the State Board, voting systems for the purpose of conducting a usability test, which will establish the minimum number of voting machines required in each polling place and the maximum number of voters that can vote on one voting machine during the course of an ordinary 15-hour election day. The ballots to be used for this test shall include both primary and general election ballots, with ample candidate selection options and ballot proposal selections. For the purposes of the usability test, voting shall occur by utilizing all the devices which a voter may use to make their selections. If a vendor has previously performed a usability test on the same or similar voting system which meets the requirements of this section, the State Board may consider the findings of same. Whenever the State Board is satisfied that a voting machine or system's usability analysis has provided adequate and accurate information relative to the requirements of Election Law, section 7-203.2, then the State Board may, in its discretion, accept such documentation as satisfaction of the usability test required by these regulations.

17. (9) Voter demonstration test.

    1. (i) The purpose of this test is to provide, in a simulated election day environment, a public demonstration of the usability and accuracy of such systems or machines.
    2. (ii) Vendor must submit, in a quantity to be determined by the State Board, additional voting systems or equipment that have been submitted for certification. These additional systems or equipment will be returned to the vendor upon the completion of voter demonstration testing.
    3. (iii) The State Board shall make available to the public, all non-proprietary documentation submitted by the vendor.

18. (10) Certification.

    1. (i) The State Board shall escrow a complete copy of all certified software that is relevant to functionality, setup, configuration, and operation of the voting system, including but not limited to, a complete copy of the source and executable code, build scripts, object libraries, application program interfaces, and complete documentation of all aspects of the system including, but not limited to, compiling instructions, design documentation, technical documentation, user documentation, hardware and software specifications, drawings, records, and data. Documentation shall include a list of programmers responsible for creating the software and a sworn affidavit that the source code includes all relevant program statements in low-level and high-level languages. The State Board may require that additional items be escrowed. If any vendor contracts to escrow additional items, those items shall be subject to the provisions of this section.
    2. (ii) The vendor shall immediately notify the State Board of any change in any item required to be escrowed by subparagraph (i) of this paragraph, and shall provide an updated version for deposit.
    3. (iii) The chief executive officer of the vendor shall sign a sworn affidavit that the source code and other material in escrow is the same being used in its voting systems in the State. The chief executive officer shall have an ongoing obligation to ensure the statement is true.
    4. (iv) The vendor shall promptly notify the State Board and each county board using its voting system of any decertification of the same system in any state, of any defect in the same system known to have occurred anywhere, and of any relevant defect known to have occurred in similar systems.
    5. (v) Upon completion of testing, reports shall be produced by the ITA and State Board staff, and a recommendation either for or against certification shall be made to the State Board's commissioners.
    6. (vi) If the State Board determines that a system meets the requirements of this Part, and is determined to be suitable for use by voters, it shall certify such system. A notice of provisional certification shall be prepared and forwarded to the vendor, forthwith. The vendor shall ensure that the voting system's software has been escrowed as set forth in Election Law, section 7-208, and the vendor has updated any affidavit and complied with the affidavit requirements, as set forth in section 6209.4(h) of this Part.
    7. (vii) Upon compliance with the provisions set forth above, a notice of certification shall be awarded to the vendor. Notice of such certification shall also be provided to all county boards.
    8. (viii) If the State Board fails to certify a system, the vendor shall be so notified.
    9. (ix) Once a certified system is selected for purchase by a county board, that system's software shall be provided to the county board by the State Board, and not the vendor.