When the tests are due to be carried out, it becomes necessary to specify the tests in more detail. This automatically leads to the question, whether a certain test has been specified in a proper manner. This question addresses the right definition and parameterization of the planned tests.
For this purpose, the (test) specification shall include information about the following items:
- The parameters to be tested must be specified. It is important that the parameters are in line with the scenarios the ADF will encounter in its ODD. Therefore, it must be analysed before the test, which situations and parameters occur in the ADF’s ODD. Verifying the completeness of the tests remains a challenge, since there will be unknown situations and parameters. A first step is to analyse the known area of the test space in a transparent manner. This means to describe the data and information on which the test criteria are defined. In the second step a process should be defined which ensures a continuous reduction of the unknown areas in test space.
- Depending on the test, the test amount (e.g. number of repetitions, number of test persons, driven mileage, driven time) needs to be defined. It is important that the amount of testing is chosen in a way that it ensures sufficient data to run an authoritative analysis. The test amount also covers the duration of each test.
- The success criteria for a test must be defined. This could be a single criterion or multiple criteria. The conditions under which a test needs to be repeated or re-run, shall also be defined.
- It shall be defined, which data and information of the test must be documented and how the data are stored.
- If reference data are required or measured in the test, these reference data shall be clearly defined. This includes information, which data should be used as a reference and how they are collected respectively and by which tool they are measured.
It shall be checked for the different tests, whether privacy aspects are relevant and how these can be ensured during testing.
Main Question
Is each single test of the (test) concept specified properly? (Test planning)
Alternative Questions:
- Are the test parameters defined for each test (e.g., start velocity, reaction of other vehicles, etc.)?
- Are the test parameters in line with the situations that the ADF will encounter in its ODD?
- Is it defined, how many test repetitions / test persons / mileage driven / time driven is/are required?
- Are guidelines for the conduction of tests defined?
- Are criteria defined that specify the required data quality?