Pass the ISTQB ISTQB AI Testing CT-AI Questions and answers with CertsForce

The ISTQB CT-AI syllabus explains inSection 4.4 – Experience-Based Testing for AI Systemsthat AI-based systems frequently suffer frominsufficient specifications, unpredictable model behavior, andtest oracle problems, especially when outputs depend on probabilistic or learned patterns. The syllabus explicitly states thatexploratory testingis especially valuable in such contexts because it allows testers to investigate the system interactively, observe unexpected behavior, and evaluate system responses that cannot be fully predicted beforehand. Thus, OptionCaccurately reflects the role and justification of exploratory testing for AI systems.

Option A describes data analysis rather than intuitive test design. Option B is incorrect because checklist-based testing does not dynamically adapt test cases; instead, it follows predetermined checklists. Option D incorrectly defines “tour-based testing”; tours refer to structured exploratory approaches, not biased datasets.

Therefore,Option Cis the syllabus-aligned correct statement.

According to the syllabus,conventional AIsystems are limited to specific, pre-defined tasks and do not have generalized intelligence:

"Conventional AI systems are limited in their scope and typically only perform specific tasks within the domain for which they have been designed. They do not exhibit general AI behavior."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 1.2)

 A. Using a pixel comparison of the GUI before and after the change to check the differences.

Pixel comparison is a traditional method and does not involve AI. It compares images at the pixel level, which can be effective but is not an intelligent approach. It is not considered an AI usage and is the least likely usage of AI for detecting GUI changes.

 B. Using computer vision to compare the GUI before and after the test object changes.

Computer vision involves using AI techniques to interpret and process images. It is a likely usage of AI for detecting changes in the GUI.

 C. Using vision-based detection of the GUI layout changes before and after test object changes.

Vision-based detection is another AI technique where the layout and structure of the GUI are analyzed to detect changes. This is a typical application of AI.

 D. Using a ML-based classifier to flag if changes in GUI are to be flagged for humans.

An ML-based classifier can intelligently determine significant changes and decide if they need human review, which is a sophisticated AI application.

The question asks which test is least likely to be performed during the ML model testing phase. Let's consider each option:

Testing the accuracy of the classification model (A): Accuracy testing is a fundamental part of the ML model testing phase. It ensures that the model correctly classifies the data as intended and meets the required performance metrics.

Testing the API of the service powered by the ML model (B): Testing the API is crucial, especially if the ML model is deployed as part of a service. This ensures that the service integrates well with other systems and that the API performs as expected.

Testing the speed of the training of the model (C): This is least likely to be part of the ML model testing phase. The speed of training is more relevant during the development phase when optimizing and tuning the model. During testing, the focus is more on the model's performance and behavior rather than how quickly it was trained.

Testing the speed of the prediction by the model (D): Testing the speed of prediction is important to ensure that the model meets performance requirements in a production environment, especially for real-time applications.

The syllabus describes exploratory testing as especially useful when there are poor specifications or test oracle problems:

“Exploratory testing is especially useful when there are poor specifications or test oracle problems, which is often the case for AI-based systems.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.6, page 70 of 99)

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

The ISTQB CT-AI syllabus (Section4.3 – Test Design for AI-Based Systems) highlights pairwise testing as an effectivetest-case reduction techniquefor systems with many input parameters. Lane-keeping assist systems typically include environmental, sensor, and vehicle-dynamic parameters, making exhaustive testing infeasible. Pairwise testing significantly reduces the number of test cases while still capturingall 2-way interactions, which are responsible for a large proportion of software defects.

OptionBaligns with this syllabus description: pairwise testing reduces otherwise extremely large parameter combinations, making test effort manageable.

Option A overstates feasibility guarantees; the syllabus never claims pairwise testing always makes testing “typically feasible.” Option C is unsupported and incorrect because pairwise testing doesnotrestrict parameters to driver-controlled ones. Option D is incorrect because, although some defects arise from higher-order interactions, pairwise testing captures many relevant defects and is widely recognized as a pragmatic compromise.

Thus,Option Bis the correct statement.

The syllabus explains that supervised learning requires correctly labeled data so the algorithm can learn the relationship between input features and output labels:

"In supervised learning, the algorithm creates the ML model from labeled data during the training phase. The labeled data is used to infer the relationship between the input data and output labels."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.1.1)

The ISTQB CT-AI syllabus explains inSection 4.5 – Testing AI-Specific Risksthat adversarial testing is a structured test activity in which testers applyadversarial attacks—crafted or perturbed inputs—to intentionally expose weaknesses in the ML model. The purpose is to identify vulnerabilities that could be exploited throughdata poisoning,evasion attacks, orinput manipulation. OptionCcorrectly reflects this syllabus definition: adversarial testing is aboutusing attacks to locate weaknesses so they can be removed or mitigated.

Option A is incorrect because regression testing does not verify data sourcing policies; it verifies unchanged functionality after modifications. Option B is incorrect because adversarial examples are oftenaddedto training datasets to improverobustness(a practice called adversarial training), not excluded. Option D is incorrect because AIB testing is not described as superior to exploratory data analysis in outlier detection; both have different purposes, and EDA remains essential for data quality assessment.

Thus,Option Cis consistent with syllabus-defined adversarial testing.

Defect prediction models aim to identify parts of the software that are likely to contain defects by analyzing historical data and code quality metrics. The primary goal is to use this predictive information to allocate testing and maintenance resources effectively. Let's break down why option D is the correct choice:

Understanding Classification Models:

Classification models are a type of supervised learning algorithm used to categorize or classify data into predefined classes or labels. In the context of defect prediction, the classification model would classify parts of the code as either "defective" or "non-defective" based on the input features.

Input Data - Code Quality Metrics:

The input data for these classification models typically includes various code quality metrics such as cyclomatic complexity, lines of code, number of methods, depth of inheritance, coupling between objects, etc. These metrics help the model learn patterns associated with defects.

Historical Data:

Historical versions of the code along with their defect records provide the labeled data needed for training the classification model. By analyzing this historical data, the model can learn which metrics are indicative of defects.

Why Option D is Correct:

Option D specifies using a classification model to predict the presence of defects by using code quality metrics as input data. This accurately describes the process of defect prediction using historical bug data and quality metrics.

Eliminating Other Options:

A. Identifying the relationship between developers and the modules developed by them: This does not directly involve predicting defects based on code quality metrics and historical data.

B. Search of similar code based on natural language processing: While useful for other purposes, this method does not describe defect prediction using classification models and code metrics.

C. Clustering of similar code modules to predict based on similarity: Clustering is an unsupervised learning technique and does not directly align with the supervised learning approach typically used in defect prediction models.