Pass the iSQI AI Testing CT-GenAI Questions and answers with CertsForce

A/B testing, also known as split testing, is a systematic empirical method used to compare two versions of a prompt (Version A and Version B) to determine which one performs better based on predefined evaluation metrics. In the realm of LLMs, where outputs can be stochastic (probabilistic), A/B testing is essential for mitigating inconsistency. When a team encounters vague or varying results for a user story, simply modifying the prompt iteratively (Option B) may improve the result but does not provide a statistical or objective basis for why one version is superior. Byrunning A/B tests, testers can evaluate prompts against specific KPIs such as accuracy, relevance, format adherence, or the absence of hallucinations. This process involves sending the same input data through both prompt versions multiple times and scoring the outputs. The version that consistently yields the "stronger wording" or more precise testware is then selected as the production standard. This data-driven approach is a cornerstone of prompt engineering in professional environments, ensuring that the most effective linguistic structures are utilized to maximize the model's performance and reliability.

In professional Generative AI applications, thesystem prompt(sometimes called the system message) is the foundational set of instructions that defines the AI's persona, boundaries, and overall behavior. In a testing workflow, this is typically defined by atester or test engineerwho is configuring the AI assistant for a specific project. Unlike the user prompt, which changes with every interaction, the system prompt remains relatively static and acts as a "guardrail" to ensure the model stays in its role (e.g., "You are an expert in ISO 26262 automotive testing standards"). By defining the system prompt, the tester ensures that the model consistently uses specific terminology, adheres to data privacy constraints, and formats its output according to the team's requirements. While end users (Option B) provide the task-specific input, they do not usually have the permissions or technical need to alter the underlying system-level instructions. Similarly, while CI servers (Option C) might trigger the prompt, they do not "define" the human-centric logic contained within it. Properly crafting the system prompt is a core part of setting up an AI-augmented test environment.

As Generative AI becomes integrated into the software testing lifecycle, the role of the tester shifts from manual authoring to the "orchestration" of AI models. Mastering prompt engineering is the primary competency required to effectively steer LLMs. Prompt engineering involves the deliberate design of inputs—incorporating roles, context, instructions, and constraints—to elicit the most accurate and "on-policy" outputs from the model. In a testing context, "on-policy" refers to testware that adheres to organizational standards, security protocols, and specific project requirements. While technical skills like network configuration or low-level programming (Options B, C, and D) are valuable in specific engineering domains, they do not directly influence the communicative interface between the human and the AI. A tester proficient in prompt engineering can utilize techniques like "Chain-of-Thought" or "Few-shot prompting" to ensure the LLM understands the nuances of a test plan, thereby reducing hallucinations and ensuring the generated test cases are actionable, relevant, and compliant with the project's quality gates.

In the context of LLM inference,Temperatureis a hyperparameter that controls the randomness or "creativity" of the model's output. When the temperature is set high, the model's probability distribution is "flattened," meaning it is more likely to select less-probable tokens, leading to more diverse and sometimes unpredictable text. For software testing, where precision and repeatability are paramount,lowering the temperature(Option C) is the standard practice. A temperature of 0.0 makes the model "deterministic," meaning it will consistently choose the token with the highest probability. This narrows the sampling distribution and significantly reduces variability between runs. While a larger context window (Option D) allows the model to process more information, it does not directly control the randomness of token selection. Similarly, the "learning rate" (Option B) is a parameter used during thetrainingorfine-tuningphase, not during inference. For generating test cases or scripts that must follow strict logic, a lower temperature ensures that the model remains focused and produces consistent results.

ISO/IEC 42001:2023is the international standard for an AI Management System (AIMS). It is designed to help organizations develop, provide, or use AI systems responsibly by providing a certifiable framework of requirements and controls. In a software testing context, this standard is vital for establishing governance, ensuring that GenAI tools are used consistently and ethically across the lifecycle.NIST AI RMF 1.0(Option B) is a highly respected framework, but it is a set of voluntary guidelines for managing risk, not a "requirement standard" for a management system.ISO/IEC 23053:2022(Option C) provides a general framework for AI using machine learning but lacks the comprehensive "management system" scope found in 42001. Finally, theEU AI Act(Option D) is a regulation (law), not a technical standard. For a test organization looking to align its GenAI strategy with international best practices and achieve formal certification, ISO/IEC 42001 is the definitive standard to follow, as it covers the organizational processes, data handling, and risk management necessary for high-quality AI operations.

Differentiating between prompting techniques is essential for a tester to select the right tool for the task.Few-shot promptingis characterized by providing the model with a few examples of inputs and desired outputs, allowing it to learn the pattern and format.Prompt Chaininginvolves breaking a complex task into a sequence of smaller, interconnected prompts, where the output of one step becomes the input for the next (e.g., first extract requirements, then generate test cases from those requirements).Meta-promptingis a more advanced technique where the user asks the LLM to help design, write, or refine the prompt itself—essentially using the AI as a "prompt engineer" to optimize the instructions. Option D correctly identifies these core characteristics. Options A, B, and C contain fundamental mischaracterizations: for instance, Few-shotrequiresexamples (contradicting A), and Chaining is theoppositeof a single prompt (contradicting A). Mastering these distinctions allows testers to move from simple "chatting" to sophisticated AI orchestration that can handle complex, multi-stage testing workflows with high reliability.

A mature GenAI strategy for software testing must move beyond "hype" and focus on tangible value and operational feasibility. Selecting an LLM based onmeasurable test outcomes(such as reduction in test design time, increase in defect detection, or script accuracy) ensures that the AI investment directly supports the organization’s Quality Assurance goals. Furthermore, the model must becompatible with current infrastructure. This includes considerations for data security (on-prem vs. cloud), API integration capabilities, and cost-per-token efficiency. While vendor visibility (Option A) can be a factor, it is not a guarantee of task-specific performance. Prioritizing creativity over compliance (Option B) is highly risky for testing, where precision and policy adherence are paramount. Similarly, while broad functionality (Option C) is useful, it often results in "jack-of-all-trades" models that may not perform as well as specialized or instruction-tuned models on specific testing tasks. Strategic alignment requires a balance between model performance, organizational security requirements, and clear KPIs.

Tokenizationis the foundational process by which an LLM breaks down raw text into smaller, manageable units called "tokens." These tokens can represent individual words, parts of words (sub-words), or even punctuation marks. This is a critical step because LLMs do not "read" words like humans do; they process numerical representations of these tokens. The way text is tokenized directly impacts the model's efficiency and its ability to understand complex technical terminology used in software testing. For example, a rare technical term might be broken into several sub-word tokens. This process is closely linked to theContext Window(Option C), which is the maximum number of tokens a model can "remember" or process at one time. WhileEmbeddings(Option B) are the numerical vectors that represent the meaning of these tokens, and theTransformer(Option A) is the underlying architecture that processes them, tokenization is the specific mechanism for initial text decomposition. Understanding tokenization is vital for testers when managing long requirement documents to ensure they do not exceed the model's limits.

Generative AI is exceptionally capable of creating structured and unstructured data, but its role is limited to "generation" and "transformation," not infrastructure management or direct database administration. Creating production database backups (Option A) is a physical data management task involving the copying of actual stateful data from a server to storage; this is handled by database management systems (DBMS) and DevOps pipelines, not LLMs. Conversely, LLMs excel at the logic-based tasks listed in the other options. They can analyze requirements to identify and set boundary values (Option B) for input validation. They are also highly effective at creating combinatorial data (Option C), such as pairwise or all-combinations tables, by understanding the relationships between variables. Finally, one of the most powerful uses of GenAI in testing is generating synthetic datasets (Option D)—creating "fake" but realistically structured data that mimics production patterns without exposing Sensitive Personally Identifiable Information (SPII), thereby supporting privacy-compliant testing.

The effectiveness of an LLM is heavily dependent on the specificity of itsOutput Format. While role definition (Option C) and technical instructions (Option D) are helpful, the most significant "value add" for a test lead is receiving information that is directlyactionable. By expanding the output format to include structuredrisk predictions, severity levels, and recommended actions(Option B), the tester is forcing the LLM to perform a deeper level of analysis. Instead of just "flagging trends," the model must now synthesize the data to determinewhya trend is a risk andwhatthe team should do about it. This aligns with the "Advanced Prompting" section of the CT-GenAI syllabus, which emphasizes using AI for decision support. A structured report that includes a "timeline for intervention" allows the human tester to quickly validate the AI's logic and make informed decisions, transforming the LLM from a simple data summarizer into a strategic predictive tool that actively supports the maintenance of release quality and schedule adherence.