Microsoft AI-300 (Operationalizing Machine Learning and Generative AI Solutions) Practice Test 20

Question 190

A predictive modeling project must produce unbiased performance estimates. The immediate design goal is to learn normalization or imputation statistics only from training data before applying them to validation. Which approach addresses that goal?

  1. Time-based split
  2. Preprocessing fitted on training only
  3. Label leakage audit
  4. Group-aware split

Explanation

Correct Answer

B:

Fit-on-train preprocessing avoids importing held-out distribution information into model construction. Fitting on the full dataset creates a subtle leakage path.

Incorrect Answers

A:

Temporal splitting reflects future deployment conditions and avoids seeing later events during training; random row splits can leak future information; the outstanding need is to learn normalization or imputation statistics only from training data before applying them to validation.

C:

Post-outcome features can make evaluation unrealistically strong while being unavailable at inference time; feature provenance and availability timing must be checked; the needed capability is to learn normalization or imputation statistics only from training data before applying them to validation.

D:

Group-aware splitting reduces entity-level leakage when many rows share a source; ordinary random sampling can overestimate generalization to unseen customers; the requirement, by contrast, is to learn normalization or imputation statistics only from training data before applying them to validation.

Question 191

A predictive modeling project must produce unbiased performance estimates. The team is required to reserve a final evaluation dataset that is not consulted during hyperparameter tuning. Which option is the best fit?

  1. Label leakage audit
  2. Preprocessing fitted on training only
  3. Time-based split
  4. Immutable test set

Explanation

Correct Answer

D:

A locked test set provides a less biased estimate after model selection. Using it iteratively for development turns it into another validation set.

Incorrect Answers

A:

Post-outcome features can make evaluation unrealistically strong while being unavailable at inference time; feature provenance and availability timing must be checked; the requirement, by contrast, is to reserve a final evaluation dataset that is not consulted during hyperparameter tuning.

B:

Fit-on-train preprocessing avoids importing held-out distribution information into model construction; fitting on the full dataset creates a subtle leakage path; the team would still need to reserve a final evaluation dataset that is not consulted during hyperparameter tuning.

C:

Temporal splitting reflects future deployment conditions and avoids seeing later events during training; random row splits can leak future information; the specified outcome instead requires the team to reserve a final evaluation dataset that is not consulted during hyperparameter tuning.

Question 192

An operational forecasting model performs exceptionally in random validation but degrades when predicting future events. The audit finds rows from later dates in training and several features computed only after the target outcome occurred. Which TWO corrections directly eliminate these two sources of leakage? Choose TWO.

  1. Stratified sampling
  2. Label leakage audit
  3. Preprocessing fitted on training only
  4. Group-aware split
  5. Time-based split

Explanation

Correct Answers

B:

Post-outcome features can make evaluation unrealistically strong while being unavailable at inference time; feature provenance and availability timing must be checked. Here, post-outcome columns are excluded from every training feature view.

E:

Temporal splitting reflects future deployment conditions and avoids seeing later events during training; random row splits can leak future information. Here, the evaluation represents forward-looking deployment instead of letting later rows leak into training.

Incorrect Answers

A:

Stratification helps create representative partitions when labels are imbalanced; it cannot resolve leakage from duplicate entities in both splits; the immediate requirement instead concerns the two observed paths for future information to leak into historical training.

C:

Fit-on-train preprocessing avoids importing held-out distribution information into model construction; fitting on the full dataset creates a subtle leakage path; this mechanism leaves unresolved the two observed paths for future information to leak into historical training.

D:

Group-aware splitting reduces entity-level leakage when many rows share a source; ordinary random sampling can overestimate generalization to unseen customers; it does not by itself address the two observed paths for future information to leak into historical training.

Question 193

The model team evaluates a rare-event classifier used for operational triage. The immediate design goal is to measure how often alerts predicted positive actually correspond to true incidents. Which approach addresses that goal?

  1. Precision metric
  2. Recall metric
  3. Confusion matrix
  4. F1 score

Explanation

Correct Answer

A:

Precision is true positives divided by all predicted positives. It matters when false alarms create substantial human review cost.

Incorrect Answers

B:

Recall is true positives divided by all actual positives; it matters when missed positive cases are particularly costly; the outstanding need is to measure how often alerts predicted positive actually correspond to true incidents.

C:

A confusion matrix exposes actual error types and supports cost-based threshold decisions; a single accuracy percentage hides which errors occurred; this approach leaves the team unable to measure how often alerts predicted positive actually correspond to true incidents.

D:

F1 is the harmonic mean of precision and recall; it can conceal threshold and subgroup tradeoffs, so both constituent metrics remain useful; the needed capability is to measure how often alerts predicted positive actually correspond to true incidents.

Question 194

The model team evaluates a rare-event classifier used for operational triage. The team is required to measure the fraction of actual incidents caught by the classifier. Which option is the best fit?

  1. Recall metric
  2. Precision metric
  3. PR AUC
  4. F1 score

Explanation

Correct Answer

A:

Recall is true positives divided by all actual positives. It matters when missed positive cases are particularly costly.

Incorrect Answers

B:

Precision is true positives divided by all predicted positives; it matters when false alarms create substantial human review cost; the requirement, by contrast, is to measure the fraction of actual incidents caught by the classifier.

C:

Area under the precision-recall curve is sensitive to positive-class performance; it is often more informative than overall accuracy for rare events; the team would still need to measure the fraction of actual incidents caught by the classifier.

D:

F1 is the harmonic mean of precision and recall; it can conceal threshold and subgroup tradeoffs, so both constituent metrics remain useful; the specified outcome instead requires the team to measure the fraction of actual incidents caught by the classifier.

Question 195

The model team evaluates a rare-event classifier used for operational triage. An operator is tasked to balance precision and recall in one summary when both missed incidents and false alarms matter. Which mechanism provides that capability?

  1. ROC AUC
  2. PR AUC
  3. F1 score
  4. Recall metric

Explanation

Correct Answer

C:

F1 is the harmonic mean of precision and recall. It can conceal threshold and subgroup tradeoffs, so both constituent metrics remain useful.

Incorrect Answers

A:

ROC AUC summarizes true-positive versus false-positive tradeoffs across thresholds; severe imbalance may require complementary precision-recall analysis; the specified outcome instead requires the team to balance precision and recall in one summary when both missed incidents and false alarms matter.

B:

Area under the precision-recall curve is sensitive to positive-class performance; it is often more informative than overall accuracy for rare events; the implementation under review needs to balance precision and recall in one summary when both missed incidents and false alarms matter.

D:

Recall is true positives divided by all actual positives; it matters when missed positive cases are particularly costly; this alternative does not achieve the requirement to balance precision and recall in one summary when both missed incidents and false alarms matter.

Question 196

The model team evaluates a rare-event classifier used for operational triage. The immediate design goal is to compare ranking behavior across classification thresholds when both positive and negative classes are represented. Which approach addresses that goal?

  1. Confusion matrix
  2. ROC AUC
  3. Recall metric
  4. Precision metric
  5. F1 score

Explanation

Correct Answer

B:

ROC AUC summarizes true-positive versus false-positive tradeoffs across thresholds. Severe imbalance may require complementary precision-recall analysis.

Incorrect Answers

A:

A confusion matrix exposes actual error types and supports cost-based threshold decisions; a single accuracy percentage hides which errors occurred; this alternative does not achieve the requirement to compare ranking behavior across classification thresholds when both positive and negative classes are represented.

C:

Recall is true positives divided by all actual positives; it matters when missed positive cases are particularly costly; the outstanding need is to compare ranking behavior across classification thresholds when both positive and negative classes are represented.

D:

Precision is true positives divided by all predicted positives; it matters when false alarms create substantial human review cost; this approach leaves the team unable to compare ranking behavior across classification thresholds when both positive and negative classes are represented.

E:

F1 is the harmonic mean of precision and recall; it can conceal threshold and subgroup tradeoffs, so both constituent metrics remain useful; the needed capability is to compare ranking behavior across classification thresholds when both positive and negative classes are represented.

Question 197

The model team evaluates a rare-event classifier used for operational triage. The team is required to evaluate a rare-positive event model using the precision-recall relationship across operating thresholds. Which option is the best fit?

  1. Recall metric
  2. ROC AUC
  3. Confusion matrix
  4. PR AUC

Explanation

Correct Answer

D:

Area under the precision-recall curve is sensitive to positive-class performance. It is often more informative than overall accuracy for rare events.

Incorrect Answers

A:

Recall is true positives divided by all actual positives; it matters when missed positive cases are particularly costly; this approach leaves the team unable to evaluate a rare-positive event model using the precision-recall relationship across operating thresholds.

B:

ROC AUC summarizes true-positive versus false-positive tradeoffs across thresholds; severe imbalance may require complementary precision-recall analysis; the needed capability is to evaluate a rare-positive event model using the precision-recall relationship across operating thresholds.

C:

A confusion matrix exposes actual error types and supports cost-based threshold decisions; a single accuracy percentage hides which errors occurred; the requirement, by contrast, is to evaluate a rare-positive event model using the precision-recall relationship across operating thresholds.

Question 198

The model team evaluates a rare-event classifier used for operational triage. An operator is tasked to inspect counts of true positives, false positives, true negatives, and false negatives at a chosen threshold. Which mechanism provides that capability?

  1. Confusion matrix
  2. ROC AUC
  3. Recall metric
  4. Precision metric

Explanation

Correct Answer

A:

A confusion matrix exposes actual error types and supports cost-based threshold decisions. A single accuracy percentage hides which errors occurred.

Incorrect Answers

B:

ROC AUC summarizes true-positive versus false-positive tradeoffs across thresholds; severe imbalance may require complementary precision-recall analysis; the specified outcome instead requires the team to inspect counts of true positives, false positives, true negatives, and false negatives at a chosen threshold.

C:

Recall is true positives divided by all actual positives; it matters when missed positive cases are particularly costly; the implementation under review needs to inspect counts of true positives, false positives, true negatives, and false negatives at a chosen threshold.

D:

Precision is true positives divided by all predicted positives; it matters when false alarms create substantial human review cost; the requested task remains to inspect counts of true positives, false positives, true negatives, and false negatives at a chosen threshold.

Question 199

Deployed models must remain useful as real-world inputs change over time. The immediate design goal is to identify changes in input feature distributions even when current ground-truth labels are delayed. Which approach addresses that goal?

  1. Concept drift analysis
  2. Label-based performance decay
  3. Covariate drift detection
  4. Reference window selection

Explanation

Correct Answer

C:

Covariate drift compares feature distributions between reference and recent observations. It does not automatically prove that predictive accuracy has degraded.

Incorrect Answers

A:

Concept drift concerns the underlying predictive relationship; simple feature distribution comparisons can miss it when marginal distributions appear stable; the requested task remains to identify changes in input feature distributions even when current ground-truth labels are delayed.

B:

True performance monitoring requires observed outcomes linked to predictions; a drift score alone is only a proxy for possible quality deterioration; this alternative does not achieve the requirement to identify changes in input feature distributions even when current ground-truth labels are delayed.

D:

Reference windows define what the monitor treats as normal; choosing an abnormal baseline can create misleading alerts even with valid statistical tests; the outstanding need is to identify changes in input feature distributions even when current ground-truth labels are delayed.

Question 200

Deployed models must remain useful as real-world inputs change over time. The team is required to investigate a change in the relationship between features and the target outcome. Which option is the best fit?

  1. Reference window selection
  2. Prediction distribution tracking
  3. Concept drift analysis
  4. Label-based performance decay

Explanation

Correct Answer

C:

Concept drift concerns the underlying predictive relationship. Simple feature distribution comparisons can miss it when marginal distributions appear stable.

Incorrect Answers

A:

Reference windows define what the monitor treats as normal; choosing an abnormal baseline can create misleading alerts even with valid statistical tests; the outstanding need is to investigate a change in the relationship between features and the target outcome.

B:

Prediction monitoring reveals output shifts that merit investigation; it does not directly measure precision without corresponding labels; this approach leaves the team unable to investigate a change in the relationship between features and the target outcome.

D:

True performance monitoring requires observed outcomes linked to predictions; a drift score alone is only a proxy for possible quality deterioration; the requirement, by contrast, is to investigate a change in the relationship between features and the target outcome.

Operationalizing Machine Learning and Generative AI Solutions (AI-300) Practice Tests