mantispy.metrics.diagnose_testing

mantispy.metrics.diagnose_testing#

mantispy.metrics.diagnose_testing(adata, groupby='Metadata_Perturbation', reference='negcon', block='Metadata_Plate', n_draws=8, alpha=0.05, seed=0, n_permutations=200)#

Check whether differential testing is calibrated on this screen.

Parameters:

  • adata (AnnData) – Well-level profiles after the normalization and transform you plan to test with, since the results depend on both.

  • groupby (str (default: 'Metadata_Perturbation')) – As in differential_features().

  • reference (str | None (default: 'negcon')) – As in differential_features().

  • block (str | None (default: 'Metadata_Plate')) – As in differential_features().

  • n_draws (int (default: 8)) – Pseudo-treatments drawn from the controls for the empirical null. More draws resolve the false positive rate better and take longer.

  • alpha (float (default: 0.05)) – Nominal rate the null is compared against.

  • seed (int (default: 0)) – Seed for choosing which control wells stand in for a treatment. The two hit callers’ permutation nulls are seeded by the draw index instead, so they are identical across calls that differ only in seed.

  • n_permutations (int (default: 200)) – Null size for the two hit callers; smaller is faster and coarser.

Return type:

DataFrame

Returns:

A frame with columns check, value, expected, verdict and note, one row per check that ran, where a FAIL verdict means the check does not hold on this data. The empirical-null rows are absent when every null p-value came back non-finite, and the two hit-caller rows need at least eight reference wells.

Raises:

  • ValueError – The object is annotated at cell resolution, which none of these checks describe.

  • ValueError – No treatment has two wells, or there are fewer than four reference wells, leaving nothing to measure a null against.

Notes

The checks and what each one detects:

null p < 0.05 / null p < 0.01

Control wells relabeled as treatments of the size yours have. The rate should match the nominal one. Heavy tails distort small p-values first, so a test can be calibrated at 0.05 and not at 0.01, which is closer to the range a false discovery rate works in.

null discoveries

How many of those null p-values survive Benjamini-Hochberg. A count above zero means the q-values on the real data are optimistic by roughly that much.

hit_calling null rate / edistance null rate

The same relabeling applied to the two hit callers, counted over n_draws draws. Both are permutation tests that are not fully calibrated at small control counts, so the count is compared against the upper tail of Binomial(n_draws, alpha) instead of a fixed rate. At eight draws the smallest non-zero rate is 0.125, and a threshold below that would fail a calibrated screen a third of the time. Raising n_draws sharpens the answer and moves the cutoff with it.

rank test resolution

The smallest p-value a Mann-Whitney test can return at your replication, compared with what multiple-testing correction requires. With three wells against 14 reference wells the floor is 2.9e-03 whatever the effect size, and effect_size() then silently calls nothing.

excess kurtosis

How far the features are from the normality a t-test assumes. It predicts the null checks above but is not a verdict on its own, since heavy tails matter less with enough wells per group.

wells per treatment and treatments sharing a {block} with the reference

The replicate structure the other checks depend on. A treatment whose wells share no block with the reference cannot be tested.