SmartX LabHKUSTstRSC
GitHub

A noise-corrected metric for evaluating spatial transcriptomics predicted from histology

Jiarui Ouyang†, Yihang Gao†, Cheng Jin, Yihui Wang, Yingxue Xu, Ling Liang, Fengtao Zhou, Zhiyuan Cai and Hao ChenThe Hong Kong University of Science and Technology · † Equal contribution

stRSC evaluates methods that predict spatial transcriptomics from H&E-stained histology. Compute the reproducible-signal correlation (RSC) of your own predictions in the browser or with the Python toolkit, and compare your scores with the evaluations in the paper.

Prediction methods are usually evaluated by the Pearson correlation between predictions and measured RNA expression. Across whole sections, the correlation overstates prediction quality by crediting methods for recognizing tissue regions. In the HER2ST breast cancer dataset, a class-mean baseline that predicts one value per tissue region reaches a whole-section Pearson r of 0.111, above the median of 11 published methods (0.108). At fine scales, the correlation understates prediction quality because sampling noise dominates sparse RNA counts.

RSC largely avoids overstating and understating prediction quality. The measured RNA molecules are split at random into two halves, which share the true expression and carry independent sampling noise. RSC scores a prediction against the signal that both halves share, at each spatial scale and within tissue regions or cell classes.

Calculate RSC on your dataResults of the paper
Noise correction across scalesFig. 1c, three colorectal cancer patients
0.40.60.81.0Correlation<128128–256256–5120.5–1k1–2k2–4k≥4kSpatial scale (µm)RSC at <128 µm: 0.997RSC at 128–256 µm: 0.996RSC at 256–512 µm: 0.996RSC at 0.5–1k µm: 0.996RSC at 1–2k µm: 0.996RSC at 2–4k µm: 0.996RSC at ≥4k µm: 0.996Pearson r with measured RNA expression at <128 µm: mean 0.516, patients 0.441 to 0.569Pearson r with measured RNA expression at 128–256 µm: mean 0.780, patients 0.704 to 0.847Pearson r with measured RNA expression at 256–512 µm: mean 0.921, patients 0.875 to 0.944Pearson r with measured RNA expression at 0.5–1k µm: mean 0.980, patients 0.965 to 0.993Pearson r with measured RNA expression at 1–2k µm: mean 0.995, patients 0.992 to 0.998Pearson r with measured RNA expression at 2–4k µm: mean 0.999, patients 0.997 to 0.999Pearson r with measured RNA expression at ≥4k µm: mean 1.000, patients 1.000 to 1.000RSC 0.997Pearson r 0.44–0.57

At the finest scale, sampling noise lowers the Pearson r between the true expression and measured RNA expression to 0.44–0.57, whereas RSC stays at 0.997.

Downsampled countsExtended Data Fig. 1d, Ridge–Phikon-v2 in nine Xenium samples
0.20.30.40.50.6Correlation11/21/41/8Fraction of molecules keptWhole-section Pearson r, 1 of the molecules: 0.509Whole-section Pearson r, 1/2 of the molecules: 0.481Whole-section Pearson r, 1/4 of the molecules: 0.448Whole-section Pearson r, 1/8 of the molecules: 0.408Fine-scale RSC, 1 of the molecules: 0.337Fine-scale RSC, 1/2 of the molecules: 0.354Fine-scale RSC, 1/4 of the molecules: 0.369Fine-scale RSC, 1/8 of the molecules: 0.386Fine-scale split-half reliability, 1 of the molecules: 0.416Fine-scale split-half reliability, 1/2 of the molecules: 0.360Fine-scale split-half reliability, 1/4 of the molecules: 0.305Fine-scale split-half reliability, 1/8 of the molecules: 0.2470.509Whole-section Pearson r0.337Fine-scale RSC0.416Fine-scale split-halfreliability

Removing molecules at random adds sampling noise to the counts while the predictions stay the same. Whole-section Pearson r falls from 0.509 to 0.408, whereas fine-scale RSC remains largely stable (0.337 to 0.386).

All figures and their data in Results

How RSC is computed

  1. Split

    Each RNA molecule is assigned at random to one of two halves. For a gene with N molecules in a spot, bin or cell, NA ~ Binomial(N, 0.5) and NB = N − NA. The split uses raw counts, before normalization.

  2. Decompose

    The prediction and both halves are decomposed into spatial scales, from broad tissue regions to neighboring spots or cells. The decomposition uses a Fourier basis on a regular grid, a neighbor graph on spots and a cell graph on single cells.

  3. Correlate

    At each scale b, C is the mean covariance of the prediction with the two halves, and Vmodel is the variance of the prediction. VRNA is the covariance between the two halves. All three are summed over genes and locations, and RSC is averaged over repeated splits.

RSCb = C / √(Vmodel · VRNA)

RSC computes the Pearson correlation of the prediction with the two halves but replaces the RNA variance with the split-half covariance VRNA, which estimates the variance of the reproducible signal. An RSC of 1 indicates that the prediction accounts for all reproducible variation at that scale, and an RSC of 0 reflects chance agreement. The correlation between the two halves, VRNA/√(VAVB), is the split-half reliability ρb. For halves of equal variance, the Pearson r of a prediction with the mean of the halves is RSCb · √(2ρb/(1 + ρb)). Pearson r therefore falls at fine scales, where counts are sparse and the split-half reliability is low. RSC follows the idea of Fourier shell correlation in cryo-electron microscopy, which compares two half-maps of the same structure. More in the user guide

Evaluate your own methods

Figure 1 of the paper, with an overview of RSC, the whole-section Pearson r of a class-mean baseline and 11 published methods in HER2ST, RSC and Pearson r with measured RNA expression of the true expression across spatial scales, and a benchmark of fine-scale predictions across spatial scales, tissue regions, cell classes and biological analyses.
Fig. 1 | Reproducible-signal correlation (RSC) largely avoids overstating and understating prediction quality. a, Overview of RSC. The measured RNA molecules are split at random into two halves with independent Poisson noise (1), the prediction and both halves are decomposed into broad, mid and fine scales (2), and the correlation of the prediction with each half is divided by the square root of the split-half reliability (3). b, Pitfall of whole-section correlation. Along a transect across tumor and other regions, a class-mean baseline predicts one value per region (left). Over 785 genes in eight HER2ST patients, the baseline reaches a whole-section Pearson r of 0.111, above the median of 11 published methods (0.108; right). c, Noise correction across scales. With the true expression as the prediction, Pearson r with measured RNA expression falls at fine scales because of sampling noise, whereas RSC stays near 1 from below 128 µm to 4,096 µm and above. Lines are means of three colorectal cancer patients. d, Benchmark of fine-scale predictions. In colorectal cancer, the RSC of Ridge–Phikon predictions falls from 0.800 at the broadest scale to 0.298 at the finest (spatial scales; Fig. 3). Normalized pairwise accuracy falls toward the 0% of a majority-order baseline once spot pairs are matched or compared within regions (tissue regions; Fig. 2). Fine-scale RSC within malignant cells separates three single-cell methods (cell classes; Fig. 4). Single-cell methods differ in the enrichment of proliferation hotspots, and the RSC of a spot-level model is lower within tertiary lymphoid structures (TLS) than between regions (biological analyses; Figs. 4e and 5f).

Results of the paper

  1. Fig. 2

    Tissue regions

    Whole-section correlation overstates prediction quality within tissue regions. Normalized pairwise accuracy is 0% for a majority-order baseline and 100% for the agreement between two RNA halves. In HER2ST, the median of 11 published methods falls from 28.5% on unmatched spot pairs to 6.6% on matched pairs within the same region. Method rankings within regions are nearly unrelated to rankings by whole-section correlation (Kendall's τb = 0.091).

    Fig. 2 in Results
  2. Fig. 3

    Spatial scales

    RSC shows that spot-level predictions generally capture broad regional expression patterns but miss most fine-scale differences between nearby spots. In colorectal cancer, the RSC of Ridge–Phikon predictions falls from 0.800 at scales of 4,096 µm and above to 0.298 below 128 µm. In the Xenium samples of breast, pancreatic and lung cancer, 10 of the 11 spot-level models that we trained score lower at the finest than at the broadest scale in all nine samples.

    Fig. 3 in Results
  3. Fig. 4

    Cell classes

    Within malignant cells, RSC distinguishes single-cell methods that capture part of the cell-to-cell variability from methods whose scores across all cells are inflated by cell-type composition. At the fine scale within malignant cells, GHIST reaches an RSC of 0.242, compared with 0.042 for DeepSpot2Cell and 0.020 for sCellST. Across all cells, a class-mean baseline outscores sCellST (fine-scale RSC 0.081 versus 0.036).

    Fig. 4 in Results
  4. Fig. 5

    Tertiary lymphoid structures

    A high AUROC for locating tertiary lymphoid structures (TLS) can overstate how well a method predicts expression inside the structures. Ridge–Phikon-v2 separates TLS from other spots with an AUROC of 0.93 to 0.99, and a class-mean baseline reaches 0.99 to 1.00. Within TLS, the RSC of Ridge–Phikon-v2 is 0.25 to 0.29, compared with 0.59 to 0.79 between regions.

    Fig. 5 in Results
  5. Fig. 6

    Pairwise difference loss

    Guided by RSC, we designed a pairwise difference loss that supervises expression differences between neighboring spots. Across 11 spot-level models in nine Xenium samples, mean fine-scale RSC rises from 0.194 with standard training to 0.205 with the loss. On neighboring spots, the loss raises fine-scale RSC by 0.009 more than on the same number of non-neighboring spot pairs.

    Fig. 6 in Results