This vignette is made for users that are already familiar with the basic condiments workflow described in the first vignette. Here, we will show how to modify the default parameters for the first two steps of the workflow

# For analysis
library(condiments)
library(slingshot)
set.seed(21)

Toy dataset

We rely on the same toy dataset as the first vignette

data("toy_dataset", package = "condiments")
df <- toy_dataset$sd
rd <- as.matrix(df[, c("Dim1", "Dim2")])
sds <- slingshot(rd, df$cl)

The topologyTest function

By default, the topologyTest function requires only two inputs, the sds object and the condition labels. To limit run time for the vignette, we also change the default number of permutations used to generate trajectories under the null by setting the rep argument to \(10\) instead of the default \(100\). As such, the test statistics might be more variable.

top_res <- topologyTest(sds = sds, conditions = df$conditions, rep = 10)
## Generating permuted trajectories
## Running KS-mean test
knitr::kable(top_res)
method thresh statistic p.value
KS_mean 0.01 0 1

Changing the method or the threshold

The topologyTest function can be relatively slow on large datasets. Moreover, when changing the method used to test the null hypothesis that a common trajectory should be fitted, the first permutation part of generating rep trajectories under the null is identical. Therefore, we allow users to specify more than one method and one value of the threshold. Here, we will use both the Kolmogorov-Smirnov test test(Smirnov 1939) and the classifier-test(Lopez-Paz and Oquab 2016).

top_res <- topologyTest(sds = sds, conditions = df$conditions, rep = 10,
                        methods = c("KS_mean", "Classifier"),
                        threshs = c(0, .01, .05, .1))
## Generating permuted trajectories
## Running KS-mean test
## Running Classifier test
## Loading required package: lattice
## Loading required package: ggplot2
knitr::kable(top_res)
method thresh statistic p.value
KS_mean 0 0.0070000 1.0000000
KS_mean 0.01 0.0000000 1.0000000
KS_mean 0.05 0.0000000 1.0000000
KS_mean 0.1 0.0000000 1.0000000
Classifier 0 0.3816667 1.0000000
Classifier 0.01 0.4066667 0.9999972
Classifier 0.05 0.3333333 1.0000000
Classifier 0.1 0.2583333 1.0000000

To see all methods avaible, use /usr/local/lib/R/site-library/condiments/help/topologyTest and look at the methods argument.

Passing arguments to the test method

For all methods but the KS test, additional paramters can be specified, using a custom argument: args_classifier, args_wass or args_mmd. See the help file for given test more information on those parameters. For example, since the default test based on the wasserstein distance and permutation test is quite slow, we can pass a fast argument.

top_res <- topologyTest(sds = sds, conditions = df$conditions, rep = 10,
                        methods = "wasserstein_permutation",
                        args_wass = list(fast = TRUE, S = 100, iterations  = 10^2))
## Generating permuted trajectories
## Running wassertsein permutation test
knitr::kable(top_res)
method thresh statistic p.value
wasserstein_permutation NA 1.525219 0.72

Using parallelisation

For now, the first part of the topologyTest has been designed for parallelisation using the BiocParallel package. For example, to run with 4 cores, you can run the following command

library(BiocParallel)
BPPARAM <- bpparam()
BPPARAM$progressbar <- TRUE
BPPARAM$workers <- 4
top_res <- topologyTest(sds = sds, conditions = df$conditions, rep = 100, 
                        parallel = TRUE, BPPARAM = BPPARAM)
knitr::kable(top_res)

Differential progression and differentiation

The tests for the second test are much less compute-intensive, therefore there is no parallelisation. However, the other changes introduce in the previous section are still possible

Default

prog_res <- progressionTest(sds, conditions = df$conditions)
knitr::kable(prog_res)
lineage statistic p.value
All 6.665207 0
dif_res <- differentiationTest(sds, conditions = df$conditions)
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
## 
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
knitr::kable(dif_res)
pair statistic p.value
1vs2 0.6521622 2.9e-06

Changing the method and / or threshold

prog_res <- progressionTest(sds, conditions = df$conditions, method = "Classifier")
knitr::kable(prog_res)
lineage statistic p.value
All 0.5800901 0.0093097
dif_res <- differentiationTest(sds, conditions = df$conditions, thresh = .05)
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
## 
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
knitr::kable(dif_res)
pair statistic p.value
1vs2 0.6211712 0.0001764

Passing more parameters to the test methods

prog_res <- progressionTest(sds, conditions = df$conditions, method = "Classifier",
                            args_classifier = list(method = "rf"))
## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range

## Warning in randomForest.default(x, y, mtry = param$mtry, ...): invalid mtry:
## reset to within valid range
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
knitr::kable(prog_res)
lineage statistic p.value
All 0.5845946 0.0064207
dif_res <- differentiationTest(sds, conditions = df$conditions)
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
## 
## note: only 1 unique complexity parameters in default grid. Truncating the grid to 1 .
knitr::kable(dif_res)
pair statistic p.value
1vs2 0.6206306 0.0001764

Conclusion

For all of the above procedures, it is important to note that we are making multiple comparisons. The p-values we obtain from these tests should be corrected for multiple testing, especially for trajectories with a large number of lineages.

That said, trajectory inference is often one of the last computational methods in a very long analysis pipeline (generally including gene-level quantification, gene filtering / feature selection, and dimensionality reduction). Hence, we strongly discourage the reader from putting too much faith in any p-value that comes out of this analysis. Such values may be useful suggestions, indicating particular features or cells for follow-up study, but should generally not be treated as meaningful statistical quantities.

If some commands and parameters are still unclear after going through this vignette, do not hesitate to open an issue on the condiments Github repository.

Session Info

## R Under development (unstable) (2021-04-05 r80145)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.2 LTS
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] parallel  stats4    stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] caret_6.0-86                 ggplot2_3.3.3               
##  [3] lattice_0.20-41              slingshot_1.99.3            
##  [5] TrajectoryUtils_0.99.2       SingleCellExperiment_1.13.12
##  [7] SummarizedExperiment_1.21.2  Biobase_2.51.0              
##  [9] GenomicRanges_1.43.4         GenomeInfoDb_1.27.9         
## [11] IRanges_2.25.6               S4Vectors_0.29.12           
## [13] BiocGenerics_0.37.1          MatrixGenerics_1.3.1        
## [15] matrixStats_0.58.0           princurve_2.1.6             
## [17] condiments_0.99.14           knitr_1.31                  
## 
## loaded via a namespace (and not attached):
##   [1] colorspace_2.0-0          deldir_0.2-10            
##   [3] ellipsis_0.3.1            class_7.3-18             
##   [5] rprojroot_2.0.2           XVector_0.31.1           
##   [7] fs_1.5.0                  proxy_0.4-25             
##   [9] spatstat.data_2.1-0       prodlim_2019.11.13       
##  [11] fansi_0.4.2               lubridate_1.7.10         
##  [13] sparseMatrixStats_1.3.7   codetools_0.2-18         
##  [15] splines_4.1.0             cachem_1.0.4             
##  [17] polyclip_1.10-0           jsonlite_1.7.2           
##  [19] pROC_1.17.0.1             kernlab_0.9-29           
##  [21] spatstat.linnet_2.1-1     spatstat.sparse_2.0-0    
##  [23] compiler_4.1.0            Matrix_1.3-2             
##  [25] fastmap_1.1.0             htmltools_0.5.1.1        
##  [27] tools_4.1.0               igraph_1.2.6             
##  [29] gtable_0.3.0              glue_1.4.2               
##  [31] GenomeInfoDbData_1.2.4    RANN_2.6.1               
##  [33] reshape2_1.4.4            dplyr_1.0.5              
##  [35] Rcpp_1.0.6                spatstat_2.1-0           
##  [37] jquerylib_0.1.3           pkgdown_1.6.1            
##  [39] vctrs_0.3.7               debugme_1.1.0            
##  [41] nlme_3.1-152              DelayedMatrixStats_1.13.5
##  [43] iterators_1.0.13          timeDate_3043.102        
##  [45] gower_0.2.2               xfun_0.22                
##  [47] stringr_1.4.0             Ecume_0.9.1              
##  [49] lifecycle_1.0.0           goftest_1.2-2            
##  [51] zlibbioc_1.37.0           MASS_7.3-53.1            
##  [53] scales_1.1.1              ipred_0.9-11             
##  [55] spatstat.core_2.0-0       ragg_1.1.2               
##  [57] spatstat.utils_2.1-0      yaml_2.2.1               
##  [59] memoise_2.0.0             pbapply_1.4-3            
##  [61] sass_0.3.1                rpart_4.1-15             
##  [63] stringi_1.5.3             highr_0.8                
##  [65] desc_1.3.0                randomForest_4.6-14      
##  [67] foreach_1.5.1             e1071_1.7-6              
##  [69] BiocParallel_1.25.5       lava_1.6.9               
##  [71] rlang_0.4.10              pkgconfig_2.0.3          
##  [73] systemfonts_1.0.1         bitops_1.0-6             
##  [75] evaluate_0.14             purrr_0.3.4              
##  [77] tensor_1.5                recipes_0.1.15           
##  [79] transport_0.12-2          tidyselect_1.1.0         
##  [81] plyr_1.8.6                magrittr_2.0.1           
##  [83] R6_2.5.0                  generics_0.1.0           
##  [85] DelayedArray_0.17.10      mgcv_1.8-34              
##  [87] pillar_1.5.1              withr_2.4.1              
##  [89] survival_3.2-10           abind_1.4-5              
##  [91] RCurl_1.98-1.3            nnet_7.3-15              
##  [93] tibble_3.1.0              crayon_1.4.1             
##  [95] utf8_1.2.1                spatstat.geom_2.0-1      
##  [97] rmarkdown_2.7             grid_4.1.0               
##  [99] data.table_1.14.0         ModelMetrics_1.2.2.2     
## [101] digest_0.6.27             textshaping_0.3.3        
## [103] munsell_0.5.0             bslib_0.2.4

References

Lopez-Paz, David, and Maxime Oquab. 2016. Revisiting Classifier Two-Sample Tests.” Arxiv, October, 1–15. http://arxiv.org/abs/1610.06545.
Smirnov, Nikolai V. 1939. “On the Estimation of the Discrepancy Between Empirical Curves of Distribution for Two Independent Samples.” Bull. Math. Univ. Moscou 2 (2): 3–14.