Answering my own question. First, colAUC has parameter alg which allows options of "Wilcoxon" or "ROC". The "ROC" option computes the AUC by integrating the ROC curve using the trapezoid rule, which is what I would expect, and it does not give an error for larger samples, e.g. > colAUC(runif(1000000),...

Unless you have some implementation bug (test your code with synthetic, well separated data), the problem might lay in the class imbalance. This can be solved by adjusting the missclassification cost (See this discussion in CV). I'd use the cost parameter of fitcsvm to increase the missclassification cost of the...

You can use the add = TRUE argument the plot function to plot multiple ROC curves. Make up some fake data library(pROC) a=rbinom(100, 1, 0.25) b=runif(100) c=rnorm(100) Get model fits fit1=glm(a~b+c, family='binomial') fit2=glm(a~c, family='binomial') Predict on the same data you trained the model with (or hold some out to test...