Repeated balanced kfold cross validation using caret in R
By : apfrogerio
Date : March 29 2020, 07:55 AM
will help you The answer is yes. If the method = "repeatedcv" it calls on the function createMultiFolds, which internally calls on createFolds, but n times as specified in repeats = n

Test set and train set for each fold in Caret cross validation
By : Ashwini Zinavo
Date : March 29 2020, 07:55 AM
Hope that helps I tried to understand the 5 fold cross validation algorithm in Caret package but I could not find out how to get train set and test set for each fold and I also could not find this from the similar suggested questions. Imagine if I want to do cross validation by random forest method, I do the following: code :
str(rfmodel)
names(rfmodel)
# [1] "method" "modelInfo" "modelType" "results" "pred"
# [6] "bestTune" "call" "dots" "metric" "control"
# [11] "finalModel" "preProcess" "trainingData" "resample" "resampledCM"
# [16] "perfNames" "maximize" "yLimits" "times" "levels"
# [21] "terms" "coefnames" "xlevels"
# Indexes of Hold Out Sets
rfmodel$control$indexOut
# Indexes of Train Sets for above hold outs
rfmodel$control$index

How does Caret generate an OLS model with Kfold cross validation?
By : Steven K
Date : March 29 2020, 07:55 AM
this will help Let's say I have some generic dataset for which an OLS regression is the best choice. So, I generate a model with some firstorder terms and decide to use Caret in R for my regression coefficient estimates and error estimates. , Here's reproducible data for your example. code :
library("caret")
my_data < iris
k10_cv < trainControl(method="cv", number=10)
set.seed(100)
ols_model < train(Sepal.Length ~ Sepal.Width + Petal.Length + Petal.Width,
data = my_data, trControl = k10_cv, method = "lm")
> ols_model$results
intercept RMSE Rsquared MAE RMSESD RsquaredSD MAESD
1 TRUE 0.3173942 0.8610242 0.2582343 0.03881222 0.04784331 0.02960042
> (ols_model$resample)
RMSE Rsquared MAE Resample
1 0.3386472 0.8954600 0.2503482 Fold01
2 0.3154519 0.8831588 0.2815940 Fold02
3 0.3167943 0.8904550 0.2441537 Fold03
4 0.2644717 0.9085548 0.2145686 Fold04
5 0.3769947 0.8269794 0.3070733 Fold05
6 0.3720051 0.7792611 0.2746565 Fold06
7 0.3258501 0.8095141 0.2647466 Fold07
8 0.2962375 0.8530810 0.2731445 Fold08
9 0.3059100 0.8351535 0.2611982 Fold09
10 0.2615792 0.9286246 0.2108592 Fold10
> mean(ols_model$resample$RMSE)==ols_model$results$RMSE
[1] TRUE
coef(lm(Sepal.Length ~ Sepal.Width + Petal.Length + Petal.Width, data = my_data))
(Intercept) Sepal.Width Petal.Length Petal.Width
1.8559975 0.6508372 0.7091320 0.5564827

How to get predictions for each fold in 10fold crossvalidation of the best tuned hyperparameters using caret package i
By : user2961092
Date : March 29 2020, 07:55 AM
may help you . One way to achieve your goal is to subset fit.svm$pred using the hyper parameters in fit.svm$bestTune, and then aggregate the desired measure by CV replicates. I will perform this using dplyr: code :
library(tidyverse)
library(caret)
fit.svm$pred %>%
filter(sigma == fit.svm$bestTune$sigma & C == fit.svm$bestTune$C) %>% #subset
mutate(fold = gsub("\\..*", "", Resample), #extract fold info from resample info
rep = gsub(".*\\.(.*)", "\\1", Resample)) %>% #extract replicate info from resample info
group_by(rep) %>% #group by replicate
summarise(rmse = RMSE(pred, obs)) #aggregate the desired measure
# A tibble: 3 x 2
rep rmse
<chr> <dbl>
1 Rep1 4.02
2 Rep2 3.96
3 Rep3 4.06
fit.svm$pred %>%
filter(sigma == fit.svm$bestTune$sigma & C == fit.svm$bestTune$C) %>%
separate(Resample, c("fold", "rep"), "\\.") %>%
group_by(rep) %>%
summarise(rmse = RMSE(obs, pred))
fit.svm$pred %>%
filter(sigma == fit.svm$bestTune$sigma & C == fit.svm$bestTune$C) %>%
write.csv("predictions.csv")

Regarding the Caret package in R when apply K fold cross validation
By : user3077919
Date : March 29 2020, 07:55 AM
Hope that helps It can be done by cross tabulating the observed and predicted values as follows, code :
table((mod_fit$pred)$obs,(mod_fit$pred)$pred)
Down Up
Down 125 477
Up 151 497
overall missclassification = (125+497)/250 = 0.4976
sensitivity = 497/(151+497) = 0.7770

