Prediction Accuracy from Stability Assessment Results

Description

Function to compute the prediction accuracy from an object of class “stablelearner” or “stablelearnerList” as a parallel to the similarity values estimated by stability in each iteration of the stability assessment procedure.

Usage

  accuracy(x, measure = "kappa", na.action = na.exclude, 
       applyfun = NULL, cores = NULL) 

Arguments

x	an object of class “stablelearner” or “stablelearnerList”.
measure	a character string (or a vector of character strings). Name(s) of the measure(s) used to compute accuracy. Currently implemented measures are “diag” = percentage of observations on the main diagonal of a confusion matrix, “kappa” = “diag” corrected for agreement by chance (default), “rand” = Rand index, and “crand” = Rand index corrected for agreemend by chance (see also classAgreement).
na.action	a function which indicates what should happen to the predictions of the results containing NAs. The default function is na.exclude.
applyfun	a lapply-like function. The default is to use lapply unless cores is specified in which case mclapply is used (for multicore computations on platforms that support these).
cores	integer. The number of cores to use in multicore computations using mclapply (see above).

Details

This function can be used to compute prediction accuracy after the stability was estimated using stability.

Value

A matrix of size 2*B times length(measure) containing prediction accuracy values of the learners trained during the stability assessment procedure.

See Also

stability

Examples

library("stablelearner")



library("partykit")
res <- ctree(Species ~ ., data = iris)
stab <- stability(res)
accuracy(stab)

[[1]]
          [,1]      [,2] [,3]      [,4] [,5]      [,6]      [,7]      [,8] [,9]
[1,] 1.0000000 0.6796657    1 0.9211618    1 0.8301887 0.9196787 0.9418345    1
[2,] 0.8496241 0.8639053    1 0.9211618    1 0.9162791 0.9206349 0.9418345    1
         [,10]     [,11]     [,12]     [,13]    [,14]     [,15] [,16]     [,17]
[1,] 0.7378641 0.9006211 0.9159292 0.9375000 0.923913 0.8543046     1 0.9260563
[2,] 0.8235294 1.0000000 0.8362069 0.8740157 0.923913 0.8543046     1 1.0000000
         [,18]     [,19]     [,20]     [,21]     [,22]     [,23]     [,24]
[1,] 0.8967742 0.9303797 0.9420935 0.7309942 0.8139535 0.9081081 0.9058824
[2,] 0.8967742 0.9303797 0.9420935 0.7982456 0.9053254 0.9081081 0.7192982
         [,25] [,26]     [,27]     [,28] [,29]     [,30]     [,31] [,32]
[1,] 0.8224299     1 0.7257143 0.8815789     1 0.8712871 0.8834081     1
[2,] 1.0000000     1 1.0000000 0.8815789     1 1.0000000 0.8834081     1
         [,33] [,34]     [,35]     [,36] [,37]     [,38]     [,39]     [,40]
[1,] 0.9247312 0.875 0.9285714 1.0000000     1 0.8269231 0.9130435 0.9283276
[2,] 0.9247312 0.875 0.9285714 0.8888889     1 0.8235294 1.0000000 0.8571429
         [,41]     [,42]     [,43]     [,44] [,45]     [,46]     [,47]    [,48]
[1,] 0.7283951 0.9305994 0.9396135 0.8967742     1 0.8000000 0.9114583 0.862500
[2,] 0.7300613 1.0000000 0.8798077 0.8967742     1 0.8979592 0.7343750 0.863354
         [,49] [,50]     [,51]     [,52]     [,53]     [,54]     [,55]
[1,] 1.0000000     1 0.8278146 0.8818182 1.0000000 0.8285714 0.9233449
[2,] 0.9148936     1 0.9423503 0.8818182 0.9166667 0.8285714 0.8493151
         [,56]     [,57]     [,58]     [,59] [,60] [,61]     [,62]     [,63]
[1,] 0.9233716 1.0000000 0.7443946 0.9000000     1     1 1.0000000 1.0000000
[2,] 0.9233716 0.9150943 0.9147982 0.8045977     1     1 0.9344729 0.9322034
     [,64]     [,65] [,66]     [,67]     [,68]     [,69] [,70] [,71]     [,72]
[1,]     1 0.7866667     1 0.8849558 0.9166667 1.0000000     1     1 0.8839286
[2,]     1 0.7866667     1 0.8849558 0.8333333 0.8101266     1     1 0.7719298
     [,73] [,74]     [,75]     [,76]     [,77]     [,78]     [,79] [,80] [,81]
[1,]     1     1 0.9285714 0.9191489 0.9205021 0.9206349 0.9422222   0.9     1
[2,]     1     1 0.7885906 0.9191489 1.0000000 0.9206349 0.9422222   1.0     1
        [,82] [,83]     [,84]     [,85]     [,86] [,87] [,88]     [,89]
[1,] 0.941573     1 0.8455882 0.8235294 0.9208333     1 0.936 0.8389831
[2,] 1.000000     1 0.8455882 0.8235294 1.0000000     1 0.875 0.8389831
        [,90]     [,91]     [,92]    [,93]     [,94] [,95]     [,96] [,97]
[1,] 0.931677 0.9055556 0.8606061 0.933526 0.9208333     1 0.9413093     1
[2,] 1.000000 1.0000000 0.9319527 0.933526 0.8423237     1 0.8828829     1
         [,98]     [,99]    [,100]    [,101]    [,102]    [,103] [,104]
[1,] 0.9182879 0.7678571 0.8152174 0.9317507 0.9357326 0.9370079      1
[2,] 0.9182879 0.8839286 0.8152174 1.0000000 0.8746867 0.8740157      1
        [,105]   [,106]    [,107]    [,108] [,109] [,110] [,111]    [,112]
[1,] 0.8736842 0.765625 0.8616352 0.8636364      1      1      1 0.9065934
[2,] 0.8736842 0.765625 0.8616352 0.8636364      1      1      1 0.9065934
     [,113] [,114]    [,115]    [,116]    [,117] [,118]    [,119]    [,120]
[1,]      1      1 0.9068627 0.9252669 0.8607595      1 0.9222222 0.8987342
[2,]      1      1 0.9068627 0.8500000 0.8607595      1 0.9210526 0.8987342
        [,121]    [,122]    [,123]   [,124]    [,125] [,126]    [,127]
[1,] 0.9188034 0.9285714 0.9373368 0.924812 0.9156118      1 0.8303571
[2,] 1.0000000 0.8571429 1.0000000 0.924812 0.8319328      1 0.8303571
        [,128]    [,129]    [,130]    [,131]    [,132]    [,133]    [,134]
[1,] 1.0000000 0.9348442 0.8095238 0.9285714 0.8541667 0.8309859 0.9278351
[2,] 0.8709677 0.8050847 0.8095238 1.0000000 0.7789474 0.9142857 1.0000000
        [,135]    [,136]    [,137] [,138]    [,139]    [,140]    [,141]
[1,] 0.9212598 0.8507463 0.9166667      1 0.9307479 0.9278689 0.7904762
[2,] 1.0000000 0.8507463 0.9166667      1 0.9307479 0.8589744 0.7904762
        [,142]    [,143]    [,144] [,145] [,146] [,147]    [,148]    [,149]
[1,] 0.7477876 1.0000000 1.0000000      1      1      1 0.8472222 0.9071038
[2,] 0.9132420 0.9142857 0.9104478      1      1      1 0.7659574 0.9071038
        [,150]    [,151]    [,152]    [,153]   [,154]    [,155] [,156]   [,157]
[1,] 0.7275542 0.8847007 1.0000000 0.9210526 0.915493 0.9301587  0.888 0.923913
[2,] 0.7828947 0.8274336 0.9307229 0.9210526 0.915493 0.9301587  0.888 0.923913
        [,158] [,159]    [,160]    [,161]    [,162] [,163]    [,164] [,165]
[1,] 0.9076923      1 0.9344262 0.9060773 0.9269103      1 1.0000000      1
[2,] 0.9076923      1 0.9344262 0.8068182 0.8533333      1 0.9222615      1
        [,166] [,167]  [,168]    [,169]    [,170]    [,171]    [,172]    [,173]
[1,] 0.8923077      1 0.90625 0.8743719 0.8541667 0.9166667 1.0000000 0.8662791
[2,] 0.8923077      1 0.90625 0.8780488 0.8541667 0.9166667 0.9384236 0.8662791
       [,174]    [,175]    [,176]    [,177]    [,178] [,179] [,180]    [,181]
[1,] 0.922179 0.9081081 1.0000000 0.9301587 0.7261905      1 0.8625 1.0000000
[2,] 0.922179 0.9081081 0.9173913 0.8580645 0.7988338      1 0.8625 0.9190283
        [,182] [,183] [,184]    [,185]    [,186]    [,187]  [,188]    [,189]
[1,] 0.9130435      1      1 0.9090909 0.9044944 1.0000000 0.90625 0.8301887
[2,] 1.0000000      1      1 0.9090909 1.0000000 0.8551724 0.90625 0.8301887
        [,190] [,191]    [,192]    [,193] [,194]    [,195]    [,196] [,197]
[1,] 0.8039773      1 0.9247312 0.8315789      1 0.9342466 0.8888889      1
[2,] 0.8039773      1 0.9252669 0.9183673      1 0.8702703 1.0000000      1
     [,198]    [,199] [,200]    [,201]    [,202]    [,203]    [,204]    [,205]
[1,] 0.8625 0.8056338      1 0.7313433 0.8217822 0.9073171 0.8837209 1.0000000
[2,] 0.8625 0.8050847      1 0.7313433 0.9016393 0.8199052 0.8837209 0.8056338
     [,206]    [,207]   [,208]    [,209]    [,210]    [,211]    [,212]
[1,]      1 0.8860759 0.915493 0.8588957 0.9348958 0.9346591 0.9188034
[2,]      1 1.0000000 0.915493 0.8606061 0.8697917 0.9346591 0.9188034
        [,213]    [,214]    [,215]    [,216]    [,217]    [,218]    [,219]
[1,] 0.9184549 0.9285714 0.8589744 0.9015544 0.8473282 0.7435897 0.9393204
[2,] 0.8403361 0.8589744 0.7904762 0.9015544 0.8473282 0.7435897 0.9393204
        [,220] [,221]    [,222]    [,223] [,224]    [,225]    [,226] [,227]
[1,] 0.9283388      1 0.8988095 0.8936170      1 0.9278351 0.9263158      1
[2,] 0.8589744      1 1.0000000 0.8979592      1 0.7849829 0.9263158      1
        [,228]    [,229]    [,230]    [,231] [,232]    [,233]    [,234]
[1,] 1.0000000 0.9227799 0.9510703 0.9368687      1 0.8389831 0.9428008
[2,] 0.9159292 1.0000000 0.9025875 0.9368687      1 0.8389831 0.9428008
        [,235]   [,236] [,237]   [,238]    [,239]    [,240]    [,241]    [,242]
[1,] 0.9314642 1.000000      1 0.934555 0.9170306 0.5065789 0.8478261 0.7615063
[2,] 0.8633540 0.915493      1 0.934555 1.0000000 0.5833333 0.8478261 0.9211618
        [,243]    [,244]    [,245]    [,246]   [,247]    [,248] [,249]
[1,] 0.9333333 0.8562092 0.8285714 0.8888889 0.923913 0.9162791      1
[2,] 0.9333333 0.8562092 0.8333333 0.8888889 0.923913 0.8317757      1
        [,250]   [,251] [,252]    [,253]  [,254]    [,255]    [,256]    [,257]
[1,] 1.0000000 0.863354      1 0.9041916 0.84375 1.0000000 0.9095745 0.8685714
[2,] 0.8773585 0.863354      1 0.7090909 0.84375 0.9114583 0.8210526 0.8662791
        [,258] [,259] [,260] [,261]    [,262]    [,263]    [,264]    [,265]
[1,] 0.9399038      1      1      1 0.8191489 0.9400480 0.9270833 0.9240924
[2,] 0.8798077      1      1      1 0.8191489 0.8201439 0.8531469 0.8476821
        [,266] [,267]    [,268] [,269] [,270]    [,271]    [,272] [,273]
[1,] 0.7348837      1 0.9196787      1      1 0.9215686 0.8389831      1
[2,] 0.6576577      1 0.9196787      1      1 0.9215686 0.9198312      1
        [,274]    [,275]    [,276]    [,277]    [,278]   [,279]    [,280]
[1,] 0.8693182 0.8594249 0.9441118 0.9109948 0.9317507 0.863354 0.9205021
[2,] 0.8005780 0.8589744 0.8387716 1.0000000 0.9317507 0.863354 0.8429752
        [,281]    [,282]    [,283]   [,284]    [,285]    [,286]    [,287]
[1,] 0.7883436 0.8074866 1.0000000 0.915493 0.9230769 0.9218750 0.7628458
[2,] 0.8588957 0.8074866 0.9368421 1.000000 0.9230769 0.8406375 0.8449612
     [,288]    [,289]    [,290]    [,291]    [,292]    [,293]    [,294]
[1,]      1 1.0000000 0.9261745 0.9275862 0.9242424 0.9245283 0.8362069
[2,]      1 0.9385343 0.9261745 0.7842466 0.8484848 0.9245283 0.8362069
        [,295] [,296]    [,297]    [,298]    [,299]    [,300]    [,301]
[1,] 0.9230769      1 0.8753247 0.8730159 1.0000000 0.9346591 0.9169675
[2,] 0.9230769      1 0.9375000 0.8730159 0.9349593 0.9346591 0.9178571
        [,302]    [,303]    [,304] [,305]    [,306] [,307]    [,308]    [,309]
[1,] 0.9053254 0.8823529 0.8541667      1 1.0000000      1 1.0000000 1.0000000
[2,] 0.9053254 0.8823529 0.7781690      1 0.9236641      1 0.9301587 0.9208333
     [,310]    [,311]    [,312] [,313] [,314]    [,315]    [,316] [,317]
[1,]      1 0.8217822 0.8654971      1      1 0.9443299 0.9370079      1
[2,]      1 0.8260870 0.8654971      1      1 0.9443299 0.9370079      1
        [,318]    [,319]    [,320]    [,321] [,322]    [,323]    [,324] [,325]
[1,] 0.9170124 0.9365079 0.9136364 0.9326146      1 0.8412409 1.0000000      1
[2,] 0.9170124 1.0000000 1.0000000 0.9326146      1 0.7890909 0.8896552      1
        [,326] [,327] [,328]    [,329]    [,330]    [,331]    [,332]    [,333]
[1,] 0.8507463      1    0.8 0.9247312 0.9018405 0.9260563 0.7159763 0.9151786
[2,] 0.8507463      1    0.8 0.9247312 0.9018405 0.8541667 0.7159763 1.0000000
        [,334]    [,335] [,336]    [,337]    [,338]    [,339]    [,340]
[1,] 0.9384236 0.8844444      1 0.9114583 0.9208333 1.0000000 0.8134715
[2,] 0.9384236 0.8844444      1 1.0000000 1.0000000 0.9206349 0.9090909
        [,341]    [,342]    [,343] [,344]    [,345]    [,346]    [,347]
[1,] 0.8024691 0.9320113 0.8000000      1 0.8571429 0.9109948 0.7547170
[2,] 0.8024691 0.9320113 0.9027027      1 1.0000000 0.9109948 0.6176471
        [,348]    [,349]    [,350]    [,351] [,352] [,353] [,354]    [,355]
[1,] 0.9119171 0.8730159 1.0000000 0.9294872      1      1      1 0.8743455
[2,] 1.0000000 0.8730159 0.9150943 0.8607595      1      1      1 0.9373368
        [,356]    [,357]    [,358]    [,359]    [,360] [,361]    [,362] [,363]
[1,] 0.9109948 1.0000000 0.8841871 0.8796296 0.9310345      1 0.8982036  0.808
[2,] 0.9081081 0.9201681 0.9423503 1.0000000 0.8616352      1 1.0000000  0.808
     [,364]    [,365]    [,366] [,367]    [,368]    [,369]    [,370]    [,371]
[1,]      1 0.8616352 0.8487973 0.8625 0.8551724 0.9119171 0.9230769 1.0000000
[2,]      1 0.8616352 0.9230769 0.8625 0.9260563 0.9119171 0.9230769 0.9191489
        [,372]    [,373] [,374]    [,375] [,376] [,377]    [,378]    [,379]
[1,] 0.8836207 1.0000000      1 0.8285714      1      1 1.0000000 0.9204545
[2,] 0.7711864 0.9278351      1 1.0000000      1      1 0.9036145 0.9204545
        [,380] [,381] [,382]    [,383]    [,384] [,385]    [,386] [,387]
[1,] 0.9365079      1      1 0.8571429 0.9005848      1 0.8571429      1
[2,] 0.9365079      1      1 0.8571429 0.8089888      1 0.8571429      1
        [,388]    [,389]    [,390]    [,391]    [,392]  [,393]    [,394]
[1,] 0.9285714 0.7232704 0.9105263 0.9116279 0.7124183 0.79375 0.9081081
[2,] 1.0000000 0.8625000 0.9086022 0.9116279 0.7843137 0.79375 0.9039548
        [,395]    [,396]    [,397]    [,398] [,399]    [,400]    [,401]
[1,] 0.9114583 0.9104478 0.9184549 0.8739496      1 0.9191489 0.8507463
[2,] 0.8229167 0.9104478 0.9184549 0.7321429      1 0.8403361 0.8507463
        [,402]    [,403]    [,404]    [,405]    [,406]    [,407] [,408] [,409]
[1,] 0.9208333 0.9245283 0.6818182 0.6571429 0.8736842 0.7744361      1      1
[2,] 0.9208333 0.8484848 0.6793893 0.6571429 1.0000000 0.8490566      1      1
        [,410]    [,411]    [,412]    [,413]    [,414]    [,415]    [,416]
[1,] 0.9245283 0.9060773 0.8607595 0.9303797 1.0000000 0.9383562 0.7357513
[2,] 0.9242424 1.0000000 0.9294872 0.9303797 0.8551724 0.9383562 0.8210526
        [,417]    [,418]    [,419]    [,420]    [,421] [,422]   [,423]
[1,] 0.8449612 0.9270833 0.8235294 0.9441118 0.9314642      1 1.000000
[2,] 0.8449612 0.9270833 0.9104478 0.9441118 0.9314642      1 0.915493
        [,424] [,425] [,426]    [,427]    [,428]    [,429]    [,430]    [,431]
[1,] 0.9239544      1      1 0.7785235 0.9360614 0.9177489 0.8910506 0.8347826
[2,] 0.8473282      1      1 1.0000000 1.0000000 1.0000000 0.8910506 0.9147982
        [,432] [,433] [,434]    [,435]    [,436]    [,437]    [,438] [,439]
[1,] 0.8333333      1      1 1.0000000 0.8011527 0.8467153 0.9211618      1
[2,] 0.9150943      1      1 0.9329446 0.9325513 0.9222222 0.8416667      1
        [,440]    [,441]    [,442]    [,443]    [,444]    [,445] [,446]
[1,] 0.9095745 1.0000000 0.8743719 0.9414414 1.0000000 1.0000000      1
[2,] 0.9095745 0.9058824 0.8768473 1.0000000 0.9030303 0.9183673      1
        [,447]   [,448]   [,449]    [,450]    [,451]    [,452]    [,453] [,454]
[1,] 0.9211618 0.915493 1.000000 1.0000000 0.9166667 0.8285714 0.9191489      1
[2,] 0.9211618 0.915493 0.862069 0.9090909 1.0000000 0.9150943 1.0000000      1
     [,455] [,456]    [,457]    [,458]    [,459]    [,460] [,461]    [,462]
[1,]      1      1 0.9420935 0.7945205 0.8191489 0.9394673      1 0.9047619
[2,]      1      1 1.0000000 0.6000000 0.8191489 0.9394673      1 1.0000000
        [,463]   [,464]    [,465]    [,466]    [,467]    [,468]    [,469]
[1,] 0.9109948 1.000000 0.9319527 1.0000000 0.8700565 0.9211618 0.9285714
[2,] 0.9109948 0.913242 0.8654971 0.9081081 0.8056338 1.0000000 0.9285714
     [,470]    [,471]    [,472]    [,473]    [,474]    [,475]    [,476] [,477]
[1,]      1 0.9183673 0.8790698 0.8979592 0.9308176 0.9278351 1.0000000      1
[2,]      1 1.0000000 0.8823529 0.7972973 0.8611987 0.7827586 0.9255319      1
        [,478] [,479]    [,480]    [,481]    [,482] [,483]    [,484]    [,485]
[1,] 0.9393204      1 0.9296636 0.9333333 0.9188034      1 1.0000000 1.0000000
[2,] 0.8783455      1 0.7976540 0.9333333 0.9188034      1 0.9173913 0.9310345
        [,486]    [,487]    [,488]    [,489]    [,490]    [,491]    [,492]
[1,] 0.9511041 1.0000000 0.9278351 0.9417040 0.9184549 0.7668919 0.9472727
[2,] 0.9025157 0.9250936 1.0000000 0.9422222 0.9184549 0.7668919 0.7319778
        [,493]    [,494]    [,495]    [,496]    [,497]    [,498]    [,499]
[1,] 0.9227799 0.8743455 0.9236641 0.8571429 0.8309859 0.8607595 0.9290323
[2,] 0.9227799 0.8743455 0.9236641 0.9243986 0.9154930 0.9297125 0.9290323
     [,500]
[1,]      1
[2,]      1

attr(,"measure")
[1] "kappa"