Prediction Accuracy from Stability Assessment Results

Description

Function to compute the prediction accuracy from an object of class “stablelearner” or “stablelearnerList” as a parallel to the similarity values estimated by stability in each iteration of the stability assessment procedure.

Usage

  accuracy(x, measure = "kappa", na.action = na.exclude, 
       applyfun = NULL, cores = NULL) 

Arguments

x	an object of class “stablelearner” or “stablelearnerList”.
measure	a character string (or a vector of character strings). Name(s) of the measure(s) used to compute accuracy. Currently implemented measures are “diag” = percentage of observations on the main diagonal of a confusion matrix, “kappa” = “diag” corrected for agreement by chance (default), “rand” = Rand index, and “crand” = Rand index corrected for agreemend by chance (see also classAgreement).
na.action	a function which indicates what should happen to the predictions of the results containing NAs. The default function is na.exclude.
applyfun	a lapply-like function. The default is to use lapply unless cores is specified in which case mclapply is used (for multicore computations on platforms that support these).
cores	integer. The number of cores to use in multicore computations using mclapply (see above).

Details

This function can be used to compute prediction accuracy after the stability was estimated using stability.

Value

A matrix of size 2*B times length(measure) containing prediction accuracy values of the learners trained during the stability assessment procedure.

See Also

stability

Examples

library("stablelearner")



library("partykit")
res <- ctree(Species ~ ., data = iris)
stab <- stability(res)
accuracy(stab)

[[1]]
          [,1] [,2]      [,3]      [,4]      [,5]      [,6] [,7]      [,8] [,9]
[1,] 1.0000000    1 0.7904762 1.0000000 0.9340974 0.9415584    1 0.7510917    1
[2,] 0.9109948    1 0.9297125 0.9032258 0.9340974 0.9415584    1 0.8362069    1
         [,10]     [,11]     [,12]     [,13]     [,14] [,15]    [,16]     [,17]
[1,] 0.9422222 0.8860759 0.9305994 0.9159292 0.9166667     1 0.915493 1.0000000
[2,] 0.9422222 0.8860759 0.9305994 0.9159292 1.0000000     1 0.915493 0.7986577
         [,18]     [,19] [,20]     [,21] [,22] [,23]     [,24]     [,25]
[1,] 0.9119171 0.8958333     1 0.8571429     1     1 0.8923077 0.8393881
[2,] 0.8229167 0.8958333     1 0.9285714     1     1 0.8923077 1.0000000
         [,26]     [,27]     [,28] [,29] [,30]     [,31]     [,32]     [,33]
[1,] 0.9370079 0.8662791 1.0000000     1     1 0.9090909 0.8188406 0.8571429
[2,] 0.9370079 0.9327485 0.9162791     1     1 0.9090909 1.0000000 0.8622754
         [,34] [,35]     [,36] [,37]     [,38]    [,39]    [,40]     [,41]
[1,] 0.9100529     1 0.9314642  0.84 0.9283489 0.924812 0.915493 0.8561644
[2,] 1.0000000     1 0.8633540  1.00 0.9283489 0.924812 0.915493 1.0000000
         [,42]     [,43]     [,44]     [,45]     [,46]     [,47]     [,48]
[1,] 0.9233716 0.9105691 0.9236641 0.9250936 0.8333333 0.7789474 0.9177489
[2,] 0.7718631 0.9105691 0.8461538 0.9250936 0.8333333 0.8510638 0.9177489
         [,49]     [,50] [,51]     [,52]     [,53]     [,54] [,55]     [,56]
[1,] 0.8761905 0.7994652     1 0.8285714 0.9105263 0.9342466     1 0.9230769
[2,] 0.8761905 0.7994652     1 0.9126214 0.9105263 0.8736842     1 0.7741935
         [,57]     [,58]     [,59]     [,60]     [,61]     [,62]     [,63]
[1,] 0.8839286 0.8210526 0.9283276 0.8896552 0.9100529 1.0000000 0.9064327
[2,] 0.8839286 0.8210526 1.0000000 0.8000000 0.9100529 0.9423077 0.9064327
     [,64]     [,65]     [,66]     [,67]     [,68]     [,69]     [,70]
[1,]     1 0.8928571 1.0000000 1.0000000 0.8152174 0.9346591 0.9247312
[2,]     1 1.0000000 0.8807339 0.9170306 0.8152174 1.0000000 1.0000000
         [,71]     [,72] [,73]     [,74] [,75]     [,76]     [,77]     [,78]
[1,] 0.9242424 1.0000000     1 0.8979592     1 0.8737864 0.8095238 0.7647059
[2,] 0.9248120 0.8792271     1 0.8979592     1 1.0000000 0.9036145 0.8449612
         [,79] [,80]     [,81]     [,82]     [,83] [,84] [,85]     [,86] [,87]
[1,] 0.7884615   0.9 0.8219895 0.8737864 0.8432836     1     1 0.9206349     1
[2,] 0.7884615   0.9 1.0000000 1.0000000 1.0000000     1     1 0.9206349     1
         [,88]     [,89] [,90]     [,91] [,92]     [,93]     [,94]     [,95]
[1,] 0.8461538 0.6691176     1 0.9230769     1 0.9222222 0.9132420 0.9370079
[2,] 0.8461538 0.7000000     1 1.0000000     1 1.0000000 0.8240741 0.8733509
         [,96] [,97]     [,98]     [,99]    [,100]    [,101]    [,102] [,103]
[1,] 1.0000000     1 0.8786408 0.8333333 0.7931034 0.8813559 0.7125749      1
[2,] 0.9325843     1 0.8786408 0.7464789 0.7223975 0.8813559 0.7125749      1
        [,104] [,105]    [,106]   [,107] [,108]    [,109]    [,110] [,111]
[1,] 0.8403361      1 0.8730159 0.940367      1 0.9346591 0.9346591      1
[2,] 0.8423237      1 1.0000000 1.000000      1 0.9346591 0.9346591      1
        [,112]    [,113]    [,114]    [,115]    [,116]    [,117] [,118] [,119]
[1,] 0.8798077 0.7162162 0.9344729 0.9259259 0.9218750 0.8285714      1      1
[2,] 0.8798077 0.8531469 0.9337176 0.9259259 0.8449612 0.8181818      1      1
        [,120] [,121] [,122]    [,123]    [,124]    [,125]    [,126]    [,127]
[1,] 0.8931298      1      1 0.9236364 0.9263158 0.9278351 0.9260563 0.7678571
[2,] 1.0000000      1      1 1.0000000 1.0000000 0.9278351 1.0000000 0.9417040
        [,128]    [,129]    [,130]    [,131]    [,132]    [,133] [,134] [,135]
[1,] 0.7455357 0.7956656 1.0000000 0.9173913 0.9247312 0.8541667      1      1
[2,] 0.7455357 0.8641975 0.8340611 1.0000000 0.8478261 0.8541667      1      1
        [,136]    [,137]    [,138] [,139]    [,140] [,141]    [,142]    [,143]
[1,] 1.0000000 0.8576271 0.8507463  0.936 0.9211618      1 0.8589744 0.8693182
[2,] 0.9208333 0.9283276 0.9245283  1.000 0.9211618      1 1.0000000 0.8693182
     [,144]    [,145]    [,146]    [,147]    [,148]    [,149]    [,150]
[1,]      1 0.8288288 0.9271523 0.9399038 0.8702703 0.8317757 0.9349593
[2,]      1 0.8288288 0.9271523 0.9399038 0.8129870 0.8317757 1.0000000
        [,151]   [,152]    [,153]    [,154]    [,155]    [,156]    [,157]
[1,] 1.0000000 0.915493 0.9346591 0.8979592 0.9281046 0.9358289 0.8994413
[2,] 0.9283276 0.915493 1.0000000 0.8979592 0.9281046 0.9358289 0.8994413
        [,158] [,159] [,160]    [,161]    [,162]    [,163] [,164]    [,165]
[1,] 0.8709677      1  0.875 1.0000000 0.8198614 0.9071038      1 0.9250936
[2,] 0.8695652      1  0.875 0.8993289 0.8785047 0.9071038      1 0.9250936
        [,166] [,167]    [,168] [,169]    [,170] [,171]    [,172]    [,173]
[1,] 0.8461538      1 0.9159292      1 0.9432773      1 1.0000000 0.9086294
[2,] 0.8437500      1 0.8333333      1 0.9432773      1 0.8551724 0.9062500
       [,174]    [,175] [,176]   [,177]   [,178] [,179]    [,180]    [,181]
[1,] 1.000000 0.9270833      1 0.890625 1.000000      1 0.9342857 0.7917981
[2,] 0.921875 0.8551724      1 1.000000 0.915493      1 1.0000000 0.7917981
        [,182]    [,183]    [,184]    [,185] [,186]    [,187]    [,188]
[1,] 0.8217822 1.0000000 0.7928177 0.8666667      1 0.7662338 1.0000000
[2,] 0.9117647 0.7607656 0.8595506 0.8016529      1 0.8258065 0.9030303
        [,189]    [,190]    [,191]    [,192]    [,193]    [,194]    [,195]
[1,] 0.7590361 0.9208333 0.9119171 0.9305136 0.9219858 0.9348442 0.8571429
[2,] 0.9190283 1.0000000 0.7384615 0.9305136 0.7066667 0.9344729 0.7878788
        [,196]    [,197]    [,198]    [,199] [,200]    [,201] [,202]    [,203]
[1,] 0.7969231 0.9076923 0.9349593 0.9191489      1 0.9047619      1 0.8243243
[2,] 0.9318885 1.0000000 0.8080000 0.8410042      1 0.9047619      1 0.7709251
        [,204]    [,205] [,206]    [,207]    [,208]    [,209]    [,210]
[1,] 0.9242424 0.9278351      1 0.9333333 0.8589744 0.9036145 0.8256881
[2,] 0.9242424 1.0000000      1 0.9333333 0.9297125 0.9058824 0.8256881
        [,211]  [,212]    [,213] [,214]    [,215]    [,216]    [,217]    [,218]
[1,] 0.7391304 0.79375 0.9177489      1 0.9292605 0.9242424 0.8403361 1.0000000
[2,] 0.8235294 0.79375 0.9177489      1 0.8513514 0.9242424 0.9201681 0.9177489
       [,219]    [,220]    [,221]    [,222] [,223]    [,224] [,225]    [,226]
[1,] 0.915493 0.9203187 0.9153226 0.9301587      1 1.0000000      1 1.0000000
[2,] 0.915493 0.9203187 1.0000000 1.0000000      1 0.8552632      1 0.9370277
     [,227]    [,228] [,229]   [,230]    [,231]    [,232]    [,233]    [,234]
[1,]      1 1.0000000      1 0.915493 0.8625000 1.0000000 0.8693182 0.8131868
[2,]      1 0.9270833      1 0.915493 0.9292605 0.9201681 0.8685714 1.0000000
        [,235]    [,236]    [,237]    [,238]    [,239] [,240]    [,241]
[1,] 0.9275862 1.0000000 0.7464789 0.9183673 0.8473282      1 0.9208333
[2,] 0.9275862 0.8863636 0.7464789 0.9215686 0.8473282      1 1.0000000
        [,242]    [,243]    [,244]    [,245]    [,246]   [,247]    [,248]
[1,] 0.9151786 0.9278351 0.9206349 0.8979592 0.9290123 0.915493 1.0000000
[2,] 0.9151786 0.9278351 0.9206349 1.0000000 0.9290123 0.915493 0.9314642
     [,249]    [,250]    [,251]    [,252]   [,253] [,254]    [,255]    [,256]
[1,]      1 0.8589744 0.8017241 1.0000000 0.893617      1 0.7709091 1.0000000
[2,]      1 0.7891374 0.8017241 0.8551724 0.893617      1 0.9195402 0.9105263
        [,257]    [,258]    [,259] [,260]    [,261] [,262]    [,263]    [,264]
[1,] 0.9128440 0.9260563 0.8589744      1 0.9342857      1 1.0000000 0.5692308
[2,] 0.8303571 0.9239130 0.8589744      1 0.9342857      1 0.9184549 0.6769231
     [,265]    [,266] [,267] [,268]    [,269] [,270]   [,271]    [,272]
[1,]      1 0.9283276      1      1 0.9095745      1 0.931677 0.9201521
[2,]      1 0.9283276      1      1 0.9095745      1 0.931677 0.9201521
        [,273]    [,274]    [,275]    [,276]    [,277]    [,278]    [,279]
[1,] 0.9260563 0.8229167 1.0000000 0.9210526 1.0000000 0.9211618 0.8074866
[2,] 0.9260563 0.8229167 0.9011628 0.9210526 0.9422222 0.9211618 0.8074866
     [,280]    [,281]    [,282]    [,283] [,284] [,285]    [,286]    [,287]
[1,]      1 1.0000000 0.9263158 0.8068182      1    0.9 0.8396624 0.8678161
[2,]      1 0.8376068 0.8500000 1.0000000      1    0.8 1.0000000 0.9340974
        [,288]    [,289]    [,290]    [,291] [,292] [,293]   [,294]    [,295]
[1,] 1.0000000 1.0000000 1.0000000 1.0000000      1      1 0.915493 0.9081081
[2,] 0.8693182 0.8576271 0.9245283 0.8685714      1      1 0.915493 0.9081081
        [,296]    [,297] [,298] [,299] [,300]   [,301]    [,302]  [,303]
[1,] 0.9342857 1.0000000      1      1      1 0.734375 0.9142857 1.00000
[2,] 0.9335260 0.8551724      1      1      1 0.734375 0.9142857 0.90625
       [,304]    [,305]    [,306]    [,307] [,308]    [,309] [,310]  [,311]
[1,] 0.843750 0.7835052 0.7532468 0.8516129      1 0.9201681      1 1.00000
[2,] 0.921875 0.7820069 0.9194915 0.8516129      1 1.0000000      1 0.93125
        [,312]    [,313]    [,314] [,315]    [,316] [,317]    [,318] [,319]
[1,] 0.8473282 0.8473282 0.7086093      1 0.9099526      1 0.8674352      1
[2,] 0.9227799 0.8473282 0.7086093      1 0.9073171      1 0.9335260      1
        [,320]    [,321] [,322]    [,323]    [,324]    [,325] [,326] [,327]
[1,] 1.0000000 0.9142857      1 0.8285714 1.0000000 0.9325513      1      1
[2,] 0.9205021 0.8269231      1 0.8285714 0.6734694 0.9325513      1      1
        [,328]    [,329]    [,330]    [,331]    [,332]    [,333]    [,334]
[1,] 0.8622754 0.8026316 0.9154930 0.9270833 0.8673469 0.9323529 0.7545455
[2,] 0.8622754 0.8026316 0.8309859 0.9270833 0.7500000 0.9323529 0.6785714
        [,335]    [,336]    [,337]    [,338]    [,339] [,340]    [,341]
[1,] 0.8403361 0.9042553 0.9400480 0.8513514 0.8496241      1 0.8625000
[2,] 0.7625000 0.9042553 0.8798077 1.0000000 0.8496241      1 0.8552632
        [,342] [,343]    [,344]    [,345]    [,346]    [,347]    [,348] [,349]
[1,] 0.8444444      1 0.7877814 0.9388753 0.9198312 0.9301587 0.9367089  1.000
[2,] 0.9239130      1 0.7877814 0.9373434 0.9198312 0.9301587 1.0000000  0.875
     [,350]   [,351]    [,352]    [,353]    [,354]    [,355]    [,356]
[1,]      1 0.795858 0.9255319 0.7945205 0.8931298 0.8000000 0.9180328
[2,]      1 0.795858 1.0000000 0.8979592 0.8931298 0.8972603 0.9180328
        [,357]   [,358] [,359]    [,360]    [,361] [,362] [,363] [,364] [,365]
[1,] 0.7532468 0.912844 1.0000 0.9239544 0.8851351  1.000   0.92      1      1
[2,] 0.8362069 0.912844 0.9375 1.0000000 0.6812500  0.875   0.92      1      1
        [,366] [,367]    [,368] [,369]    [,370]    [,371] [,372] [,373]
[1,] 0.9105263      1 0.8336756      1 0.8000000 0.7789474      1    0.8
[2,] 0.9105263      1 0.8881988      1 0.9018405 0.8510638      1    1.0
        [,374]    [,375]    [,376]    [,377]    [,378]    [,379]    [,380]
[1,] 0.9278351 0.8400000 0.8360656 1.0000000 0.7329843 0.8689459 0.9194915
[2,] 0.8571429 0.8449612 0.8360656 0.9433962 0.7329843 0.9344729 1.0000000
        [,381]    [,382]    [,383] [,384]    [,385]    [,386] [,387] [,388]
[1,] 0.9245283 0.8818182 0.8588957      1 0.9344262 1.0000000      1      1
[2,] 0.8490566 0.7636364 0.7278107      1 0.8074866 0.7862595      1      1
        [,389]    [,390]    [,391]    [,392]    [,393] [,394]    [,395] [,396]
[1,] 0.7911392 0.9365482 0.9319527 0.7377049 0.8923077      1 0.8688525    1.0
[2,] 0.7898089 0.8768473 0.8005780 0.8021978 0.8923077      1 0.8709677    0.9
     [,397]    [,398]    [,399]    [,400]    [,401] [,402]    [,403] [,404]
[1,]      1 0.9173913 0.7261905 0.8390805 0.8641975      1 0.7981651      1
[2,]      1 0.9173913 0.9298780 0.7846154 1.0000000      1 0.7981651      1
        [,405]    [,406]   [,407]    [,408]    [,409] [,410]   [,411]    [,412]
[1,] 0.7244898 0.9183673 0.781250 0.9011628 0.8403361      1 1.000000 0.7750000
[2,] 0.8977273 0.9183673 0.890625 1.0000000 0.8403361      1 0.915493 0.7721519
      [,413]    [,414]    [,415]    [,416] [,417] [,418] [,419]    [,420]
[1,] 0.91133 0.9188034 0.9406393 0.8531469      1      1      1 0.9018405
[2,] 1.00000 1.0000000 1.0000000 0.9255319      1      1      1 1.0000000
        [,421]    [,422]    [,423]    [,424]    [,425]    [,426]    [,427]
[1,] 1.0000000 1.0000000 0.7058824 0.9348442 1.0000000 0.9443299 1.0000000
[2,] 0.9184549 0.9107981 0.7058824 0.8700565 0.9422222 0.9443299 0.9314642
        [,428]    [,429] [,430]    [,431] [,432]    [,433]    [,434]    [,435]
[1,] 0.8798077 0.9391727      1 0.9285714      1 0.9312500 1.0000000 0.8756477
[2,] 0.8798077 0.8774510      1 1.0000000      1 0.8616352 0.9318885 0.8756477
        [,436]    [,437]    [,438] [,439] [,440] [,441] [,442] [,443] [,444]
[1,] 0.8191489 0.9136364 0.8720000      1      1      1      1      1      1
[2,] 0.9081081 0.9136364 0.9338843      1      1      1      1      1      1
        [,445]    [,446]    [,447]    [,448]    [,449]    [,450]    [,451]
[1,] 0.9215686 0.8333333 0.7816092 0.9242424 0.7916667 0.9230769 0.7835052
[2,] 0.8461538 1.0000000 0.5410628 1.0000000 0.8936170 0.9230769 0.8561644
        [,452] [,453] [,454]    [,455]    [,456]    [,457]    [,458]    [,459]
[1,] 0.7857143      1      1 0.9252669 0.9373368 0.8479087 0.9285714 0.9140271
[2,] 0.7857143      1      1 1.0000000 0.9373368 0.9227799 1.0000000 1.0000000
     [,460] [,461]    [,462]   [,463]    [,464] [,465]    [,466]    [,467]
[1,]      1      1 1.0000000 0.931677 0.9514867    1.0 0.8496241 0.8285714
[2,]      1      1 0.8562092 1.000000 0.9514867    0.7 0.8496241 0.9154930
        [,468]    [,469]    [,470]    [,471] [,472]    [,473]    [,474]  [,475]
[1,] 0.9354839 0.8139535 0.9086022 0.8496241      1 0.6865672 0.9159292 1.00000
[2,] 0.9354839 0.9058824 0.9086022 0.8496241      1 0.7878788 1.0000000 0.91133
        [,476]    [,477]    [,478]    [,479]    [,480]    [,481] [,482]
[1,] 0.9166667 0.9278351 0.9198312 0.9297125 0.8940397 0.8410042      1
[2,] 0.9166667 0.9278351 0.9198312 0.9297125 0.8940397 0.9191489      1
        [,483] [,484] [,485]    [,486]    [,487]    [,488]    [,489]    [,490]
[1,] 1.0000000      1      1 0.9119171 0.8510638 0.9381188 1.0000000 0.8210526
[2,] 0.9305994      1      1 1.0000000 0.8510638 1.0000000 0.9230769 0.9114583
        [,491]    [,492]    [,493]    [,494]    [,495]    [,496]    [,497]
[1,] 0.9162791 0.8704663 0.8318584 0.9343832 0.8312236 0.8210526 0.8736842
[2,] 0.9162791 1.0000000 0.8318584 0.9343832 0.9180328 0.8210526 0.8723404
     [,498]    [,499] [,500]
[1,]      1 0.6724138   0.85
[2,]      1 0.9177489   0.85

attr(,"measure")
[1] "kappa"