Results

Rater Reliability

Instructions

___________________________________________________________________________________

1. The R package irr is described in the page.

2. Feature requests and bug reports can be made on my GitHub.

___________________________________________________________________________________

Interrater reliability
	n	rater	statistic	z	p
Light's Kappa	11	40	0.605	NaN	NaN

[3]

Fleiss Kappa
	n	rater	statistic	z	p
Fleiss' Kappa	11	40	0.603	105	< .001

[3]

Category-wise kappas
Category	Kappa	Z	p
1	0.728	67.5	< .001
2	0.242	22.4	< .001
3	0.611	56.6	< .001
4	0.631	58.4	< .001
5	0.820	76.0	< .001

[3]

Intraclass correlation coefficient
	Subjects	Raters	Subject variance	Rater variance	Residual variance	Consistency	Agreement
Value	11	40	1.57	0.00935	0.158	0.909	0.904
Note. The analysis was performed by 'psy::icc' function.

[4]

Bootstrap agreement
	95% CI
	Lower	Upper
Agreement	0.752	0.952

Intraclass Correlation Coefficient(ICC)
Model	Type	Unit	Subjects	Raters	ICC
oneway	agreement	single	11	40	0.904
Note. The analysis was performed by 'irr::icc' function.

[3]

F test for ICC
					95% CI
ICC	F	df1	df2	p	Lower	Upper
0.904	377	10.0	429	0.000	0.818	0.967
Note. H0: ICC = 0; H1: ICC > 0

[3]

Partial plot

[5]

Spearman Correlation

Spearman Correlation
	rater1	rater2	rater3	rater4	rater5	rater6	rater7	rater8	rater9	rater10	rater11	rater12	rater13	rater14	rater15	rater16	rater17	rater18	rater19	rater20	rater21	rater22	rater23	rater24	rater25	rater26	rater27	rater28	rater29	rater30	rater31	rater32	rater33	rater34	rater35	rater36	rater37	rater38	rater39	rater40
rater1	1.000	0.928	0.976	0.971	0.945	0.985	0.971	0.971	0.907	0.963	0.893	0.971	0.933	0.834	0.971	0.980	0.907	0.928	0.947	0.951	0.949	0.985	0.904	0.971	0.928	0.892	0.985	0.915	0.859	0.933	0.966	0.933	0.933	0.971	0.862	0.928	0.957	0.868	0.948	0.966
rater2	0.928	1.000	0.881	0.976	0.976	0.911	0.952	0.952	0.956	0.969	0.904	0.928	0.912	0.874	0.976	0.952	0.956	0.870	0.959	0.904	0.957	0.911	0.893	0.952	0.940	0.893	0.954	0.964	0.842	0.937	0.959	0.937	0.937	0.976	0.952	0.940	0.898	0.940	0.914	0.959
rater3	0.976	0.881	1.000	0.928	0.905	0.959	0.952	0.952	0.857	0.921	0.843	0.928	0.912	0.790	0.928	0.952	0.857	0.920	0.911	0.916	0.910	0.959	0.842	0.952	0.881	0.893	0.954	0.865	0.842	0.887	0.959	0.887	0.887	0.928	0.808	0.881	0.910	0.821	0.914	0.959
rater4	0.971	0.976	0.928	1.000	0.964	0.966	0.980	0.980	0.968	0.993	0.944	0.961	0.943	0.856	1.000	0.971	0.968	0.908	0.966	0.944	0.966	0.966	0.924	0.980	0.976	0.916	0.985	0.975	0.891	0.963	0.985	0.963	0.963	1.000	0.934	0.976	0.937	0.904	0.948	0.985
rater5	0.945	0.976	0.905	0.964	1.000	0.916	0.935	0.935	0.956	0.957	0.875	0.906	0.892	0.903	0.964	0.945	0.956	0.844	0.916	0.947	0.960	0.916	0.872	0.935	0.917	0.881	0.934	0.964	0.803	0.922	0.945	0.922	0.922	0.964	0.909	0.917	0.940	0.929	0.891	0.945
rater6	0.985	0.911	0.959	0.966	0.916	1.000	0.966	0.966	0.903	0.959	0.932	0.966	0.928	0.801	0.966	0.985	0.903	0.924	0.961	0.932	0.909	1.000	0.899	0.966	0.959	0.935	0.980	0.911	0.914	0.928	0.961	0.928	0.928	0.966	0.829	0.959	0.938	0.815	0.943	0.961
rater7	0.971	0.952	0.952	0.980	0.935	0.966	1.000	1.000	0.927	0.973	0.920	0.980	0.963	0.822	0.980	0.971	0.927	0.948	0.966	0.920	0.947	0.966	0.924	1.000	0.952	0.916	0.985	0.935	0.916	0.943	0.985	0.943	0.943	0.980	0.905	0.952	0.937	0.880	0.948	0.985
rater8	0.971	0.952	0.952	0.980	0.935	0.966	1.000	1.000	0.927	0.973	0.920	0.980	0.963	0.822	0.980	0.971	0.927	0.948	0.966	0.920	0.947	0.966	0.924	1.000	0.952	0.916	0.985	0.935	0.916	0.943	0.985	0.943	0.943	0.980	0.905	0.952	0.937	0.880	0.948	0.985
rater9	0.907	0.956	0.857	0.968	0.956	0.903	0.927	0.927	1.000	0.968	0.907	0.887	0.866	0.829	0.968	0.907	1.000	0.800	0.903	0.907	0.936	0.903	0.832	0.927	0.956	0.877	0.921	0.992	0.837	0.907	0.943	0.907	0.907	0.968	0.923	0.956	0.916	0.857	0.859	0.943
rater10	0.963	0.969	0.921	0.993	0.957	0.959	0.973	0.973	0.968	1.000	0.951	0.954	0.908	0.807	0.993	0.963	0.968	0.918	0.959	0.951	0.947	0.959	0.906	0.973	0.969	0.906	0.978	0.968	0.862	0.928	0.978	0.928	0.928	0.993	0.910	0.969	0.923	0.872	0.912	0.978
rater11	0.893	0.904	0.843	0.944	0.875	0.932	0.920	0.920	0.907	0.951	1.000	0.895	0.863	0.729	0.944	0.922	0.907	0.896	0.932	0.912	0.834	0.932	0.904	0.920	0.976	0.940	0.921	0.923	0.901	0.888	0.927	0.888	0.888	0.944	0.813	0.976	0.836	0.771	0.864	0.927
rater12	0.971	0.928	0.928	0.961	0.906	0.966	0.980	0.980	0.887	0.954	0.895	1.000	0.943	0.787	0.961	0.971	0.887	0.948	0.966	0.895	0.928	0.966	0.924	0.980	0.928	0.868	0.985	0.895	0.891	0.923	0.947	0.923	0.923	0.961	0.876	0.928	0.937	0.855	0.948	0.947
rater13	0.933	0.912	0.912	0.943	0.892	0.928	0.963	0.963	0.866	0.908	0.863	0.943	1.000	0.895	0.943	0.933	0.866	0.893	0.928	0.863	0.941	0.928	0.945	0.963	0.912	0.887	0.947	0.885	0.929	0.980	0.948	0.980	0.980	0.943	0.909	0.912	0.912	0.912	0.985	0.948
rater14	0.834	0.874	0.790	0.856	0.903	0.801	0.822	0.822	0.829	0.807	0.729	0.787	0.895	1.000	0.856	0.834	0.829	0.689	0.801	0.817	0.913	0.801	0.846	0.822	0.802	0.788	0.816	0.847	0.762	0.930	0.835	0.930	0.930	0.856	0.876	0.802	0.865	0.947	0.891	0.835
rater15	0.971	0.976	0.928	1.000	0.964	0.966	0.980	0.980	0.968	0.993	0.944	0.961	0.943	0.856	1.000	0.971	0.968	0.908	0.966	0.944	0.966	0.966	0.924	0.980	0.976	0.916	0.985	0.975	0.891	0.963	0.985	0.963	0.963	1.000	0.934	0.976	0.937	0.904	0.948	0.985
rater16	0.980	0.952	0.952	0.971	0.945	0.985	0.971	0.971	0.907	0.963	0.922	0.971	0.933	0.834	0.971	1.000	0.907	0.928	0.985	0.922	0.925	0.985	0.904	0.971	0.952	0.940	0.985	0.915	0.899	0.933	0.966	0.933	0.933	0.971	0.862	0.952	0.928	0.868	0.948	0.966
rater17	0.907	0.956	0.857	0.968	0.956	0.903	0.927	0.927	1.000	0.968	0.907	0.887	0.866	0.829	0.968	0.907	1.000	0.800	0.903	0.907	0.936	0.903	0.832	0.927	0.956	0.877	0.921	0.992	0.837	0.907	0.943	0.907	0.907	0.968	0.923	0.956	0.916	0.857	0.859	0.943
rater18	0.928	0.870	0.920	0.908	0.844	0.924	0.948	0.948	0.800	0.918	0.896	0.948	0.893	0.689	0.908	0.928	0.800	1.000	0.924	0.896	0.848	0.924	0.919	0.948	0.870	0.858	0.942	0.819	0.837	0.853	0.924	0.853	0.853	0.908	0.785	0.870	0.839	0.797	0.887	0.924
rater19	0.947	0.959	0.911	0.966	0.916	0.961	0.966	0.966	0.903	0.959	0.932	0.966	0.928	0.801	0.966	0.985	0.903	0.924	1.000	0.874	0.909	0.961	0.899	0.966	0.959	0.935	0.980	0.911	0.914	0.928	0.961	0.928	0.928	0.966	0.886	0.959	0.880	0.863	0.943	0.961
rater20	0.951	0.904	0.916	0.944	0.947	0.932	0.920	0.920	0.907	0.951	0.912	0.895	0.863	0.817	0.944	0.922	0.907	0.896	0.874	1.000	0.906	0.932	0.904	0.920	0.904	0.868	0.921	0.923	0.783	0.888	0.927	0.888	0.888	0.944	0.813	0.904	0.923	0.843	0.864	0.927
rater21	0.949	0.957	0.910	0.966	0.960	0.909	0.947	0.947	0.936	0.947	0.834	0.928	0.941	0.913	0.966	0.925	0.936	0.848	0.909	0.906	1.000	0.909	0.895	0.947	0.898	0.824	0.951	0.939	0.810	0.961	0.952	0.961	0.961	0.966	0.964	0.898	0.938	0.957	0.946	0.952
rater22	0.985	0.911	0.959	0.966	0.916	1.000	0.966	0.966	0.903	0.959	0.932	0.966	0.928	0.801	0.966	0.985	0.903	0.924	0.961	0.932	0.909	1.000	0.899	0.966	0.959	0.935	0.980	0.911	0.914	0.928	0.961	0.928	0.928	0.966	0.829	0.959	0.938	0.815	0.943	0.961
rater23	0.904	0.893	0.842	0.924	0.872	0.899	0.924	0.924	0.832	0.906	0.904	0.924	0.945	0.846	0.924	0.904	0.832	0.919	0.899	0.904	0.895	0.899	1.000	0.924	0.893	0.842	0.917	0.863	0.862	0.945	0.899	0.945	0.945	0.924	0.861	0.893	0.875	0.893	0.939	0.899
rater24	0.971	0.952	0.952	0.980	0.935	0.966	1.000	1.000	0.927	0.973	0.920	0.980	0.963	0.822	0.980	0.971	0.927	0.948	0.966	0.920	0.947	0.966	0.924	1.000	0.952	0.916	0.985	0.935	0.916	0.943	0.985	0.943	0.943	0.980	0.905	0.952	0.937	0.880	0.948	0.985
rater25	0.928	0.940	0.881	0.976	0.917	0.959	0.952	0.952	0.956	0.969	0.976	0.928	0.912	0.802	0.976	0.952	0.956	0.870	0.959	0.904	0.898	0.959	0.893	0.952	1.000	0.952	0.954	0.964	0.939	0.937	0.959	0.937	0.937	0.976	0.880	1.000	0.898	0.821	0.914	0.959
rater26	0.892	0.893	0.893	0.916	0.881	0.935	0.916	0.916	0.877	0.906	0.940	0.868	0.887	0.788	0.916	0.940	0.877	0.858	0.935	0.868	0.824	0.935	0.842	0.916	0.952	1.000	0.905	0.889	0.939	0.887	0.935	0.887	0.887	0.916	0.782	0.952	0.836	0.774	0.876	0.935
rater27	0.985	0.954	0.954	0.985	0.934	0.980	0.985	0.985	0.921	0.978	0.921	0.985	0.947	0.816	0.985	0.985	0.921	0.942	0.980	0.921	0.951	0.980	0.917	0.985	0.954	0.905	1.000	0.929	0.892	0.947	0.980	0.947	0.947	0.985	0.904	0.954	0.927	0.881	0.962	0.980
rater28	0.915	0.964	0.865	0.975	0.964	0.911	0.935	0.935	0.992	0.968	0.923	0.895	0.885	0.847	0.975	0.915	0.992	0.819	0.911	0.923	0.939	0.911	0.863	0.935	0.964	0.889	0.929	1.000	0.853	0.926	0.951	0.926	0.926	0.975	0.928	0.964	0.917	0.877	0.878	0.951
rater29	0.859	0.842	0.842	0.891	0.803	0.914	0.916	0.916	0.837	0.862	0.901	0.891	0.929	0.762	0.891	0.899	0.837	0.837	0.914	0.783	0.810	0.914	0.862	0.916	0.939	0.939	0.892	0.853	1.000	0.904	0.899	0.904	0.904	0.891	0.796	0.939	0.842	0.744	0.905	0.899
rater30	0.933	0.937	0.887	0.963	0.922	0.928	0.943	0.943	0.907	0.928	0.888	0.923	0.980	0.930	0.963	0.933	0.907	0.853	0.928	0.888	0.961	0.928	0.945	0.943	0.937	0.887	0.947	0.926	0.904	1.000	0.948	1.000	1.000	0.963	0.938	0.937	0.912	0.937	0.985	0.948
rater31	0.966	0.959	0.959	0.985	0.945	0.961	0.985	0.985	0.943	0.978	0.927	0.947	0.948	0.835	0.985	0.966	0.943	0.924	0.961	0.927	0.952	0.961	0.899	0.985	0.959	0.935	0.980	0.951	0.899	0.948	1.000	0.948	0.948	0.985	0.915	0.959	0.909	0.887	0.943	1.000
rater32	0.933	0.937	0.887	0.963	0.922	0.928	0.943	0.943	0.907	0.928	0.888	0.923	0.980	0.930	0.963	0.933	0.907	0.853	0.928	0.888	0.961	0.928	0.945	0.943	0.937	0.887	0.947	0.926	0.904	1.000	0.948	1.000	1.000	0.963	0.938	0.937	0.912	0.937	0.985	0.948
rater33	0.933	0.937	0.887	0.963	0.922	0.928	0.943	0.943	0.907	0.928	0.888	0.923	0.980	0.930	0.963	0.933	0.907	0.853	0.928	0.888	0.961	0.928	0.945	0.943	0.937	0.887	0.947	0.926	0.904	1.000	0.948	1.000	1.000	0.963	0.938	0.937	0.912	0.937	0.985	0.948
rater34	0.971	0.976	0.928	1.000	0.964	0.966	0.980	0.980	0.968	0.993	0.944	0.961	0.943	0.856	1.000	0.971	0.968	0.908	0.966	0.944	0.966	0.966	0.924	0.980	0.976	0.916	0.985	0.975	0.891	0.963	0.985	0.963	0.963	1.000	0.934	0.976	0.937	0.904	0.948	0.985
rater35	0.862	0.952	0.808	0.934	0.909	0.829	0.905	0.905	0.923	0.910	0.813	0.876	0.909	0.876	0.934	0.862	0.923	0.785	0.886	0.813	0.964	0.829	0.861	0.905	0.880	0.782	0.904	0.928	0.796	0.938	0.915	0.938	0.938	0.934	1.000	0.880	0.849	0.952	0.908	0.915
rater36	0.928	0.940	0.881	0.976	0.917	0.959	0.952	0.952	0.956	0.969	0.976	0.928	0.912	0.802	0.976	0.952	0.956	0.870	0.959	0.904	0.898	0.959	0.893	0.952	1.000	0.952	0.954	0.964	0.939	0.937	0.959	0.937	0.937	0.976	0.880	1.000	0.898	0.821	0.914	0.959
rater37	0.957	0.898	0.910	0.937	0.940	0.938	0.937	0.937	0.916	0.923	0.836	0.937	0.912	0.865	0.937	0.928	0.916	0.839	0.880	0.923	0.938	0.938	0.875	0.937	0.898	0.836	0.927	0.917	0.842	0.912	0.909	0.912	0.912	0.937	0.849	0.898	1.000	0.850	0.901	0.909
rater38	0.868	0.940	0.821	0.904	0.929	0.815	0.880	0.880	0.857	0.872	0.771	0.855	0.912	0.947	0.904	0.868	0.857	0.797	0.863	0.843	0.957	0.815	0.893	0.880	0.821	0.774	0.881	0.877	0.744	0.937	0.887	0.937	0.937	0.904	0.952	0.821	0.850	1.000	0.914	0.887
rater39	0.948	0.914	0.914	0.948	0.891	0.943	0.948	0.948	0.859	0.912	0.864	0.948	0.985	0.891	0.948	0.948	0.859	0.887	0.943	0.864	0.946	0.943	0.939	0.948	0.914	0.876	0.962	0.878	0.905	0.985	0.943	0.985	0.985	0.948	0.908	0.914	0.901	0.914	1.000	0.943
rater40	0.966	0.959	0.959	0.985	0.945	0.961	0.985	0.985	0.943	0.978	0.927	0.947	0.948	0.835	0.985	0.966	0.943	0.924	0.961	0.927	0.952	0.961	0.899	0.985	0.959	0.935	0.980	0.951	0.899	0.948	1.000	0.948	0.948	0.985	0.915	0.959	0.909	0.887	0.943	1.000

[7]

Partial plot

[5]

[6]

References

[1] The jamovi project (2023). jamovi. (Version 2.4) [Computer Software]. Retrieved from https://www.jamovi.org.

[2] R Core Team (2022). R: A Language and environment for statistical computing. (Version 4.1) [Computer software]. Retrieved from https://cran.r-project.org. (R packages retrieved from CRAN snapshot 2023-04-07).

[3] Gamer, M., Jim Lemon, J., & Singh, I. (2019). irr: Various Coefficients of Interrater Reliability and Agreement. [R package]. Retrieved from https://CRAN.R-project.org/package=irr.

[4] Falissard, B. (2012). psy: Various procedures used in psychometry. [R package]. Retrieved from https://CRAN.R-project.org/package=psy.

[5] Epskamp,S., et al. (2012). qgraph: Network Visualizations of Relationships in Psychometric Data. [R package]. Retrieved from https://CRAN.R-project.org/package=qgraph.

[6] Seol, H. (2023). seolmatrix: Correlations suite for jamovi. (Version 3.7.4)[jamovi module]. URL https://github.com/hyunsooseol/seolmatrix.

[7] Revelle,W (2019). psych: Procedures for Personality and Psychological Research. [R package]. Retrieved from https://CRAN.R-project.org/package=psych.