Inbreeding  in software engineering conferences?

A cornerstone of research is publishing: important research results must be made generally available for the research community, so that existing research can be exploited in new research and the community can share a common understanding of the research area.  Although most researchers are presumably motivated by genuine research challenges rather than by secondary reasons,  such as metrics used to assess the productivity of researchers and research institute, there is an obvious danger that researchers form inbreeding communities around publication forums, rewarding itself with publications and citations. This kind of development could take place without any conscious intentions.

We have studied the practices used in software engineering conferences for paper evaluation. Since the authority that grants publications is a program committee (PC), a key role in the evolution of a research community is PC membership. This is emphasized by the fact that the PC membership itself is often regarded as a merit. Thus, in this study we will concentrate primarily on analyzing data related to the PC memberships. Our hypothesis is that two properties characterize a healthy, non-inbreeding research community as far as PC memberships are concerned: reasonable turnover of a PC, and non-favouring acceptance of papers submitted by PC members.

We focused on the data from six software engineering (ICSE, ICSM, WCRE, CSMR, ICPC/IWPC, and GPCE) conferences in years 2004–2009. The analysis is solely based on data publically available.

The results of the analysis and our more detailed recommendations can be found here.

A shorter version (by request) of the paper has been submitted to a software engineering journal.

Metrics:

• number of submitted papers [#subm]

• number of accepted papers [#acc]

• number of accepted paper (co-)authored by PC members [#PCacc]

• number of PC members [#PCmem]

• number of new PC members (compared to the previous year) [#newPCmem]

• number of new PC members (compared to all previous years starting from 2004) [#realnewPCmem].

• acceptance rate [accRate = #acc / #subm]

• proportion of papers (co-)authored by PC member from all accepted papers [PCaccProp = #PCacc / #acc ]

• proportion of new PC members [newPCprop = #newPCmem / #PCmem ]

• proportion of real new PC members [realnewPCprop = #realnewPCmem / #PCmem ]

• review coefficient [revCoeff = #subm / #PCmem].

Metrics related to submitted and accepted papers were applied to long technical papers only. That is, short papers, tool demos, and industry track papers were out of the scope of this study.

Data collected

1. ICSE  data

rYear	2004	2005	2006	2007	2008	2009
#subm	436	313	395	334	371	405
#acc	58	44	36	49	55	50
#PCacc	14	7	5	6	11	10
#PCmem	47	43	33	33	38	41
#newPCmem	39	33	26	30	31	30
#realnewPCmem			22	12	19	11
accRate	0.13	0.14	0.09	0.15	0.15	0.12
PCaccProp	0.24	0.16	0.14	0.12	0.20	0.20
newPCprop	0.83	0.77	0.79	0.91	0.82	0.73
realnewPCprop			0.67	0.36	0.50	0.27
revCoeff	9.28	7.28	11.97	10.12	9.76	9.88

2. ICSM  data

Year	2004	2005	2006	2007	2008	2009
#subm	122	180	147	214	156	162
#acc	38	55	41	46	40	35
#PCacc	14	15	15	19	19	16
#PCmem	57	59	78	87	78	78
#newPCmem	20	31	36	24	30	45
#realnewPCmem			17	20	22	19
accRate	0.31	0.31	0.28	0.21	0.26	0.22
PCaccProp	0.37	0.27	0.37	0.41	0.48	0.46
newPCprop	0.35	0.53	0.46	0.28	0.38	0.58
realnewPCprop			0.22	0.23	0.28	0.24
revCoeff	2.14	3.05	1.88	2.46	2.00	2.08

3. ICPC/IWPC  data

Year	2004	2005	2006	2007	2008	2009
#subm	42	54	73	66	57	74
#acc	21	24	23	22	20	20
#PCacc	10	7	6	13	12	7
#PCmem	41	30	36	45	53	51
#newPCmem	19	11	14	16	17	20
#realnewPCmem			12	10	14	14
accRate	0.50	0.44	0.32	0.33	0.37	0.27
PCaccProp	0.48	0.29	0.26	0.59	0.60	0.35
newPCprop	0.46	0.37	0.39	0.36	0.32	0.39
realnewPCprop			0.33	0.22	0.26	0.27
revCoeff	1.02	1.80	2.03	1.47	1.08	1.45

4. CSMR  data

Year	2004	2005	2006	2007	2008	2009
#subm	62	81	65	85	87	70
#acc	33	33	27	29	24	22
#PCacc	5	7	13	13	10	7
#PCmem	37	48	40	36	50	51
#newPCmem	12	15	22	15	29	19
#realnewPCmem			20	11	23	11
accRate	0.53	0.41	0.42	0.34	0.28	0.31
PCaccProp	0.15	0.21	0.48	0.45	0.42	0.32
newPCprop	0.33	0.31	0.55	0.42	0.58	0.37
realnewPCprop			0.55	0.31	0.46	0.22
revCoeff	1.72	1.69	1.63	2.36	1.74	1.37

5. WCRE  data

Year	2004	2005	2006	2007	2008	2009
#subm	78	58	84	87	70	79
#acc	28	21	24	27	20	20
#PCacc	8	10	17	10	11	8
#PCmem	37	37	41	38	56	47
#newPCmem	13	11	8	8	27	14
#realnewPCmem			7	7	24	7
accRate	0.36	0.36	0.29	0.31	0.29	0.25
PCaccProp	0.29	0.48	0.71	0.37	0.55	0.40
newPCprop	0.35	0.30	0.20	0.21	0.48	0.30
realnewPCprop			0.17	0.18	0.43	0.15
revCoeff	2.11	1.57	2.05	2.29	1.25	1.68

6. GPCE  data

Year	2002	2003	2004	2005	2006	2007	2008	2009
#subm	39	62	75	87	88	62	55	62
#acc	18	21	25	25	25	19	17	18
#PCacc	1	3	2	3	4	0	4	6
#PCmem	22	20	18	17	26	25	21	29
#newPCmem		15	16	15	22	24	16	23
#realnewPCmem			14	11	14	21	10	18
accRate	0.46	0.34	0.33	0.29	0.28	0.31	0.31	0.29
PCaccProp	0.06	0.14	0.08	0.12	0.16	0	0.24	0.33
newPCprop		0.75	0.89	0.88	0.85	0.96	0.76	0.79
realnewPCprop			0.78	0.65	0.54	0.84	0.48	0.62
revCoeff	1.77	3.10	4.17	5.12	3.38	2.48	2.62	2.14