Once we had scraped our data from Google Scholar, we needed a method to quantify the relevance of a given scientific article. There are many ways you can quantify success of a paper. Here are a few we investigated:

Is our data usable?

Yes. Mostly...

Given our doubts over accuracy of Google Scholar's data, we considered it a priority to exercise caution with our search query results. This paid off: data we compiled using the search queries of IDs 5 and 2 had fatal flaws. They were rejected from further analysis (discussed below). The other data sets were found to be suitable.

Our method to examine data suitability was empirical: Random samplings of each data set were passed under human eyes. For most queries, this observation of random subsets showed acceptably low levels of "background static" (i.e. results that Google Scholar had automatically matched to the query, which were not actually relevant). These would form only a drop of error in the ocean of relevant data.

...but only mostly.

Query 5 (iGEM OR "International Genetically Engineered Machine") was found to have an unacceptably large level of static. The reason was quickly identified: Because the two quoted terms in Query ID 5 ("iGEM", "International Genetically Engineered Machine") were separated by a disjunction (OR), the query would easily match anything that contained just the acronym "IGEM"! This meant acronyms in economics such as "Inter-temporal General Equilibrium Model (IGEM)" or the British "Institution of Gas Engineers & Managers (IGEM)" and various medical terms and chemical names snuck in. The entire data set with ID 5 was dismissed from further analysis.

The lesson we took from this is not to search for short acronyms by themselves. Query 1 ((“synthetic biology” OR "genetic engineering") AND (“iGEM” OR “International Genetically Engineered Machine")) could be thought of as the "Version Two" of the problematic Query 5. It searches for the same terms, but includes a conjunction (AND) with either synthetic biology or genetic engineering, which bends results toward our iGEM. This tunes static down to an acceptable level.

Query 2 (synthetic biology) had a different problem: It was too big. The query was an attempt to capture stats for the entire field of synthetic biology, so we could statistically determine the relative influence of the iGEM competition. However, we had forgotten the 1000-result cap imposed by Google Scholar. It is impossible to retrieve results beyond this 1k "event horizon". Google does not publish information regarding how the order of results is determined. Hence these first 1000 results (out of what are likely to be 10s or 100s of thousands of papers) are all biased by some unknown force. Were more cited papers favoured? Were papers published more recently favoured? No conclusions can be drawn from a biased and small subset of the full data. We also discounted data set 2 from any further analysis.

To emphasize: Data sets 2 and 5 are not included in any further analysis.

What's is iGEM's impact?

What does "impact" mean within a scientific context? We will operate under the assumption that having impact correlates strongly with being mentioned in scholarly articles. Hence, we quantify the impact that a term is having simply by searching for that term, and summing up result counts.

Here is a chart summarising how many papers mention various terms floating about iGEM over time:

Number of teams Parts submitted (10s) Papers mentioning iGEM Papers mentioning Registry of Parts Papers mentioning specific Registry Parts

Chart 1: Summary of iGEM-related attention over time

This chart shows the amount of "attention" the iGEM competition is receiving over time, shown relative to the growth of the competition itself (quantified by amount of registered teams). "Attention" is measured by amount of papers mentioning certain keywords deemed to be relevant to iGEM or the Registry of Standard Parts. An overall positive trend in proportion with the growth of the competition can be seen.

And the answer is...

Things are looking good for iGEM! Since the first iGEM competition in 2003, more teams are participating each year and more parts are being submitted. The efforts to expose iGEM to the community has paid off - the amount of scientific papers mentioning iGEM and the Registry of Standard Parts has risen, proportional to the increase of the competition itself.

Papers mentioning a specific Registry BioBrick have only begun to appear in recent years, but the numbers show growth. We hypothesize that the Registry's contents is only now reaching the critical mass to become a useful research tool. The founders' dream (iGEM Foundation, 2012b) of "making genetics modular" is becoming reality.

Where this data comes from

Data about participating teams and number of submitted BioBricks comes from the iGEM Foundation (2012a). The other data sets come from the results Google Scholar queries with IDs 1, 3 and 4 (see query summary table above).

Is the relationship between iGEM and the Registry clear?

Expectations

One of the iGEM competition's important goals is to build up well-characterized BioBrick content in the Registry. Thus, the iGEM competition and the Registry are inherently linked. Hence we would expect publications that mention iGEM to also refer to the Registry. However, as the Registry is not only used by iGEM, we expected to also find a large number of papers that mention the Registry without mentioning iGEM.

In other words, we expected:

• a large proportion of papers that mention iGEM and the Registry,
• a large proportion that mention only the Registry and
• only a very small proportion that mention only iGEM.

Yes and No

The expected proportion of papers mentioning only the Registry was found. However, of the papers that mention iGEM, about half do not mention the Registry. This was a greater proportion than we expected.

We theorise that this may be due to the iGEM Foundation and iGEM teams under-emphasizing their importance to part standardisation. Indeed iGEM is usually presented by the Foundation and teams as "a synthetic biology competition", when really that's just half the picture. We're not just competing, but we also exist for a greater good: Make the Registry better and do our part in helping organise synthetic biology!

Just iGEM iGEM and Parts Registry Just Parts Registry

Chart 5: Proportional mentions of iGEM, Registry and Both

This chart is a stacked combination of the three charts above. It shows how often iGEM, the Registry and both are mentioned in scientific articles (i.e. how many papers from each year match our search terms). Each of the three quantities has stayed roughly equal to the others throughout the years.

We found that the iGEM competition is making a positive impact. The competition is growing in size and scope, and both iGEM and the Registry are netting a proportionally increasing amount of attention from the scientific community. We are doing well!

However, we also found that quite a number of discussions of iGEM miss the important connection between our iGEM competition and the Parts Registry. We recommend that the iGEM Foundation and future teams emphasise the iGEM competition's raison d'être clearly in the future.

We also noticed that some papers do not give sufficient or clear credit to the Registry. We interpret this as confusion as to how the Registry should be cited. We recommend that a standardised referencing should be introduced. This would support the publishing process for inexperienced undergraduates involved in the competition, gather greater attention for the iGEM Foundation and Parts Registry and hence further our aims of synthetic biology standardisation. ∎

"Ael 2" and "Vulpecula", 2012. h-index (Hirsch). Wikipedia. [image online] Available at: <http://en.wikipedia.org/wiki/File:H-index-en.svg> [Accessed Jul 27, 2012].

"Ael 2", 2012. Illustrated example for the g-index proposed by Egghe. Wikipedia [image online] Available at: <http://en.wikipedia.org/wiki/File:Gindex1.jpg> [Accessed Jul 27, 2012].

Cowell, M.L., 2008. Making iGEM Better. [web page] Available at <http://openwetware.org/wiki/User:Macowell/Making_iGEM_Better> [Accessed Jul 19, 2012].

Egghe, L., 2006. Theory and practise of the g-index. Scientometrics [online], Volume 69 (Issue 1), p.131-152. Available at: <www.springerlink.com/content/4119257t25h0852w/?MUD=MP> [Accessed Jun 7, 2012].

Harzing, A.W., 2007. Publish or Perish. [computer program] Available from <http://www.harzing.com/pop.htm>

Hirsch, J.E., 2005. An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, Volume 102 (Issue 46). [online] Available at: http://<www.ncbi.nlm.nih.gov/pmc/articles/PMC1283832/?tool=pmcentrez> [Accessed 5th Jul, 2012]

iGEM Foundation, 2012a. Previous iGEM Competitions. [web page] Available at: <https://igem.org/Previous_iGEM_Competitions> [Accessed Jul 30, 2012]

iGEM Foundation, 2012b. Press Kit. [web page] Available at: <https://igem.org/Press_Kit> [Accessed Aug 3, 2012]

Iselid, L., 2006. Research on citation search in Web of Science, Scopus and Google Scholar. One Entry to Research [blog] Available at: <http://oneentry.wordpress.com/2006/08/11/research-on-citation-search-in-web-of-science-scopus-and-google-scholar/> [Accessed Jun 20, 2012].

Péter J., 2006. Dubious hit counts and cuckoo's eggs. Online Information Review [online] Volume 30 (Issue 2) p.188-193. Available at: <http://www.emeraldinsight.com/journals.htm?articleid=1550726&show=abstract> [Accessed Jun 20, 2012].

Zhou Y., Liyan L. and Menghui L., 2012. Quantifying the influence of scientists and their publications: distinguishing between prestige and popularity. New Journal of Physics, [online] Volume 14 (March 2012) Available at: <http://iopscience.iop.org/1367-2630/14/3/033033/> [Accessed Jun 7, 2012].