Data collection Data and where to find it

Query summary

Here's a quick breakdown of what we queried for on Google Scholar and what sort of data was returned. (The ID matches the name of the data set in our data tables). The h- and g-indexes are explained just below!

Dataset ID	Plain English query	Query	Nº Papers	Nº Citations	h-index	g-index	Query date
1	Papers mentioning iGEM in context of synbio	(“synthetic biology” OR "genetic engineering") AND (“iGEM” OR “International Genetically Engineered Machine")	770	3253	26	45	17/7/2012
2	All synthetic biology	synthetic biology	1000	68482	127	214	17/7/2012
3	Papers mentioning Registry of Parts	"Registry of Standard Biological Parts" OR "partsregistry.org" OR "parts.mit.edu"	751	6442	39	69	17/7/2012
4	Papers citing a particular Part	partsregistry.org/Part:	54	263	5	16	17/7/2012
5	Papers mentioning iGEM	iGEM OR "International Genetically Engineered Machine"	1000	9095	36	64	17/7/2012
6	Papers mentioning iGEM and Registry	("iGEM" OR "International Genetically Engineered Machine") AND("Registry of Standard Biological Parts" OR "partsregistry.org" OR "parts.mit.edu")	330	2208	23	42	17/7/2012

Note Searches were capped at a maximum of 1000 results. Hence getting 1000 results for a query implies that more exist! Those first 1000 are only the ones the search engine judged most relevant.

Why we used Google Scholar

All in all, we found Google Scholar to most closely meet our analytic needs.

As WoK, Scopus and PubMed are strictly curated databases and limited in scope, they missed many obviously relevant publications. We also found their search options unsuitable: Many of them did not support either full text search (they looked at titles, keywords and abstracts only) or boolean operators. But for the following reasons, we needed a search to include both:

• iGEM cannot be expected to always be the main subject of a paper, hence full text search.
• There are many relevant terms floating about iGEM, hence boolean operators like "OR" to allow treating papers that contain "International Genetically Engineered Machine" or the acronym "iGEM" equally.

Just how much wider is Google's search scope? Here is an example: PubMed gave so few results (16 for iGEM genetic*) that we quickly discarded it. Manually merging the Web of Knowledge and Scopus results for the query iGEM AND genetic* (discarding obviously irrelevant results) gave 43 results. Then we queried Google Scholar. It gave us 770 for (“synthetic biology” OR "genetic engineering") AND (“iGEM” OR “International Genetically Engineered Machine").

Of course, Google Scholar too is but a bronze bullet: It brings its own drawbacks. It is engineered to pick up things that only seem like scholarly articles. Like Google's search results in general, the results are not curated by a human. This has been criticised in the literature (Péter J., 2006.). We found the occasional hilarious total miss. Google Scholar is also known to somewhat overestimate citation counts (Iselid, L., 2006.). However, we concluded from empirical manual examination of a random sample that the majority of the results are plausible and (most importantly) far greater in scope than searches in curated databases. Taking these aspects into consideration, we found Google Scholar best fulfilled our requirements.

Caveat! We only want statistics to identify trends. For this, large and coarse pieces of data suffice. We would discourage using this method for obtaining precise values!

Browse the data

Our tables

...are online in a nifty and very usable Google Docs folder.

An introduction is included in case you get lost or want more information.

On extraction tools

We made extensive use of Harzing Publish or Perish (Harzing, A.W., 2007.) to scrape Google Scholar results. The tool has many limitations. However, in our experience it is the best available resource for managing the mess that scientific publication data scraping tends to become.

We did try other things: We quickly found manual methods too slow. Various Firefox browser plugins simply failed outright, were extremely awkward to use or produced clearly erroneous results. The Mac OS program Papers was easy to use and found huge numbers of papers (as it could access many sources), but had unacceptably high rates of error, problems with duplicates and could not export the results into a form we could easily process. Hence Publish or Perish.

Once we had scraped our data from Google Scholar, we needed a method to quantify the relevance of a given scientific article. There are many ways you can quantify success of a paper. Here are a few we investigated:

Is our data usable?

Yes. Mostly...

Given our doubts over accuracy of Google Scholar's data, we considered it a priority to exercise caution with our search query results. This paid off: data we compiled using the search queries of IDs 5 and 2 had fatal flaws. They were rejected from further discussion (discussed below). The other data sets were found to be suitable.

Our method to examine data suitability was empirical: Random samplings of each data set were passed under human eyes. For most queries, this observation of random subsets showed acceptably low levels of "background static" (i.e. results that Google Scholar had automatically matched to the query, which were not actually relevant). These would form only a drop of error in the ocean of relevant data.

...but only mostly.

Query 5 (iGEM OR "International Genetically Engineered Machine") was found to have an unacceptably large level of static. The reason was quickly identified: Because the two quoted terms in Query ID 5 ("iGEM", "International Genetically Engineered Machine") were separated by a disjunction (OR), the query would easily match anything that contained just the acronym "IGEM"! This meant acronyms in economics such as "Inter-temporal General Equilibrium Model (IGEM)" or the British "Institution of Gas Engineers & Managers (IGEM)" and various medical terms and chemical names snuck in. The entire data set with ID 5 was dismissed from further analysis.

The lesson we took from this is not to search for short acronyms by themselves. Query 1 ((“synthetic biology” OR "genetic engineering") AND (“iGEM” OR “International Genetically Engineered Machine")) could be thought of as the "Version Two" of the problematic Query 5. It searches for the same terms, but includes a conjunction (AND) with either synthetic biology or genetic engineering, which bends results toward our iGEM. This tunes static down to an acceptable level.

Query 2 (synthetic biology) had a different problem: It was too big. The query was an attempt to capture stats for the entire field of synthetic biology, so we could statistically determine the relative influence of the iGEM competition. However, we had forgotten the 1000-result cap imposed by Google Scholar. It is impossible to retrieve results beyond this 1k "event horizon". Google does not publish information regarding how the order of results is determined. Hence these first 1000 results (out of what are likely to be 10s or 100s of thousands of papers) are all biased by some unknown force. Were more cited papers favoured? Were papers published more recently favoured? No conclusions can be drawn from a biased and small subset of the full data. We also discounted data set 2 from any further analysis.

To emphasize: Data sets 2 and 5 are not included in any further analysis.

Is iGEM getting attention?

What does "attention" mean within a scientific context? We will operate under the assumption that getting attention correlates strongly with being mentioned in scholarly articles. Hence, we quantify the attention that some term is getting iby searching for that term and summing up result counts.

Here is a chart summarising how many papers mention various terms floating about iGEM over time:

Number of teams Parts submitted (10s) Papers mentioning iGEM Papers mentioning Registry of Parts Papers mentioning specific Registry Parts

Chart 1: Summary of iGEM-related attention over time

This chart shows the amount of "attention" the iGEM competition is receiving over time, shown relative to the growth of the competition itself (quantified by amount of registered teams). "Attention" is measured by amount of papers mentioning certain keywords deemed to be relevant to iGEM or the Registry of Standard Parts. An overall positive trend in proportion with the growth of the competition can be seen.

And the answer is...

Things are looking good for iGEM! Since the first iGEM competition in 2003, more teams are participating each year and more parts are being submitted. The efforts to expose iGEM to the community has paid off - the amount of scientific papers mentioning iGEM and the Registry of Standard Parts has risen, proportional to the increase of the competition itself.

Papers mentioning a specific Registry BioBrick have only begun to appear in recent years, but the numbers show growth. We hypothesize that the Registry's contents is only now reaching the critical mass to become a useful research tool. The founders' dream (iGEM Foundation, 2012b) of "making genetics modular" is becoming reality.

Where this data comes from

Data about participating teams and number of submitted BioBricks comes from the iGEM Foundation (2012a). The other data sets come from the results Google Scholar queries with IDs 1, 3 and 4 (see query summary table above).

Is the relationship between iGEM and the Registry clear?

• 

  Papers published in that year Times those papers were cited

• 

  Papers published in that year Times those papers were cited

• 

  Papers published in that year Times those papers were cited

These three graphs show the amount of papers published each year containing certain search queries, as well as the number of times these papers were cited. All graphs show positive tendencies: the competition is becoming more wide-spread and more iGEM-related papers are being published and recognized. The search queries were chosen to show which part of iGEM is usually cited: the iGEM competition, the Registry of Standard Parts, or both. The data shows that only around half of papers will cite both elements. It should be noted that some outlying data points were ignored as they are obvious mis-searches.

Expectations

One of the iGEM competition's important goals is to build up well-characterized BioBrick content in the Registry. Thus, the iGEM competition and the Registry are inherently linked. We would expect publications that mention iGEM to also refer to the Registry. However, as the Registry is not only used by iGEM, we expected to also find a large number of papers that mention the Registry without mentioning iGEM.

In other words, we expected:

• a large proportion of papers that mention iGEM and the Registry,
• a large propotion that mention only the Registry and
• only a very small proportion that mention only iGEM.

Yes and No

The expected proportion of papers mentioning only the Registry was found. However, of the papers that mention iGEM, about half do not mention the Registry. This was a greater proportion than we expected.

We theorise that this may be due to the iGEM Foundation and iGEM teams under-emphasizing the importance of part standardisation. Indeed iGEM is usually presented by the Foundation and teams as "a synthetic biology competition", when really that's just half the picture. We think the Registry is just as important!

Just iGEM iGEM and Parts Registry Just Parts Registry

Chart 5: Proportional mentions of iGEM, Registry and Both

This graph shows how often iGEM and the Registry of Standard Parts are mentioned in scientific articles, according to terms mentioned. A smaller proportion of papers than expected mention both iGEM and the Registry, most papers will only mention one or the other.

Data collection

Choosing a data set

In order to analyse how exactly iGEM and the Registry is being cited, we decided to manually examine a set of papers. Earlier, we discarded Scopus and Web of Knowledge when carrying out wide-range data collection. For this, the result of the search term “iGEM” gave us a random selection of iGEM-related papers, which suited perfectly.

We have manually deleted all the publications which mentioned the search terms in the irrelevant context. Duplicates were deleted from the combined dataset.

The keyword “iGEM” gave 41 publications, which we looked at in closer detail. Of these, we discarded 16 publications due to various reasons: 5 texts were non-research articles, such as magazine articles; 2 were articles on bioethics that mentioned iGEM only in passing; 1 article cited an articles with “iGEM” in the title but was itself unrelated; 6 were articles we couldn’t locate or access with the University of St Andrews account; and 2 were in French.

This left 25 articles remaining, which we examined for how they cited iGEM and the Registry. Out of these, we found 11 articles to cite iGEM and/or BioBricks to our satisfaction. We had previously defined sufficient citation as at least either citing the Registry, or naming the specific BioBricks used (when appropriate) and citing either the Knight or Endy paper related to early BioBrick assembly.

The standard form of citation was:

5 articles did not cite the Registry or BioBricks sufficiently according to our demands. For one of them, we question its content relevancy. The remaining 4 show examples of simple in-text citation of the Registry, with or without a hyperlink, no mention of the specific BioBrick used, or no mention of BioBricks at all. Surprisingly, each of these papers was associated with an iGEM team.

There are 9 additional papers we found in the journal IELTS Synthetic Biology, all of which show vast variations in quality of content, research, and citation. However, we assume this journal asked all of the 2006 iGEM finalists to submit a paper based on their research, resulting in the some sub-par articles. Due to this, we found their quality of citation to be irrelevant to our overall data set.

Recommendation

Reviewing the data, we have concluded that there is a standard method of citation that the scientific community is using. In order to simplify the publishing process and uphold referencing standards within synthetic biology, this method of referencing should be recommended on the Registry and iGEM website. A similar tutorial to the one on the Registry website (iGEM Foundation, 2012) should be introduced to iGEM as a whole. Such a change has been proposed since 2007 (Cowell, 2007), but hasn’t been carried through. We see this as a high priority.

iGEM teams are mostly made up of undergraduates, often with very little experience in publishing. An accessible citation method would support those teams attempting to publish. Additionally, a standard method of citation will motivate all Registry data extraction to be cited correctly. As the Registry receives more citations, it will also garner more attention within the field. Thus, the Registry will ultimately become a more valuable resource.

Overall, we found that the iGEM competition is making a positive impact. The competition is growing both in size and scope, and it is netting a proportionally increasing amount of attention from the scientific community. However, we found that quite a number of papers are not naming both the iGEM competition and the Registry, and that there are papers that do not give any credit to the Registry. Thus, we recommend that a standardised referencing should be introduced. This will support the publishing process for inexperienced undergraduates, and result in greater attention for the iGEM Foundation overall.

"Ael 2" and "Vulpecula", 2012. h-index (Hirsch). Wikipedia. [image online] Available at: <http://en.wikipedia.org/wiki/File:H-index-en.svg> [Accessed Jul 27, 2012].

"Ael 2", 2012. Illustrated example for the g-index proposed by Egghe. Wikipedia [image online] Available at: <http://en.wikipedia.org/wiki/File:Gindex1.jpg> [Accessed Jul 27, 2012].

Egghe, L., 2006. Theory and practise of the g-index. Scientometrics [online], Volume 69 (Issue 1), p.131-152. Available at: <www.springerlink.com/content/4119257t25h0852w/?MUD=MP> [Accessed Jun 7, 2012].

Harzing, A.W., 2007. Publish or Perish. [computer program] Available from <http://www.harzing.com/pop.htm>

Hirsch, J.E., 2005. An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, Volume 102 (Issue 46). [online] Available at: http://<www.ncbi.nlm.nih.gov/pmc/articles/PMC1283832/?tool=pmcentrez> [Accessed 5th Jul, 2012]

iGEM Foundation, 2012a. Previous iGEM Competitions. [web page] Available at: <https://igem.org/Previous_iGEM_Competitions> [Accessed Jul 30, 2012]

iGEM Foundation, 2012b. Press Kit. [web page] Available at: <https://igem.org/Press_Kit> [Accessed Aug 3, 2012]

Iselid, L., 2006. Research on citation search in Web of Science, Scopus and Google Scholar. One Entry to Research [blog] Available at: <http://oneentry.wordpress.com/2006/08/11/research-on-citation-search-in-web-of-science-scopus-and-google-scholar/> [Accessed Jun 20, 2012].

Péter J., 2006. Dubious hit counts and cuckoo's eggs. Online Information Review [online] Volume 30 (Issue 2) p.188-193. Available at: <http://www.emeraldinsight.com/journals.htm?articleid=1550726&show=abstract> [Accessed Jun 20, 2012].

Zhou Y., Liyan L. and Menghui L., 2012. Quantifying the influence of scientists and their publications: distinguishing between prestige and popularity. New Journal of Physics, [online] Volume 14 (March 2012) Available at: <http://iopscience.iop.org/1367-2630/14/3/033033/> [Accessed Jun 7, 2012].