Legal Research when Relying on Open Access: a Primer

• 1. Introduction

  Relevant sources are required to properly embed a research question, or to answer one. However, sources are not always available or accessible. Various legal researchers, law students included, face some sort of restriction when it comes to finding relevant sources. In some parts of the world, researchers are completely blocked or suffer serious restrictions regarding access to recent books and subscription-based journals. My experience, which is supported by anecdotal evidence from students, is that the reliance on open-access information when searching for information in English and outside of one’s jurisdiction is at least common in various countries on the African, Asian and South American continent as well as in Eastern Europe. However, to my knowledge, no materials are available that guide researchers who conduct legal research under such circumstances. What is therefore missing in the current debate is knowledge on how to identify sources and on what search strategies to use in an environment where researchers necessarily rely on open-access sources because books and subscriptions to legal journals and databases are lacking.¹x Tilburg University. The author thanks Lukas Dziedzic, Marie-Claire Menting, Zihan Niu, ­Marnix Snel, Eric Tjong Tjin Tai and three anonymous reviewers for their valuable comments on a previous version of this article. Parts of section 2 and section 3 can also be found in Gijs van Dijck, ‘Eerste hulp bij juridisch bronnenonderzoek: waar te zoeken en hoe relevante bronnen te selecteren op het internet?’ (2015) Surinaams Juristenblad 29 (in Dutch). For a general overview of research strategies, see <https://olinuris.library.cornell.edu/content/skill-guides> (last accessed 26 April 2016). Existing publications on research methods in law describe and discuss the various types of research (e.g. doctrinal, comparative law, empirical),²x For example, Dawn Elizabeth Watkins and Mandy Burton (eds), Research Methods in Law (­Routledge 2013); T. Hutchinson, Researching and writing in law (3 edn, Thomson Reuters 2010); I. Curry-Sumner and F.G.H. Kristen, Onderzoeksvaardigheden - Research Skills (Ars Aequi Libri 2010); Mike McConville and Wing Hong (Eric) Chui (eds), Research Methods for Law (Edinburgh University Press 2007). with some also describing the steps of the research process (e.g. formulating a research topic and research question, conducting a literature review).³x Hutchinson (supra note 1). See also Curry-Sumner and Kristen (supra note 2).
  Furthermore, a significant amount of information is available on theoretical foundations, goals and position of legal (academic) research.⁴x For example, Jan M. Smits, The Mind And Method Of The Legal Academic (Edward Elgar Publishing 2012); Mark Van Hoecke (ed) Methodologies of legal research – which kind of method for what kind of discipline? (Hart Publishing 2011); Bart Van Klink and Sanne Taekema (eds), Law and method – Interdisciplinary Research into Law (Mohr Siebeck 2011). All publications provide many further references. Information can also be found on how and where to find legal sources,⁵x For example, Stephen Elias and Nolo Editors, Legal Research: How to Find & Understand the Law (Nolo Press 2012); Morris L. Cohen and Kent Olsen, Legal Research in a Nutshell (Thomson west 2010); Steven M. Barkan, Roy M. Mersky and Donald J. Dunn, Fundamentals of Legal Research (Foundation Press 2009). See also Hutchinson. or on how to interpret certain sources (e.g. case law, statutes).⁶x Peter L. Strauss, Strauss’ Legal Methods- Understanding and Using Cases and Statutes (3 edn, West Academic Publishing 2014). These publications are relevant both from an academic point of view and from a legal practitioner’s perspective, but the majority of the publications tend to focus on legal sources in the United States that generally require a subscription or a licence (e.g. Westlaw, LexisNexis).
  This article focuses on the knowledge gap that exists with respect to where and how to identify relevant scholarly sources in a research environment where the researcher, students included, is highly dependent on open-access materials. This often concerns situations where a researcher outside of a jurisdiction desires to gather knowledge, for example a researcher outside of the United States who desires to conduct comparative research with the United States but does not have access to Westlaw or HeinOnline, or a researcher who wants to conduct research in the field of human rights law but who does not have access to subscription-based human rights law journals.
  This article discusses where and how legal researchers can find as much relevant information as possible when they mostly or only rely on open-access sources. It provides information on where to find sources without being charged, how to select relevant sources from a large number of ‘hits’, and what search strategies researchers can adopt. Consequently, this article is of interest for at least two groups of researchers. It is relevant to researchers who rely on materials that are freely accessible because they lack access to books and to subscription-based journals outside of their own jurisdiction. The section on search strategy is relevant for legal researchers, in general, and novices, in particular, who aim to identify sources in an effective and efficient way.
  Although the information presented in the section on search strategies applies equally to legal academic research in general as well as to legal scholars who are not dependent on open access, I have not found an instruction that provides an overview on various search strategies researchers can use to identify relevant sources in an effective and efficient way.⁷x Information on search strategies generally focuses on searching databases with keywords (e.g. <http://guides.is.uwa.edu.au/c.php?g=324800&p=2177569>; <http://subjectguides.york.ac.uk/law/search_strategy>) or on a single search strategy (e.g. <http://guides.library.harvard.edu/law/researchstrategy>, mainly discussing the use of secondary sources) (all websites last accessed 26 April 2016). Moreover, the question of how to identify relevant sources in an effective and efficient way is closely related to and a crucial part of conducting legal academic research when relying on open access, particularly in situations where a researcher will be confronted with large amounts of possibly relevant materials. This is why this article includes a discussion on what are effective search strategies. Experienced researchers may consider skipping this section. Conversely, those who do have a proper amount of books and access to subscription-based journals may directly turn to section 4.
  This article focuses on books, journal articles and working papers that are studied with the aim of answering a legal research question. Consequently, it concentrates on sources regarding literature rather than legislation, statutes, case law, government documents or newspaper articles. Additionally, the resources, selection tools and search strategies explained in this article focus on finding open-­access sources in English. It is assumed here that sources in English are commonly sought when searching for sources outside one’s jurisdiction.
  The next section explores what sources are available to legal researchers who have access to the Internet but who have no or limited access to offline information and have no or limited access to subscriptions-based information (section 2). This section is followed by a discussion of how to select relevant publications (section 3). Subsequently, it will be explained what strategies researchers can use that are likely to yield good coverage of the most relevant sources in an effective and efficient way (section 4).

• 2. Available Sources

  No access or limited access to books and journals imposes serious restrictions on conducting legal research. On the other hand, open-access journals are more common and more popular than before, book chapters are made more frequently available free of charge and even complete books can sometimes be found and accessed online.⁸x For example, Carol A. Parker, ‘Institutional Repositories and the Principle of Open Access: Changing the Way We Think About Legal Scholarship’ (2010) 37 New Mexico Law Review 431 (discussing the emergence of open-access repositories in the legal field and reporting that approximately 40% of U.S. law schools have some form of institutional repository that is indexed by Internet search engines). Hutchinson distinguishes between various types of research materials that can be found online, including hardcopy books, journals, government websites, university and institutional repositories, and Legal Information Institutes (LII) repositories.⁹x Terry Hutchinson, ‘Valé Bunny Watson? Law Librarians, Law Libraries, and Legal Research in the Post-Internet Era’ (2014) 106 Law Library Journal 579, 587-588. Because open-access repositories often concern materials written in English, this section focuses on sources written in English.
  Google Scholar is one source that can be useful for retrieving sources. The search process works similarly to Google, except that Google Scholar focuses on scholarly literature (and even case law). Google Scholar claims ‘to rank documents the way researchers do, weighing the full text of each document, where it was published, who it was written by, as well as how often and how recently it has been cited in other scholarly literature’.¹⁰x <https://scholar.google.com/intl/en/scholar/about.html> (last accessed 26 April 2016). Although Google Scholar has been praised for its familiarity (‘googling’), for its user interface and for the fact that it comes up with results that may otherwise not have been found,¹¹x For example, Martin Kesselman and Sarah Barbara Watstein, ‘Google Scholar(tm) and libraries: point/counterpoint’ (2005) 33 Ref Serv Rev 380 (‘find some resources they can use rather than be frustrated by a database’s search screen’); Jim Giles, ‘Science in the web age: Start your engines’ (2005) 438 Nature 554; Greg R. Notess, ‘Scholarly Web searching: Google Scholar and Scirus’ (2005) <http://www.infotoday.com/Online/jul05/OnTheNet.shtml> (last accessed 26 April 2016). it has also been criticized for various reasons. The criticism includes the unreliability of advanced searches and for the results it produces, for the lack of transparency about the search process, for being susceptible to manipulate the system in order to get more citations to an article (‘gaming’), and for using the h-index¹²x According to Hirsch, researchers have an index h if h of their total of N publications have at least h citations in other publications and if the other (N-h) publications are not cited more than h times, see J.E. Hirsch, ‘An index to quantify an individual’s scientific research output’ (2005) 102 PNAS 16569. This means that someone with an h index of 10 will have 10 publications that have been cited at least 10 times. as main impact indicator,¹³x Mary Shultz, ‘Comparing test searches in PubMed and Google Scholar’ (2007) 9 J Med Libr Assoc 442, Table 1 (providing an overview); ’4 reasons why Google Scholar isn’t as great as you think it is’, <http://blog.impactstory.org/googe-scholar-profiles-fail/> (last accessed 26 April 2016). although the latter concern is not a concern in legal research as the h-index is not a common metric to evaluate legal research.
  The Social Science Research Network (www.ssrn.com) is another valuable source of information. SSRN was founded in 1994 and is one of the largest open-access databases worldwide.¹⁴x Number 2 according to the Ranking Web of Repositories <http://repositories.webometrics.info/en/search/Rankings/ssrn> (last accessed 26 April 2016), number 1 according to the same ranking for July 2012 <http://www.ssrn.com/update/general/ssrn_faq.html> (last accessed 26 April 2016). Its objective is ‘to provide rapid worldwide distribution of research to authors and their readers and to facilitate communication among them at the lowest possible cost’. It does so by allowing authors to upload without charge. The SSRN Library contains abstracts and full text papers of published articles, book chapters and working papers. Most papers can be downloaded for free. It is not required to have a subscription or account to download papers.
  What is appealing about SSRN is that many scholars contribute to open access by uploading their work. Sometimes, they upload pdf files of how the paper appeared or will appear in the journal it was or will be published in. Unlike Google Scholar (or Google), SSRN focuses only on academic papers. Additionally, it also provides search articles based on keywords. The search engine consists of the options to search publication titles, publication abstracts, and author name(s). SSRN also allows the search results to be limited according to the publication period (e.g. last week, last year, last 3 years).
  An important element to keep in mind is that the papers in the SSRN database can contain various types of uploads. Legal academics who upload papers may upload their publications, but they can also upload working papers. As the name suggests, working papers are work in progress. Working papers that are uploaded can have various versions. Authors may upload working papers in preparation for a presentation at a conference, to collect comments or to ‘claim’ a topic or perspective. They can be a first version of an article or a version that is, or will be, submitted to a journal. SSRN does have a review process regarding each version that is uploaded or modified, but it is not clear what this review process exactly entails.
  As a rule of thumb, one should not refer to a working paper. The problem with working papers is that they can be revised. Consequently, it is possible, and perhaps probable, that the final version of a paper (the version that is generally or, hopefully, published) differs from a previous version. However, if one comes across arguments, perspectives, claims or findings that will be used in one’s own publication, a reference to the working paper is required. Referring to thoughts and ideas laid down in a working paper is particularly required if the insights are new and cannot be found in existing publications. In contrast, it can be possible to track references in a working paper, to subsequently check the references and to use them in one’s own publication without referring to the paper.¹⁵x However, this becomes tricky if a substantial part of the sources in the working paper are used or if, for example, the working paper is a literature review on a certain topic. A reference might be in place in such instances. It is better to refer to a source one time too many than not to refer to a source where one should.
  Authors can add a reference to the publication, indicating the name of the journal, the year the paper was published, the volume number, the page numbers and other relevant information. This information can be added at the time of the upload, or at a later time.
  The popularity of a paper can be determined according to the number of downloads of an article or the number of abstract views. This is, however, tricky. The number of downloads can be easily inflated, for example by prescribing a paper in a course for a large number of students. Download numbers only say so much. They also depend on whether the author is known, on the journal the publication was published in and on whether the topic is popular, specialized. The number of views or downloads therefore does not seem to be a very reliable indicator of quality. One should consider other elements also to determine the potential relevance of a publication.
  Authors’ personal or company websites can also contain uploads of publications one is looking for. If a publication cannot be found through Google, Google Scholar or SSRN, the author’s website can be a good resort to find the relevant publication. Alternatively, one can go to an author’s website at the start of the process of identifying relevant literature. This is particularly effective if that author is an authority on the topic. Reviewing his or her website can result in finding a recent relevant publication that will contain many references to other publications.¹⁶x See also section 4. The website may include a link to the publication. Furthermore, it is always possible to contact an author and to request the author to email the full text of the publication that one desires or requires. The author may be willing to meet the request.
  An author’s personal website does not necessarily have to be restricted to his or her institutional website. Several platforms exist that facilitate the exchange of knowledge between scholars. Examples are ResearchGate¹⁷x <www.researchgate.com> (last accessed 26 April 2016). and Academia.edu.¹⁸x <www.academia.edu> (last accessed 26 April 2016). These platforms allow searching for researchers and their publications as well as for papers or publications.
  Finally, relevant information can be retrieved by searching websites such as Google Books. Characteristically, Google Books leaves out sections in order to prevent Internet users from downloading complete books free of charge. However, important parts of a book may be available and consequently be accessed without charge.

• 3. Selection of Sources

  The previous section discussed where to find relevant sources online. This section focuses on how to select relevant sources from search results. When applying the strategies and consulting the databases laid out in the foregoing sections, Internet searches can yield too many hits.¹⁹x For example, Stephanie Davidson, ‘Way Beyond Legal Research: Understanding the Research Habits of Legal Scholars’ (2010) 102 Law Library Journal 561, 563 (‘advances in technology and publishing formats that have vastly increased the scholar’s access to an ever-growing volume of legal information’). This particularly applies to searches produced with Google Scholar, and, to a lesser extent, to SSRN searches. Additionally, keyword searches can lead to a list of results that are not necessarily relevant to the question the researcher is interested in.²⁰x Lee F. Peoples, ‘The Death of the Digest and the Pitfalls of Electronic Research: What Is the Modern Legal Researcher to Do?’ (2005) 97 Law Library Journal 661, 664 (‘[c]omputers are good in indexing but fail at classifying information in a meaningful way’). Finally, keywords may be used in contexts that are not of interest to the researcher.²¹x Barbara Bintliff, ‘From Creativity to Computerese: Thinking Like a Lawyer in the Computer Age’ (1996) 88 Law Library Journal 338, 346 (giving the example of burden of proof, which ‘can be frustrating because of the many different ways these words are used in cases’). As a result, one may come up with sources that are not relevant for embedding a research question or answering one.
  Ultimately, it is the content that matters, but it is often not feasible to read everything at the start of a research – one simply does not have enough time to go through all materials to determine what is relevant and what is not. Moreover, only after reading more and more publications can one adequately evaluate which sources are the most authoritative, or start to adequately evaluate this.
  Formal criteria can serve as an effective means to select relevant publications, at least in the initial stages of the research.²²x See also <http://guides.library.cornell.edu/criticallyanalyzing> (last accessed 26 April 2016). One formal criterion that is useful is publication status. Articles and books that are published are preferred to non-published works, such as working papers. Additionally, reflective books may be preferred to textbooks, particularly when one is looking for a debate, viewpoints or perspectives rather than for what the law is in a certain jurisdiction. Recency is another formal criterion. It can be worthwhile to start reading recent work rather than older publications (e.g. publications on environmental law of 30 years ago).
  More tricky formal criteria are the publication outlet, author status and the length of the publication. Rankings concern the question of whether the article or book was published by a prestigious publisher (e.g. select Oxford University Press over Wolf Legal Publishers) or in a prestigious journal (e.g. Harvard Law Review over the Utrecht Law Review). The difficulty, particularly for novice students, is to determine which journals are more prestigious than others.
  This is not straightforward, since there is a lack of official general rankings or measures that provide for such information, especially in the legal field. Rankings exist mainly in the minds of the researchers. For example, publishers such as Cambridge University Press, Oxford University Press, Hart Publishing, and Edward Elgar Publishing are well-respected book publishers, but certainly not the only ones.
  For U.S. journals, the Washington and Lee Ranking may provide for a ranking.²³x See <http://lawlib.wlu.edu/LJ/> (last accessed 26 April 2016). However, this ranking is biased towards legal journals based in the U.S. For example, the well-respected European Law Journal is ranked well below the Sports Lawyers Journal. Other rankings include Google Scholar Metrics, the ExpressO ranking, the Siems World Law Journal Ranking 2011 and SCImago.²⁴x The rankings can be easily found online. See Rob van Gestel, ‘Sense and non-sense of a European ranking of law schools and law journals’ (2014) 35 Legal Studies 165 for the advantages and limitations of law journal rankings. Alternatively, one can look up the impact factors of the various journals, if available. Journal Impact Factors (JIFs) generally refer to how often a journal article gets cited during a certain period. However, the problem with JIFs is that they provide information about how often articles in a given journal are cited, but no information on how often a specific article is cited by others. As a result, publication X in journal Y may seem relevant because of a high number of citations of journal Y articles, but the question may be raised whether publication X frequently has been referred to in other journals.
  Importantly, journal rank is not an absolute quality indicator. Not all articles in top journals are top publications. One can find interesting and important information in, for example, lower- ranked journals. Sometimes such journals provide for more relevant information than top journals, particularly if the journal is a specialized journal. Moreover, Yoon, after analysing over 25,000 law journal articles, demonstrated an editorial bias.²⁵x Albert H. Yoon, ‘Editorial Bias in Legal Scholarship’ (2013) 5 Journal of Legal Analysis 309. He found that law reviews publish more articles from their own faculty than papers from other faculties, especially in higher-ranked law reviews. The assumption that therefore seems more justified is that top journals will, on average, contain better publications than other journals.
  Another formal quality indicator is author status. One strategy is to find authors who have experience on the topic and have published previously on the particular area of research. Moreover, publications by well-known researchers get more ­interest and weight than those of novices.²⁶x Van Hoecke, 11 (supra note 4). An indication of author quality is author affiliation. The idea here is similar to selecting top publishers or publications in top journals: authors that are affiliated with top law schools or well-known law schools are more likely to publish high-quality articles and books compared with researchers who have positions at schools that are less well known. Moreover, publications by professors may be considered more authoritative than publications by Ph.D. students.
  As with journal ranking, one should not rely too much on author status. Good papers written by relatively unknown authors or by authors who are not affiliated with a top-tier law school may easily be deemed irrelevant although they can actually be highly interesting and relevant. Moreover, the affiliation of an author is no guarantee for excellence. There are many good researchers at various faculties and institutes around the world.
  Finally, article length may also influence the decision to select an article. This particularly applies in cases where one aims to track references. In such instances, longer publications are preferred over very short publications.

• 4. Search Strategies

  Since various resources are available, how to search for legal academic research in a meaningful way? This section discusses how to ensure a proper coverage of relevant sources out there in an efficient way. The role of apologies in the law will serve as an example to illustrate the various strategies and sources available to researchers. Imagine a researcher who is interested in the question of how tort law should deal with apologies: Should courts be able to order an apology in tort law cases? Should apologies be kept out of the courtroom because they could otherwise serve as evidence to establish fault or liability? And what effects do apologies have on case outcomes, the number of settlements, the settlement amount, the verdict and so on? An automatic response of many novice researchers (worldwide) is to search on the Internet, for example Google, using keywords such as ‘apologies’, ‘legal’ and ‘law’.²⁷x Those who have access to them may start with searching a database such as Westlaw or Hein­Online. This strategy will produce many hits. This is also a remarkable strategy, since it deviates from what researchers would do when identifying relevant case law or when they would like to start building general knowledge on a certain topic (e.g. the right to be forgotten). In such instances, one generally looks for an authoritative publication, for example a handbook, and then possibly looks for other sources (e.g. publications, case law) on the topic. The hits on the first page on Google yield possibly interesting papers, including those by Carroll,²⁸x Robyn Carroll, ‘Apologies as a Legal Remedy’ (2013) 35 Sydney Law Review 317. Ho,²⁹x Benjamin Ho and Elaine Liu, ‘Does Sorry Work? The Impact of Apology Laws on Medical Malpractice’ (2011) 43 Journal of Risk and Uncertainty 141. Rachlinski et al.³⁰x Jeffrey J. Rachlinski, Chris Guthrie and Andrew J. Wistrich, ‘Contrition in the Courtroom: Do Apologies Affect Adjudication?’ (2013) 98 Cornell Law Review 1189. and ­Robbennolt³¹x Jennifer K. Robbennolt, ‘Apologies and Legal Settlement: An Empirical Examination’ (2003) 102 Mich L Rev 460. . However, there are many pages that follow that may (and do) also provide relevant publications. Because of this, it is often confusing to novices what the next step should be. Some start to look for additional publications by going to the next page with hits on Google; others start a new search with different keywords, or they start reading the ‘hits’ on the first page.
  Although it can be a very good idea to start ‘googling’ for relevant publications, particularly when one is exploring a topic, it is not recommended if no thought has been given to the search strategy, or if ‘googling’ is the sole strategy.³²x In some countries, such as China, to ‘google’ is not an option, since Google is blocked. The latter especially applies to situations where a researcher is looking for specific information about a topic, which is the case if a researcher already has a particular research question that he or she wants to answer.³³x As opposed to situations where a researcher conducts a literature review to come up with a research question. The issue here is that the search for publications is not systematic. Consequently, the results of the literature search, and hence the research findings, may be random, since they depend on what the researcher ‘accidentally’ finds using certain keywords in a specific database (e.g. Google).
  A different, more systematic, strategy increases the likelihood of finding as many relevant sources as possible. The use of three common techniques can, especially when combined, provide a better and more reliable overview of the field. The three techniques are:
  

  1. Identifying Relevant Experts

  2. Reference Tracking (or snowballing)

  3. Independent Literature Search.

  
  Finding the first piece of relevant information is often the biggest challenge when working on a given topic.³⁴x Cohen and Olsen, 13 (supra note 5). Identifying relevant experts in the field as one of the first steps may be counter-intuitive for researchers. The purpose of searching literature is to find relevant publications, not relevant authors. However, relevant publications are written by relevant authors, and the most popular hits on Google may not include the most recent articles on a topic. The first hits found on Google may therefore not provide a complete or accurate picture of what is out there. However, the first hits may be a good starting point to see whether the authors are experts on the topic. If the authors are experts, it can be worthwhile to research what other publications the authors have produced on the topic and whether any of those other publications are more recent than the ones found on Google.³⁵x See section 2 for information on how to determine who is an expert.
  In the example of apologies and the law, one would identify several experts based on the first hits in Google. Reviewing authors’ personal websites would reveal that Robbennolt³⁶x For example, Jennifer K. Robbennolt, ‘Apologies and Reasonableness: Some Implications of Psychology for Torts’ (2010) 59 DePaul Law Review 489; Jennifer K. Robbennolt, ‘Attorneys, Apologies, and Settlement Negotiation’ (2008) 13 Harv Negot L Rev; Jennifer K. Robbennolt, ‘Apologies and Settlement Levers’ (2006) 3 Journal of Empirical Legal Studies 333; Jennifer K. Robbennolt, ‘What We Know and Don’t Know about the Role of Apologies in Resolving Health Care Disputes’ (2005) 21 Ga St U L Rev 1009; Robbennolt, ‘Apologies and Legal Settlement: An Empirical Examination’ (2003). and Carroll³⁷x For example, Robyn Carroll, ‘When ‘Sorry’ is the Hardest Word to Say, How Might Apology Legislation Assist?’ (2014) 44 Hong Kong Law Journal; Carroll, ‘Apologies as a Legal Remedy’ (2013); Robyn Carroll, ‘You Can’t Order Sorriness, so Is There any Value in an Ordered Apology? An Analysis of Apology Orders in Anti-Discrimination Cases’ (2010) 32 University of New South Wales Law Journal 360; Alfred Allan, Dianne McKillop and Robyn Carroll, ‘Parties’ Perceptions of Apologies in Resolving Equal Opportunity Complaints’ (2010) 17 Journal of Psychiatry, Psychology and Law 538; Robyn Carroll, ‘Beyond Compensation: Apology as a Private Law Remedy’ in Jeff Berryman and Rick Bigwood (eds), The Law of Remedies: New Direction in the Common Law (Irwin Law 2010). have published extensively and more recently on the topic than what the initial search on Google shows. The additional publications (recent and older) that are found by the identification of relevant authors can already point the researcher to publications that are relevant and possibly more recent than the ones found in the initial Google search.
  The danger of relying solely on identifying relevant authors based on an initial Google search is that other relevant authors and studies may be overlooked. The mere fact that studies do not show up in Google (or another search engine) does not imply that those studies are not relevant. A technique to overcome this limitation, at least partly, is to track the references of the studies that are found. This involves simply going through the references of the parts of the article that are, or seem, relevant. Depending on the time and resources available, one can go through all of the initial studies that were identified. Alternatively, a selection of publications can be made. An approach to make this selection is to at least include the publications that first showed up in the Google search as well as the most recent publications, because the latter publications will likely contain the most recent publications in the field.
  Reference tracking can be used to identify other leading researchers and relevant sources. The references of those sources can subsequently be tracked, which could again lead to new authoritative researchers and sources. After a while the list of relevant sources can grow significantly. If the same publications or names are repeatedly referred to in various publications, this can suggest that these publications or authors are essential in the field. This strategy is called the saturation principle: one keeps tracking references until no or hardly any new publications of interest are found.
  In the apologies example, reference tracking of recent publications results in identifying other researchers that could possibly be authorities in the field. Tracking the references of recent Carroll publications results in a new name, Allan, who has also produced several publications on the topic.³⁸x For example, Alfred Allan and others, ‘Apology in Restorative and Juvenile Justice’ (2013) Psychiatry, Psychology and Law; Debra J. Slocum, Alfred Allan and Maria M. Allan, ‘An Emerging Theory of Apology’ (2011) 62 Australian Journal of Psychology 83; Alfred Allan, ‘Functional Apologies in Law’ (2008) 15 Psychiatry, Psychology and Law 369; Alfred Allan, ‘Apology in Civil Law: A Psycholegal Perspective’ (2007) 14 Psychiatry, Psychology and Law 5. However, tracking the references of the Rachlinski and Robbennolt articles also leads to many new interesting studies on apologies and the law, some of which focus on the effects of apologies,³⁹x For example, Mark Bennett and Deborah Earwaker, ‘Victim’s responses to apologies: The effects of offender responsibility and offense severity’ (1994) 134 The Journal of Social Psychology 457; M.G. Rumsey, ‘Effects of defendant background and remorse on sentencing judgments’ (1976) 6 Journal of Applied Psychology 64. and other studies on whether apologies should get evidentiary protection.⁴⁰x For example, Jonathan R. Cohen, ‘Legislating Apology: The Pros and Cons’ (2002) 70 U Cin L Rev 819.
  The concern with identifying authoritative scholars in combination with reference tracking is that one can overlook important parts of the debate. A group of authors may refer only to authors within their network because they disagree with other streams of research, have opposing viewpoints or are simply unaware of other studies on the same topic. In the example of apologies and the law, it turns out that one would overlook the insights produced by Australian researchers such as Carroll and Allan if only the names of U.S. scholars such as Rachlinski and Robbennolt were found in an initial search. This is why it is worthwhile to conduct an independent literature search, that is, to look for literature based on certain keywords and in databases that are available. In the apology example, keywords such as ‘apology’ in combination with ‘law’ or ‘legal’ may be used. Additionally, one will discover new sets of keywords after reading more publications. For instance, studying the sources that have already been identified will reveal that ‘apology protection laws’ and ‘apology legislation’ are common concepts in the field of apology and the law. These, and possibly other keywords, can be used to identify additional relevant sources, which can, in turn, reveal new authorities or new relevant publications.

• 5. Conclusion

  This article discusses how to identify and select relevant publications when relying mainly on open access. It is argued that an effective search strategy relies on a combination of three techniques: identifying relevant experts, reference tracking (or snowballing) and conducting an independent literature search. Google Scholar, Google Books, SSRN, ResearchGate and Academia.edu are search engines or databases that can or do provide for academic publications without charge. However, one may need to narrow down the search results. Several heuristics are available to make such a selection without having to read every single publication. These heuristics include looking at publication ranking, publication recency (in what year was the publication published?), publication outlet, author status and article length. However, heuristics cannot replace reading. In the end, it is about content.

Noten

• 1 Tilburg University. The author thanks Lukas Dziedzic, Marie-Claire Menting, Zihan Niu, ­Marnix Snel, Eric Tjong Tjin Tai and three anonymous reviewers for their valuable comments on a previous version of this article. Parts of section 2 and section 3 can also be found in Gijs van Dijck, ‘Eerste hulp bij juridisch bronnenonderzoek: waar te zoeken en hoe relevante bronnen te selecteren op het internet?’ (2015) Surinaams Juristenblad 29 (in Dutch). For a general overview of research strategies, see <https://olinuris.library.cornell.edu/content/skill-guides> (last accessed 26 April 2016).

• 2 For example, Dawn Elizabeth Watkins and Mandy Burton (eds), Research Methods in Law (­Routledge 2013); T. Hutchinson, Researching and writing in law (3 edn, Thomson Reuters 2010); I. Curry-Sumner and F.G.H. Kristen, Onderzoeksvaardigheden - Research Skills (Ars Aequi Libri 2010); Mike McConville and Wing Hong (Eric) Chui (eds), Research Methods for Law (Edinburgh University Press 2007).

• 3 Hutchinson (supra note 1). See also Curry-Sumner and Kristen (supra note 2).

• 4 For example, Jan M. Smits, The Mind And Method Of The Legal Academic (Edward Elgar Publishing 2012); Mark Van Hoecke (ed) Methodologies of legal research – which kind of method for what kind of discipline? (Hart Publishing 2011); Bart Van Klink and Sanne Taekema (eds), Law and method – Interdisciplinary Research into Law (Mohr Siebeck 2011). All publications provide many further references.

• 5 For example, Stephen Elias and Nolo Editors, Legal Research: How to Find & Understand the Law (Nolo Press 2012); Morris L. Cohen and Kent Olsen, Legal Research in a Nutshell (Thomson west 2010); Steven M. Barkan, Roy M. Mersky and Donald J. Dunn, Fundamentals of Legal Research (Foundation Press 2009). See also Hutchinson.

• 6 Peter L. Strauss, Strauss’ Legal Methods- Understanding and Using Cases and Statutes (3 edn, West Academic Publishing 2014).

• 7 Information on search strategies generally focuses on searching databases with keywords (e.g. <http://guides.is.uwa.edu.au/c.php?g=324800&p=2177569>; <http://subjectguides.york.ac.uk/law/search_strategy>) or on a single search strategy (e.g. <http://guides.library.harvard.edu/law/researchstrategy>, mainly discussing the use of secondary sources) (all websites last accessed 26 April 2016).

• 8 For example, Carol A. Parker, ‘Institutional Repositories and the Principle of Open Access: Changing the Way We Think About Legal Scholarship’ (2010) 37 New Mexico Law Review 431 (discussing the emergence of open-access repositories in the legal field and reporting that approximately 40% of U.S. law schools have some form of institutional repository that is indexed by Internet search engines).

• 9 Terry Hutchinson, ‘Valé Bunny Watson? Law Librarians, Law Libraries, and Legal Research in the Post-Internet Era’ (2014) 106 Law Library Journal 579, 587-588.

• 10 <https://scholar.google.com/intl/en/scholar/about.html> (last accessed 26 April 2016).

• 11 For example, Martin Kesselman and Sarah Barbara Watstein, ‘Google Scholar(tm) and libraries: point/counterpoint’ (2005) 33 Ref Serv Rev 380 (‘find some resources they can use rather than be frustrated by a database’s search screen’); Jim Giles, ‘Science in the web age: Start your engines’ (2005) 438 Nature 554; Greg R. Notess, ‘Scholarly Web searching: Google Scholar and Scirus’ (2005) <http://www.infotoday.com/Online/jul05/OnTheNet.shtml> (last accessed 26 April 2016).

• 12 According to Hirsch, researchers have an index h if h of their total of N publications have at least h citations in other publications and if the other (N-h) publications are not cited more than h times, see J.E. Hirsch, ‘An index to quantify an individual’s scientific research output’ (2005) 102 PNAS 16569. This means that someone with an h index of 10 will have 10 publications that have been cited at least 10 times.

• 13 Mary Shultz, ‘Comparing test searches in PubMed and Google Scholar’ (2007) 9 J Med Libr Assoc 442, Table 1 (providing an overview); ’4 reasons why Google Scholar isn’t as great as you think it is’, <http://blog.impactstory.org/googe-scholar-profiles-fail/> (last accessed 26 April 2016).

• 14 Number 2 according to the Ranking Web of Repositories <http://repositories.webometrics.info/en/search/Rankings/ssrn> (last accessed 26 April 2016), number 1 according to the same ranking for July 2012 <http://www.ssrn.com/update/general/ssrn_faq.html> (last accessed 26 April 2016).

• 15 However, this becomes tricky if a substantial part of the sources in the working paper are used or if, for example, the working paper is a literature review on a certain topic. A reference might be in place in such instances. It is better to refer to a source one time too many than not to refer to a source where one should.

• 16 See also section 4.

• 17 <www.researchgate.com> (last accessed 26 April 2016).

• 18 <www.academia.edu> (last accessed 26 April 2016).

• 19 For example, Stephanie Davidson, ‘Way Beyond Legal Research: Understanding the Research Habits of Legal Scholars’ (2010) 102 Law Library Journal 561, 563 (‘advances in technology and publishing formats that have vastly increased the scholar’s access to an ever-growing volume of legal information’).

• 20 Lee F. Peoples, ‘The Death of the Digest and the Pitfalls of Electronic Research: What Is the Modern Legal Researcher to Do?’ (2005) 97 Law Library Journal 661, 664 (‘[c]omputers are good in indexing but fail at classifying information in a meaningful way’).

• 21 Barbara Bintliff, ‘From Creativity to Computerese: Thinking Like a Lawyer in the Computer Age’ (1996) 88 Law Library Journal 338, 346 (giving the example of burden of proof, which ‘can be frustrating because of the many different ways these words are used in cases’).

• 22 See also <http://guides.library.cornell.edu/criticallyanalyzing> (last accessed 26 April 2016).

• 23 See <http://lawlib.wlu.edu/LJ/> (last accessed 26 April 2016).

• 24 The rankings can be easily found online. See Rob van Gestel, ‘Sense and non-sense of a European ranking of law schools and law journals’ (2014) 35 Legal Studies 165 for the advantages and limitations of law journal rankings.

• 25 Albert H. Yoon, ‘Editorial Bias in Legal Scholarship’ (2013) 5 Journal of Legal Analysis 309.

• 26 Van Hoecke, 11 (supra note 4).

• 27 Those who have access to them may start with searching a database such as Westlaw or Hein­Online. This strategy will produce many hits. This is also a remarkable strategy, since it deviates from what researchers would do when identifying relevant case law or when they would like to start building general knowledge on a certain topic (e.g. the right to be forgotten). In such instances, one generally looks for an authoritative publication, for example a handbook, and then possibly looks for other sources (e.g. publications, case law) on the topic.

• 28 Robyn Carroll, ‘Apologies as a Legal Remedy’ (2013) 35 Sydney Law Review 317.

• 29 Benjamin Ho and Elaine Liu, ‘Does Sorry Work? The Impact of Apology Laws on Medical Malpractice’ (2011) 43 Journal of Risk and Uncertainty 141.

• 30 Jeffrey J. Rachlinski, Chris Guthrie and Andrew J. Wistrich, ‘Contrition in the Courtroom: Do Apologies Affect Adjudication?’ (2013) 98 Cornell Law Review 1189.

• 31 Jennifer K. Robbennolt, ‘Apologies and Legal Settlement: An Empirical Examination’ (2003) 102 Mich L Rev 460.

• 32 In some countries, such as China, to ‘google’ is not an option, since Google is blocked.

• 33 As opposed to situations where a researcher conducts a literature review to come up with a research question.

• 34 Cohen and Olsen, 13 (supra note 5).

• 35 See section 2 for information on how to determine who is an expert.

• 36 For example, Jennifer K. Robbennolt, ‘Apologies and Reasonableness: Some Implications of Psychology for Torts’ (2010) 59 DePaul Law Review 489; Jennifer K. Robbennolt, ‘Attorneys, Apologies, and Settlement Negotiation’ (2008) 13 Harv Negot L Rev; Jennifer K. Robbennolt, ‘Apologies and Settlement Levers’ (2006) 3 Journal of Empirical Legal Studies 333; Jennifer K. Robbennolt, ‘What We Know and Don’t Know about the Role of Apologies in Resolving Health Care Disputes’ (2005) 21 Ga St U L Rev 1009; Robbennolt, ‘Apologies and Legal Settlement: An Empirical Examination’ (2003).

• 37 For example, Robyn Carroll, ‘When ‘Sorry’ is the Hardest Word to Say, How Might Apology Legislation Assist?’ (2014) 44 Hong Kong Law Journal; Carroll, ‘Apologies as a Legal Remedy’ (2013); Robyn Carroll, ‘You Can’t Order Sorriness, so Is There any Value in an Ordered Apology? An Analysis of Apology Orders in Anti-Discrimination Cases’ (2010) 32 University of New South Wales Law Journal 360; Alfred Allan, Dianne McKillop and Robyn Carroll, ‘Parties’ Perceptions of Apologies in Resolving Equal Opportunity Complaints’ (2010) 17 Journal of Psychiatry, Psychology and Law 538; Robyn Carroll, ‘Beyond Compensation: Apology as a Private Law Remedy’ in Jeff Berryman and Rick Bigwood (eds), The Law of Remedies: New Direction in the Common Law (Irwin Law 2010).

• 38 For example, Alfred Allan and others, ‘Apology in Restorative and Juvenile Justice’ (2013) Psychiatry, Psychology and Law; Debra J. Slocum, Alfred Allan and Maria M. Allan, ‘An Emerging Theory of Apology’ (2011) 62 Australian Journal of Psychology 83; Alfred Allan, ‘Functional Apologies in Law’ (2008) 15 Psychiatry, Psychology and Law 369; Alfred Allan, ‘Apology in Civil Law: A Psycholegal Perspective’ (2007) 14 Psychiatry, Psychology and Law 5.

• 39 For example, Mark Bennett and Deborah Earwaker, ‘Victim’s responses to apologies: The effects of offender responsibility and offense severity’ (1994) 134 The Journal of Social Psychology 457; M.G. Rumsey, ‘Effects of defendant background and remorse on sentencing judgments’ (1976) 6 Journal of Applied Psychology 64.

• 40 For example, Jonathan R. Cohen, ‘Legislating Apology: The Pros and Cons’ (2002) 70 U Cin L Rev 819.