Law and Method, september 2023

Progress in Legal Methodology – A Methodological Assessment of Six PhD Theses

Sanne Taekema en Bart van Klink

1 Introduction

In some academic disciplines, one can identify clear advances in methodology. DNA sequencing in biology or randomized controlled trials in medicine have fundamentally changed the way research is done in those fields. The idea that there is progress in scientific methodology thus has intuitive appeal. In the context of legal research, however, we lack such clearly recognizable advances in methods of research. This makes the question that prompted the writing of this article a difficult one: Can we identify advances in legal methodology? Moreover, for this Special Issue, we were asked to identify recent scholarly works that exemplify methodological progress, which is even more difficult. There is also reason to doubt the premise of this Special Issue when it comes to methodology, because the idea of identifying single works that have changed the field does not really reflect scholarly practice. In our view, methodological change – if there is such a thing – would mean a change of the practice of legal research more broadly: methodology is an aspect of research that depends on developing standards of research in the scientific community, which depends on critical reflection, gradual improvement and educating researchers.
For these reasons, the question we set ourselves for this contribution is slightly different: to what extent has the methodology debate in legal scholarship improved the practice by PhD researchers of justifying their methodology? Over the past twenty years, there has been much more consideration and discussion of legal methods, especially in Dutch academia. The focus on Dutch academia seems warranted, because as far as we are able to ascertain, other countries have not experienced such an intense discussion on legal methodology. A consequence is that a number of our sources are in Dutch, although we have tried to refer to English-language publications as much as possible. Taking this Dutch debate as our starting point, we argue that it has led to a normative framework with which the methodology of legal research can be assessed. Formulating a set of topics and questions that form the core of this framework, we apply it to a set of six fairly recent PhD dissertations. The reasons for this choice are twofold: first, the Dutch methodology debate has had dissertations as a focal point to study the (lack of) methodological awareness of legal researchers, and second, graduate education is a primary site for teaching methods, which makes it likely that changes, if any, are implemented there first and should be visible in dissertations. We are, moreover, not the first to focus on dissertations: Herweijer (2003) and Tijssen (2006) also studied legal methodology by looking at PhD theses.¹x Van Gestel and Vranken (2007) took a different approach by assessing legal academic articles. They applied a set of three criteria: quality of the research question/problem statement, use of sources and drawing of conclusions.
In the following, we first briefly discuss the Dutch methodology debate and present the main result we derive from this debate, a normative framework for assessment of legal methodologies (Section 2). Subsequently, we explain our approach in applying the framework to PhD dissertations and present our findings (Section 3). We should already highlight here that this application is an explorative endeavour, to get a first sense of which aspects of the framework have been addressed, and in which way. Finally, we present some tentative observations based on our analysis (Section 4); we do not pretend that these are firm conclusions about the quality of legal dissertations.
2 From the Dutch Methodology Debate to the Assessment Framework

The methodology debate has a clear starting point: Carel Stolker’s article about the academic character of legal scholarship in 2003. Stolker responded to the sceptical view that the discipline of law cannot really be regarded as scientific, with its overlap with legal practice and lack of methodological clarity.²x An adaptation of the 2003 article is part of Stolker’s book Rethinking the Law School (2014, pp. 200-230). That article sparked a series of further contributions, discussing what makes legal scholarship scientific and what methodological criteria need to be met by legal scholarly work. The debate on legal scholarship was lively, leading to interesting discussions about normativity, empirical work and the special character of doctrinal-legal scholarship (e.g. Van Hoecke, 2011; Van Gestel, Micklitz & Rubin, 2017). Part of the debate calls for more methodological awareness and better methodological justification of legal research, which received mixed responses (compare Tijssen, 2006, pp. 30-33). Nonetheless, opinions seem to have evolved: whereas the first points of convergence in the early 2000s were only on the need for a clear research question and for methodological justification (Barendrecht et al., 2004), gradually other quality criteria were introduced (see e.g. Snel, 2016). In recent years, more concrete instructional texts have been published, which converge on a number of guidelines and criteria for legal methodology (Curry-Sumner et al., 2010; Kestemont, 2018; Van Dijck, Snel & Van Golen, 2018; Vols, 2021).
When reviewing these publications and developments, certain criteria appear both as part of the instructional texts and earlier assessments of dissertations. The emphasis on specific criteria differs: for instance, from an emphasis on research questions (Curry-Sumner, 2010) and on research aims (Kestemont, 2018), to a stress on the interaction between literature review and research question (Van Dijck, Snel & Van Golen, 2018). To assess legal dissertations from a methodological point of view, we have constructed a framework consisting of the following standards: research aims, research question, theoretical framework, normative framework, methodological choices and demarcation, and social and academic relevance. The specific criteria included in each category are explained in the Annex to this article. The breakdown of methodological standards into these categories resembles earlier work, but also differs. The theoretical framework, for instance, is sometimes used as an umbrella category, which includes a normative framework in case of a normative research aim (Kestemont, 2018; Van Dijck et al., 2018). Others only use a conceptual framework and normative framework as separate categories (Herweijer, 2003, p. 28) or include a normative framework as part of the methods of research (Tijssen, 2006, p. 108). We follow the recent literature in using both theoretical and normative framework to denote concepts and theories versus normative standards (Taekema, 2018). The broadest category we use is methodological choices and demarcation which includes choices regarding both sources and methods, and discussion of the scope and limitations of the research (categories IV to VII in the Appendix). In other work, sources and methods are treated separately (Tijssen, 2006, pp. 65-66). Methodology also concerns determination of research scope, operationalization of concepts and the choice of exemplars, i.e. specific cases or illustrations, all of which are not strictly methods of research, but do co-determine the way the research is executed. Justification of choices is also involved in the question, sources, framework and methods. Ultimately, all these aspects need to be aligned with each other in order to provide a thorough methodology: (in)consistency between, for instance, research aims and methods is one of the clearer indicators of methodological quality.
The strong emphasis on justification in the assessment framework is connected to a difficulty in methodological evaluation: the relationship of methodology to the substance of the research. Although methodological aspects of research can be discussed separately, the quality of a methodology is intertwined with the substance of the research. How sources are used and how particular methods are applied determine the quality of the content of the research. For a full assessment of such aspects, knowledge of the substantive area of research is needed. For instance, how academically relevant a publication is, depends on its content and the relation to the existing substantive work in the field. This limits the possibility for separate assessment of methodologies, without in-depth knowledge of content. Our assessment in this article is thus limited to transparency and justification; we have therefore not used the full set of criteria listed in the attached framework. In our explorative study, we have not used the criteria of the framework that concern use of sources and methods and have assessed academic and social relevance only marginally, because these depend on substantive quality as well.
In this article, we explore the idea of methodological progress by assessing six dissertations on the basis of the assessment framework for methodology. We choose this approach to build on the work of Herweijer and Tijssen in the early 2000s. Herweijer studied five Dutch dissertations to make an inventory of aims and methods and to evaluate them on the basis of general scientific demands.³x The selection criterion for the five dissertations was that they had been judged ‘excellent’; Herweijer does not mention any other criteria. They are all doctrinal, mainly private law, studies. They date from 1993 to 1999. Regarding the latter, his most significant conclusion is that the methodological justification of the dissertations was weakly developed (Herweijer, 2003, p. 28). They showed limited separate attention for methodological aspects of the research. A similar conclusion is reached by Tijssen in his more extensive study of the methodological justification of ninety dissertations (2006). Tijssen (2006, pp. 211-214) uses three dimensions of justification: of the research problem, of sources, of methods, and scores the extent of justification of multiple aspects of these. He concludes that only the research problem is adequately explained and justified, but that sources and methods are not (ibid., p. 206). One question underlying our assessment is therefore whether methodological justification has moved beyond the limited attention it received in the early 2000s. In order to assess the extent and kind of justification offered now, it seems necessary to move beyond the three dimensions of Tijssen. By distinguishing the theoretical and normative frameworks and academic and social relevance as separate aspects of assessment, it becomes easier to discover whether the studies reflect the more recent developments in legal methodological literature. Thus, we see our exploration as assessing the next step in methodological justification of dissertations rather than as replicating the study by Tijssen. Although this detracts from the comparability to his study, it adds to the understanding of the interplay between theoretical developments in legal methodology and the application of such developments in the practice of PhD research.
Like Tijssen, we selected dissertations primarily on the basis of research fields. However, where he used areas such as criminal law and private law, we distinguished them at a more general level of the disciplinary approach taken. We chose two doctrinal theses, two empirical-legal theses and two theoretical-legal theses. In our view, the recent rise in empirical-legal and interdisciplinary work warrants a shift away from an emphasis on doctrinal dissertations. It also makes it possible to consider whether the disciplinary character of the research influences the attention given to methodology. Moreover, we chose dissertations written in English, in order to make the process more accessible and verifiable for a broader (non-Dutch) audience. However, we focus on dissertations written at Dutch universities, because we expect the methodology debate just sketched to have had some influence there, especially in the training offered within legal graduate schools. Like Tijssen, our selection is diverse as to home universities, although we do not cover them all. The six dissertations are by Lisa Ansems (2021, Utrecht), Lianne Boer (2017, VU Amsterdam), Alice Bosma (2019, Tilburg), Ruben de Graaff (2020, Leiden), Ekaterina Pannebakker (2016, Rotterdam), and Tamar de Waal (2017, UvA Amsterdam). Ansems and Bosma did empirical-legal research, De Graaff and Pannebakker did doctrinal-legal research and Boer and De Waal did interdisciplinary legal research with a theoretical focus.
3 Methodological Justification in PhD Theses

As discussed in the previous section (see also the Appendix), we have devised an assessment framework for evaluating the methodological quality of legal research (including legal-doctrinal, empirical-legal and meta-juridical research). This framework is based on previous studies (among which are Herweijer, 2003; Tijssen, 2006) and our own experience with methodological assessments as lecturers and assessors. As explained above, we do not use the parts of the framework that require substantive knowledge of the topic of the dissertations.⁴x Concretely, this means our analysis here uses the following parts of the framework: I (Section 3.1), II (Section 3.2), III (Sections 3.3 and 3.4), V, VI and VII (together in Section 3.5) and VIII (Section 3.6). In our methodological assessment of the selected six dissertations, we focus on the following assessment criteria: (1) research aims, (2) research questions, (3) theoretical framework, (4) normative framework, (5) methodological justification and (6) academic and societal relevance.

3.1 Research Aims

Considering the research aims, the six theses can be placed on a sliding scale from purely descriptive to predominantly normative. To begin with, De Graaff and Boer solely (or predominantly) have a descriptive purpose. De Graaff (2020, p. 4) wants to develop a ‘scheme of analysis’ by means of which the relation between national law and EU law can be described: ‘By developing this scheme of analysis, the book purports to provide a complete and nuanced account of the impact of the laws of the European Union and their interaction with the national systems of legal protection’. More specifically, he wants to show how concurrence between rules governing private relations is solved in national and EU law. Boer’s aim is

to show ‘what happens’, to point out precisely where it happens, and what it does to practice legal scholarship in this way: to make legal scholarship as tangible as possible in order to provide new insights into legal knowledge construction. (Boer, 2017, p. 28; original italics)

Through a ‘close reading’ of cyberwar discourse, she intends to offer a ‘precise description’ (ibid., p. 20) of how international law in the field of cyber law is constructed (and not merely found, as positivist legal scholars claim). However, she engages in ‘precise description’ not for its own sake; ultimately, it serves a critical purpose. Her ambition is ‘to point out those sites that are not usually the subject of close (critical) scrutiny, and to show how the practices employed there do in fact matter to what we know’ (ibid., pp. 26-27). Especially, she is interested – in the line of Foucault – in the way discourse affects power relations.
Subsequently, the theses of Pannebakker, Bosma and Ansems mainly have a descriptive aim too, but in addition, they give recommendations for improving the law. Pannebakker (2016, p. 5) focuses on the ‘impact’ a letter of intent may have on the ‘general regime of negotiations’. In particular, she is interested in the question to what extent international regulation takes into account ‘the practice of contractually organizing or “privatizing” negotiations made through a letter of intent’ (Pannebakker, 2016, p. 6). In her conclusion, she gives recommendations to the national legislature how to integrate soft law instruments in the domestic law and to legal practitioners how to apply these instruments to the international letter of intent. Moreover, she gives concrete suggestions how the UNIDROIT Principles of International Commercial Contracts (UPICC) could be amended (ibid., pp. 288-289). In her empirical-legal study, Bosma wants to improve our understanding of the ‘underlying processes of secondary victimization by laypersons as well as legal professionals through examining their attitudes toward emotional victims of crime’ (Bosma, 2019, p. 185). Under the heading of ‘implications’, she gives some recommendations, the most important of which is raising awareness among professionals and lay persons about victim stereotypes and how these stereotypes could implicitly affect their responses (ibid., p. 193). The main purpose of Ansems’ thesis is to put previous empirical research into the role of perceived procedural justice to a critical test by studying procedural justice perceptions among defendants in criminal cases. As she puts it: ‘By critically examining the role of perceived procedural justice in these ways, the current dissertation puts procedural justice on trial’ (Ansems, 2021, p. 2). Moreover, she intends to translate her empirical insights into perceived procedural justice to the normative domain of law. After a thorough discussion of the fact-value gap, she gives – hesitantly and carefully – some suggestions how in legal procedures the people’s perception of procedural justice could be enhanced.
Finally, De Waal’s thesis aims at explaining and evaluating integration policies. For that purpose, she combines empirical and normative research methods:

The general methodology of the study is both explanatory and normative. Although I do not conduct empirical research myself, I analytically explain and normatively assess integration policies based on insights gleaned from empirical case studies. As such, one of the objectives of this research project is to show how fruitful the interaction between political philosophy, legal research and social science can be for all these disciplines. (De Waal, 2017, p. 17)

In the introduction, the two research aims – explanation and evaluation – are presented as equivalent. However, in the following chapters (in particular Chapters 4 to 6), evaluation appears to be her main aim. De Waal critically assesses the integration requirements in EU Member States and proposes a new solution, the so-called ‘firewall model’, which divides ‘(mandatory) integration strategies from laws that regulate the allocation of residency and citizenship rights to refugees and family migrants’ (ibid., p. 166).
In general, we can see that in the theses discussed, the research goals are sufficiently made clear. In two cases (Pannebakker and Bosma), the theses go beyond the declared purpose of mere describing or understanding their subject matter by also offering recommendations. In one case (De Waal), the balance between explanation and evaluation is different from what is initially indicated, since the research aim appears to be mainly evaluative.

3.2 Research Questions

The importance of formulating a central research question is generally acknowledged by the authors. With one exception, the empirical study by Ansems,⁵x The statement closest to a question is: ‘Specifically, I assess whether defendants in criminal cases care about procedural justice during their court hearings, what makes them feel treated fairly, and how they respond to experiences of fair and unfair treatment’ (Ansems, 2021, p. 2). In the following, we use this as the central question of her study. all the dissertations contain a clearly designated central question. In the literature, the criteria to assess research questions centre on clarity, scope and feasibility, and alignment with the research aims and problem statement. These criteria overlap, which implies that the following discussion needs to be read as covering the various aspects in connection.
In most cases, the formulation of the research question was understandable in light of the introductory remarks made before. However, there is variation in the clarity of the question itself. Sometimes this is due to ambiguous terms: for instance, what does ‘can’ imply in the research question? ‘To what extent can the law, particularly international regulation, accommodate the practice of contractually organising or “privatising” negotiations made through a letter of intent?’ (as used by Pannebakker, 2016, p. 6). Or it may be a lack of precision in the concepts used (which already goes to the issue of scope as well, see the examples below). While they are all clear enough in the context of the introduction to understand what the research is about, there is still room for improvement in the use of terminology. However, we might ask whether a further focus on the perfect research question would do much to improve the research project as such.
When reviewing the central questions in terms of scope, there is some variation: some of the questions are slightly more general than the focus of the study. For instance, De Graaff also uses a wider scope in his central question: ‘Whether the scheme of analysis conceived and fostered in the context of the national systems of private law can be valued as a source of understanding of the laws of the European Union’ (De Graaff, 2020, p. 13), In his conclusion, however, De Graaff indicates that question concerns ‘whether law permits the interested party to elect the rule of his choice’ (ibid., p. 162), narrowing it down more. Boer formulates the central question as ‘How is legal knowledge constructed in the academic discourse on cyberwar and international law?’ (Boer, 2017, p. 18). Although the term ‘legal knowledge’ is broad, it is clear that the research concerns a specific part of international law scholarship. Overall, the dissertations show sufficient attention for the need to delineate the scope of the research through the research question.
An interesting aspect is the issue of alignment with research aims. As we discussed, the six dissertations vary in their research aims, ranging from descriptive and explanatory aims to normative assessment and recommendations. Interestingly, even the two dissertations that are presented as empirical work, by Ansems and Bosma, include normative statements and recommendations.⁶x As discussed in Section 4.1. These normative parts are not reflected in the central research question formulated at the start of the work. Both of them focus their central question on the descriptive part of the study, leaving out the explanatory framework and the normative assessment.⁷x See footnote 5 for Ansems’ research question. Bosma’s central question is: ‘In which way and to what extent do emotional expressions of victimization trigger victim-oriented strategies in laypersons versus legal professionals?’ (Bosma, 2019, p. 19). Here, we could say that the research questions are not sufficiently general, being limited to only part of the research aims. Similarly, the character of the research by De Waal – ‘This is a dissertation in legal and political philosophy’ (De Waal, 2017, p. 16) – is not recognizable in the central research question: ‘How to explain and evaluate the growth of integration requirements in multiple EU Member States over the last two decades?’ (ibid., p. 15), which could be a purely policy-oriented question. The question is very clear, however, about the combined explanatory and normative aims of the dissertation. In the two more doctrinal studies by De Graaff and Pannebakker, this is left open in the research question, although it is clear from the introductions that they both also combine explanatory and normative analyses.⁸x Respectively: ‘… whether the scheme of analysis conceived and fostered in the context of the national systems of private law can be valued as a source of understanding of the laws of the European Union?’ (De Graaff, 2020, p. 13) and: ‘To what extent can the law, particularly international regulation, accommodate the practice of contractually organising or “privatising” negotiations made through a letter of intent?’ (Pannebakker, 2016, p. 6).
While the importance of central questions is acknowledged, the picture is more diverse when it comes to subquestions. Only three out of six – De Graaff (2020, pp. 13-14), Pannebakker (2016, p. 8) and De Waal (2017, p. 15) – formulate subquestions. If we take the aim of subquestions to be to create a clear subdivision of research tasks to perform in order to answer the central question, it seems that the three who do not use subquestions find other ways to divide up the research. In case of the empirical studies, this seems to be managed by justifying the choice of specific empirical studies in relation to the literature. Here, the fact that these chapters were published as separate articles may also play a role in downplaying the need for subquestions. For the three dissertations with subquestions, different roles are given to these questions. For De Waal, the subquestions are an initial way to explore more specific directions and she speaks of them in the past tense ‘When I began this study, the sub-questions I had … in mind’ (De Waal, 2017, p. 15). She does not explicitly answer them, or return to them in the conclusions. In Pannebakker’s study, it is striking that the comparative nature of the study is only revealed in the subquestions, not in the main research question (Pannebakker, 2016, pp. 7-8), while this is the main methodology of the dissertation. De Graaff formulates subquestions for the first part of the research, but leaves them implicit for the subsequent parts (De Graaff, 2020, pp. 13-14). Overall, subquestions do not appear to be regarded as an essential feature of a research design.

3.3 Theoretical Framework

Since all dissertations have a descriptive research goal solely or to a large extent, they can be expected to offer a theoretical framework in which the theoretical approach or approaches taken to achieve this goal are clarified, including the central concepts. Not every thesis contains an elaborated theoretical framework. In her study on the letter of intent, Pannebakker uses a legal-doctrinal approach. This approach is only indicated without any further explanation: ‘In drawing its conclusions, this research will rely primarily on classical legal hermeneutics: analysis of legal scholarship and discussion of the internal coherence and logic of a given legal framework’ (Pannebakker, 2016, p. 13). The central concepts ‘letter of intent’ and ‘contract’ are discussed briefly. Pannebakker explains why she opts for a restrictive definition which conceives of letters of intent as ‘contractual agreements preliminary to the conclusion of the final contract or contracts’ (ibid., pp. 1-3). De Waal (2017, p. 16) presents her research as ‘interdisciplinary work’. However, a methodological justification of the interdisciplinary approach taken is lacking. As a consequence, it remains unclear how she intends to combine the various (legal, theoretical and empirical) perspectives. She indicates that her analysis involves what Kymlicka has called a ‘mid-level theory’ which does not address ‘foundational philosophical questions’ or ‘highly detailed and intricate case studies’ (ibid., p. 17). Instead, she intends to provide ‘correct legal exegeses of EU and domestic laws’, ‘sound empirical descriptions of the standard conceptualizations of “integration”’ and ‘constructive normative analyses’ (ibid.). However, she does not explain how she is going to do that, by means of which theoretical approaches. So it is not clear how she wants to achieve her first research goal of explaining integration policies. Moreover, it is not clear what exactly needs to be explained. She does discuss, albeit briefly, her key concept of ‘integration’ (ibid., pp. 17-18).
In the theses of De Graaff, Boer, Bosma and Ansems, more elaborated theoretical frameworks can be found. For his ‘scheme of analysis’, De Graaff needs a theory by means of which the relation between concurrent rights and duties in national law and EU law can be described. First, he discusses the hierarchical Kelsenian model and some institutional models (based on, for instance, the Code Civil or the Bürgerliches Gesetzbuch). In addition, he explains why he considers them unsuited for his purpose:

If we wish to understand the relationship between concurrent rights and duties, our focus should be on the legal relations between persons and not on the institutional or hierarchical structure of the legal system. In other words, we should view the law from the perspective of the individuals involved. (De Graaff, 2020, p. 9)

After that, he introduces Hohfeld’s theory of legal relations and discusses extensively its central concepts ‘claims’ and ‘powers’. So De Graaff makes clear why he chooses this theoretical approach and what it consists of. As Boer (2017, p. 20) indicates, her inquiry ‘stands in a critical tradition of international legal scholarship, as expressed in the works of, for example, Anne Orford and Fleur Johns’. It aligns with the ‘turn to practice’ in international law and follows earlier structuralist analyses of international legal arguments provided by Koskenniemi (among others). For the linguistic analysis of cyberwar discourse, Boer draws in particular on the works of Hyland. She discusses his view on disciplinary discourses briefly in her introduction. In each of the following chapters, she clarifies the linguistic methods used for analysing the cyberwar discourse. We notice that Boer pays much attention to her theoretical framework and connects it to other approaches. For readers unfamiliar with the various sociological and linguistic theories referred to, the text may sometimes be difficult to follow. In Bosma’s study on emotive justice, the theoretical framework consists mainly of the Belief in a Just World Theory. This theory is discussed briefly in the introduction (Bosma, 2019, p. 12 ff.) and elaborated in the first part of her research (Chapters 2 to 5). The key concept ‘emotion’ is discussed, but not very extensively (ibid., pp. 43-44). Finally, Ansems discusses at length the central concepts ‘procedural justice’ and ‘fair process effect’ and shows how these concepts are defined in previous research (Ansems, 2021, pp. 3-11). She builds on the current approach to perceived procedural justice, developed by Taylor and others, but also wants to put its central claim that perceived procedural justice has a positive effect on people’s attitudes and behaviours to a critical test.
As we can see, in most cases, the thesis contains a theoretical framework and defines, to a greater or lesser extent, its central concepts. Remarkably, the concept of law is discussed in almost none of the PhD theses. Since ‘law’ is an essentially contested concept in legal scholarship, some clarification could have been expected. It seems that the standard positivist conception of law, according to which law is equated with official enacted law, is taken for granted by most scholars. De Graaff discusses in length what ‘legal relations’ are but not what makes them ‘legal’ in the first place. He simply states that he studies the legal systems in various countries (the Netherlands, Germany, France and the United Kingdom) and the European Union (including regulations as well as directives). In her first chapter, De Waal discusses integration requirements in EU directives, international treaties, domestic laws in various EU Member States and case law. Ansems and Bosma depart in their empirical-legal studies from the official criminal law. The only exception is Pannebakker, who includes in her research not only ‘hard’ law but also ‘soft’ law. As she argues, international law consists to a large extent of soft law instruments. Although soft law lacks binding force, it ‘possesses high persuasive authority due to the reputation and experience of its drafters’ (that is, ‘renowned academics’; Pannebakker, 2016, p. 3). Boer’s conception of law would also fit with this interactionist conception of law. She studies how international cyber law is construed from an internal perspective and adopts the authoritative conception of law as postulated by legal experts in the field (among whom Michael N. Schmitt). She does, however, not refer to the notion of soft law.

3.4 Normative Framework

As a distinct part of the theoretical framework, we discuss the normative framework separately. If a thesis only contains descriptive statements and no evaluative claims, no normative framework is required. This seems to be the case with Boer and De Graaff. With respect to Boer, however, it could be argued that – since her ‘precise description’ ultimately serves a critical purpose – she has to make explicit on the basis of what standard(s) she assesses international legal scholarship. In her view, international legal scholarship should be more critical and self-critical about how it produces knowledge (see further below), but it is not elaborated where this ‘should’ exactly come from. Pannebakker, Bosma and Ansems, who mainly have a descriptive aim, also give recommendations for improving the law or the legal practice. For that purpose, a normative framework is needed which provides the standards from which the recommendations may be derived. In the studies of Pannebakker and Bosma, an explicit normative framework is lacking. In her conclusion, Pannebakker gives concrete suggestions how the UPICC could be amended, without explaining why this would be desirable. Moreover, she gives recommendations how legal practitioners could apply soft law instruments to the international letter of intent. These recommendations seem mostly of a practical nature, as Pannebakker (2016, p. 289) indicates, ‘to ascertain the legal effects of the letter of intent’. In the final part of her conclusion, Bosma presents some general ‘practical implications’, which seems to downplay the fact that is actually giving normative, and possibly controversial, recommendations. According to her, it is important that professionals and lay persons become more aware of victim stereotypes and how these stereotypes could implicitly affect their responses. A possible ‘implication’ of this increased awareness is that legal professionals could or should show more empathy towards victims. As Bosma notices, professionals such as judges and prosecutors are often hesitant to do so, because they do not want to compromise their impartiality. She does not, however, discuss on a principal level the possible tension between showing empathy and keeping one’s impartiality and how to deal with this tension. Without further explanation, she takes the prevention of secondary victimization as the highest value and bases her normative recommendations on this standard.
By contrast, Ansems offers an elaborated reflection on her normative framework, in particular on how normative recommendations can be derived from her empirical findings. She is well aware of the is-ought-gap: ‘empirical findings can have important practical implications, but one needs to recognise that empirical findings with regard to the way things are do not in themselves warrant normative conclusions about how things ought to be’ (Ansems, 2021, p. 13, see also p. 132). In order to ‘bridge the gap’, two conditions must be fulfilled: ‘I suggest that the translation of empirical findings into normative conclusions needs to be (1) explicit and (2) underpinned by arguments’ (ibid., p. 134). Accordingly, she gives an extended justification for the recommendations she derives from her empirical research. After having established on an empirical level that perceived procedural justice does matter for victims, she posits it as her highest value on a normative level. In her view, perceived procedural justice is important, not only for instrumental reasons (because victims benefit from it) but also as a matter of principle: ‘it could be argued that having people feel treated fairly has value in itself and that decision-makers “are morally obligated to treat [decision] recipients in a humane, respectful manner”’ (ibid., p. 137, citing Brockner and Wiesenfeld). Moreover, she links the goal of enhancing procedural justice explicitly to generally accepted goals within positive law, legal practice and legal theory (in particular, to the notion of ‘responsive law’). She phrases the recommendations drawn from her empirical research very carefully, using the conditional form (if/then) or the potentialis (can/could).⁹x For instance: ‘Thus, my findings suggest that it can be relevant for legal practitioners to try to enhance defendants’ procedural justice perceptions …. For instance, legal practitioners who aim to enhance perceptions of procedural fairness would do well to focus on conveying neutrality, because perceived neutrality appeared to play a key role in shaping defendants’ fairness perceptions’ (Ansems, 2021, p. 129). She also discusses counter-arguments against her focus on perceived procedural justice. She does not refute these counter-arguments (or only occasionally, using again the potentialis), but takes them as a warning to be cautious with giving recommendations:

The preceding sections suggest that, in addition to various possible reasons one might want to enhance people’s perceptions of procedural fairness, there may also be reasons for exercising some restraint in this regard. I think it is important to keep these reasons in mind. (ibid., p. 139)

From an academic point of view, this methodological rigour and carefulness can only be applauded. From a practical point of view, it may have some disadvantages (see further below).
In our view, De Waal’s main research goal is evaluation. She assesses critically the integration requirements in EU Member States and proposes an alternative approach, the so-called ‘firewall solution’. In her introduction, she presents her normative framework in rather general terms. It is not based on a ‘specific conception of liberal democracy’, but on the ‘core values of the EU’:

[m]y argumentation is grounded in the core values of the EU of ‘human dignity, freedom, democracy, equality, the rule of law and respect for human rights including the rights belonging to minorities’ that, according to the Treaty of Lisbon ‘are common to the Member States in a society in which pluralism, non-discrimination, tolerance, justice, solidarity and equality between women and men prevail’. (De Waal, 2017, p. 17)

What these core values exactly entail and how they relate to her topic, she does not explain. She also indicates that she intends to combine analyses of domestic and EU legal frameworks with ‘normative political and legal theory’ (ibid., p. 19), without specifying which theories she intends to discuss and for what reason. In the following chapters, she does offer arguments for her normative claims. For instance, she defends her ‘firewall solution’ on ‘a combination of principled and pragmatic arguments’.¹⁰x See De Waal (2017, p. 140): ‘First, I argue that the firewall is normatively desirable because it prevents states from misusing the clear asymmetry in power relations between them and residing refugees and family migrants. Second, I defend that a benefit of the firewall is that it curbs the ability of states to reinforce the reward paradigm of naturalization, by framing rights as something to be earned through demonstrating the ability to pass the hurdles of integration requirements. Third, I explain that if the firewall would be in force, receiving EU countries will be inclined to adopt a broader perspective on the purposes of integration policies.’ It is not fully clear, however, what exactly the grounds are – e.g., legal norms, policy considerations, principles or values – on which these principled and pragmatic arguments rest. In response to possible criticism, De Waal invokes the notion of citizenship. She defends a broad conceptualization of this notion, for which she offers a fairly general justification:

Liberal-democratic states (committed to individual freedom, equality, the rule of law, etc.), are, first and foremost, inclusive and protective political projects. This means that their core ambition is to offer individuals maximum protection in terms of rights, even if their individual characteristics and competences do not coincide or even contrast with those of the majority. The best understanding of citizenship is therefore that certain levels of knowledge, skills, language levels and dispositions are unquestionably important to factually exercise citizenship. (ibid., p. 161)

In general, her reasoning seems more pragmatic and policy based (given this goal, these are the best means) than principle based (given these values, this is the most just approach).
We conclude that in most of the theses that offer an evaluation of the existing law or legal practice¹¹x Or, in Boer’s case, legal scholarship. and/or give recommendations for improving the law or the legal practice, an elaborated normative framework is lacking. The only exception is Ansems’ thesis which contains, among other things, a comprehensive theoretical reflection on how to translate empirical findings into normative recommendations and how to avoid possible pitfalls.

3.5 Methodological Choices and Demarcation

The diversity of approaches to methodology, the justification of methodological choices and their limitations is striking across scientific disciplines. To a significant extent, this diversity is also apparent when looking at the doctrinal, theoretical and empirical approaches of these dissertations. Where the theoretical interdisciplinary studies pay attention to theoretical and normative underpinnings of their frameworks, they are relatively silent about the specific methods used and do little to justify the choices for particular theoretical approaches or case studies. They explain the general approach and what they do, rather than why they do it and the particulars of how this is done. On the other end of the spectrum, the two empirical studies are very much in line with the practice of empirical social sciences to justify methodological choices in relation to the state of the art and the theoretical framework and to address the limitations of the study. The two doctrinal works focus on justifying certain choices: Pannebakker (2016, pp. 11-16) extensively discusses and justifies the comparative method; De Graaf explains and justifies the role of the conceptual schemes he uses for systematization and interpretation, as was discussed in the section about the theoretical framework. De Graaff (2020, pp. 15-18) explains the choice of legal sources and jurisdictions to compare, but not the choice of academic sources, while Pannebakker (2016, p. 10) explains how she researches the letter of intent as a practice, through secondary negotiation studies, and discusses the sources of law in the studied jurisdictions and relevant international instruments (ibid., pp. 11-13), but has only general remarks on further doctrinal sources and methods (ibid., p. 14).
When it comes to methodological choices, this disciplinary diversity is apparent with the exception of conceptual clarification. All studies pay attention to problematizing or operationalizing their central concepts. At the level of the choice of a doctrinal or more interdisciplinary approach, the doctrinal studies take that characteristic for granted: they state that this is what they do in a specified sense, but do not justify why. Pannebakker (2016, p. 7) implies this, for instance, when she says: ‘The research addresses general contract law …’. De Graaff (2020, p. 14) states this as part of his methodology: ‘This book is not, therefore, interdisciplinary in its approach. As a product of legal doctrinal research, the book analyses formal legal materials with the objective of revealing statements relevant to understand the legal questions raised’. The only parts of the approach that seem to merit justification are the non-doctrinal elements and the particular version of comparative law. The empirically oriented studies justify why further empirical work is needed, by claiming that ‘legal professionals have received little attention’ (Bosma, 2019, p. 15) and ‘acknowledging both the strengths and the weaknesses accompanying individual research methods, the current dissertation adopts a mixed methods approach’ (Ansems, 2021, p. 14). The most explicit in terms of approach are the theoretical studies, which justify the combination of theoretical framing and the interdisciplinary approach more extensively (see Boer especially, who devotes almost her whole first chapter to this issue). Only the empirical studies pay attention to the concrete research design, and the choice of courts, respondents and the like: for instance, Ansems justifies why she does two field studies rather than an experiment only (ibid.: 12), and Bosma has the explicit aim of improving measurement techniques and justifies the vignette study and questionnaires in this context (‘I would like to improve the measurement of strategies in BJW research’, Bosma, 2019, p. 18). This is a clear difference: the other four are not concerned with concrete methodological advancement the way the empirical studies are. For instance, Boer chooses three particular discourses within the international cybersecurity debate as ‘close-ups’, which are methodologically key, because ‘together they deal with the “what” and “who” of legal knowledge construction in the cyberwar discourse’ (Boer, 2017, p. 29) The character and scope of these close-ups, other than their linguistic focus and the focus on ‘what’ and ‘who’, remain vague. It seems the reason to choose these as exemplars is mainly content-driven.¹²x For instance, regarding the first close-up Boer (2017, p. 67) writes that it ‘looks at those academic pieces in which the boundary of force is related to the exercise of economic coercion.’ Moreover, the justification for investigating the texts by Michael N. Schmitt in the third close-up seems to lie mainly in Boer’s interest in Schmitt as ‘the gatekeeper of who gets to be “in” and “out” of the game; who gets to have a say in matters of international law’ (p. 116). On a more general level, Boer can certainly also be seen as interested in methodological advancement: she proposes a new approach. The same is true of De Waal, who claims an innovative approach (De Waal, 2017, p. 17) and uses standard methods in an implicit fashion for each of the contributing disciplinary perspectives in her work. The two doctrinal dissertations focus their methodological justification on comparative method (Pannebakker) and conceptual framework (De Graaff) respectively. They seem to reflect the sense that only potentially problematic or unexpected choices merit justification.
In the context of scientific integrity, researchers are expected to reflect upon the limitations of their study and their own positioning. While an issue such as the generalizability of findings is flagged in empirical research practices, this is less prominent in theoretical and doctrinal work. That division is also clear within this group of dissertations. Both Ansems and Bosma explicitly address generalizability in terms of limitations: for instance, to what extent their empirical findings are generalizable across cultures and legal contexts (Ansems, 2021, pp. 122-123) or are representative of a larger group (Bosma, 2019, p. 191). In the other projects, generalizability is not an explicit issue, in part perhaps because it is not something that directly arises in most doctrinal or theoretical projects that do not include elements, such as case studies, that need to be given broader meaning. However, claims are made in the comparative law context by both De Graaff (2020, p. 17) and Pannebakker (2016, p. 11) that the selected jurisdictions represent relevant legal traditions of common law and civil law. The limitations of this selective approach are only marginally discussed or not discussed at all.¹³x Pannebakker (2016, p. 18) discusses the objection of bias towards western jurisdictions. De Waal discusses the limitations of particular normative approaches (De Waal, 2017, p. 81), other than her own, while Boer is clearly self-reflective about the research process itself (Boer, 2017, pp. 155-56), and thus very aware of possible biases, but less interested in an issue such as generalizability. On these issues too then, the empirical studies are the only ones to address them systematically. One possible explanation for this is that these studies are not primarily legal studies in the first place: they align themselves with the research traditions of empirical social science, in which methodological justification and discussion of limitations are part of virtually every project description. So the extent to which these studies show progress in methodological awareness is therefore questionable: they follow the conventions of another discipline.

3.6 Societal and Academic Relevance

Academic legal research is nowadays expected to be relevant both in the academic and societal respects. In two of the six theses, this issue is addressed in a very short and general way. In her introduction, Pannebakker (2016, p. 6) states:

The research is addressed to legal practitioners and scholars. A better understanding of private regulation of negotiations may contribute to academic knowledge on international trade usages. In practice, it may also enhance legal certainty and financial security of the parties at the beginning of an international business relationship.

No further explanation is given, so the academic and societal relevance of her study remains unclear. What exactly does it contribute to the existing body of knowledge? And how can it help parties in an international business relation? According to De Graaff (2020: 172), his findings are relevant for scholars and practitioners of private law and EU law, since it is important for them ‘to understand how questions of concurrence are debated and solved’. Moreover, they may also be of interest ‘for those participating in the ongoing debate about the nature of the Union legal order’. The conclusion is, however, that issues of concurrence remain complicated: ‘… the question as to who is to decide the ultimate boundaries of Union competences remains contentious’ (ibid., p. 173). As a result, his scheme of analysis can only offer some very general topical questions how these issues could be solved.
Ansems pays attention both to the academic and the societal relevance of her study. With a reference to Popper, she aims to put current theories of procedural justice to a critical test. For that purpose, she uses innovative methods which include participants who tend to be underrepresented in behavioural sciences, namely defendants in criminal cases (in Chapters 2 and 3) and people with a non-western ethnic background (in Chapters 3 and 4). However, this critical examination seems to ultimately serve a social purpose: ‘Given the stakes involved in real-life courtroom contexts, studying procedural justice perceptions among defendants in criminal cases allows for a critical examination of the role of perceived procedural justice. This is my main aim in the current dissertation’ (Ansems, 2021, p. 2). In order to enhance perceived procedural justice, she draws some normative recommendations from her empirical findings. However, due to her many caveats and careful formulations, the societal relevance of her research is somewhat downplayed. In our view, the academic relevance is more convincingly elaborated, as reflected in the critical testing of current theories on procedural justice by means of innovative research methods.
In the remaining three theses, the academic relevance gets the most emphasis. Boer (2017, p. 153) aims to ‘shed new light on what knowledge construction looks like in legal scholarship and what the cost is of the practices we employ’. Her ‘close reading’ of cyberwar discourse by means of linguistic methods is innovative and, ultimately, has a critical goal. As she indicates in her conclusion, Boer intends to make international legal scholarship more critical and self-critical about the way it produces knowledge. More specifically, she appeals to the community of international legal scholars ‘to be careful with the use of consensus claims in our arguments, and to be aware of the effects of citations’ (ibid., p. 155). By using these means, some voices are included in the discourse, whereas others are excluded. Remarkably and refreshingly, Boer’s research does not pursue any societal relevance. On the contrary, she warns doctrinal lawyers in general and scholars involved in the cyberwar discourse in particular about the ‘cost of application-orientation’ because of its ‘distributive effect’ (ibid., p. 154).
In her introduction, De Waal (2017, p. 19) stresses the academic relevance of her study:

This study attempts to bridge academic discussions on immigration, integration and citizenship that often take place in isolation from one another …. That said, this study aims to further the academic debate by combining analyses of domestic and EU legal frameworks concerning the integration of newcomers with normative political and legal theory …. However, as we will see, each of these fields of academic research provides relevant points and arguments, but, none of them, as currently elaborated, has succeeded in fully integrating the legal, philosophical and empirical debates about the growth of (mandatory) integration requirements for newcomers in European states. This has created a gap in the literature, lacking a comprehensive evaluation of these policies that discusses their legal form, engages in applied normative reasoning and proposes possible institutional reforms for EU states that are feasible on the short-term and have the potential of leading to better (disaggregated) integration outcomes.

Not fully clear is what this alleged ‘gap in the literature’ precisely consists of and how a ‘comprehensive evaluation’ could help to fill this gap. In her conclusion, the societal relevance gets more attention:

the findings of this study are highly critical of the current developments concerning integration in Europe, on multiple levels: regarding the political and rhetorical tendency to operate from individualised understandings of integration, as well as the societal ramifications of the implementation of public policies based on this understanding. (ibid., p. 168)

In our view, the relevance of De Waal’s thesis should primarily be located on a societal level. She seems to be more interested in influencing migration policies in EU Member States than in developing normative legal and political theory. As a critique on current integration requirements in Europe, it offers a valuable contribution to the public debate.
Bosma (2019, p. 15) is very explicit about the academic relevance of her research:

How criminal justice professionals react to victims, which strategies they use, and under what circumstances they use particular strategies, is currently understudied. In general, we know that legal experts may have similar biases as non-experts, but that some of these biases are reduced through an expertise effect or safeguards in the field …. The extent to which legal professionals are motivated by the BJW [Belief in a Just World] mechanism is currently unknown. This dissertation will provide a first step to address this gap in the literature.

It is clear to us what gap she wants to fill. Whether it really is a gap, or a gap worth filling, we as outsiders cannot assess. About the societal relevance of her research, Bosma is less explicit. She hints upon it in conditional terms: ‘The societal relevance of the crossover between BJW and the legal sphere depends on the magnitude of the problem of secondary victimization within the legal practice’ (ibid.). In addition, several opinions on this issue are discussed, but no clear answer is given. The ‘practical implications’ she presents in her conclusion, aimed at ‘raising awareness’, are rather vague and, therefore, will contribute little to the social impact of her research.
The overall picture is somewhat mixed. In all theses, the academic relevance of the research is discussed to a greater or lesser extent. In three cases (Ansems, Boer and Bosma), it was clear to us what the thesis intended to contribute the existing literature and what could make it innovative. Generally speaking, less attention was paid to the societal relevance. In three cases (Pannebakker, De Graaff and Bosma), it is indicated very briefly. In one case (Boer), it is not addressed at all, or only in a critical way. In the case of Ansems, the practical usefulness of her recommendations is seriously hampered by the many caveats she makes. An exception is De Waal’s thesis, which contains a strong criticism of the current integration requirements in EU Member States and offers an alternative solution. (The question whether this constitutes a viable alternative falls outside the scope of this article.)
4 Progress in Legal Methodology

So the question is: can we, in the end, see any progress in the methodological justification of legal research? Due to the limited size of our sample (we only looked into six PhD theses which were defended during the past ten years at some faculties of law in the Netherlands), we cannot give a conclusive answer to this question. In order to make generalizable claims, of course much more research is required. What we can do, however, is to give an indication of some general tendencies within our small batch. Let us start with the positive points. First, in all dissertations, the research goal was sufficiently made clear (albeit in a few cases something more or something else was done than initially was indicated). Second, all the dissertations contain a clearly designated central question. Moreover, the formulation of the research question is mostly understandable given the preceding introductory remarks. Third, in most cases, the thesis contains a theoretical framework and defines, to a greater or lesser extent, its central concepts. Fourth, the two empirical-legal theses justify extensively the methodological choices made, clarify the methods used and specify the limitations of their research. In the doctrinal studies, some of the methodological choices are justified, in particular regarding the methods used. Finally, in half of the cases, the academic relevance of the study is well argued for.
At some other points, however, the theses do not or do not fully comply with the methodological requirements. To begin with, it strikes us that in most theses, no attention is paid to the concept of law, since it is an essentially contested concept. The interpretative methods for understanding the law is not discussed either. Subsequently, the central research question is not always clear due to the use of ambiguous or vague terms. Some research questions are phrased in rather general terms in the introduction and are eventually specified in the conclusion. If giving recommendations is part of the research goal, this is not always reflected in the research question. Overall, subquestions are not included in the research design and do not serve to structure the subsequent research as they are supposed to do. Moreover, in most of the theses that offer an evaluation of the existing law or legal practice and/or give recommendations for improving the law or the legal practice, an elaborated normative framework is lacking. As a consequence, it is not always clear on what standards the evaluation or recommendations given are based, how the standards used are understood, why these standards are used and/or how they relate to other (in particular legal) standards. In particular in the two theoretical interdisciplinary studies, the methodological choices are justified only to a very limited extent. Finally, in three cases, the academic relevance of the research is not convincingly demonstrated and even in four cases,¹⁴x We have not included Boer’s thesis in our count, because she offers a good reason for being critical on the ‘application-orientation’ within legal-doctrinal and international legal scholarship (see above). the societal relevance is underdeveloped.
Building on these cases, we can observe that some progress is made from a methodological point of view, compared with the situation described by Tijssen almost 14 years ago. As Tijssen (2006, 186 ff.) concludes from his quantitative analysis of 90 dissertations, legal academic research scores moderately to poorly on the following five points: (1) the problem statement; (2) the selection and justification of sources; (3) the elaboration and operationalization of assessment frameworks (or what we call ‘normative frameworks’) used; (4) the comparative law method and (5) the justification of non-legal, empirical methods. Based on our limited qualitative analyses of six theses, we can see some progress in particular with respect to the description and justification of comparative law method and the empirical-legal methods used, the description of the theoretical framework (including the operationalization of key concepts) and the demarcation of the central research question. However, much remains to be desired, especially regarding the justification of the normative framework and the methodological choices made, the clarity of the research question, the relation between the central research question and the research aims, and the relevance of the research, both on the academic and societal level. The debate on legal methods that started with Stolker’s article (see Section 1), has undoubtedly contributed to a higher awareness of the importance of justifying methodological choices among legal scholars. In our view, however, it has not led to a significantly better practice of methodological justification, at least not on all assessment criteria. The normative framework, for instance, still is an important point for attention. In most of the cases we examined, it was not sufficiently clear on which standards the evaluation of the existing law or legal practice and the recommendations were based, why these standards were selected and/or how they were operationalized.
We like to conclude with two caveats. First, we have assessed the dissertations exclusively from a methodological point of view. Although we deem a methodological justification important, we do not mean to disqualify the theses if they show some shortcomings in this regard. We do believe that the six theses discussed are valuable and interesting pieces of scholarly work. We appreciate very much – to name a few things – Ansems’ methodological rigour and carefulness, Boer’s meticulous and merciless dissection of how international cyber law is constructed, Bosma’s discussion of the Belief in a Just World Theory, De Waal’s forceful criticism of integration requirements in EU Member States, De Graaff’s conceptual framework for analysing concurrence between legal rules, Pannebakker’s attention for soft law in international private law relations, and so on. Like all products of human creation, they are of course fallible and susceptible for improvement. Second, we acknowledge that, even if all methodological requirements would be met, the resulting research would not necessarily be good or interesting from an academic point of view (see Tijssen, 2006, p. 191). In fact, academic publications would become unreadable, if they would discuss all methodological issues extensively.¹⁵x The experts interviewed by Tijssen also warned against ‘thick unreadable research reports’ (Tijssen, 2006, p. 150, our translation). So some restraint is needed here. As a rule of thumb, we would suggest that only those issues have to be discussed that are controversial or not self-evident within the community of scholars for which the publication is written.¹⁶x This is in line with what the experts interviewed by Tijssen advised. As Tijssen (2006, p. 148, our translation) writes, they are ‘more inclined to assume that the more is deviated from what is customary within the field, the greater the need for accountability’. In the legal community, for instance, as we argued, the concept of law cannot be taken for granted nor the interpretative methods by which the law is understood.¹⁷x Traditional legal scholars would probably be inclined to consider this ‘olifantenpaden’ or customary shortcuts, as one of the experts interviewed by Tijssen (2006, p. 147) put it. However, since the choices made here often remain implicit, it cannot be taken for granted that everyone takes the same shortcut regarding the conception of law and the applicable interpretative methods. Conversely, publications that are heavily criticized on methodological grounds (such as the works of Freud, Heidegger and Agamben), can still be very interesting and valuable from the viewpoint of knowledge development. Methods matter, but they are not everything.
The Six Dissertations
Ansems, L. F. M. (2021). Procedural justice on trial: A critical test of perceived procedural justice from the perspective of criminal defendants (dissertation Utrecht), Alblasserdam: Ridderprint.
Boer, L. J. M. (2017). International law as we know it: Cyberwar discourse and the construction of knowledge in international legal scholarship (dissertation Vrije Universiteit Amsterdam), privately published (a revised version was published by Cambridge University Press in 2021).
Bosma, A. (2019). Emotive justice: Laypersons’ and legal professionals’ evaluation of emotional victims within the just world paradigm (dissertation Tilburg). Tilburg: Wolf Legal Publishers.
De Graaff, R. (2020). Concurrence in European private law (dissertation Leiden). Den Haag: Eleven International Publishing.
De Waal, T. M. (2017). Conditional belonging: A legal-philosophical inquiry into integration requirements for immigrants in Europe (dissertation University of Amsterdam). www.researchgate.net/publication/338112924_CONDITIONAL_BELONGING_A_LEGAL-PHILOSOPHICAL_INQUIRY_INTO_INTEGRATION_REQUIREMENTS_FOR_IMMIGRANTS_IN_EUROPE (a revised version was published by Bloomsbury in 2021).
Pannebakker, E. (2016). Letter of intent in international contracting (dissertation Rotterdam). Mortsel: Intersentia.
Barendrecht, J. M. et al. (2004). Methoden van rechtswetenschap: Komen we verder? Nederlands Juristenblad, 79(28), 1419-1428.
Curry-Summer, I. et al. (2010). Onderzoeksvaardigheden: Instructies voor juristen/Research skills: Instructions for lawyers. Nijmegen: Ars Aequi Libri.
Herweijer, M. (2003). Juridisch onderzoek. In J. Broeksteeg & E. Stamhuis (Eds.), Rechtswetenschappelijk onderzoek: Over object en methode (pp. 23-33). Den Haag: Boom Juridische Uitgevers.
Kestemont, L. (2018). Handbook on legal methodology: From objective to method. Cambridge: Intersentia.
Snel, M. (2016). Meester(s) over bronnen: Een empirische studie naar kwaliteitseisen, gevaren en onderzoekstechnieken die betrekking hebben op het brongebruik in academisch juridisch-dogmatisch onderzoek. Den Haag: Boom Juridisch.
Stolker, C. (2003). “Ja, geleerd zijn jullie wel!”. Over de status van de rechtswetenschap. Nederlands Juristenblad, 78(15), 766-778, https://www.researchgate.net/publication/28641012_%27Ja_geleerd_zijn_jullie_wel%27_Over_de_status_van_de_rechtswetenschap.
Stolker, C. (2014). Rethinking the law school: Education, research, outreach and governance. Cambridge: Cambridge University Press.
Taekema, S. (2018). Theoretical and normative frameworks for legal research: Putting theory into practice. Law and Method, 2018 (2), 1-17, https://doi.org/10.5553/REM/.000031.
Tijssen, H. E. B. (2006). De juridische dissertatie onder de loep: De verantwoording van methodologische keuzes in juridische dissertaties (dissertation Tilburg). Den Haag: Boom Juridische Uitgevers.
Van Dijck, G., Snel, M. & van Golen, T. (2018). Methoden van rechtswetenschappelijk onderzoek. Den Haag: Boom Juridisch.
Van Gestel, R. & Vranken, J. (2007). Rechtswetenschappelijke artikelen: Naar criteria voor methodologische verantwoording. Nederlands Juristenblad 82(24), 1448-1461.
Van Gestel, R., Micklitz, H.-W. & Rubin, E. (Eds.) (2017). Rethinking legal scholarship: A transatlantic dialogue. Cambridge: Cambridge University Press.
Van Hoecke, M. (Ed.) (2011). Methodologies of legal research: Which kind of method for what kind of discipline? Oxford: Hart.
Vols, M. (2021). Legal research: One hundred questions and answers. The Hague: Eleven.

Appendix: Assessment framework

I. PROBLEM DESCRIPTION AND RESEARCH AIMS

Is it clear what the central (social and/or academic) problem is that the study intends to address?

Is this a relevant, important and/or interesting problem? (This question is closely related to the societal and academic relevance of the study.)

Is it sufficiently clear what the main research aim(s) of the study is (are)?

II. RESEARCH QUESTION(S)

Is there a central research question?

If so, is the central research question well connected to the problem description and the main research aim(s)?

Is the central research question formulated clearly? (requirement of clarity)

Is it possible to answer the central research question within the scope of this publication and by means of legal and/or legal-empirical methods? (requirement of feasibility)

(If there are sub-questions) Are the sub-questions formulated clearly?

Are the sub-questions well connected to the central research question? (requirement of connection)

Are there sufficient questions and/or not too many questions? (requirement of fit)

III. THEORETICAL AND NORMATIVE FRAMEWORK

Are the key concepts in the study defined, described or operationalized clearly? For instance, it should be clear which concept of law is used in the study.

Is the theoretical approach taken sufficiently connected to current theories? (requirement of embeddedness)

(If applicable) Is it clear by means of which normative standards the law at hand is evaluated and/or recommendations are given?

Are these normative standards described or defined clearly?

IV. JUSTIFICATION OF SOURCES

Is the selection of (primary and secondary) sources justified sufficiently? Primary sources are sources that originate from legal practice (such as legislation, treaties, court decisions). Secondary sources offer a reflection on these primary sources, usually but not necessarily from an academic, legal and/or empirical-legal, perspective (in books, volumes, articles, etc.).

Is the selection of sources sufficient? That is, are the most relevant sources mentioned and is the selection up-to-date?

V. JUSTIFICATION OF METHODS

Is it clear by which methods the research question(s) are answered?

Are these methods suited for answering the central research question(s)?

Do these methods clearly reflect the approach, e.g. are they legal-dogmatic, meta-juridical and/or legal-empirical methods?

Are these methods explained and justified sufficiently?

If the study involves legal interpretation: is it clear by which methods the law at hand is interpreted or analysed?

If the study involves comparative law: How is the comparison between various jurisdictions carried out? How does the comparison help to answer the central research question(s)? On what grounds have the countries involved been selected?

If the study involves meta-juridical methods: is it clear which meta-juridical methods and (if applicable) how these methods are connected to other (e.g., legal-dogmatic) methods?

In case of ELS: how is the empirical research carried out? Is the study reliable and (internally) valid, are the findings generalizable (that is, externally valid)? Is it clear (if applicable) how normative recommendations are drawn from the empirical data?

VI. JUSTIFICATION OF OTHER CHOICES

Are the choices concerning the scope of the research explained and justified?

Is the specific operationalization of central concepts justified?

Is the choice of methodological approach, i.e. the overall (sub)discipline or methodology, clear?

Are choices concerning the exemplars used in case studies, court cases, illustrations, etc., justified?

In case of comparative law: are the jurisdictions to compare clearly justified? Is the comparative law approach, e.g. functional, contextual, explained?

VII. LIMITATIONS (ACADEMIC INTEGRITY)

Does the study give a fair account of its limitations (in terms of, for instance, its bias, validity and generalizability)?

VIII. SOCIETAL AND ACADEMIC RELEVANCE

Is the study relevant from a social point of view? Does it address a topical and important social problem?

Is the study relevant from an academic / scientific point of view? What does the study (aim to) add to the existing knowledge? Is it innovative with regard to, for instance, its topic, theoretical approach and/or methods?