CRI Open Science Course

Course materials for the co-design of (open) science research projects - Version 2

At the Center for Research and Interdisciplinarity (CRI) in Paris the Master students of the Master students of the digital, learning and life sciences take a joint course on open science in their first year. After a two-day kick-off workshop, the course 2020-2021 was designed around project-based learning, in which interdisciplinary teams of 4-6 students run their own small research project from start to finish over the course of 12 weeks. To facilitate their work they are accompanied by fortnightly group sessions and the course materials we are sharing here.

The overall topic or “challenge” for the course research projects, in this case, was about learning processes at CRI, but these research design materials can be adapted for other topics and areas.


 * Authors: Enric Senabre Hidalgo, Bastian Greshake Tzovaras, Ignacio Atal & Ariel Lindner
 * Date: March 2021
 * Source: Materials based on previous work from authors, Profs Chercheurs project and Research co-design toolkit. For complementary sources see the legal notes.

= How to use this (living) document?=

This version of open course materials allow to copy and paste it completely or any of its components (sections, subsections, tables, paragraphs, etc), in order to adapt them to the specific needs for other research courses of activities. It’s a first tested version (and a work in progress through future similar courses) which can of course be improved so feedback welcome!

Basic principles

 * 1) Format: This is a template with a suggested sequence of specific steps that can guide you as a team to collaboratively design a research project. It is based on previous materials from researchers at CRI, and requires moving things around and completing different sections.
 * 2) Participation: Each team has a template like this one for the same general challenge (but deal autonomously and independently with their own research project on learning). At some parts you will be invited to contribute individually here with your own ideas, comments and suggestions, followed by sections and ways to decide as a group what to select and prioritize. Please try to consider and respect all contributions equally, and move with collective intelligence and agreements at each stage (and also having fun, when possible).
 * 3) Discussion: One tip is to also generate notes and discussions anywhere during the process, to expand the discussion and other relevant considerations as you move (in parallel to the course meetings). This way, you will also document your learning and research process.

=Identify research topics / concerns=

The following discrete list of topics and potential areas of inquiry represent a first, incomplete but series of motivating issues prepared from the course facilitators at CRI in a short brainstorming session preceding the open science course (in this case, with a focus on learning-related experiences and challenges). Please consider them as departing points or inspiration, so following the next sections you can try to “adopt” or “adapt” them, but also think about different ones which motivate you as a team…


 * 1) Time spent on learning...
 * 2) Virtual vs physical
 * 3) Time of the day
 * 4) Time between classes
 * 5) Sleep patterns during the course...
 * 6) Resources used for learning...
 * 7) Collaborative networks...
 * 8) Stress / mood analysis when studying...
 * 9) Language uses...
 * 10) Learning styles...
 * 11) Phone / screen time (distractions)...
 * 12) ...what else?

=Develop the research question(s)= A research question is what any research project wants to answer. Figuring out and choosing a research question is an essential part of any type of scientific process (open or not). Afterwards the investigation will require data collection and analysis, for which the choice of a methodology is also critical (but we will get to this later).

How to contribute to the development of your team’s research question (RQ)?

 * 1) Think about the brainstorming round based on previous examples of learning challenges and topics, and which specific questions could be derived from them;
 * 2) Write at least one question individually (we recommend anonymously), following the structure suggested below;
 * 3) Become a “peer-reviewer” of the rest of questions following the template, just adding your name to the “Selection criteria” columns to show support (and comments if needed);
 * 4) Originality: Regarding the impression you have that the question has been addressed before, or not, or not properly (good moment for a quick literature review/check on Dimensions, Semantic Scholar, Google Scholar or similar). Remember that all this is about radical openness, so the way you can make things open science also counts here.
 * 5) Feasibility: Remember that you will work together to answer the selected question! Consider if it maybe leads to a very complex process, or some variables will be too difficult, or finding useful data is going to be a hard mission, etc. Here is also a good moment to think about openness, because doing things openly does not usually mean doing things easily.
 * 6) Impact: Here it can be tricky because impact can mean many things. We would like you to think basically about (1) “academic” impact (that is, in relation to originality and what you assume is the state of the art) and (2) “social” impact (which relates to how the project can help to improve or change things for good in a real-life given context).
 * 7) As a general rule, add maximum 10 times your name on RQ table below (so each participants has in total a maximum of “10 votes”) and do not vote as reviewer for your own question(s), in order to allow the most coherent selection process possible.

Selection of main research question (and hypothesis)
Here in this second step, after checking results from the previous section and another round discussing them (if needed), please use the template to indicate the first or main research question your team would like to work on, considering that:


 * 1) You can still “merge” some of the previous questions or modify and improve a given one together, as result of your deliberation.
 * 2) You can also pick up one or two additional questions from the previous stage in case you think they are interrelated, or there’s important subquestions (or you think you may need a “B plan” at some point).
 * 3) Derived from that, we also invite you to describe a hypothesis in relation to the question(s). It is not always necessary (and your project may not need it) but in case you think about one remember that...

...which means that, during this stage, you may need to go back to “reading mode” and invest some time in checking more literature and studies :)

In this section you will also find additional areas with estimation of effort or time for answering the RQ, or where to reflect previous literature and findings, etc. Please use them under your best criteria, and only if you consider them useful!

=Research design and protocol= This phase will help you to reflect how the protocol of the data collection process will take place, as well as other needed logistics for your research.

Diagram of the research process
First of all, think about the key elements to include (from the main icons in the subsections below): research question(s) and hypothesis, concepts or units of analysis, people and groups involved, methods to apply, etc. Select everything that your team thinks would be necessary, discuss it and then organize it as a sequence following a temporary order (from left to right).

Diagram canvas
In order to reflect the overall approach of your research, you have to work on a shared presentation where to reflect the different areas to cover according to your selected research questions and / or hypothesis in the previous stage. Consider this canvas as a basic “collage” which reflects (1) the overall concepts to be addressed, as well as the sampling population of participants to involve; (2) the “flow” of data collection and data analysis methods; and finally (3) some logistics or needed tasks to take into account as well. These important ingredients of your research project range from more conceptual to more practical, and have an implicit sequence or progression you have to consider as well.



In order to fill this canvas by “copy and paste” from the options below, your team has a dedicated template like this one, where you can add more information or details regarding each selected icon, in order to make more clear for you and rest of the course participants what the research plan is about, in general terms. This won't cover more precise considerations or even changes of plans as you move through the data collection and analysis phases in the next stages (and sections of this document), but consider it as a sort of visual point of departure.

Research elements to consider
The following lists contain the main elements or “ingredients” for your research diagram on the canvas, as icons you can directly copy and paste and put in a sequential order on your slides document. It is important, as mentioned, that you add titles or short descriptions to them once selected and reflected on the diagram, for a better general understanding of the research process. Although the list is long, you only need to select a few of them!

High-level concepts related to the research challenge This icon is for questions regarding learning (in relation to this specific topic of the course), as already discussed in your team brainstorming in connection with the selected RQ or hypothesis. It is simply a reminder of key concepts derived from it (it can be many things, according to what you plan to do: accessibility, equality, formal education, stress, learning materials, concentration, motivation or any other of the many derived issues you are already wondering about).

Sampling population / participants Use this icon (more than one time, if needed) to reflect the type of participants you need to observe, ask or obtain data from / about during the research. Think about your team as “self-researchers” but additionally the broad learning and teaching community: students of different types, teachers or mentors, family, education institutions, etc. Depending on the ambition and needs of the research you are planning, now is a good moment to be as specific about them as possible! + Info on sampling

Methods for data collection

We show below a short list of possible data collection methods and techniques among the big diversity of methods usually used for research. Consider it also as a set of recommendations, based on this type of research challenge (about learning experiences) and the specific research questions you plan to answer empirically. You will probably just need one or two of them, since the more you implement the more complex your research will probably be.

If you prefer to use other data collection methods or tools, you can check for more for example at the Guide to research and research methods for Master’s (M.A.) students at the Faculty of Humanities of University of Jyväskylä (Finland), here. Also, for finding the corresponding icons to reflect them, we recommend you to use TheNounProject platform, with all types of icons like these ones, available under Creative Commons licenses.

Methods for data analysis

Below there’s another list of methods to start to consider at this stage of your research project design, in close relation to the ones suggested above. That is, several possible ways to analyze the data you obtain, regarding the scope and objectives of your research question. Again, it is just a series of suggestions for you to consider, which can also influence the previous choices regarding methods for data gathering. For this reason, on your research design diagram they should be placed in close connection to the data gathering methods. Although there could be different things to know in advance and take into account when analyzing your data, we will cover that part of the project in other sections of this document.

If you prefer to explore and use other data analysis methods, you can look for more here.

Research logistics

Finally, another order of things to consider in your research has to do with very practical tasks and skills needed for making it possible. From dissemination to data management, and as something especially important in the way these could be more transparent and coherent with open science principles and practices.

Research design examples
Below we provide an example of the Keating Memorial, a current research project under development at the CRI Peer-Produced Research Lab, which summarises in a visual way (using the same canvas as icons) the project’s alignment of research questions, concepts, methods and some of the “logistics” required. The main aim of the Keating Memorial project is understanding individual processes when doing personal citizen science, as well as the group dynamics taking place in these communities of practice and the role of technological infrastructure during the process. As the base of participants and sample population, it’s focused on self-researchers engaged in the Open Humans community.



As you can see above, two of the main research questions addressed in that project are: (RQ1) “What do individuals learn when participating in personal science and citizen science, and what are their motivations?”; (RQ2) “How are community interactions experienced by people engaged in personal science?”. The methods CRI researchers are currently applying in this case are content analysis of web interactions and semi-structured interviews, previous to a community survey, to get a typology of participants and relevant categories according to the three mentioned concepts (motivations, learning processes and peer support). As mentioned, the research diagram above is for the purpose of an example at this stage of the course, but based on a long-term project with many other specifics (if you want to know more, you can find additional information and related details of the research project on GitHub).

Another example could be self-research done by two members of the Open Humans community regarding the effects of the first covid lockdown at the individual level, specifically regarding productivity and physical activity (for more info see Paula Neonova’s blog and Bastian Greshake Tzovaras’ blog). Although slightly different, the concrete aim of both projects (which we summarize as a single process below) was driven by the challenges derived from personal situations when people were forced to drastically change their daily routines, usually shifting to a home office approach.



The concrete research questions of both self-research projects was to study measurable effects of the confinement in the researcher's behavior and physical activity. The data collection methods used were mainly self-tracking wearables and mobile apps, followed by data visualizations and comparisons.

Research protocol
A research protocol or proposal is a document describing the objectives, design, methodology, statistical considerations and organization of a research project. The research protocol also covers how you will ensure the integrity of the data collected. Here you can find an example of one of the research protocols from the Peer Produced Lab, for the Transbiome project about the exploration of the microbial diversity in the neovaginal microbiome.

At this stage, it is important that derived from your research diagram and the research questions (as well as additional material like references, overall challenge, data management, etc), you complete the following protocol document template with the detailed description of your project, prior to starting the data collection process and rest of research tasks.

=Data collection and analysis=

Project deployment and outreach
As the next stage of your research project, you should consider the different supports, communication strategies and dissemination channels for getting participants to provide data that you can analyse afterwards. For this, the following sections invite you to consider important elements of content, as well as tools needed for the best possible outreach and deployment of the study.

Project title and outline

Rather than the detailed approach needed for the research protocol, in this section you should elaborate a plain, easy to understand short text explaining the general purpose or challenge of the research (but avoiding as much as possible details which can induce bias or affect the expected results). Also add information about the project team, some contact details and mentioning anonymization in data sharing. In case of specific requirements or characteristics for the type of participants needed for the study, this should also be clearly stated in this section. This information will be used for the landing page of your project (at the end of this stage), with a consistent URL at the CRI projects page.

Project background image

The landing page of your project at the CRI directory allows for a representative image as background, for that more “creative” part of the layout you should consider images from repositories like Creative Commons or Wikimedia Commons with copyleft licenses.

Data gathering tool

Whereas a survey or other required tool for data gathering, right after the project description and invitation to participate, you should include the link where participants can provide the data. For surveys you could consider easy and usable tools like LimeSurvey, Google Forms or Typeform. The header text there can also repeat some of the basic information provided on the project's landing / info page, as well as an estimation of approximate time required for filling in the requested information.

Project dissemination

This section refers to all the communication strategies and channels you can consider for reaching out and getting participants to visit your project landing page (providing the specific link). The following list is a first suggestion of possible ones, but feel free to add more or avoid some according to your plan. You can check first as an example of possible content and style the communication templates used for the Covid open survey project.

Data gathering
Following the principle of peer support in the context of the course, as an initial data gathering process we invite the following 3 “clusters” of student teams to fill in the surveys (and other methods, when applied) of each other’s projects:


 * Teams 1-4-9
 * Teams 2-3-5
 * Teams 6-7-8

For this, you have to access the course main document to find the title and links to each project landing page (or survey instead) on the project’s table, where these clusters are also indicated on the right column. Please first make sure you check and update the basic information per each team.

Data analysis
This part of the research process should allow you to start answering the research questions and confirming (or not) your initial hypothesis. As a non-linear process on many occasions, this phase of the research can be started (or “tested”) while the data gathering is still underway, so you can have some initial insights and preliminary results.

Regardless of the stage of data gathering for your project, at this stage of the course, we invite you to start doing some preliminary visualizations below about your progress and possible approaches to analyse your data. In relation to the different alternatives, you can check again the section above “Methods for data analysis” and use one of these main possible tools:


 * Google Forms - How to make a graph in Google Sheets
 * Raw Graphs (open source) - For more elaborated visualizations
 * Gephi (open source) - Even more sophisticated

Prototyping visualizations

In the form below, reflect at least 3 possible approaches to visualize the data gathered through your survey and the selected tool, adding some text for each visualization to reflect its main value or related results.

Important: In case you have not yet gathered enough data via your survey or other methods, for the purpose of this part of the course (and corresponding session) you can also “simulate” the data, in a way that even if you cannot still derive real initial results, you are exploring the possible visualizations and analyses.

=Results, publishing and dissemination=

In this part of the course, we have a session dedicated exclusively to present each team’s results, followed by a round of comments and questions after each project.

Feedback about research results
Below you will find the first feedback from the teacher’s team for the preliminary results presentation session.

Research manuscript
This part of the research process refers to the necessary steps to publish and share your research, following a series of standard practices and formats in Academia (and also new possible open ones). For this, once you have completed the previous stages above, and discussed possible approaches within your team, the best strategy is to start drafting a “manuscript” that puts together all the previous elements you have worked with.

The following is a common format in academic papers (the IMRD model) which we invite you to follow as the last part of your research process. Here, instead of starting things from scratch, you will mostly need to consider the previous sections and content (research questions, state of the art, protocol, visualizations, etc) and “reuse” it in a coherent, easy to follow order.

Title (Up to 16 words)


 * Your title should be a clear premise, carrying the main result of your research and context, strictly confined to what your data supports.
 * Bad example: Gender of teachers determined career outcome
 * Good example: Higher professional esteem of men over women when using identical course narrative by European graduate students

Abstract (Up to 120 words)


 * Carries the crux of your paper. Must include clearly defined phrases on:
 * Framing the context
 * Hypothesis
 * Methodology
 * Main results
 * Conclusion
 * Perspective
 * Here’s an example of how to develop a Nature introductory paragraph
 * We invite you to add a visual abstract (you can also consider that possibility, as an additional form of open dissemination) as often requested in leading journals.

Introduction


 * This section should contain:
 * A) WHY is the area of research important? (1 Paragraph)
 * B) WHAT is the GAP in scientific understanding? (2-4 paragraphs)
 * C) Your hypothesis and HOW do the proposed General and Measurable hypotheses FILL the gap in understanding? (1-3 paragraphs)
 * D) Overall plan (e.g., to this end we set…. And find that…)

Methods


 * Clearly describe your methodology and its choice (including references to precedent use)
 * Include protocol (with link to the actual survey etc)

Results


 * Analysed data in table/graph format
 * Link to raw data

Discussion


 * Interpretation of results (What does it mean?)
 * Any shortcoming
 * Direct implications for the research question at hand

Speculation


 * Free non-peer-reviewed paragraph where you could suggest wider implications to the field and speculate further about the validity and implication of your results in other contexts

=Self-reflection about the research process=

The following questions and sections are for you as a team to reflect your main impressions and learning or findings regarding the course process. Since you developed as students a research project together, from different backgrounds and previous experiences, try to answer the following from a perspective of “students as researchers”, and in order of priority for each section.

Author contributions
Please consider completing the following information with each author’s initials at the end of your manuscript. Reflecting author and researcher contributions is another open science practise that allows to understand the development of a project in more detail, who to address in case of doubts, and have a more precise way to attribute contributions. Feel free to remove types of contributions which don’t apply to your case, or add additional ones if you consider it necessary for your specific research project (like dissemination, translation, testing, etc).

=Bonus track!=

After your research process and teamwork during the course, as part of this “infinite play” of doing (open) science, we would like you to take a good look at the following manual (and leave at any moment your impressions if you want accordingly!).

=Annex: feedback on research process= “Rereading” the generic description and details of the research project proposed by the team can help them to progressively produce a description that is understood in the same way by the rest of course participants, as well as different people and education actors (in this specific research topic case), even if they come from different backgrounds. The objective of the reviews or feedback is to support team members in a benevolent way to help them improve the description and development of their open science research, by pointing out elements that could be improved, modified or could need more clarification. Especially, to signal and discuss opportunities for “opening up” each stage of the research process.

The templates in this section are to be used in specific parts of the course by facilitators in parallel to each research process and stages. Except when indicated, the review should be done by people outside the team, and these cycles should be done by different reviewers to maximize the chances that the text will be assessed by people from diverse backgrounds. Feedback templates (for course facilitators) This additional section contains two samples of the feedback forms ideated for course facilitators, which are used in some sections above but can also be personalized and adapted to give specific feedback on other parts of the research documentation of every student's team.

''Add an X to “yes” or “no” to the different questions. If the answer to any of the questions is “no,” add comments directly to the description text to help co-authors improve it. The point is not to say whether what has been written is "good" or "bad" but to kindly help the team members as co-authors to clarify what needs to be improved. ''

Feedback on research diagram and protocol
=Legal notes=
 * 1) For editable version of this document (.odt format) and PDF version you can check the corresponding Zenodo files.
 * 2) The list of research questions and the publishable versions of the different projects can be shared afterwards by participants under a CC BY-NC-SA license. In this sense, you should not share personal or sensitive data (on this and the derived documents) about participants or the people you work with.
 * 3) The writing traces left in this document (different versions, drafts, comments, etc.) could be used by the organisation developing the course and by the rest of participants and students for research purposes, in a completely anonymous manner.
 * 4) The materials reflected in this document contain icons (uploaded to Wikiversity with the corresponding copyleft attributions) from Kyle Miller, Iconstock, sevgenjory, Vectors Point, Adrien Coquet, Siipkan Creative, DinosoftLab, Nithinan Tatah, Arif Arif, Misbahul Munir, Eucalyp, ProSymbols, H Alberto Gongora, Nanda Ririz, ProSymbols, vigorn, Alvaro Cabrera, developed for TheNounProject under a Creative Commons license.
 * 5) The text for research methods listed in section 4.1.2 has been adapted in some examples, as indicated, from the original content on "Guide to research and research methods for Master’s (M.A.) students", developed at the Faculty of Humanities of University of Jyväskylä (Finland), licensed under a Creative Commons Attribution - NonCommercial - ShareAlike 3.0 Unported License.