Rochelle E Tractenberg

Shared under a CC-BY-NC-ND 4.0 (Creative Commons By Attribution Non Commercial No Derivatives 4.0 International) license. Presented at World Statistics Congress 2023 (Toronto). To appear in: In, H. Doosti, (Ed.). (2024). Ethics in Statistics: Opportunities & Challenges. Cambridge, UK: Ethics International
Artificial Intelligence (AI) arises from computing and statistics, and as such, can be developed and deployed ethically when the ethical practice standards of each of these fields are followed. The Toronto Declaration was formulated in 2018 specifically to ensure that machine learning and AI could be held accountable for respecting, and promoting, universal human rights. The Code of Ethics and Professional Conduct of the Association of Computing Machinery (ACM, 2018) and the Ethical Guidelines for Statistical Practice of the American Statistical Association (ASA, 2022) describe the ethical practice standards for any person at any level of training or job title who utilizes computing (ACM) or statistical practices (ASA). These three reference documents can together define "what is ethical AI". All development, deployment, and use of computing is covered by the ACM Code; the ASA defines statistical practice to "include activities such as: designing the collection of, summarizing, processing, analyzing, interpreting, or presenting, data; as well as model or algorithm development and deployment." Just as the Toronto Declaration describes universal human rights protections, the ACM and ASA ethical practice standards apply to professionals, individuals with diverse background or jobs that include computing and statistical practices at any point, and employers, clients, organizations, and institutions that employ or utilize the outputs from computing and statistical practices worldwide. The ACM Code of Ethics has four Principles, including one specifically for Leaders with seven elements. The ASA Ethical Guidelines include eight principles and an Appendix; one Guideline Principle (G. Responsibilities of Leaders, Supervisors, and Mentors in Statistical Practice) with its five elements and the Appendix (Responsibilities of organizations/institutions) with its 12 elements are specifically intended to support workplace engagement with, and support of, ethical statistical practices, plus, the specific roles and responsibilities of those in leadership positions. These ethical practice standards can support both individual practitioners', and leaders', meeting their obligations for ethical AI worldwide.

Shared through CC-By Attribution-NonCommercial-NoDerivatives 4.0 International license.

Degrees of Freedom Analysis (DoFA) is a method originally published in 1975 that combines qualitative analysis to summarize narrative data with quantitation to summarize alignment or support of the qualitative results for theory building. This paper discusses recent adaptations of the method to facilitate decision making and prediction when theory building is not the investigator's focus. Eleven applications of the method across a variety of disciplines and materials are discussed. These examples highlight the flexibility and utility of DoFA in rigorous and reproducible analyses that involve qualitative materials that would otherwise be challenging to analyze and summarize.

Shared under a CC-BY-NC-ND 4.0 (Creative Commons By Attribution Non Commercial No Derivatives 4.0 International) license.

A change theory is a model, typically conceptual, that frames and describes relationships between conditions that can lead to change - and the outcomes that represent that change. One important type of change is the uptake of "innovation", whether it is in the form of new technology, new ideas, or new results from relevant research. In biomedical arenas, specific instances of uptake are called "implementation" of research or "translation" of knowledge, while in educational realms the same type of focused initiatives are driven by a specific "theory of change". In contrast to the delimited translation or implementation of change in single projects or initiatives, the theory that enables predictions and the identification of outcomes (what will change, and why) across contexts is a "grand" theory. A major conceptual model supporting change theory as well as theories of change is the Diffusion of Innovation (DoI), also the title of a book describing the phenomenon, originally published in 1962. Importantly, DoI has its origins in communication - i.e., the communication of a new idea or about technology.  While DoI is a popular change theory model, most presentations treat the elements of DoI as sequential, without imposing any causal influence onto the sequence. This paper explores the role of cognition in those receiving the communication (about the new idea); the complexity of both the information entailed in the new idea and that required by end users to understand and correctly interpret the new information; and the causal influences of elements of the DoI as a grand change theory. Three examples are given that the new version of the DoI model explains: the misuse of a novel statistical model in teacher performance evaluation; the underrepresentation of systems biology in algorithmic genomic predictions; and the uptake of curriculum and instructional development guidelines for higher education and training. In each of these examples, treating the DoI change theory as explicitly causal, and adding in cognition and complexity, predicts failures in all three examples. One recent example of recommendations for improving short-form training shows how the augmented DoI (aDOI) model is actionable and evaluable, to encourage diffusion of any innovation.

Shared under CC-By Attribution-NonCommercial-NoDerivatives 4.0 International. Translational science is alternatively considered a research domain and a driver of research culture. The US National Institutes of Health introduced a translation-incentivizing paradigm 20 years ago with a new funding mechanism to promote an integrating perspective on basic (cellular/molecular), pre-clinical (animal), and human (clinical, public health, and policy) science. In this paper, a five-stage model of a translational continuum is explored to identify as-yet unrealized opportunities to build translational science research programs, training of translational scientists, and examination of how statistical, computational, and data science methodologies can contribute to both. A new dimension of translation is introduced, with "lateral translation" at one end and "knowledge translation"/implementation at the other end. Knowledge translation, also called "implementation", is recognized as the use of a method or knowledge developed in/for one discipline in the same or a different discipline, used or implemented according to its original intended purpose. On the other end of this dimension is the adaptation of a method, i.e., the lateral translation of that method or technique from one discipline to another. This new dimension can address some criticisms of the clinical-translational continuum, while also highlighting underappreciated challenges of translational science. Conceptualizing lateral translation can lead to new opportunities for biomedical scientists and statistical scientists to ensure their science is rigorous and reproducible, while also encouraging investigators to forge programs that feature methodological work that can further disciplinary and multidisciplinary work.

CC-By Attribution-NonCommercial-NoDerivatives 4.0 International license. Practitioners in quantitative fields can encounter challenges when they need to communicate about their work. Transparency, the hallmark of ethical quantitative practice, requires specificity in levels of (un)certainty, assumptions, sensitivity analyses, and replicability. These considerations add to the increasingly complex computational, mathematical, and statistical modeling methods that must also be coherently reported for stakeholders. This paper describes the augmented Diffusion of Innovation (aDOI) Change Theory as a way for quantitative practitioners to comprehend and align the complexity of their message with the cognitive complexity capabilities of the hearer of the message. Kotter's Theory of Change Management (ToCM), a longstanding model for workplace change initiative leadership, is reenvisioned with the cognitive complexities of the change made explicit. Together, aDOI describes how ToCM can be facilitated by 1) creating evidence-based structure for how communication of new information or a change initiative can be optimized when cognitive complexity of the message and the hearer are leveraged; and 2) recognizing new opportunities for engagement and buy-in that emerge when the specific elements of aDOI are mapped onto the ToCM activities and phases. The combination of aDOI and ToCM outlines the importance of a nuanced understanding of how individuals and organizations internalize and apply change. Combining the theoretical underpinnings of how innovations spread with a practical, stepwise approach to implementing change offers a powerful tool for leaders to create change, and to manage transitions effectively. It also offers opportunities for individuals to create their own "change initiatives", and to document the accomplishment of subgoals along the way. The combination of aDOI and ToCM can help individuals and leaders not only advocate for change but also navigate the intricate process of engaging others as they co-navigate the stages of acceptance and execution.

Shared under CC BY-NC-ND license: Attribution-NonCommercial-NoDerivs 4.0 This material is sharable as long as the author is credited appropriately; no changes permitted in any way and no commercial use is permitted.

A steward of the discipline was originally defined as an individual to whom “we can entrust the vigor, quality, and integrity of the field”, and more specifically, as “someone who will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application” [8]. Originally articulated for doctoral education, in 2019 the construct of stewardship was expanded so that it can also be applied to non-academic practitioners in any field, and can be initiated earlier than doctoral education [18]. In this paper, we apply this construct to the context of mathematics, and argue that even for those early in their training in mathematics, stewardly practice of mathematics can be introduced and practiced. Postsecondary and tertiary education in mathematics — for future mathematicians as well as those who will use math at work — can include curriculum-spanning training, and documented achievement in stewardship. Even before a formal ethical practice standard for mathematics is developed and deployed to help inculcate math students with a “tacit responsibility for the quality and integrity of their own work”, higher education can begin to shape student attitudes towards stewardly professional identities. Learning objectives to accomplish this are described, to assist math instructors in facilitating the recognition and acceptance of responsibility for the quality and integrity of their own work and that of colleagues in the practice of mathematics.

Shared under CC BY-NC-ND license: Attribution-NonCommercial-NoDerivs 4.0 This material is sharable as long as the author is credited appropriately; no changes permitted in any way and no commercial use is permitted.

Objective: To study the reliability and validity of an odor identification test. Methods: The data come from an epidemiological cohort including 1146 non-Hispanic Caucasian, 86 Hispanic, and 12 other participants at the baseline visit (73.4% female). We tested the fit of each of three neurobiologically plausible models (validity) for responding on a 12-item odor identification task using confirmatory factor analyses (CFA); five model fit indices were assessed for each run. CFA testing fit over time (reliability) was planned for the measurement model that was found to fit across groups at the baseline visit. If a model was not found for the baseline visit, the test would be deemed "not invariant" over group, and not tested over time. In this case, we planned a post hoc Rasch analysis to further study test validity; and a multi-trait, multi-method analysis (MTMM) of the entire test battery to study reliability in terms of other, valid, cognitive and neuropsychological functional assessments. Results: Nearly 70% of the variability in odor identification scores is error, a result that was replicated over four independent samples at the baseline visit. A core of 30% of "signal" from the task was identified over time (via Rasch modeling) but was explained fully by global cognition (replicated over time). Conclusions: "Odor identification" as a construct cannot be reliably or validly measured over time or group. Multiple hypothesis-driven methods and replications show that this test provides no information that a global cognition score does not also (more validly and reliably) provide.

Shared with the license cc- by attribution- non commercial – no derivatives 4.0 international. White paper prepared for NSF-sponsored conference, held May 2022, on promoting accessible and effective career spanning learning in the life sciences (National Science Foundation (NSF) grant DRL#2027025). Catalytic learning is defined as enduring learning that objectively - measureably - prepares the learner to continue to learn and implement new knowledge, and that positions learners for future self-directed learning. Catalytic learning is an explicit learning-based schema, so the learner needs to understand something about their own metacognition, sustainable learning, and cognitive schema. This means that the instructor also needs this background information, in order to integrate these cognitive and metacognitive features into instruction and make catalytic learning-of whatever material-possible. Catalytic learning is promoted by instructors through the inclusion of an explicit metacognitive component to learning. Catalytic learning involves challenging learners to deconstruct how they think about the new knowledge of the training, and reconstruct this to include metacognitive awareness of both the new knowledge (within an evolving schema) and how to set and monitor progress towards new thinking and learning goals. This manuscript describes catalytic learning in terms of the constituent cognitive and metacognitive constructs that are common throughout adult learning and the learning sciences literature, including metacognition, schema, transfer, and sustainable learning. These are features that instructors need to understand at some level, in order to efficiently integrate them into teaching so that catalytic learning is possible. This white paper focuses on adult learners, and while examples may feature short-form training, the cognitive and metacognitive constructs are useful for college/university instruction as well. Thus, the implications for adult instruction in different contexts are discussed.

UPDATED Dec 2022: Shared with the license cc- by attribution- non commercial – no derivatives 4.0 international. In submission at Proceedings of JSM 2020 (Philadelphia, PA/Virtual).
Ten simple rules for integrating ethics content/training in ethical practice into every/any statistics and data science course are presented. These rules are intended to support instructors who seek to encourage ethical conduct in (throughout) the practice of science, whether it involves statistics, data science, or qualitative analysis; as well as throughout the employment of tools, methods, and techniques from these domains. Truly integrated ethical training can also promote awareness of the fact that every member of a research – or practice -  team has a specific role, with attendant obligations and priorities, relating to the use of statistics, data science, and qualitative analytic approaches. Even if individuals are not going to be the ‘designated statistician/analyst’ on a project, understanding the roles and responsibilities of team members can strengthen the sense of responsibility and accountability of each member of a science or practice team. True integration of ethical training is not simple to achieve, but the ten rules are based on educational and cognitive sciences, as well as a recognition of the fact that additional content, without furthering a course’s existing learning objectives, greatly dampens enthusiasm for, and the likelihood of, integration of ethical training into quantitative courses. Assumptions for readers of these ten simple rules are: the instructor wants to have something that can be graded/evaluated after the students engage with the case; and that one objective the reader has is to teach how to reason & make decisions ethically as students go about practicing or using statistics. The overarching message of the ten rules is that true integration can benefit from leveraging existing structural features that both streamline learning outcomes and increase the chance of successfully embedding ethical practice standards into existing courses. Success is defined as the creation of reproducible, gradable work from students that signal whether or not the ethics instruction had its intended effects; and the documentation of ongoing (sustained) engagement with the ethics training beyond the end of the course.

Sharing these slides under CC BY NC ND international license. These slides were prepared for use during a panel convened to discuss “Balderdash, Codswallop, and Malarkey” during the 2020 JSM Conference – planned for Philadelphia but held virtually 2-6 August 2020. The panel had composed a set of questions for discussion and I chose one of these, “How do you differentiate among statistical, logical, ethical, and critical thinking?”, and composed these slides. The slides reflect my own work/thoughts, so errors are solely attributable to me; I updated the slides 29 July 2020 <and yet, there will undoubtedly still be errors!>. The slides do not reflect the opinions of the other panelists, the ASA, or Georgetown University.

Shareable under a CC - by attribution - noncommercial - no derivatives 4.0 license.

It is common to create courses for the higher education context that accomplishes content-driven teaching goals, and then develop assessments (quizzes, exams) based on the target content. However, content-driven assessment can tend to support teaching- or teacher- centered instruction. Adult learning and educational psychology theories suggest that instead, assessment should be aligned with learning, not teaching, objectives. To support the alignment of assessments with instruction in higher education, the Assessment Evaluation Rubric (AER) was developed. The AER can be utilized to guide the development and evaluation/revision of assessments that are already used. The AER evaluates four features of an assessment: its general alignment with learning goal(s); whether the assessment is intended to/effective as formative or summative; whether some systematic approach to cognitive complexity is reflected; and whether the assessment (instructions as well as results) itself is clearly interpretable. Each dimension (alignment; utility; complexity; clarity) has four questions that can be rated as present/absent (or yes/no), or, using a three-level ordinal scale describing “present-useable”, “possibly present - needs clarification”, and “absent”. Other rating methods can also be conceptualized for the answers to the AER’s 16 questions, depending on the user’s intent. Any instructor can use the AER to evaluate their own assessments and ensure that they -or new assessments in development - will promote learning and learner centered teaching. Originally published (unedited) in Open Archive of the Social Sciences (SocArXiv) 2020, 10.31235/osf.io/bvwhn.

This white paper was published in the SocArXiv 2 April 2020 and is shared here under the CC-By Attribution-NonCommercial-NoDerivatives 4.0 International license.


Curriculum development in higher education should follow a formal process. Although the focus in formal curriculum theory is on long-term programs of study, the theoretical and practical considerations are also applicable to shorter-form learning experiences (single courses, lessons, or training sessions). With these considerations in mind, we discuss here an iterative model of curriculum design, the starting point of which (indeed, in the construction of any learning experience), is the articulation of the target learning outcomes: everything follows from these, including the selection of learning experiences and content, the development of assessments, and evaluation of the resulting curriculum. We discuss how the iterative process can be used in curriculum and instructional development, and provide a set of practical guidelines for curriculum and course preparation.

This preprint is sharable under a CC BY-NC-ND  international license.
The Mastery Rubric is a curriculum development and evaluation tool. It articulates the knowledge, skills, and abilities (KSAs) of a given curriculum, together with the developmental trajectory that learning these KSAs is intended to follow. Mastery Rubrics have focused on graduate and post-graduate curricula, and utilize the European Guild Structure for staging growth and development of KSAs. Bloom’s taxonomy is also essential for describing the performance, and performance levels, in each stage. A defining characteristic of the Mastery Rubric is the Master level: the Master is qualified, with evidence, to take a learner from novice through to Master. However, the transition from competent independent performer of a set of KSAs to Master is not addressed in any of the Mastery Rubrics to date. This article describes three levels through which any instructor can progress in order to generate evidence they are qualified at the Master level for any Mastery Rubric, even those that have already been published to include a (single) Master level. These three levels describe the evidence that can be observed to represent early, middle, and late Master capabilities in terms of teaching, and assessing learning, in students and trainees. Two new Mastery Rubrics (MRs) have recently been completed, and neither has a Master level: one for Bioinformatics (MR-Bi) and one for the Nurse Practitioner (NR-NP). Although this new Mastery Rubric for the Master Level (MR-ML) can be used with all of the existing Mastery Rubrics to characterize the development of the Master’s engagement with theories and practicalities of learning, we use the MR-Bi and MR-NP to illustrate how the MR-ML can work with these two new MRs, and how individuals in any field can compile their evidence of the specific abilities to diagnose problems exhibited by those at earlier stages, devise remediating activities for those problems, and assess the result.

CC-BY-NC-ND 4.0 International license

This article builds on the concept of disciplinary and professional stewardship, to discuss the ethical practice guidelines from two professional associations and a method that you can learn to use in order to implement those guidelines throughout a professional career. The steward is an individual who practices in a field in a manner that invites and warrants the trust of the public, other practitioners, and employers to uphold and maintain the integrity of that field. It is important to your sense of professional identity - and also your profession - to cultivate a sense of stewardship; and one of the foundational aspects of stewardly behavior is to understand professional practice guidelines and the types of behaviors that are expected by practitioners in a given field. Therefore, this article presents two sets of guidelines that can support professionalism, ethical practice, and the development of a coherent professional identity for the statistician and data scientist. The American Statistical Association (ASA) and the Association of Computing Machinery (ACM) are large professional organizations with international membership. An overall objective of each of these organizations is to promote excellence in and by their members and all those who practice in their respective – sometimes shared/joint – domains. It can be helpful to consider the field of ‘statistics and data science’ to be a hybrid of, or co-dependent on, these two fields, which is one reason why the two organizations are presented together. Another reason is that both organizations take ethical practice very seriously, and both engaged in lengthy projects to carefully revise their respective ethical guidelines for professional practice in 2018.  Not only does engagement with the guidelines support you initiating, and beginning to demonstrate, your commitment to this particular professional identity, but also exploring the ethical guidelines for professional practice (through ASA or ACM) is a first step towards documenting your commitment to stewardly work as a data scientist. Ethical reasoning, the third focus of this article, helps deepen the understanding of the guidelines and can be useful to generate evidence of stewardly development.

CC-BY-NC-ND 4.0 International license

Citation: Tractenberg, RE. (2019, April 23). Becoming a steward of data science. https://doi.org/10.31235/osf.io/j7h8t 
This article introduces the concept of the steward: the individual to whom the public, and other practitioners, can entrust the integrity of their field. The concept will be defined, particularly with respect to what about stewardship can be demonstrated by the practitioner so that others – including other stewards – can recognize this professional identity. Stewardship is an important aspect of professionalism, and although data science is a very new profession, its growth in terms of the number of practitioners should also include growth in the commitment to integrity in practice. Although an undergraduate program may seem early to begin understanding what this commitment means, and how to generate evidence of that commitment for yourself, those with a strong understanding of stewardship and how to recognize it will be better able to select jobs in contexts where this commitment to integrity is nurtured and valued. Learning about stewardship engages students in taking responsibility for their role in the profession, and so taking responsibility for the profession and the professional community. Once the construct is understood, learners can focus on the nature of the evidence they can compile - as well as the types of activities that can generate that kind of evidence- and on why this is meaningful over their career.

CC-BY-NC-ND 4.0 International license

Tractenberg, R. E. (2019, April 23). Teaching and Learning about ethical practice: The case analysis. https://doi.org/10.31235/osf.io/58umw

Statistics, biostatistics, and data science are unique disciplines/a unique discipline in the sciences: anyone with an Internet connection and computing device can utilize the methods from these disciplines –irrespective of preparation to do so. Most empirical and all experimental sciences require some form of data analysis, including qualitative methods. However, even those in degree or formal educational programs learning about statistics/biostatistics/data science do not receive training in what constitutes “ethical practice”. The American Statistical Association (ASA) maintains, and recently (2018) updated, Ethical Guidelines for Statistical Practice. Understanding and being able to utilize these Guidelines (GLs) is relevant for all applications of statistical and data science methodologies – whether for true “research” (following the scientific method) or for business or other predictive/decision-making support. Thus, students who will go on to be statisticians and non-statisticians alike need to learn about ethical statistical practice, including those who seek to apply these methods in marketing, policy, and higher education. This article describes how to employ the case study method to teach the ASA GLs, using simple vignettes and a specific tool called a “stakeholder analysis template”. The template is introduced as a method for understanding the harms and benefits, as well as the stakeholders, in each of a series of tasks common to the collection, analysis/manipulation, and drawing of inferences or conclusions based on data in any shape or size. The ASA Ethical Guidelines are discussed with respect to their potential to guide data collection and munging (two specific tasks), with three learning objectives: 1. describe how different individuals (“stakeholders”) may be affected by decisions and actions; 2. enumerate harms and benefits that are most clearly relevant for each stakeholder with respect to the activity; and 3. identify which ASA GL Principles (and/or specific elements) seem most relevant to this activity. The stakeholder analysis template is intended to facilitate teaching and learning – and the ultimate utility – of the ASA Ethical Guidelines for Statistical Practice.

CC-BY-NC-ND 4.0 International license
ABSTRACT:
2 April 2019, University of Hawaii, Manoa: This annotated presentation will describe and briefly discuss metacognition as a learnable, improvable skill set that can be taught within, and strengthened across, courses throughout a higher education curriculum. Leveraging specific ideas from the disciplines of cognitive science and educational psychology can strengthen teaching and assessment activities so that the targeted learning happens and is deepened with the incorporation of reflection and metacognition. For undergraduates, graduates, and post-graduate/professional contexts, defining the “learning enterprise” as a joint effort, by the instructor and the individual learner, orients all participants in moving the learner along a specific pre-articulated path of growth and development. A focus on cognitive psychological principles in the development, evaluation, and revision of teaching and learning opportunities emphasizes the strengths this focus brings for the instructor and the learner.

CC-BY-NC-ND 4.0 International license
This paper will appear in the Proceedings of the 2018 JSM (by end of Dec 2018).

Patient reported epidemiological data are becoming more widely available for the application of data scientific methodologies and other investigations. One new such dataset, the Fox Insight project, was launched in 2017 by the Michael J. Fox Foundation to encourage researchers to engage in the study of Parkinson's disease. The Fox Insight (FI) dataset will be released for public access late in 2018, and while it is longitudinal in nature, and while dedicated statistics and psychometrics cores have supported the scientific advisory committee in their considerations of which variables to collect, diseases like Parkinson's do not have unambiguous states (e.g., " mild " vs. " severe ") or changes (e.g., " stable " vs. " worse "). Assessing these states can be complicated when there are medical comorbidities that may contribute to, compound, or be conflated with the symptoms of the disease of interest (e.g., cerebrovascular disease; muscle weaknesses due to stroke or aging; depression). This paper describes the development and proposed validation of two new variables in the FI data set that are intended for use as " outcomes " , which would be available for interested researchers who download the FI data when it becomes publicly available. One represents " cognitive change " and the other represents the " off " syndrome of Parkinson's where symptoms suddenly become unresponsive to medication (that works otherwise) for short periods of time. We discuss how new outcomes like these can be developed from patient reported epidemiologic data like the FI set using theory, and validated using international consensus criteria (COSMIN). These results are useful for planning analyses of the FI dataset, but also may support future designs for similar patient reported epidemiological data sets, so that they will be designed to include outcome variables or variables that can be used to demonstrate alignment of derived outcomes with the COSMIN validity criteria.

CC-BY-NC-ND 4.0 International license
Appearing in Proceedings of JSM2018 (Vancouver, BC): Based on a presentation I gave in a session on quantitative literacy that I organized for the Meetings.

Statistical literacy is critically underappreciated across scientific disciplines, and is not part of the training in graduate programs even when “statistics” or “statistical methods” are. It is common to teach “introductory statistics” courses to promote engagement with key concepts and methods in the domain, but statistical literacy involves the selection of appropriate methods from among competing plausible alternatives, as well as the correct application of the methods, and then reasoning with those methods and their results. A new, developmental, model of statistical literacy that is specific for the scientific practitioner was outlined by Tractenberg (2017-a), based on prior models of statistical and scientific thinking. In this model, there are nine elements to statistical literacy: 1) Define a problem based on critical literature review; 2) Identify or choose – and justify - the measurement system; 3) Design the collection of data; 4) Pilot, analysis and interpretation; 5) Discern “exploratory”, “planned”, and “unplanned” data analysis; 6) Hypothesis generation based on planned & unplanned analyses; 7) Interpretation of results; 8) Drawing and contextualize conclusions; and 9) Communication. These knowledge, skills, and abilities (KSAs) are important for an individual to execute when they are doing scientific work, but these KSAs must also be developed to a sufficient level to permit an individual to assess and consume research articles and arguments that use data and inferences.

The nine KSAs that comprise statistical literacy for the practicing scientist each require considerably more cognitive capacity than “engagement with key concepts and methods”. The aim of this paper is to discuss the relationship between purposeful development of these KSAs and Bloom’s taxonomy of educational outcomes (cognitive) so that statistical literacy can be clearly seen as a learnable, improvable skill set that can be introduced in the context of a journal club, among other options, and can continue to be used and developed by individuals and instructors.

CC-BY-NC-ND 4.0 International license

A 2017 article in the Proceedings of the National Academy of Science (PNAS) reported that short-term, intensive “bootcamp” and other training opportunities (“train the user”) did not yield results that the training is intended to achieve. The results are predicted by cognitive psychological theories and findings from educational psychology. However, many “bootcamp” (short and intense preparation) type training opportunities – especially train the trainer - have some anecdotal evidence of their success and impact. Data and Software Carpentries (“The Carpentries”) is an organization that offers short/intensive training workshops around the world similar to some of those discussed in the 2017 paper. They train “users” of data and software, and they also train trainers of these users. In their response to this 2017 paper, The Carpentries acknowledge that one point raised in that paper, that training spaced over time is more successful than shorter/more intense training, cannot be circumvented. However, short and intense training is popular, and is sometimes all that is feasible; both their train the user and train the trainer sessions are short and intense. In their response, The Carpentries described their own strategies for achieving more positive outcomes for short and intense training than were described in this 2017 PNAS article. Two of these strategies are: “meet learners where they are” and “explicitly address motivation and self efficacy”. These strategies may not be functioning as well as they could be. To clarify what might be impeding these strategies, this white paper compares and contrasts features of training those who will train others (“train the trainer”) with training for “new users”. These types of programs are short and intense, but differ in fundamental ways. Understanding these differences can be leveraged to improve the outcomes of train the user training; recommendations for doing so is presented. The recommendations are embedded in descriptions of those features of training opportunities that can be leveraged purposefully to promote sustainable learning, even when training is short and intense. It is hoped that the model can support the success of the two Carpentries strategies to promote the achievement of Carpentries goals –and those of all who offer short/intensive training opportunities.

CC-BY-NC-ND 4.0 International license

Demand for training life scientists in bioinformatics methods, tools and resources and computational approaches is urgent and growing. To meet this demand, new trainers must be prepared with effective teaching practices for delivering 15 short hands-on training sessions—a specific type of education that is not typically part of professional preparation of life scientists in many countries. A new Train-the-Trainer (TtT) programme was created by adapting existing models, using input from experienced trainers and experts in bioinformatics, and from educational and cognitive sciences. This programme was piloted across Europe from May 2016 to January 2017. Preparation included drafting the training materials, organizing sessions to pilot them and studying this paradigm for its potential to support the development and delivery 20 of future bioinformatics training by participants. Seven pilot TtT sessions were carried out, and this manuscript describes the results of the pilot year. Lessons learned include (i) support is required for logistics, so that new instructors can focus on their teaching; (ii) institutions must provide incentives to include training opportunities for those who want/need to become better instructors;  (iii) formal evaluation of the TtT materials is now a priority; (iv) a strategy is needed to recruit, train and certify new instructor trainers (faculty); and (v) future evaluations must assess utility. Additionally, defining a flexible but rigorous and reliable process of TtT 'certification' may incentivize participants and will be considered in future.  1. Via A*, Fernandes P, Morgan S, Schneider MV, Palagi, P, Rustici G, Attwood T, Tractenberg RE†*. (in press, 2017). A new pan-European Train the Trainer Program for bioinformatics: Pilot results on feasibility, utility, and sustainability of learning. Briefings in Bioinformatics.

CC-BY-NC-ND 4.0 International license

Citation: Tractenberg, R. E. (2017, October 6). The Mastery Rubric: A tool for curriculum development and evaluation in higher, graduate/post-graduate, and professional education. https://doi.org/10.31235/osf.io/qd2ae
A Mastery Rubric is a curriculum development and evaluation tool – for higher, graduate and professional, and post-graduate education. It brings structure to a curriculum by specifying the desired knowledge, skills and abilities (KSAs) that the curriculum will provide, together with performance levels that characterize the learner, on each KSA, as the individual moves through stages of development. This tool unites a developmental implementation of Bloom’s taxonomy with curriculum objectives, to move learners along the articulated path from novice towards independence and expertise with built-in features of psychometric assessment validity. This tool promotes development in the target KSAs as well as assessment that demonstrates this development, and encourages reflection and self-monitoring by learners and instructors throughout individual courses and the entire curriculum. A Mastery Rubric represents flexible, criterion-referenced, definitions of “success” for both individuals and the program itself, promoting alignment between the intended and the actual curricula, and fosters the generation of actionable evidence for learners, instructors, and institutions. These properties are described through the seven examples that have been completed to date. The methods that are used to create a Mastery Rubric highlight the theoretical and practical features; the effort required; as well as potential benefits to learners, instructors, and the institution.  Published to the preprint archive SocArXiv:
10.17605/OSF.IO/QD2AE
SocArXiv 3 November 2017
https://osf.io/preprints/socarxiv/qd2ae

CC-BY-NC-ND 4.0 International license

Qualitative data are commonly collected in higher, graduate, and post-graduate education; however, perhaps especially in the quantitative sciences, utilization of this qualitative data for decision-making can be challenging. A method for the analysis of qualitative data is the Degrees of Freedom Analysis, published in 1975. Given its origins in political science and its application in mainly business contexts, the degrees-of-freedom analysis method (DoFA) is unlikely to be discoverable or used to understand survey or other educational data obtained from teaching, training, or evaluation. This paper therefore introduces and demonstrates the DoFA with modifications specifically to support educational research and decision-making with examples in bioinformatics. DoFA identifies and aligns theoretical or applied principles with qualitative evidence. The demonstrations include two hypothetical examples, and a case study of the role of scaffolding in an independent project (“capstone”) of a graduate course in biostatistics. Included to promote inquiry, inquiry-based learning, and the development of research skills, the capstone is often scaffolded (instructor-supported and therefore, formative), although it is actually intended to be summative. The case analysis addresses the question of whether the scaffolding provided for a capstone assignment affects its utility for formative or summative assessment. The DoFA is also used to evaluate the relative efficacies of other models for scaffolding the capstone project. These examples are intended to both explain this method and to demonstrate how it can be used to make decisions within a curriculum or for bioinformatics training.

CC-BY-NC-ND 4.0 International license

The difference between a patient-reported outcome and a patient-centered outcome is often blurred. In fact, “patient-reported” and “patient-centered” are not synonymous; a patient-centered outcome is a subtype of patient-reported outcomes (PROs). A new model of how to create a patient-centered outcome was developed: it starts with a focus on what the patients experience or what they prioritize about their experience, and not the patients’ report on what clinicians or investigators prioritize about the patients’ experiences. This approach results in a qualitatively different patient-reported outcome than one that does not follow this new framework. The recent emphasis on patient-centeredness and the inclusion of PROs in clinical and comparative effectiveness research has important implications for the translation from the “bench” to the bedside, whereas the differences between patient-centered and non-patient-centered PROs further complicate the translation from bedside to community. These problems adversely affect both analysis and interpretation of results as well as decisionmaking that relies on these results. This talk will explore relationship(s) between PROs and more basic science along the translational continuum, to promote effective translation and the integration of the patient’s perspective throughout the research continuum.

CC-BY-NC-ND 4.0 International license

TALK ABSTRACT: Statistics is a dynamic and broad discipline, and its practitioners continually strive to improve scientific methods, mechanisms, and outputs as they relate to statistical methods. The American Statistical Association has (and in 2016 updated) its Ethical Guidelines for Professional Practice. These Guidelines state that they exist “to help statistics practitioners make decisions ethically and communicate their rationale (s) honestly”.  By hewing to these Guidelines, both those whose primary occupation is statistics and those in all other disciplines who use statistical methods in their professional work can act to increase the value of science to society. Moreover, sound statistical practice is essential providing the best – most rigourous, most ethically-done -  scientific evidence to policymakers, community leaders, and citizens. The ASA Ethical Guidelines are not binding, but a commitment to ethical statistical practice marks the professional statistician as well as the ethical scientist. Reliance on statistical practitioners who are overtly and demonstrably committed to upholding the ASA Ethical Guidelines can support confidence by policy makers in the scientific evidence they use in their decisionmaking.

SESSION SYNOPSIS: Synopsis: Achievement of the goal of "advancing the practice of science"- and specifically "to increase the value of science to society" should include increasing the level of ethical behavior across the sciences, in order to both a) serve Science; and b) limit the potential for unethical behaviors to undermine the value of science to society. Policies both within and outside science should be informed by the best available evidence – obtained in the most reliable, and ethical, way possible. Science can support the translation of  knowledge into viable policy options, but only when that science is conducted and reported in a manner that explicitly supports and respects cultural values. Three speakers representing scientific disciplines with differing levels of explicit ethical frameworks will each discuss the role of ethics and professional practice within the culture specific to their discipline (Engineering, Statistics, and Economics). Our discussant will then address how discipline-specific ethics and promoting a culture of professionalism can act to continually improve scientific methods, mechanisms, and outputs to increase the value of science to society.

CC-BY-NC-ND 4.0 International license

PHENOMENON: The purpose of “systematic” reviews/reviewers of medical and health professions educational research is to identify best practices. This qualitative paper explores the question of whether systematic reviews can support “evidence informed” teaching, and contrasts traditional systematic reviewing with a knowledge-translation approach to this objective.
APPROACH: Degrees of Freedom Analysis is used to examine the alignment of systematic review methods with educational research and the pedagogical strategies and approaches that might be considered with a decision-making framework developed to support valid assessment. This method is also used to explore how knowledge translation can be used to inform teaching and learning.
FINDINGS: The nature of educational research is not compatible with most (11/14) methods for systematic review. The inconsistency of systematic reviewing with the nature of educational research impedes both the identification and implementation of ‘best-evidence’ pedagogy and teaching. This is primarily because research questions that do support the purposes of review do not support educational decision-making. By contrast to systematic reviews of the literature, both a Degrees of Freedom Analysis (DOFA) and knowledge translation (KT) are fully compatible with informing teaching using evidence. A DOFA supports the translation of theory to a specific teaching or learning case, so could be considered a type of KT. The DOFA results in a test of alignment of decision options with relevant educational theory and KT leads to interventions in teaching or learning that can be evaluated. Examples of how to structure evaluable interventions are derived from a knowledge-translation approach that are simply not available from a systematic review.
INSIGHTS: Systematic reviewing of current empirical educational research is not suitable for deriving or supporting best practices in education. However, both “evidence-informed” and scholarly approaches to teaching can be supported as knowledge translation projects, which are inherently evaluable and can generate actionable evidence about whether the decision or intervention worked for students, instructors, and the institution. A Degrees of Freedom Analysis can also support evidence- and theory-informed teaching to develop an understanding of what works, why, and for whom. Thus, knowledge translation, but not systematic reviewing, can support decision-making around pedagogy (and pedagogical innovation) that can also inform new teaching and learning initiatives; it can also point to new avenues of empirical research in education that are informed by, and can inform, theory.

In Press (Jan 2017) at Teaching and Learning in Medicine.

CC-BY-NC-ND 4.0 International license

Statistical literacy is essential to an informed citizenry; and two emerging trends highlight a growing need for training that achieves this literacy. The first trend is towards " big " data: while automated analyses can exploit massive amounts of data, the interpretation—and possibly more importantly, the replication—of results are challenging without adequate statistical literacy. The second trend is that science and scientific publishing are struggling with insufficient/ inappropriate statistical reasoning in writing, reviewing, and editing. This paper describes a model for statistical literacy (SL) and its development that can support modern scientific practice. An established curriculum development and evaluation tool—the Mastery Rubric—is integrated with a new, developmental, model of statistical literacy that reflects the complexity of reasoning and habits of mind that scientists need to cultivate in order to recognize, choose, and interpret statistical methods. This developmental model provides actionable evidence, and explicit opportunities for consequential assessment that serves students, instructors, developers/reviewers/accreditors of a curriculum, and institutions. By supporting the enrichment, rather than increasing the amount, of statistical training in the basic and life sciences, this approach supports curriculum development, evaluation, and delivery to promote statistical literacy for students and a collective quantitative proficiency more broadly.

CC-BY-NC-ND 4.0 International license

Interest in the 2016 revised American Statistical Association (ASA) Ethical Guidelines for Statistical Practice is keen across the ASA membership and leadership, but as of the 2013-14 academic year, only 35% of US universities required any ethics content for even some of their students in statistics and biostatistics programs. The Guidelines are complex – as is statistical consulting – and the ASA Guidelines require both instruction and practice, particularly to prioritize its principles during statistical consultation. The last 15 years have also seen growing interest in "pro bono statistics"-volunteer statistical consulting as a social service. In the United States, two ASA-based organizations engage directly in pro bono statistics: Statistics Without Borders works with clients globally, while members of the Statistics in the Community (StatCom) Network work at the local and state levels. In this paper, we discuss how to engage in ethical reasoning using the 2016 revised ASA Ethical Guidelines; examples arising from consulting (using actual experiences) are employed, although the examples are applicable or adaptable to any statistical work. The purpose is to demonstrate ethical reasoning both for instructors interested in adding this feature to consulting courses and for students or consultants who wish to build experience, and evidence of engagement, with the ASA Ethical Guidelines.

CC-BY-NC-ND 4.0 International license

Interest in the revised ASA Ethical Guidelines for Professional Practice is keen across the ASA membership and leadership, but as of the 2013-14 academic year, only 35% of US universities required any ethics content for *at least some* of their students in statistics and biostatistics programs. Since data analysis is becoming important across disciplines, the Guidelines can serve to introduce all students to critical concepts of responsible data analysis, interpretation, and reporting. The Guideline principles interact, and sometimes must be prioritized. Therefore, neither the simple distribution of –nor an encouragement to memorize-the Guidelines can promote the necessary level of awareness. The Guidelines contain elements that are suitable, and important, components of training for undergraduates and graduates whether or not they are statistics majors, to prepare them for ethical quantitative work. To achieve this preparation, and to support responsibility in the conduct of research involving data and its analysis, the Guidelines should be incorporated into every quantitative course. This paper discusses why and how this can be accomplished.

CC-BY-NC-ND 4.0 International license

Interest in the new Guidelines is keen across the ASA membership and leadership, but as of the 2013-14 academic year, only 35% of US universities required any ethics content for *at least some* of their students in statistics and biostatistics programs. Two barriers to increasing this to 100% of universities and students are: 1. Time and effort: either a new course is needed or time within existing courses must be carved out for new material. 2. Content: “ethics training” may be perceived as required for just those who violate norms for ethical practice (or the law). Since data analysis is becoming important across disciplines, the Guidelines can serve to introduce all students to critical concepts of responsible data analysis, interpretation, and reporting. The Guideline principles interact, and sometimes must be prioritized. Therefore, memorization of, or simply distributing, the Guidelines is unlikely to promote the needed and desired level of awareness. The Guidelines contain elements that are suitable, and important, components of training for undergraduates and graduates whether or not they are statistics majors, to prepare them for ethical quantitative work.  Introducing ethical reasoning in any undergraduate or graduate program, and perhaps integrating a respect for ethical data practice into all engagement with data (Big, big, or small), could be important for improving the reproducibility of science across disciplines.

CC-BY-NC-ND 4.0 International license

A steward of a discipline is "someone to whom we can entrust the vigor, quality, and integrity of the field." Mentorship experience can generate evidence that and how an individual is a true steward of their discipline (if they are or want to be a mentor), and mentees should seek <and the discipline should promote their finding!> true stewards to be their mentors.  Stewardship is an active, purposeful activity. Mentorship in stewardship is focused - but flexible - and promotes the vigor/integrity of the discipline as well as the growth of the mentor and the mentee.

CC-BY-NC-ND 4.0 International license

Human resource development (HRD) “is about lifelong learning”. However, “lifelong learning” can sometimes be defined as simply maintaining competency with respect to the “state of the science” or “keeping up to date”– i.e., continuing to learn new things relevant to the profession. Such “lifelong learning” is typically represented or achieved by attending workshops, reading materials and/or answering multiple-choice questions on this content, or completing other, similarly generic, work throughout a career. While continuously mastering new information associated with a target topic may in fact represent “lifelong learning” – it does not necessarily represent development, e.g., of greater sophistication in thinking about that material. In this sense, lifelong learning is not the same as “continued professional development”. “Sustainable learning” has been defined as learning that continues after teaching ends and extends beyond the course content; this implicitly involves transfer, known to be important for learning, but difficult to achieve. A focus on “sustainable” learning could be an empirically supportable approach to HRD that can change “lifelong learning” into “continued professional development”. This paper describes the application of ongoing empirical research into sustainable learning during professional development (in PhD students preparing for research careers) to the current challenge in Human Resource, faculty, and workforce development where new material continues to be published/created with increasing frequency. The framework involves a focus on metacognition around the target knowledge to engage the learner explicitly in not just learning the material, but in seeking to deepen their sophistication in transferring that learning across their work context.

A very long but super important article published in Vox (see URL) by Julia Belluz, Brad Plumer, and Brian Resnick on July 14, 2016. I couldn't figure out a way to share this other than post the link - and do not mean to imply I was involved in any way- only that I think this is important for people to share with colleagues and especially their mentees.

* I am not an author or coauthor on this document!!! I wanted to post this URL in a place for maximum exposure to it, and the only way I could figure to do it is to "upload" this .pdf. PLEASE CHECK OUT THE URL http://scholarlyoa.com!! * Open source publication is SUPER important, but we have to be wary!
Criteria for Determining Predatory Open-Access Publishers
The work is by Jeffrey Beall and this .pdf is the 3rd edition as of January 1, 2015.

For more information on predatory publishers, including lists of
publishers and standalone journals that meet these criteria,
please visit http://scholarlyoa.com

Reviews of two books that could be useful to statisticians interested in ethics and statistics. Tractenberg RE. (2015). (Book Review). Reviews of AT Panter, SK Sterba (Eds.) “Handbook of Ethics in Quantitative Methodology” and National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, “On Being A Scientist: A Guide to Responsible Conduct in Research. 3E.” The American Statistician. Doi: 10.1080/00031305.2015.1068620

An emphasis on the application of data mining and other “big data” techniques has raised concerns about ethical use, analysis, and interpretation of large amounts of data obtained from a variety of sources. In 2014, the Committee on Professional Ethics of the American Statistical Association (ASA) initiated a revision of their Ethical Guidelines for Professional Practice, which was completed in 2015. Although interest in these new Guidelines is keen across the ASA membership and leadership, as of the 2013-2014 academic year, only 35% of universities in the United States required any ethics content for “at least some” of their students in statistics and biostatistics programs. There are two main barriers to increasing this to 100% of universities and 100% of students. First is time and effort: either a new (additional) course is needed or time within existing courses must be carved out to accommodate new material. Similarly, teaching and learning about ethics, or professional practice, is qualitatively different- particularly in terms of assessment – than it is around statistics and biostatistics/data analysis. A second barrier is content: it can sometimes seem that “ethics training” is only required for those who violate norms for ethical practice (or the law –by falsifying data, plagiarising, or committing fraud in scientific research). Moreover, most faculty, if they have received training in ethics or in the “responsible conduct of research”, have experienced a major focus on memorization of rules or guidelines and possibly an emphasis on the protection of human subjects and their privacy. However, “statistical practice” involves a great deal more than just the consideration of the human (or animal) subjects in a research study –as does responsible conduct in research.
The Guideline principles interact, and sometimes must be prioritized due to their potentially conflicting applicability in any given situation. Therefore, memorization of the Guidelines - or their simple distribution to students or faculty – is unlikely to promote the awareness of their use and importance that is desired by the ASA and the Committee on Professional Ethics that has maintained and revised them. This article outlines elements of the 2015 revision of the ASA Ethical Guidelines for Statistical Practice that are suitable – and important – components of training all undergraduates and graduates whether or not they are statistics majors. It also contains recommendations reflecting current research on how best to promote adult learning - using a constructivist approach consistent with principles of andragogy, and supporting the promotion of the development of expertise, or at least its initiation. Methods for assessment of student work across level (undergrad/grad/post doc/faculty) and context (within major/non-stats majors) are also discussed.

In 2001, the Carnegie Foundation for the Advancement and Scholarship of Teaching instituted a 5-year, in-depth review of doctoral training in the United States, the Carnegie Initiative on the Doctorate (CID). To frame that initiative, the project leaders defined a "steward of the discipline" as "someone who will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application." Although formulated as the purpose of doctoral education, disciplinary stewardship can support a more general model for “education and preparation”, and need not be limited to doctoral training. This chapter argues that stewardship can support the integration of ethical reasoning into preparation to work in/with Big Data. Three dimensions of stewardship that can be introduced earlier than doctoral education are the ideas that: 1) disciplines and fields are dynamic, and require stewardship; 2) the quality and integrity of disciplines must be actively preserved and conserved; and 3) there are particular habits of mind that characterize “those to whom we can entrust” the core features of a discipline or field. In order to formulate clear objectives that will support the explicit and evaluable integration of ethical reasoning across an institution of higher education, doctorally-trained faculty need to perceive its relevance to their own disciplinary stewardship. Efforts to prepare students for participation to work in/with Big Data – whether at or prior to the doctoral program - cannot presume that all possible future situations will have been conceptualized. Preparation for participation to work in/with Big Data must include the initiation of awareness of –and readiness for- the potential to engage with complex ethical, social, and legal implications (ELSI) that have not yet been recognized or encountered. Since Big Data draws practitioners from a wide range of disciplines, it might also be impossible to integrate training for reasoning through ELSI in Big Data into a few specific programs. Thus, a specific and evaluable institutional objective would be “one instructor in every degree-granting program will teach one required course in ethical reasoning for <their discipline>”. The creation of that course is supported by a published developmental framework for instruction in ethical reasoning that can either comprise the single course or could be a first step to initiate career-spanning engagement with ethical reasoning. A course built with these recommendations can support ongoing development in ethical reasoning –both within, and independent of, the target discipline.

CC-BY-NC-ND 4.0 International license

This 50-minute talk, delivered at the University of Glasgow Department of Mathematics and Statistics on 9 September 2015, addressed the following points:
What is the/a Mastery Rubric?
Examples of Mastery Rubrics and their functioning
The clinical research certificate (pub’d 2010)
The MR-ER (ethical reasoning) (pub’d 2012)
The newest MR-> for statistical literacy <non-majors> (in prep)
How does a Mastery Rubric support educational research?
Pros and cons of MR development and use

Two scientific domains that are crucial in “Biomedical Big Data”, computing and statistics, do not typically require “training in the responsible conduct of research” or research ethics. While “responsible conduct of research” (RCR) comprises interactions with subjects (human and non-human), it also involves interactions with other scientists, the scientific community, the public, and in some contexts, research funders. Historically, the development or emergence of disciplines and professions tend to involve a semi-simultaneous emergence of professional norms and/or codes of conduct. However, Biomedical Big Data is not emerging as a single discipline or profession, and engages practitioners from many diverse backgrounds. Moreover, the place of the data analyst or the computer scientist developing analytic algorithms seems to be too granular to be considered specifically within the activities that comprise “responsible research and innovation” (RRI). Current legal and policy-level considerations of Biomedical Big Data and RRI are implicitly assuming that scientists carrying out the research and achieving the innovations are exercising their scientific freedom – i.e., conducting research – responsibly. The assumption is that all scientists are trained to conduct research responsibly. In the United States, federal agencies funding research require that training in RCR be included – some of the time. Because the vast majority of research that was federally funded has not included Biomedical Big Data, RCR training paradigms have emerged over the past 20 years in US institutions that are not particularly relevant for Big Data. While it would be efficient to utilize such established, well-known, easily-documented RCR training programs, this chapter discusses how and why this is less likely to support the development of professional norms that are relevant for Biomedical Big Data. This chapter will describe an alternative approach that can support ongoing reflection on professional obligations, which can be used in a wide range of ethical, legal, and social implications (ELSI), including those that have not yet been identified. This may be the greatest strength of this alternative approach for preparing practitioners for Biomedical Big Data, because the ability to apply prior learning in ethics to previously unseen problems is especially critical in the current era of dynamic and massive data accumulation. To support the development of normative ethical practices among practitioners in Biomedical Big Data, this chapter reviews the guidelines for professional practice from three statistical associations (American Statistical Association; Royal Statistics Society; International Statistics Institute) and from the Association of Computing Machinery. These can be leveraged to ensure that, in their work with Biomedical Big Data, participants know and understand the ethical, legal, and social implications of that work. Formal integration of these (or other relevant) guidelines into the preparation for practice with data (big and small) can help in dealing with ethical challenges currently arising with Big Data in biomedical research; moreover, this integration can also help deal with challenges that have not yet arisen. These outcomes, which are consistent with recent calls for the institutionalization of reflection and reasoning around ELSI across scientific disciplines, in Europe, are only possible as long as the integration effort does not follow a currently-dominant paradigm for training in RCR.  Preparing scientists to engage competently in conversations around ethical issues in Biomedical Big Data requires purposeful, discipline-relevant, and developmental training that can come from, and support, a culture of ethical biomedical research and practice with Big Data.

The American Statistical Association (ASA) Ethical Guidelines (ASA, 1999, http://www.amstat.org/committees/ethics/) address eight general topic areas: Professionalism; Responsibilities to Funders, Clients, and Employers; Responsibilities in Publications and Testimony; Responsibilities to Research Subjects; Responsibilities to Research Team Colleagues; Responsibilities to Other Statisticians or Statistical Practitioners; Responsibilities Regarding Allegations of Misconduct; and  Responsibilities of (those) Employing Statistical Practitioners. The National Institutes of Health (NIH) has a very similar list with nine topics (NIH NOT-OD-10-019; NIH, 2009).

Both are lists of factual information with which trainees should become familiar. However, both are also static - they neither support nor suggest increasing or changing responsibility over a career. That is, mentors and instructors in the responsible conduct of research are indistinguishable from trainees; technically at the end of a course (whether it is 1 hour, week or semester long) the trainee has as much information as the instructor. Moreover, as new areas of concern arise, additional topical training is required –but rarely completed.

Although the NIH requires that all trainees who receive NIH funding also receive training in the responsible conduct of research, to integrate the ASA Guidelines for Ethical Statistical Practice into training, two things are needed: 1. A semester course syllabus; and, 2. A method of documenting the qualifications of instructors to serve as mentors for such training. We describe both in this paper.

The use of Big Data – however the term is defined – involves a wide array of issues and stakeholders, thereby increasing numbers of complex decisions around issues including data acquisition, use, and sharing. Big Data is becoming a significant component of practice in an ever-increasing range of disciplines; however, since it is not a coherent “discipline” itself, specific codes of conduct for Big Data users and researchers do not exist.  While many institutions have created, or will create, training opportunities (e.g., degree programs, workshops) to prepare people to work in and around Big Data, insufficient time, space, and thought have been dedicated to training these people to engage with the ethical, legal, and social issues in this new domain. Since Big Data practitioners come from, and work in, diverse contexts, neither a relevant professional code of conduct nor specific formal ethics training are likely to be readily available. This normative paper describes an approach to conceptualizing ethical reasoning and integrating it into training for Big Data use and research. Our approach is based on a published framework that emphasizes ethical reasoning rather than topical knowledge. We describe the formation of professional community norms from two key disciplines that contribute to the emergent field of Big Data: computer science and statistics. Historical analogies from these professions suggest strategies for introducing trainees and orienting practitioners both to ethical reasoning and to a code of professional conduct itself. We include two semester course syllabi to strengthen our thesis that codes of conduct (including and beyond those we describe) can be harnessed to support the development of ethical reasoning in, and a sense of professional identity among, Big Data practitioners.

CC-BY-NC-ND 4.0 International license

PURPOSE: Training in the responsible conduct of research (RCR) is necessary, but RCR training typically targets those conceptualizing the experiments, and is not prioritized for those who analyze the data. This approach and bias cannot encourage development in ethical reasoning for quantitative scientists, and it does not support the identification of quantitative students with a professional code of ethics.

METHOD: A published model for lifelong learning of RCR, based on ethical reasoning skills that underpin research integrity generally, was combined with the ASA ethical guidelines. The model is based on a Mastery Rubric, a tool for curriculum development and evaluation. A Mastery Rubric is created to describe the knowledge, skills and abilities that the curriculum is intended to target (the ASA guidelines and ethical reasoning in this example), as well as concrete but flexible descriptions of performance across a continuum of developmental levels from more novice to more expert exhibition of the curricular goals. Here, a new approach to teaching the ASA Ethical Guidelines for Statistical Practice, and documentation of their performance with portfolios, is described.

RESULTS: A semester course combining ethical reasoning that supports the responsible conduct of research (in quantitative as well as other sciences) with the ASA Ethical Guidelines is outlined. It promotes sustainable learning with a developmental trajectory for reasoning and a sense of professionalism for quantitative scientists. The model also has implications for training and certification of mentors for statistical practice.

CONCLUSIONS: Synthesizing the ASA Guidelines with a published developmental trajectory can support instruction in the Guidelines and also accomplish the RCR training that federally-funded students must have. Together, these can lead to stronger professional identity for our students and (future) ASA members. The use of a portfolio, like the ASA PStat® accreditation application, can be used to document a wide variety of experiences and skills, including growth in reasoning, responsibility in the conduct of research, and professional identity.

Today we published the article,Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results, by Wicherts et al. In the manuscript, the authors show that weaker evidence in a psychology paper published in a journal requiring signed agreements to share data is associated with a failure to comply with this signed agreement to share.In the following opinion piece, Rochelle Tractenberg, the academic editor who handled the peer review of Wicherts et al., discusses the ways in which “research ethics” and the “responsible conduct of research” go beyond human subjects protection, to include data sharing, professional conduct, and the careful, correct and complete reporting of all analyses in published research.  Rochelle is a biostatistician and research methodologist at Georgetown University.

Rochelle Tractenberg, the academic editor who handled the peer review of Wicherts et al., is a biostatistician and research methodologist at Georgetown University. She interpreted the association between weak evidence and failures to comply with data sharing requirements in a different way. Instead of supporting mandatory archiving of data, Dr. Tractenberg concludes that readers and reviewers of manuscripts and grant proposals should be notified of the author‟s/applicant‟s history of compliance with data sharing mandates and policies.

Annotated slides shared under CC BY-NC-ND 4.0 -ahead of my Invited talk for the Pak Institute of Statistical Training and Research  (PISTAR), 2 March 2024. In an era where data permeates every facet of modern society, "data literacy" is both essential and enigmatic. Recognized as a critical requirement for informed societal participation in 2024, data literacy is difficult to teach, and to assess, due to the lack of a single accepted definition. This talk aims to unravel this paradox by exploring interpretations of data literacy and identifyng commonalities and differences to distinguish it from related literacies in statistics, information, and artificial intelligence, as well as proficiencies in statistics and data science. By juxtaposing literacy against proficiency in domains such as statistical reasoning and information technology, we will navigate towards a coherent and defensible definition of data literacy. We will probe the nuanced relationship between statistical literacy and data literacy, uncovering how this connection can guide us using existing pedagogical and evaluative resources. Drawing on the cognitive complexity framework of Bloom's taxonomy, we will explore how a refined understanding of data literacy can inform effective teaching strategies and assessment methods. We can leverage Bloom's taxonomy to inform data literacy objectives at educational stages from undergraduate to post-graduate training. This presentation aims to shed light on what it means to be data literate, giving educators a foundation to cultivate and measure this critical competency in an increasingly data-centered world. The Mastery Rubric for Statistical Literacy can be found here: http://www.mdpi.com/2227-7102/7/1/2/htm

annotated lecture for Swarthmore Math/Stats Colloquium Shared under CC BY-NC-ND license: Attribution-NonCommercial-NoDerivs 4.0 This material is sharable as long as the author is credited appropriately; no changes permitted in any way and no commercial use is permitted.

A typical undergraduate math curriculum is "the dominant paradigm" - it isn't bad, but it doesn't necessarily or always make you feel like you are an important member of a community. A typical undergraduate program in any field can make you feel like a generic bucket getting filled up with - super valuable - knowledge. While this is a historically generally successful paradigm, it might be due for some tweaks. Swarthmore has many opportunities to *apply* math and stats to solve social and community based challenges. Since the steward of a discipline is an individual "to whom we can entrust the vigor, quality, and integrity of the field", it can create opportunities for solving issues within the field - ultimately strengthening it and its applications/applicability. This looks different depending on the stage of your career and also your job title/responsibilities. In this talk we will learn about stewardship of the discipline - for researchers - and of the profession - for those who use math or statistics at work but don't do research in these fields. As we explore how stewardship can be manifested at any career stage, we will also uncover ways that it can subvert the dominant paradigm: leveraging the important knowledge, skills, and abilities this program aims to develop for all students, while making your undergraduate experience more personally meaningful, documenting and strengthening your evidence of stewardly practice, and helping you to establish a professional identity you can carry forward after college.

Shared with the license cc- by attribution- non commercial – no derivatives 4.0 international. This Professional Development Seminar was created for the GStat holders professional development series, given virtually 19 May 2021. The seminar outlines the application requirements (as of May 2021) for Professional Accreditation (PStat) by the American Statistical Association.

Sharing these slides under CC BY NC ND. I prepared these notes for the Birds of a Feather meetup (virtual) at JSM 2020 with the theme, “integrating ethics training into any quantitative course”. A blog post for the Data Science Education Blog, “Teach Data Science” is in preparation (27 July 2020), as is a manuscript outlining Ten Simple Rules for integrating ethics training into any quantitative course. The blog post will focus on encouraging instructors to use structure: the ASA Ethical Guidelines for Statistical Practice (https://www.amstat.org/ASA/Your-Career/Ethical-Guidelines-for-Statistical-Practice.aspx) and Guidelines for curriculum and course development in higher education and training (https://osf.io/preprints/socarxiv/7qeht) and ethical reasoning (https://link.springer.com/article/10.1007/s11948-014-9613-1), in order to optimize existing course/teaching materials so that articulated learning objectives - plus specific learning objectives about ethical reasoning for statistics and data science - have the greatest chances of being achieved.

These are annotated slides from a talk I gave at JSM 2018 in Vancouver, BC Canada. I am sharing these under a Creative Commons Attribution-NonCommercial-NoDerivs (BY-NC-ND) license. These ideas were summarized in the Proceedings paper,  Tractenberg RE. (2018). Training with the Mastery Rubric for Statistical Literacy to promote rigor and reproducibility across scientific disciplines: Making the journal club educational. Proceedings of the 2018 Joint Statistical Meetings, Vancouver, BC. pp. 1604-1629. This Proceedings paper is also available here on Academia.edu.

This is one talk in a session on communication and statistics that I organized at JSM 2018. This talk describes a model for developing statistical literacy that I published in 2017. This kind of literacy is important for the statistics as well as not-statistics students who study, or propose to utilize, arguments with or from data. Statistical literacy is also an urgent requirement for reviewers of what is submitted for publication. If statistics instructors – both in and outside of the discipline/major - adopt a pro-statistical literacy stance, increasing numbers of scientists-in-training may also adopt it, and then it will trickle to the reviewers of literature and grants, and ultimately, to journal editors and grant program officials.

This is being shared under a Creative Commons Attribution-NonCommercial-NoDerivs (BY-NC-ND) license. These are annotated slides from an invited talk at the UCI Department of Statistics in February 2018.
A 2017 paper (https://www.mdpi.com/2227-7102/7/1/3) outlined a new, developmental, model of statistical literacy that reflects the complexity of reasoning and habits of mind that practicing scientists, and those who work with data for a living, need to cultivate in order to recognize, choose, and interpret statistical methods. While automated analyses can exploit massive amounts of data, the interpretation—and, possibly more importantly, the replication—of results are challenging without adequate statistical literacy. Also, some areas of science, and scientific publishing more generally, are struggling with insufficient/inappropriate statistical reasoning in writing, reviewing, and editing. Statistical literacy that is sufficient for professional use of data for decision making or argumentation goes beyond "knowing statistics" (i.e., choosing the right method given a data type or question). It involves nine areas of knowledge, skill, and ability (KSAs). This talk will present these KSAs and emphasize how their development can be encouraged and documented, with attention to scaling some of the KSAs over time, integrating them into existing courses, and how the KSAs can be grown and purposefully developed for undergraduate and graduate students in and outside of the discipline of statistics and data science.

Here is the link to include for this 2017 paper (open source):
http://www.mdpi.com/2227-7102/7/1/3

Shared under CC BY-NC-ND  license. These are adapted and annotated notes on the Table of Test Specifications (TOTS) that I created for a workshop (in a medical school, for courses in the first (preclinical) two years of courses. The workshop has been adapted to focus attention on/discuss tables of test specifications and how/why to create them.

Shared under CC BY-NC-ND  license. These are the annotated slides I presented at the 2019 NIBLSE Annual Conference in Omaha, NE on 10 October 2019. I described the Assessment Evaluation Rubric, which is fully described in this manuscript (submitted for peer review August 2020): https://osf.io/preprints/socarxiv/bvwhn/

Shared under CC BY-NC-ND license.
Invited talk/webinar given at the Science Life Laboratory, Stockholm, Sweden 10 Sept 2019.

ABSTRACT: A Mastery Rubric requires a set of core knowledge, skills, and abilities (KSAs) together with a developmental trajectory that describes how each KSA can and should change as a result of instruction, feedback, and deliberate practice. In 2017, a cohort of bioinformatics experts who are enthusiastic and reflective teachers were convened in Sweden for a workshop that led to the Mastery Rubric for Bioinformatics (MR-Bi) currently under peer review; also available as a preprint, https:// www.biorxiv.org/content/10.1101/655456v1.full). In this hourlong seminar, the final version of the MR-Bi will be introduced, along with guidelines for using the MR-Bi to solve common problems that arise in curriculum *and course* development. The features of the MR-Bi that can support curriculum and course development (both stand-alone/training courses and courses in formal curricula) will be discussed. manuscript describing the MR-Bi and its development are available in the bioRxiv preprint repository where it has been accessed over 1700 times (abstract accessed over 4500 times) to date.

Shared under CC BY-NC-ND  license. Invited seminar, Sept 2019, in the BiG Talks – Bioinformatics and Genomics Seminar Series of the Science for Life Laboratory (SciLifeLab), Stockholm, Sweden.

ABSTRACT: Genomics and bioinformatics are multi-disciplinary domains, with influences of highly-experimental sciences (biology, genetics) and less-experimental domains (computing; data science). Much of the ethical training for life scientists can derive from the historical emphasis on ethics relating to research involving humans and animals; privacy and confidentiality; autonomy; beneficence and nonmalevolence (and sometimes, social justice). Five "other" challenges that may be unrecognized - and unaddressed - are the focus of this talk:
a) “statistical misconduct” and other disruptive research practices are ubiquitous and contrary to practice standards;
b) innovation that does not lead to reliable or reproducible results may do so because of statistical or scientific misconduct;
c) transparency in practice and communication are essential to ethical work;
d) practicing ethically may not support recognition and appropriate responses to unethical behaviors by others; and
e) doing nothing when others are not behaving ethically, is itself, unethical.
A focus on positivism in science may help to erode community tolerance for a) and b), while a focus on what constitutes ethical science is important for c) and d). However, recognition of d) and e) are absolutely essential for the genomics and bioinformatics community/-ies, particularly as they engage in multi-disciplinary modern science. In this talk we will discuss these five challenges and their importance – and how they go beyond data ethics and the other core areas of “ethics” that are typical of ‘training in ethics” in the domain.

CC BY-NC-ND
This is a seminar given at the Florey Institute Neuroscience of Neuroscience and Mental Health
Neuroscience and the Melbourne Dementia Research Centre by invitation from EMBL-ABR and the Melbourne Bioinformatics unit at the University of Melbourne on 23 August 2018. 

Neuroscience is a multi-disciplinary domain, with influences of highly-experimental sciences (biology, psychology) and more clinical work (neurology). Because much of the ethical training for neuroscientists can derive from the historical emphasis on ethics relating to research involving humans, privacy and confidentiality; autonomy; beneficence and nonmalevolence (and sometimes, social justice) are focal features in research ethics training. Five "other" challenges that may be unrecognized - and unaddressed - are the focus of this talk: a) mismatch of data and/or analysis to the decision they should support; b) sample size that is affordable/feasible rather than generalizable; c) innovation that does not lead to reliable or reproducible results; d) failure to correct for multiple comparisons; and e) failures to consider the difficulties in translating outcomes in translational research paradigms. A focus on strong experimental design and a positivist approach to hypothesis testing, together with considerations of statistical literacy and its development, can assist the neuroscience community in identifying and avoiding these ethical dilemmas.

CC BY-NC-ND
This is a talk given at the invitation of EMBL-ABR, the Florey Institute (and the University of Melbourne) as a Special Seminar in the Florey Institute’s Neuroscience Seminar Series at the University of Melbourne on 22 August 2018.

This talk explored the ways in which under-acknowledged, under-appreciated, but fundamental weaknesses in common measures and instruments actually undermine research that involves neurological, neuropsychological, and neuroscientific ("neuro*") outcomes. Measurement properties and considerations arise in the design of clinical, experimental, and observational studies – particularly with respect to the selection of endpoints or outcomes; and they can also affect the interpretability of results. Unfortunately, the software with which the data are analyzed almost always will run, because the programs do not know whether or not the data were collected using valid tools. Many people believe that, if they get “a result”, then it must be interpretable. However, if you choose an outcome based on the incorrect assumption that it measures what you want it to, the fact that the results do not represent what you intended them to may not be clear – until the next study fails to replicate earlier findings. The talk also briefly discussed approaches to the conceptualization of outcomes that can lead to stronger study design, as well as to results that are interpretable and reproducible.

CC BY-NC-ND
This talk was given by invitation from EMBL-ABR and Melbourne Bioinformatics, and Australia Research Data Commons on 21 August 2018 at the University of Melbourne in Australia as part of their regular seminar series.

In 2017, Feldon et al. published a study of the results of federally  (grant) -funded "bootcamp" type training in data science/technology that was aimed at doctoral-level graduate students in the US. Their results showed that these short/sharp training experiences did not, in fact, lead to any discernable differences between students who did, and those who did not, attend the workshops. This webinar will discuss these results as not surprising - given what is known about adult learners, cognition, and common design characteristics of both short/sharp and longer instructional opportunities. To overcome the difficulties the Feldon et al. results highlighted, we will examine lessons from education; from cognitive science; and from the Data and Software Carpentries. These lessons can be applied to any adult learning experience ("bootcamp", course, program or curriculum).

CC BY-NC-ND
This talk was given at the University of Otago-Christchurch campus by invitation on 16 August 2018.

“Clinical and Translational Research” describes research along a continuum, whether your model has two or five stages. Investigators may conduct their work within just one of the stages (without considering the continuum), they may work at the “border” of a stage where their work needs to feed forward down the continuum. However, there are under-appreciated difficulties in ensuring that outputs and effects obtained in work at one stage have clear parallels in the next (or future) stages; few investigators are prepared to determine how or whether outputs from one stage can be translated for interpretability in the next stage. The recent emphasis on patient-reported outcomes (PROs) in clinical and comparative effectiveness research in the United States has important implications for the translation from the “bench” to the bedside, whereas the differences between patient-centered and non-patient-centered PROs further complicate the translation from bedside to community. These problems can adversely affect both analysis and interpretation of results as well as decisionmaking that relies on these results. This talk will discuss the interplay between measurement, design/analysis, and interpretation along the translational continuum.

CC BY-NC-ND
This is an invited talk delivered at the University of Otago, Dunedin, NZ for the university community and Genomics Aotearoa on 15 August 2018.

Competencies for a bioinformatics workforce have been articulated, but competencies are endpoints, and the development of those competencies has not been adequately described to support their development. This webinar will present the Mastery Rubric for Bioinformatics (MR-Bi), a curriculum design and evaluation tool (MR) that can be used to support the development of the Knowledge, Skills and Abilities (KSAs) in the domain of bioinformatics from very early (novice) stage performance through to that of the fully independent (journeyman). The KSAs were derived via a cognitive task analysis, and they are aligned with those competencies that are sufficiently articulated (and they highlight which competencies are not sufficiently articulated for either design or evaluation of instruction). The performance level descriptors of each KSA have been crafted by international consensus, using formal psychometric methods for range-finding and pinpointing. Capitalizing on established, formal concepts and methods in educational and cognitive psychology, the MR-Bi is designed and intended to promote effective instruction in bioinformatics that is consistent with what is known about teaching and learning for the adult. In this webinar, the MR-Bi will be described in terms of its development and contents, and briefly discussed in terms of utility by the self-directed learner, the lab or research group director, and the instructional developer with respect to bionformatics and modern biological research practice.

CC BY-NC-ND
Invited talk at Loyola Marymount University, Los Angeles, CA in November 2014.
While higher education can tend to focus on cultivating *certainty* - in the sense of providing the right or best answer in summative assessment, this presentation will discuss two types of uncertainty that can be useful to support the development of “scientific literacy”. One type of uncertainty plays a critical role in the development of any new and complex skill: the uncertainty that is (self-) assessed throughout metacognition, the process of thinking about, and directing, one’s own thinking. We have found that metacognition is a learnable, improvable skill set, although it is difficult to teach and assess. A second type of uncertainty – possibly equally difficult to teach – is that inherent in the application and interpretation of statistics. The utility of a Mastery Rubric, a curriculum building and evaluation tool, in cultivating each of these types of uncertainty, will be discussed. Evidence on metacognitive development will be presented from a Mastery Rubric for Ethical Reasoning; and the newest Mastery Rubric, for Statistical Literacy, will be described. An argument will be outlined for the utility of the Mastery Rubric for Statistical Literacy in cultivating uncertainty in terms of both statistics and the learner’s abilities to think and reason in complex problem solving such as that which supports “scientific literacy”.

CC BY-NC-ND
I was invited to share this ongoing work at the Research Seminar in the Georgetown University Neurology Department. This talk outlines a manuscript that is in progress. A new set of patient-centered, patient reported instruments are in development to document and assess the patient experience of urinary signs and symptoms in the presence of neurogenic bladder. These instruments, Urinary Symptoms Questionnaires for Neurogenic Bladder (USQNB) were developed specifically for cohorts according to bladder management, yielding one for intermittent catheterization (USQNB-IC), one for indwelling catheters (USQNB-IUC), and one for voiders (USQNB-v). Their development followed a model that prioritizes the patient experience, but integrates the clinical perspective with a research-oriented perspective. Because it prioritizes the patient perspective and experience, this model does not follow the psychometric approach to creating measurement instruments; as such, scoring on these instruments is not amenable to traditional scoring methods. This talk will outline the different methods for scoring these instruments that the group is currently considering/writing up, discussing pros and cons according to the use and interpretation that different users may have: patient in self-management; clinician in patient management; and investigator studying patients, clinicians, interventions, or combinations of these.

CC BY-NC-ND
A 2017 article in the Proceedings of the National Academy of Science reported that short-term, intensive “bootcamp” and other training opportunities (“train the user”) did not yield results that the training is intended to achieve. The results are predicted by cognitive psychological theories and findings from educational psychology. However, many “bootcamp” (short and intense preparation) type training opportunities – especially train the trainer - have some anecdotal evidence of their success and impact. In their response to this 2017 paper, Data and Software Carpentries (which offer short/intensive training workshops around the world similar to some of those discussed in the 2017 paper) acknowledge that the timing issue that was raised cannot be circumvented. They describe their own strategies for supporting more positive outcomes than were described in this 2017 article, and two of these may not be functioning as well as is hoped (“meet learners where they are” and “explicitly address motivation and self efficacy”). This talk (delivered via webinar 2 Feb 2018) described the empirical and theoretical validity of the psychological findings in Feldon et al. 2017, explanations for anecdotal evidence of bootcamp "success", and specific strategies for purposefully leveraging longstanding lessons from educational psychology, andragogy, and psychometrics to promote sustainable learning, even when training is short and intense. It is hoped that these strategies can support the success of the Carpentries' approaches to promoting the achievement of Carpentries goals –and those of all who offer short/intensive training opportunities.

CC BY-NC-ND
Invited workshop for the Jamaican Statistical Society, 2017.
This daylong workshop will comprise two sessions. The first one will focus on current/prior experience with “training in the responsible conduct of research”, and its strengths and weaknesses, and introduce an alternative paradigm that may be better suited to statistics and data science as well as to modern scientific practice. The second session will discuss ethical statistical practice and Stewardship, a model for professionalism that prioritizes the discipline over the individual practitioner’s career, to support ethical practice and to promote the “vigor, quality, and integrity of the field”. With these sessions we will consider the dominant paradigm for ethics training and explore a new paradigm that emphasizes reasoning and decision making instead of familiarization with rules. We will use case studies in small- and whole- group discussion, to learn about ethical reasoning and to see how it works. The ASA Ethical Guidelines for Statistical Practice will be introduced and then we will explore ethical reasoning with the ASA Ethical Guidelines with two examples. After our lunch break we will learn about the stewardship construct and explore how our everyday engagement in statistical practice can represent opportunities to demonstrate and grow our stewardship roles. Finally, at the end of the first session we will have a discussion about whether (and why) statisticians may, or may seem to, be held to “a higher ethical standard”. At the end of the second session we will discuss how participants can continue to build on these introductory materials, how to promote ethical statistical practice in our home departments/institutions, and how to promote stewardly statistical practice (including the use of statistics by others).

CC BY-NC-ND
Featured Presentation at the Biennial Jamaican Statistical Symposium 2017 in Kingston, Jamaica. A “steward of the discipline” is someone to whom we can entrust the vigor, quality, and integrity of the field. This concept was originally conceptualized for doctoral students and doctoral holders, but every practitioner may strive to be that person to whom the field can be entrusted. Engagement with the discipline/professional community can support your scholarship, your stewardship -and the discipline. The stewardship model puts the field first – ahead of the individual’s CV. This can seem unreasonable, although it is simply unusual – and difficult to observe. However, it requires only a commitment to practice with integrity, to engage with professionalism, and to actively promote the prioritization of the discipline in decision-making. Stewardly statistical practice is therefore ethical statistical practice; engagement with the stewardship construct and the ASA Ethical Guidelines for Statistical Practice can help demonstrate that you are (or intend to be/become) someone to whom the community can entrust the vigor, quality, and integrity of the field.

CC BY-NC-ND
Bioinformatics training has important similarities and even more important differences as compared to bioinformatics “education” within a degree program. Typically, the training opportunities are highly focused and short, while the more formal educational opportunities are longer, embedded within other instruction, and may require assessment like tests, papers, or independent research projects. However, there are key features of adult learning and educational psychology that can help teaching in both the training and educational contexts. This workshop will define and explore teaching goals, which often feature what the instructor feels must be “covered” or conveyed; and contrast these with learning goals, which represent what the instructor intends that students will be able to do after the instruction. In the four hours we have together, we will have some lecture based foundational material with exercises around existing training course descriptions, followed by small group work to create or revise learning goals that can engage attendees in one course or sequences of courses. We will also explore how to efficiently plan a course and utilize the limited time to achieve learning goals and promote ongoing engagement with the tools and reasoning that are essential to modern biological sciences and bioinformatics. Participants are encouraged to bring course descriptions and/or learning goals to share, discuss, and work on.

CC BY-NC-ND
This talk focuses on the role of statistical literacy in journal reviewing and thereby, in scientific publishing. This type of literacy is learnable and improvable, the model of statistical literacy that is specific for the practicing scientist (or practicing scientist in training) was only published in January 2017, so is not widely known. Its recency notwithstanding, it is unlikely that any journal would be able to insist that its reviewers all possess sufficient statistical literacy to competently review potential contributions to science and their use of statistics to justify results and conclusions. An alternative would be for journal editorial staff to develop the skill set, and deploy it as needed. In the PLOS utopia, all science is peer reviewed and openly available; in MY utopia, all scientific peer review is *competent* so that whether or not the material is openly available, it is always reproducible and rigorous. I created this talk to introduce statistical literacy (in a model published in January 2017) as a learnable, improvable skill set, to the PLOS editorial staff in Cambridge, UK. We then spent some time discussing whether and how editors could start developing this skill set.

CC BY-NC-ND
Bioinformatics is a highly interdisciplinary domain. Currently there are many threads of discussions and effort globally aimed at ensuring that formal (undergraduate/graduate) training represents the biological and the computational aspects sufficiently to equip new practitioners from both of these domains to engage and participate in the field. Training opportunities are struggling less with this explicit interdisciplinarity because individuals who recognize that they require some targeted, “point of need” training are already prepared in some sense to integrate the biological and the computational aspects of the training. However, there are other options for disciplinary integration that can have immediate impact on the design and evaluation of training in bioinformatics. This talk will focus on how formal (degree programs) and informal (training/training programs) can be strengthened by explicitly integrating the disciplinary work around adult learning from the fields of education and cognitive science, with examples of successes and failures of this integration from the domain of biostatistics.

CC BY-NC-ND
This  invited talk at the Faculty Research Seminar for the Georgetown University School of Nursing and Health Studies presented three concepts that may need definitions and definitely need to be linked in order to promote meaningful educational research: stewardship, knowledge translation, and evidence informed teaching. With an example based on a recent 15-month pilot project with SNHS faculty around an "education research" question, the talk was intended to convince participants that "Translating knowledge promotes evidence-informed teaching as well as stewardly education research."

CC BY-NC-ND
This invited talk presented a new model (Tractenberg, et al. in review 2016) of developing *patient-centered* patient reported outcomes (PC-PROs), contrasting it with typical PROs, and integrated this type of model with Dr. Tractenberg's method for the analysis of change in Likert-rated items, Qualified Change.

CC BY-NC-ND
This invited presentation at the US Federal Trade Commission focused on the fact that measurement, and the measurement properties of outcomes, are essential features of the credibility of scientific research results and claims. A brief overview of different measurement constructs (definitions and methods of measurement; composite scores; practice effects; test-retest reliability; reliable change (index, methodology); normative data; validity) was provided to highlight how important each is, and their respective relevances throughout clinical research, focusing on cognitive applications.

This presentation at the 2016 BayesiaLab Users Conference in Nashville, TN (29 Sept 2016) was a collaborative effort focusing on leveraging key concepts and constructs from measurement and statistics to improve the accuracy and representativeness of a Bayesian Network (model of decisionmaking) that is created following a Define-Structure-Elicit-Verify (DSEV) approach. This presentation discussed how to use (leverage) model failures and edge cases to improve performance; how to use latent variable modeling (conceptualizing both emergent and causal forms of expert knowledge); and formally utilizing critical decision making techniques in the verification phase of the DSEV process.

This is an article by Ana Komnenic at Science Magazine posted 14 December 2016. http://www.sciencemag.org/news/2016/12/canada-case-spurs-concern-over-misconduct-secrecy

This is a really important story, for every scientist. If you publish your research, then *every single thing about that research should also be PUBLIC* - especially if it turns out you committed fraud. We must ALL insist that our institutions hold incompetent and unethical researchers responsible for their lack of competence and ethics. There is absolutely NO way to support a claim that fraud, falsification, or plagiarism do NOT clearly harm the public interest - in science. This is an especially indefensible position for research that is funded by taxpayer dollars. As I editorialized in 2011, "the responsible conduct of research" is not limited to properly obtained consent!!! http://blogs.plos.org/everyone/2011/11/02/the-%E2%80%9Cresponsible-conduct-of-research%E2%80%9D-is-not-limited-to-properly-obtained-consent/

CC BY-NC-ND
Since data analysis is becoming important across disciplines, the ASA Ethical Guidelines for Statistical Practice, which were originally approved by the ASA in 1995 and were updated in 2016, can serve to introduce all students in quantitative courses to critical concepts of responsible data analysis, interpretation, and reporting. The Guidelines contain elements that are suitable, and important, components of training for undergraduates and graduates whether or not they are statistics majors, to prepare them for ethical quantitative work. To achieve this preparation, and to support responsibility in the conduct of research involving data and its analysis, the Guidelines should be incorporated into every quantitative course. The Guideline principles interact, and sometimes must be prioritized. Therefore, neither the simple distribution of –nor an encouragement to memorize- the Guidelines can promote the necessary level of awareness. This presentation will introduce ethical reasoning as a learnable, improvable skill set that can provide an entry point to working with the 2016 revised ASA Ethical Guidelines. The purpose is to describe ethical reasoning both for instructors interested in adding this feature to quantitative courses and for students or consultants who wish to build experience, and evidence of engagement, with the ASA Ethical Guidelines.

CC BY-NC-ND
The Carnegie Initiative on the Doctorate (CID, 2001-2006) defined the “formation of stewards of the discipline” as the objective for doctoral education. “A steward is someone to whom the vigor, quality, and integrity of the field can be entrusted.” Key features of disciplinary stewardship include: the capacity to generate new knowledge and defend knowledge claims against challenges and criticism; a commitment to conserving the most important ideas and findings; and engagement in transforming knowledge by teaching well to a variety of audiences. This talk (third in a series on Higher Learning at the 2016 Baylor Symposium on Faith and Culture) discusses how the author achieves stewardship through purposeful engagement with others in research and teaching to promote stewardship.

CC BY-NC-ND
Metacognition is defined by the National Research Council (2001) as "the process of reflecting on and directing one's own thinking." It is an important -and arguably complex - skill set; as such it is both learnable and improvable. Taking advantage of these characteristics (learnable and improvable) requires a systematic and evaluable approach to an entire curriculum, but it can seem very challenging to even begin this process. Sustainable learning, defined as learning that continues beyond the end of formal instruction, might be an incentive to beginning the difficult work of formally integrating metacognition throughout a curriculum. For education to be truly formational and transformational, it requires that the learning continue beyond that educational experience. This presentation will outline the argument that teaching metacognitive skills not only can produce evidence that students can develop metacognition, but also that a focus on metacognition produces awareness in these students that these skills are learnable and improvable- and also, sustainable. Although theorists have been discussing "sustainable learning" for decades, this is the first evidence that students can perceive the sustainability of learning. Moreover, since metacognition is learnable and improvable, but sustainability is not, the results suggest that the challenges in integrating metacognition throughout a curriculum would be well worth that effort to achieve the sustainability of formational and transformational learning objectives.

These fully-annotated slides outline the results of the first (ever) 5-yearly revision effort by the ASA Committee on Professional Ethics (COPE). As of the date of this presentation (12 August 2021), the first draft of the revisions have been completed. Remaining work includes editorial work (wordsmithing) on the proposed revisions and the submission of the final draft to the ASA Board for their consideration (and approval). These slides were prepared and presented by the co-Chair of the 2021 Working Group on Revising the Guidelines (RET) for/at JSM2021, but represent the extensive and thoughtful efforts of the entire Working Group (WG), including the COPE Chair Jing Cao-co-Chair of the 2021 Revisions WG-and three other COPE members: Jason Gillikin; Matt Rotelli; and Marcia Weisman. Many of the COPE members joined the weekly meetings of the WG, but WG members were specifically responsible for marshaling input for two Guideline Principles each and both facilitating discussions and debates until a first draft of revisions were completed, and summarizing the discussion for presentation to the full COPE (and ultimately, the Board). These slides-and notes-will be summarized for the Proceedings of JSM 2021; it is uncertain if the Board will have had time to consider the proposed revisions before the Proceedings are published. Thus, this presentation (and the Proceedings article) represent the then-current report by the WG on the proposed revisions to the ASA Ethical Guidelines for Statistical Practice.

CC-BY-NC-ND 4.0 International license. Talk invited for the 2020 TAGC meeting, session 201.
It is common to create course material for the higher education context that accomplishes content-driven teaching goals, and then develop assessments (quizzes, exams) based on the target content. Content-driven assessment can tend to support teaching- or teacher- centered instruction. Adult learning and educational psychology theories suggest that instead, assessment should be aligned with learning objectives. The Genomics Education Alliance (GEA) was funded 2018-2020 to develop and curate classroom-based undergraduate research experience (CUREs) materials and part of the project is to evaluate the assessments that these materials include. To accomplish this aim, and also to support the alignment of assessments with instruction across the higher education life sciences, the Assessment Evaluation Rubric (AER) was developed. This rubric is intended to support the systematic evaluation of assessments that are included in materials that are curated by the GEA; however the AER can also be utilized to guide the development and evaluation/revision of assessments that are already used, whether or not these relate to genomics or to CUREs. The AER evaluates four features of an assessment: its general alignment with learning goal(s); whether the assessment is intended to/effective as formative or summative; whether some systematic approach to cognitive complexity is reflected; and whether the assessment (instructions as well as results) itself is clearly interpretable. Each dimension (alignment; utility; complexity; clarity) has four questions. Any assessment can be rated “present/absent” or “present/present, needs clarification/absent”, or along other dimensions, depending on the user. In the final year of the GEA funding, we are seeking to train assessment evaluators who can use the AER consistently; however, any instructor can use the AER to evaluate their own assessments and ensure that their quizzes and tests promote learning and learner centered teaching.

CC-BY-NC-ND 4.0 International license. Invited presentation at RSS 2019 4 Sept 2019 Belfast, NI.
The American Statistical Association (ASA) first established its Ethical Guidelines for Statistical Practice in 1999. In 2014, the ASA Committee on Professional Ethics set out to revise these Guidelines.  These were revised in 2016 and 2018. Lessons learned included how to balance “completeness” with shortness of attention spans; the need for applicability of the Guidelines to any person who works directly or indirectly with data; and the fact that Guideline principles may conflict in any given case, so rather than “learn the Guidelines”, ethical practitioners need to learn how to use the Guidelines. Recent worldwide interest in data science and data ethics creates challenges for teaching statistics and statistical ethics, but also creates new “data analysts” who are not, and who would not identify themselves as, “professional statisticians”. If applicability of the Guidelines is subtly shifted from "ethical statistical practice" to “ethical quantitative practice”, it can comprise data science and data ethics.

Abstract for poster and talk combination presented 25 July 2019 at ISMB/ECCB 2019 as part of the Education COSI:

Background: As the life sciences have become more computational and data intensive, the pressure to incorporate the requisite training into life-science education and training programs has increased. To facilitate curriculum development, various sets of bioinformatics competencies have been articulated; however, these have proved difficult to implement in practice. Addressing this issue, we have created a curriculum design and evaluation tool – the Mastery Rubric for Bioinformatics (MR-Bi) - to support the development of specific Knowledge, Skills and Abilities (KSAs) that promote bioinformatics practice and the achievement of competencies.

Methods: 12 KSAs were extracted, and stages along a developmental trajectory were identified. The KSAs and their performance level descriptors at each stage were formulated, ultimately yielding the MR-Bi.

Results and Conclusions: The MR-Bi prioritizes the development of independence and scientific reasoning. It can be used by practicing scientists at all career stages to direct their (and their team’s) acquisition of new, or to deepen existing, bioinformatics KSAs. It can be used to strengthen teaching and learning and for curriculum building. It can thereby contribute to the cultivation of a next generation of bioinformaticians who can design reproducible and rigorous research, and to critically analyze results from their own, and others’, work.

Invited Closing Keynote talk at Big Data & HealthCare Analytics Forum, held in Boston, MA 22-23 Oct 2018. *Abstract I was provided*: “Digital data has become the lifeblood of healthcare, touching every facet of the delivery system and informing – sometimes transforming – the way clinical decisions are made and operational strategies are developed and deployed.
And, as anyone who is a healthcare decision-maker knows all too well, there's more data than ever before – structured, unstructured, semi-structured; labs and imaging; genomic and proteomic; patent-generated data; social determinants of health – with more being amassed in electronic health records, connected devices and data lakes every day.
Thankfully, the technology used to access, analyze and put that data to work is also getting more advanced on a continuing basis. Ever more sophisticated analytics software and precise predictive algorithms are at our fingertips. Artificial intelligence and machine learning tools are finding their place and hospitals and health systems large and small.
At the same time, many more basic challenges – related to information governance, say, or simple data literacy – continue to vex many healthcare providers. But whether data beginners, or more advanced analytics innovators, everyone can learn more about how to make clinical and financial information work for higher-quality and more efficient care.”