Artificial Intelligence in Measurement and Education

Introductions - Positioning

Executive Board

  • Co-Chair: Chris Ormerod

  • Co-chair: John Whitmer

  • Secretary: Maggie Beiting-Parrish

Research Presentations

Andrew Lan, Assistant Professor at The University of Massachusetts Amherst

Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

Where to from here?

Update the Status

We have been approved by the NCME board. Two more immediate concerns are:

  • Meet with NCME board Liasons.

  • NCME Proposal.

Draft NCME SIGIME 2024 Panel Session

Applying Generative AI in Education: Demonstrations, Evidence of Impact, and Lessons Learned

Generative AI has exploded upon our culture and is having widespread impacts in many industries. In this presentation, members of the NCME Generative AI SIGIME will discuss examples of prototyping, piloting, and/or deploying education solutions using these new technologies. While this specific technology is new, measurement experts have significant experience applying innovative technologies in our work and many of our conventional measures of efficacy - namely validity, reliability, and fairness - are more relevant than ever in determining whether these innovations are ready for widescale use.

Draft NCME SIGIME 2024 Panel Session

Call for member submissions - anticipate 3-4 presentations

Paragraph-length abstract describing your application, which includes:

  1. Specific problem/use case seeking to address

  2. Technology / algorithmic approaches used (high level summary)

  3. Any custom datasets used to tune/train/refine the approach

  4. Evidence of impact

Additional Next Steps (Chris)

Furthering our Goals

  • Advertising GAIME on NCME

  • Suggestions for future speakers

  • Links with AI companies (Future speakers)

  • Advancement of Fair, Accountable, Transparent, and Ethical use of AI

Additional Next Steps (Chris)

Resources purchases ranking

  • Zoom account (webinar?)

  • Github Organization

  • Zotero w/shared space

  • Paid website domain name

  • Datasets for fine-tuning

  • Other Suggestions? What are we missing?

Note: Use of funds requires some board approval.

Publication Oppotunities

Future Shock: Grappling with the Generative AI Revolution

Journal: Harvard Data Science Review
Published: 11 Jul 2023
Deadline: 25 August 2023

URL: Call for papers

Articles clarifying the nature and limitations of foundation models, large language models (LLMs), and generative AI applications. Articles exploring the wider societal risks and impacts of foundation models, LLMs, and generative AI applications.

Publication Oppotunities

Leveraging Large Language Models for Assessment Support: Applications and Implications

Journal: Computers and Education: Artificial Intelligence (Open Access)
Published: 28 June 2023
Deadline: 30 October 2023
Contact: Prof. Rafael Ferreira Mello at rflm@cesar.org.br

URL: Call for Papers

This special issue focuses on exploring the implications of utilizing large language models (LLMs) for educational assessment, with particular emphasis on leveraging LLMs for formative feedback provision and mitigating challenges related to academic integrity. To this end, we invite researchers from diverse fields to submit papers investigating the use of LLMs to support the assessment process.