weekend ai reads for 2023-03-17

March 17, 2023

🏗️ FOUNDATIONS

  • I'm going to skip all the posts about everyone and their product owner bolting GPT-4 onto their thing. Except this screenshot (Twitter, sorry) of a AI-generated summary of a Teams meeting. Yes, please.

  • "The Difference Between Speaking and Thinking", The human brain could explain why AI programs are so good at writing grammatically superb nonsense The Atlantic

  • via Patrick:

    • A key limitation - and funding opportunity - is creation of high-quality datasets that power the LLM models. One example is the “Feedback Prize: English Language Learners” project led by NLP expert Scott Crossley. This project curated a dataset of essays written by English Language Learners and used AI to determine the level of English proficiency in the student writing. The project crowdsourced solutions on Kaggle, a major data science platform, with more than 2,000 data scientists participating. 

    • Other high-leverage datasets that need to be assembled include:

      • Tutoring moves such as giving students feedback

      • Middle school math such as understanding algebraic equations.

      • Spaced repetition in middle school math

  • Note that there’s a large role to play in agenda setting around datasets. This is important for math researchers and closely linked to large model datasets and design. High-quality labeled datasets come from human raters, which has many complications: how to adequately collect labels, how to choose an appropriate taxonomy or label dictionary, and other humanistic and product/use-inspired questions. 

  • There are other issues too. Common (benchmark) labeled datasets can lead to systems all overfitting to those training sets. Dynabench sources new cases that are crowdsourced to specifically break models’ performance on those benchmarks: a testament to human creativity in designing ML evaluation. Here is an idea for a dataset solicitation with IES that might be helpful to revisit. 

🎓 EDUCATION and AI

📊 DATA & TECHNOLOGY

🎉 FUN and/or PRACTICAL THINGS