weekend ai reads for 2024-04-26

“The first thing to recognise is that it’s very hard to really properly evaluate models in the same way it’s very hard to properly evaluate humans,” said Mike Volpi, a partner at venture capital firm Index Ventures. “If you look at one thing like ‘can you jump high or run fast?’ it’s easy. But human intelligence? It’s almost an impossible task.”

Traditional benchmarks are often static or close-ended (e.g., MMLU multi-choice QA), which do not satisfy the above requirements. On the other hand, models are evolving faster than ever, underscoring the need to build benchmarks with high separability.

We introduce Arena-Hard – a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, which is a crowd-sourced platform for LLM evals. To measure its quality, we propose two key metrics:

1. Agreement to Human preference: whether the benchmark score has high agreement to human preference.

2. Separability: whether the benchmark can confidently separate models.

“All of these benchmarks are wrong, but some are useful,” he said. “Some of them can serve some utility for a fixed amount of time, but at some point, there’s so much pressure put on it that it reaches its breaking point.”

AI is getting so clever, so fast, that many of the benchmarks used to this point are now obsolete. Indeed, researchers in this area are scrambling to develop new, more challenging benchmarks. To put it simply, AIs are getting so good at passing tests that now we need new tests – not to measure competence, but to highlight areas where humans and AIs are still different, and find where we still have an advantage.

I think that people don’t realize how much they expose by simply putting a picture out there.

When it comes to artificial intelligence, what are we actually creating? Even those closest to its development are struggling to describe exactly where things are headed, says Microsoft AI CEO Mustafa Suleyman, one of the primary architects of the AI models many of us use today. He offers an honest and compelling new vision for the future of AI, proposing an unignorable metaphor — a new digital species — to focus attention on this extraordinary moment.

What does it mean to make something totally new, fundamentally different to any invention that we’ve known before?

Now that I have my documents of an appropriate length, I will need to load them to a vector database. A vector database stores text in both its original form, but also as embeddings, which are large arrays of floating point numbers, fundamental to how large language models process language. Words, sentences or documents that have ‘close’ embeddings in multidimensional space will be closely related to each other in content.

What was encouraging for me is that we seem to be getting over that annoying moral panic phase of AI where everything seemed driven and dominated by a fear of cheating and the attempts to detect it. I believe we are in a new phase now where colleges and universities and vendors and investors are exploring what AI can do and how it might fit into education, but crucially we haven’t figured it out yet.

This database is intended to be a community resource for educators, researchers, students, and other edtech specialists looking to stay up to date.

All of the questions apply a culturally aware perspective rather than a traditional edtech adoption perspective (though the traditional perspective is another useful lens to evaluate this moment of rapid AI integration).

Kai is your Al-powered teaching assistant, designed to save you time on lesson planning, content creation, and grading. It integrates chat, file uploads, and a robust knowledge base to streamline your workflow.

While the Phi-3 family of models knows some general knowledge, it cannot beat a GPT-4 or another LLM in breadth — there’s a big difference in the kind of answers you can get from a LLM trained on the entirety of the internet versus a smaller model like Phi-3.