Google DeepMind has introduced a framework to measure progress toward Artificial General Intelligence (AGI) and launched a Kaggle hackathon to build relevant evaluations. Their new paper, "Measuring Progress Toward AGI: A Cognitive Taxonomy," provides a scientific foundation for understanding the cognitive capabilities of AI systems. Participants can design evaluations for key cognitive abilities for a chance to win from a prize pool of $200,000.
Measuring general intelligence is challenging due to the lack of empirical tools. Tracking AGI progress requires diverse methods, and cognitive science plays a crucial role. The paper identifies 10 key cognitive abilities:
- Perception: Extracting and processing sensory information
- Generation: Producing outputs like text, speech, and actions
- Attention: Focusing cognitive resources on significant matters
- Learning: Acquiring new knowledge through experience and instruction
- Memory: Storing and retrieving information
- Reasoning: Drawing valid conclusions through logical inference
- Metacognition: Knowledge and monitoring of one's cognitive processes
- Executive functions: Planning, inhibition, and cognitive flexibility
- Problem solving: Finding effective solutions to domain-specific issues
- Social cognition: Processing and interpreting social information appropriately
To evaluate AI capabilities across these cognitive abilities, a three-stage evaluation protocol is proposed:
- Evaluate AI systems across a broad suite of cognitive tasks using held-out test sets
- Collect human baselines from a demographically representative sample
- Map each AI system’s performance relative to human performance distribution
The Kaggle hackathon, "Measuring Progress Toward AGI: Cognitive Abilities," invites the community to design evaluations for five cognitive abilities with the largest evaluation gaps: learning, metacognition, attention, executive functions, and social cognition. A total prize pool of $200,000 is available, with submissions open from March 17 to April 16, and results announced on June 1.
Blogger's Review: By integrating cognitive science into AI evaluation, Google DeepMind offers a novel tool for quantifying AGI progress. This approach not only aids academia in understanding AGI construction but also provides clear directions for developing more intelligent AI systems, making it a noteworthy initiative.