Principled Role Assignment for Deliberative Reasoning with Multi-Agent LLMs in a Metacognitive Architecture
Abstract
We present a principled five-role architecture for multi-agent deliberative reasoning. Unlike existing frameworks that assign roles through ad-hoc domain metaphors or simple adversarial pairings, our system draws upon validated team-role taxonomies (Mumford et al., 2006; Belbin, 1993; Mathieu et al., 2014), dialectical reasoning (Schwenk, 1990), and metacognitive monitoring theory (Nelson & Narens, 1990) to compute a five-dimensional Metacognitive State Vector (MSV) for each agent, apply role-specific fitness functions, and assign globally optimal agent-roles via the Hungarian algorithm.
The demonstration system runs entirely offline using multiple local LLMs and provides real-time visualization of routing decisions, fitness matrices, dialectical pipeline stages, and MSV evolution.
Two principled departures distinguish this work: separating the Evaluator from the Critic to enable explicit meta-level monitoring distinct from object-level adversarial critique, grounded in Nelson & Narens' two-level metacognitive architecture; and positioning the Generalist outside the dialectical chain as a boundary-spanning orchestrator, grounded in the TREO framework.
Demo Video
The video demonstrates:
- MSV-based routing correctly discriminating simple factual queries (System 1) from complex contested queries (System 2);
- two new roles innovated as principled departures grounded in team science and dialectical philosophy;
- the full four-stage dialectical pipeline—Expert, Critic, Evaluator, Synthesizer—producing substantively different content at each stage;
- the heterogeneous fitness matrix using MSV-derived fitness functions and Hungarian-optimal role assignment; and
- explainability dashboard with MSV evolution across pipeline stages showing Correctness Evaluation increasing and Conflicting Information resolving through the dialectical process.
System Architecture
The five roles are derived from the intersection of dialectical philosophy, team role theory, and metacognitive science:
- Domain Expert (Thesis) — Mumford et al.'s Contributor: generates the initial response
- Critic (Antithesis) — Mumford et al.'s Critic: adversarial devil's advocacy at the object level
- Evaluator (Resolution) — Novel departure: meta-level monitoring of the thesis-antithesis exchange quality, grounded in Nelson & Narens (1990)
- Synthesizer (Synthesis) — Mumford et al.'s Completer: integrates refined perspectives
- Generalist (Orchestrating Observer) — Novel departure: boundary-spanning orchestrator from TREO framework (Mathieu et al., 2014)
Implementation
The system is implemented as a self-contained Python package (~2,100 lines across 12 modules) that extends an existing FastAPI/HTMX dual-process chat platform. The backend connects to local LLM instances via Ollama (currently Llama 3.2 and Qwen variants), enabling fully offline operation with no API dependencies.
External dependencies: SciPy (Hungarian algorithm), NumPy, Bokeh (visualization).
Source code: https://github.com/wpiresearch/metis-llm-src/tree/main/ijcai-msv-2026
Presentation Materials
Materials from the IJCAI-ECAI 2026 demo presentation in Bremen.
- Demo Talk Slides (10 min): [PDF]
Related Publications
- IJCAI-ECAI 2026 (this work): Ricky J. Sethi, Charles Courchaine, and Hefei Qiu, “Principled Role Assignment for Deliberative Reasoning with Multi-Agent LLMs in a Metacognitive Architecture.” Demonstrations Track, IJCAI-ECAI 2026. [PDF]
- TheWebConf 2026: Charles Courchaine*, Ricky J. Sethi*, and Hefei Qiu, “Implementation of a Metacognition Framework for Self-Awareness and Self-Regulation in Ensembles of LLMs.” ACM Proceedings of The Web Conference (TheWebConf), 2026. (* Equal contribution) [PDF]
- CIKM 2025: Ricky J. Sethi, Hefei Qiu, Charles Courchaine, and Joshua Iacoboni, “Do LLMs Dream of Electric Emotions? Towards Quantifying Metacognition and Generalizing the Teacher-Student Model Using Ensembles of LLMs.” ACM International Conference on Information and Knowledge Management (CIKM), 2025. [ACM DL] [PDF]
Funding: Supported in part by a Google Cloud Research Grant (November 2025).
Contact
For questions about this work, please contact rickys@sethi.org.