Nanowhale is a 100-million-parameter mixture-of-experts (MoE) language model that represents an experimental approach to model development through agent-driven architecture and training methodologies. The model was both pretrained and post-trained using autonomous agent systems, demonstrating novel paradigms in how artificial intelligence systems can participate in the development of other AI systems.

Nanowhale exemplifies an emerging trend in machine learning where autonomous agents take active roles in model development pipelines. Rather than relying exclusively on human-directed training processes, the model's creation involved agents making decisions about architecture design, training data selection, and optimization strategies. This agent-centric approach represents a shift toward exploring whether AI systems can effectively guide the development of successor models, potentially creating feedback loops in AI capability advancement ¹⁾.

The 100-million-parameter scale positions Nanowhale as a relatively compact model compared to larger language models in production, yet sufficient for demonstrating architectural innovations and training methodologies at a manageable computational scale.

Nanowhale employs a mixture-of-experts (MoE) architecture, which represents a departure from dense transformer models. In MoE systems, multiple specialized neural network modules (experts) process input data, with a learned gating mechanism determining which experts activate for each token or input. This approach offers potential advantages in computational efficiency by activating only a subset of parameters per forward pass, despite the model containing a larger total parameter count ²⁾.

The specific configuration of Nanowhale's experts, gating strategy, and load balancing mechanisms reflect decisions made during the agent-driven training process, making the architectural choices themselves a product of automated optimization rather than purely human design.

The distinguishing characteristic of Nanowhale is its development through agent-driven pretraining and post-training phases. During pretraining, autonomous agents guided decisions about:

- Data curation and selection from available training corpora - Hyperparameter optimization including learning rates, batch sizes, and scheduling strategies - Training objective formulation and loss function design

Post-training involved agents directing processes such as:

- Instruction tuning where agents selected or generated task distributions for behavioral refinement ³⁾. - Reinforcement learning from human feedback (RLHF) or related preference optimization techniques where agents may have participated in reward modeling or policy optimization decisions ⁴⁾.

This approach raises important questions about emergent properties in model development: whether agent-optimized training processes discover fundamentally different solutions compared to human-directed approaches, and whether iterative agent participation creates distinctive behavioral patterns or capability profiles.

Nanowhale demonstrates technical feasibility of delegating model development decisions to autonomous agents, with potential implications for:

- Scalability of model development: Agent-driven processes could potentially accelerate model iteration cycles and reduce human engineering bottlenecks - Optimization discovery: Agents may identify non-intuitive architectural or training configurations that human engineers might overlook - Research methodology: The project contributes empirical data on agent capability in optimization and decision-making domains

The 100-million-parameter scale provides a testbed for these methodologies without requiring the massive computational resources associated with larger frontier models, enabling broader experimentation with agent-driven approaches.

As of 2026, Nanowhale represents an active exploration of agent-autonomous collaboration in AI development. Key research questions include assessment of model performance relative to comparable baselines, analysis of whether agent-selected configurations exhibit unique properties, and evaluation of whether agent-driven training produces models with distinctive generalization characteristics or emergent capabilities ⁵⁾.

The project contributes to broader research into how AI systems might participate in their own development and optimization, with implications for understanding both AI agent capabilities and the nature of model architecture and training design decisions.

• Mixtral 8x22B
• Large Language Models
• AI-Accelerated Theoretical Physics