Self-Play Fine-Tuning (SPIN)

Self-Play fIne-tuNing (SPIN) is an iterative alignment method introduced by Chen et al. (2024) that converts weak language models into strong ones through a self-play mechanism, without requiring additional human-annotated preference data beyond an initial supervised fine-tuning (SFT) dataset. Inspired by self-play in game theory (e.g., AlphaGo Zero), SPIN enables an LLM to refine itself by distinguishing its own generated responses from human-written ones across successive iterations.

SPIN frames LLM alignment as a two-player game:

• Main Player: The current LLM being trained, which learns to distinguish human responses from synthetic ones.
• Opponent Player: The previous iteration of the LLM, which generates synthetic responses attempting to mimic human data.

At each iteration, the opponent generates responses to SFT prompts, and the main player is trained to prefer the original human responses over these synthetic generations. The process converges when the main player can no longer distinguish between human and self-generated responses – i.e., the model's output distribution matches the target data distribution.

The SPIN objective uses a logistic loss similar to DPO-style preference optimization:

<latex> \mathcal{L}_{\text{SPIN}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right] </latex>

Where:

• y_w is the winning (human/SFT) response
• y_l is the losing (self-generated) response
• pi_ref is the opponent (previous iteration)
• beta controls KL regularization strength
• sigma is the sigmoid function

The global optimum is provably achieved only when the LLM policy aligns with the target human data distribution.

1. Initialize with an SFT model trained on the target dataset
2. Generate synthetic responses from the current model on SFT prompts
3. Construct preference pairs: human responses (chosen) vs. synthetic responses (rejected)
4. Fine-tune via logistic loss to prefer chosen over rejected
5. Update roles: the newly trained model becomes the main player; the previous version becomes the opponent
6. Repeat until convergence (typically 2-4 iterations)

SPIN has provable convergence guarantees. The training objective reaches its global optimum when the model's generation distribution matches the human data distribution exactly. In practice, convergence typically occurs within 3 iterations, after which additional training yields diminishing returns. The logistic loss formulation prevents value function divergence, and KL regularization ensures stability between iterations.

from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
 
# SPIN iterative training loop (simplified)
def spin_iteration(model_path, sft_dataset, iteration, beta=0.1):
    # Load current model (opponent) and create training copy (main player)
    opponent = AutoModelForCausalLM.from_pretrained(model_path)
    main_player = AutoModelForCausalLM.from_pretrained(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_path)
 
    preference_pairs = []
    for example in sft_dataset:
        prompt = example["prompt"]
        human_response = example["response"]  # y_w (chosen)
 
        # Generate synthetic response from opponent
        inputs = tokenizer(prompt, return_tensors="pt")
        synthetic = opponent.generate(**inputs, max_new_tokens=512)
        synthetic_response = tokenizer.decode(synthetic[0])  # y_l (rejected)
 
        preference_pairs.append({
            "prompt": prompt,
            "chosen": human_response,
            "rejected": synthetic_response,
        })
 
    # Train main player with DPO-style logistic loss on preference pairs
    # Uses standard DPO trainer with opponent as reference model
    from trl import DPOTrainer, DPOConfig
    config = DPOConfig(beta=beta, output_dir=f"spin_iter_{iteration}")
    trainer = DPOTrainer(
        model=main_player,
        ref_model=opponent,
        train_dataset=preference_pairs,
        tokenizer=tokenizer,
        args=config,
    )
    trainer.train()
    return main_player
 
# Run 3 SPIN iterations
model_path = "alignment-handbook/zephyr-7b-sft-full"
dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train")
for i in range(3):
    model = spin_iteration(model_path, dataset, iteration=i)
    model_path = f"spin_iter_{i}"

• HuggingFace Open LLM Leaderboard: Starting from zephyr-7b-sft-full, SPIN iteration 1 matches or surpasses DPO trained with extra GPT-4 preference data
• MT-Bench: Significant improvements over SFT baselines across all categories
• Big-Bench: Consistent gains on reasoning and knowledge tasks
• SPIN achieves these results using only the existing SFT dataset, while DPO requires additional preference annotations

• Performance is upper-bounded by the quality of the initial SFT dataset
• Cannot exceed the target human data distribution
• Diminishing returns after 3-4 iterations as model-human gap closes
• Computationally expensive due to generation step at each iteration

• Chen et al. (2024) - Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• ICML 2024 Proceedings - PMLR 235:6621-6642
• Official SPIN Implementation (UCLA ML)
• Rafailov et al. (2023) - Direct Preference Optimization (DPO)

• LLM-as-a-Judge - Evaluation methodology used in SPIN experiments (MT-Bench)
• AgentBench - Benchmark for evaluating LLM agent capabilities