Graph Prompting

Graph prompting integrates graph structures – such as knowledge graphs, relational networks, and graph neural network (GNN) embeddings – into LLM prompts to enhance the model's ability to reason over structured, relational data. This addresses a key limitation of LLMs: their difficulty in precisely handling factual and relational information encoded in graph form.¹⁾

Graphs are serialized into textual representations that LLMs can process directly. Google Research's “Talk Like a Graph” explores encoding strategies including node ordering, edge notation formats, and subgraph selection methods.²⁾ Key findings:

• Larger LLMs perform better on graph reasoning tasks due to their capacity for complex patterns.
• LLMs struggle with specific graph tasks like cycle detection compared to simple algorithmic baselines.
• Encoding format significantly impacts performance.

Graph Neural Prompting combines GNNs with LLMs through a multi-step process:³⁾

1. Subgraph retrieval: Relevant subgraphs are retrieved from a knowledge graph based on query entities.
2. GNN encoding: A graph neural network encodes the subgraph nodes into embeddings.
3. Cross-modality pooling: Relevant nodes are selected through cross-attention with the text query.
4. Domain projection: A projector aligns graph embeddings with the LLM's text embedding space.
5. Prompt construction: The resulting graph neural prompt (a soft prompt) is prepended to text embeddings.

This produces instance-specific prompts per query, unlike dataset-level methods like standard prompt tuning.

GraphICL uses structured prompt templates to capture graph structure in Text-Attributed Graphs, enabling in-context learning without training. It outperforms specialized graph LLMs in resource-constrained settings.⁴⁾

Knowledge graphs provide factual and structural knowledge to augment LLMs through retrieval-augmented mechanisms:

• Subgraphs are retrieved based on query entities from questions and answer options.
• Graph-derived information is encoded into prompts via soft prompts or textual serialization.
• This plug-and-play integration avoids retraining LLMs.

Integration approaches fall into four categories:

• GNNs as prefixes: Graph encodings prepended to LLM input.
• LLMs as prefixes: LLM features used to enhance graph processing.
• Full integration: Joint training of GNN and LLM components.
• LLMs only: Graph information serialized as text for pure LLM processing.

Graph Neural Prompting achieved significant improvements:⁵⁾

• +13.5% accuracy over baselines (e.g., prompt tuning) on frozen LLMs across six commonsense and biomedical datasets.
• +1.8% accuracy improvement when LLMs are additionally fine-tuned.
• Outperformed both standard prompt tuning and LoRA on multiple benchmarks.
• Ablation studies confirmed the contribution of each component (GNN, pooling, projector).

GraphICL outperformed specialized graph LLMs and GNNs on out-of-domain text-attributed graph benchmarks.

• Commonsense reasoning: Grounding LLM responses in knowledge graph facts.
• Biomedical question answering: Leveraging medical knowledge graphs for clinical QA.
• Node classification: Classifying entities in social or citation networks.
• Link prediction: Predicting missing relationships in knowledge graphs.
• Graph reasoning: Tasks like cycle detection, edge existence checking, and shortest path finding.

• Graph quality dependency: Performance relies on the coverage and accuracy of the underlying knowledge graph.
• Alignment challenge: Bridging graph and text embedding spaces requires careful projection.
• Scalability: Large knowledge graphs increase retrieval and encoding overhead.
• Task-specific tuning: GNP components may need retraining for different domains or graph types.

• Prompt Engineering
• Generate Knowledge Prompting
• chain_of_thought_prompting
• Few-Shot Prompting
• Program-Aided Language Models (PAL)