Skip to content

Critics Guide

Critics are the core of Sifaka's text improvement system. They analyze text and provide structured feedback for iterative refinement.

Overview

Each critic implements a specific evaluation strategy based on academic research:

Critic Best For Research Paper
SELF_REFINE General improvement Self-Refine (2023)
REFLEXION Learning from mistakes Reflexion (2023)
CONSTITUTIONAL Safety & principles Constitutional AI (2022)
SELF_CONSISTENCY Balanced perspectives Self-Consistency (2022)
SELF_RAG Fact-checking Self-RAG (2023)
META_REWARDING Self-evaluation Meta-Rewarding (2024)
N_CRITICS Multiple perspectives Based on ensemble methods
STYLE Tone & style Custom implementation

Using Critics

Single Critic

from sifaka import improve
from sifaka.core.types import CriticType

result = await improve(
    "Your text here",
    critics=[CriticType.SELF_REFINE]
)

Multiple Critics

result = await improve(
    "Your text here",
    critics=[CriticType.SELF_REFINE, CriticType.REFLEXION]
)

Custom Configuration

from sifaka.core.config import Config, CriticConfig

config = Config(
    critic=CriticConfig(
        critics=[CriticType.CONSTITUTIONAL],
        confidence_threshold=0.8
    )
)

result = await improve("Your text", config=config)

Critic Details

SELF_REFINE

General-purpose improvement focusing on clarity, coherence, and completeness.

Use when: - You need balanced, general improvement - Working with any type of content - Starting point for text refinement

Example: 

result = await improve(
    "AI is important for business.",
    critics=[CriticType.SELF_REFINE],
    max_iterations=3
)

REFLEXION

Learns from previous attempts by reflecting on what worked and what didn't.

Use when: - Complex reasoning tasks - Technical explanations - Content requiring deep analysis

Example: 

result = await improve(
    "Explain quantum computing simply.",
    critics=[CriticType.REFLEXION],
    max_iterations=4  # Benefits from more iterations
)

CONSTITUTIONAL

Evaluates text against constitutional principles for safety and ethics.

Use when: - Content needs ethical review - Safety-critical applications - Ensuring balanced perspectives

Principles evaluated: - Harmlessness - Helpfulness - Honesty - Accuracy - Nuance

Example: 

result = await improve(
    "Guide on pest control methods",
    critics=[CriticType.CONSTITUTIONAL]
)

SELF_CONSISTENCY

Generates multiple perspectives and finds consensus.

Use when: - Controversial topics - Need balanced viewpoints - Avoiding bias

Example: 

result = await improve(
    "Analysis of renewable energy policies",
    critics=[CriticType.SELF_CONSISTENCY]
)

SELF_RAG

Fact-checks and retrieves information to verify claims.

Use when: - Factual content - Academic writing - News or reports

Requires tools: 

# Tools must be configured separately
result = await improve(
    "The Great Wall of China facts",
    critics=[CriticType.SELF_RAG]
)

META_REWARDING

Evaluates its own critique quality through meta-evaluation.

Use when: - High-stakes content - Need confidence in improvements - Quality assurance

Example: 

result = await improve(
    "Medical advice disclaimer",
    critics=[CriticType.META_REWARDING]
)

N_CRITICS

Uses multiple critical perspectives in parallel.

Use when: - Comprehensive review needed - Multiple stakeholder perspectives - Final quality check

Default perspectives: - Technical Expert - General Audience - Subject Matter Expert - Editor - Skeptic

Example: 

result = await improve(
    "Product launch announcement",
    critics=[CriticType.N_CRITICS]
)

STYLE

Transforms text style and tone.

Use when: - Adapting content for audiences - Brand voice consistency - Style transformation

Configuration: 

from sifaka.critics.style import StyleCritic

critic = StyleCritic(
    style_description="Casual and friendly",
    style_examples=[
        "Hey there! Let me explain...",
        "No worries, we've got you covered!"
    ]
)

Combining Critics

Critics work well together:

# Technical accuracy + readability
result = await improve(
    text,
    critics=[CriticType.REFLEXION, CriticType.STYLE]
)

# Safety + factual accuracy
result = await improve(
    text,
    critics=[CriticType.CONSTITUTIONAL, CriticType.SELF_RAG]
)

# Comprehensive review
result = await improve(
    text,
    critics=[
        CriticType.SELF_REFINE,
        CriticType.N_CRITICS,
        CriticType.META_REWARDING
    ]
)

Performance Considerations

Speed vs Quality

Fast critics: - SELF_REFINE - STYLE

Thorough critics: - REFLEXION - N_CRITICS - META_REWARDING

Resource intensive: - SELF_RAG (requires tools) - SELF_CONSISTENCY (multiple samples)

Optimization Tips

  1. Start simple: Use SELF_REFINE first
  2. Add specificity: Layer on specialized critics
  3. Limit iterations: 2-3 for most use cases
  4. Use appropriate models: Smaller models for simple critics

Custom Critics

Create your own critic:

from sifaka.plugins import CriticPlugin
from sifaka.core.models import CritiqueResult

class DomainExpertCritic(CriticPlugin):
    def __init__(self, domain: str):
        self.domain = domain

    async def critique(self, text: str, result):
        # Your critique logic
        feedback = f"From a {self.domain} perspective..."

        return CritiqueResult(
            critic=f"domain_expert_{self.domain}",
            feedback=feedback,
            suggestions=["Add more domain-specific details"],
            needs_improvement=True,
            confidence=0.75
        )

# Use it
result = await improve(
    text,
    critics=[DomainExpertCritic("medical")]
)

Best Practices

  1. Match critic to content: Use domain-appropriate critics
  2. Start broad, get specific: SELF_REFINE → specialized critics
  3. Consider your audience: Use STYLE for audience adaptation
  4. Verify facts: Use SELF_RAG for factual content
  5. Ensure safety: Use CONSTITUTIONAL for public content
  6. Iterate wisely: More isn't always better (2-4 iterations usually sufficient)

Troubleshooting

Text not improving? - Try different critics - Increase temperature (0.7-0.9) - Add more specific critics

Too many changes? - Reduce iterations - Lower temperature - Use more focused critics

Inconsistent results? - Use SELF_CONSISTENCY - Lower temperature for stability - Set specific style guidelines with STYLE critic