Virtnet.bond/extensions/Others/SocialBase/docs/AUTO_MODERATION_HOOKS.md

Auto-Moderation Hooks System

The SocialBase messaging system includes a powerful hooking system that allows extensions to automatically moderate messages using custom filters and AI.

🎯 Overview

When a message is sent, the system dispatches a socialbase.message.moderation event that extensions can listen to. Extensions can analyze the message and return an action to take.

📋 Available Actions

Extensions can return one of the following actions:

• flag - Flag the message for manual review
• delete - Automatically delete the message
• approve - Pre-approve the message
• keep or null - Do nothing, allow the message

🔧 Implementation

Basic Example

use Illuminate\Support\Facades\Event;

// In your extension's boot() method
Event::listen('socialbase.message.moderation', function ($message) {
    // Analyze the message
    $content = strtolower($message->content);
    
    // Check for spam keywords
    $spamKeywords = ['casino', 'viagra', 'lottery', 'winner'];
    foreach ($spamKeywords as $keyword) {
        if (str_contains($content, $keyword)) {
            return [
                'action' => 'flag',
                'reason' => "Message contains spam keyword: {$keyword}",
            ];
        }
    }
    
    // Allow the message
    return ['action' => 'keep'];
});

Advanced Example with AI

Event::listen('socialbase.message.moderation', function ($message) {
    // Use OpenAI or similar to analyze message
    $analysis = analyzeMessageWithAI($message->content);
    
    if ($analysis['is_toxic'] && $analysis['confidence'] > 0.9) {
        return [
            'action' => 'delete',
            'reason' => 'Toxic content detected with high confidence',
        ];
    }
    
    if ($analysis['is_spam'] && $analysis['confidence'] > 0.7) {
        return [
            'action' => 'flag',
            'reason' => 'Possible spam detected',
        ];
    }
    
    return ['action' => 'keep'];
});

URL Filter Example

Event::listen('socialbase.message.moderation', function ($message) {
    // Check for URLs
    if (preg_match('/(https?:\/\/[^\s]+)/', $message->content, $matches)) {
        $url = $matches[1];
        
        // Check against blacklist
        $blacklistedDomains = ['spam-site.com', 'phishing-site.com'];
        foreach ($blacklistedDomains as $domain) {
            if (str_contains($url, $domain)) {
                return [
                    'action' => 'delete',
                    'reason' => "Blacklisted URL: {$domain}",
                ];
            }
        }
        
        // Flag messages with multiple URLs
        $urlCount = substr_count($message->content, 'http');
        if ($urlCount > 2) {
            return [
                'action' => 'flag',
                'reason' => "Message contains {$urlCount} URLs",
            ];
        }
    }
    
    return ['action' => 'keep'];
});

Rate Limiting Example

Event::listen('socialbase.message.moderation', function ($message) {
    if (!$message->sender_id) {
        return ['action' => 'keep']; // System messages
    }
    
    // Check messages sent in last minute
    $recentMessages = \Paymenter\Extensions\Others\SocialBase\Models\Message::where('sender_id', $message->sender_id)
        ->where('created_at', '>', now()->subMinute())
        ->count();
    
    if ($recentMessages > 10) {
        return [
            'action' => 'flag',
            'reason' => 'User sent too many messages ({$recentMessages} in 1 minute)',
        ];
    }
    
    return ['action' => 'keep'];
});

Content Length Filter

Event::listen('socialbase.message.moderation', function ($message) {
    $length = strlen($message->content);
    
    // Flag very short messages (potential spam)
    if ($length < 3) {
        return [
            'action' => 'flag',
            'reason' => 'Message too short (possible spam)',
        ];
    }
    
    // Flag extremely long messages
    if ($length > 5000) {
        return [
            'action' => 'flag',
            'reason' => 'Message exceeds maximum length',
        ];
    }
    
    return ['action' => 'keep'];
});

🚨 User Reporting System

Users can report messages directly. When a message receives 3 or more reports, it's automatically flagged for review.

Report a Message (For Extensions)

use Paymenter\Extensions\Others\SocialBase\Helpers\MessagingHelper;

MessagingHelper::reportMessage(
    messageId: 123,
    reportedBy: $userId,
    reason: 'spam',
    description: 'This message contains unwanted advertising'
);

Report Reasons

Common report reasons:

• spam - Unwanted or repetitive content
• harassment - Bullying or threatening behavior
• inappropriate - Offensive or adult content
• scam - Fraudulent or phishing attempt
• other - Other reason (requires description)

📊 Available Message Data

The $message object passed to the hook contains:

$message->id               // Message ID
$message->conversation_id  // Conversation ID
$message->sender_id        // Sender user ID (null for system messages)
$message->sender           // Sender user object
$message->content          // Message text content
$message->type             // 'text' or 'system'
$message->is_system        // Boolean
$message->system_type      // Type if system message
$message->metadata         // Additional data (array)
$message->created_at       // Timestamp
$message->conversation     // Conversation object with participants

🔔 Available Events

Message Events

// When a message is sent
Event::listen('socialbase.message.sent', function ($message) {
    // Handle message sent
});

// When a message is reported by a user
Event::listen('socialbase.message.reported', function ($message, $report) {
    // Handle message report
    // $report contains: reporter, reason, description
});

// When a message is flagged for moderation
Event::listen('socialbase.message.flagged', function ($message, $flaggedBy) {
    // Handle message flagged
});

// When a message is approved by moderator
Event::listen('socialbase.message.approved', function ($message, $moderatorId) {
    // Handle message approved
});

// When a message is deleted by moderator
Event::listen('socialbase.message.deleted', function ($message, $moderatorId) {
    // Handle message deleted
});

🎓 Best Practices

1. Return Early - Only return an action if you need to moderate the message
2. Provide Reasons - Always include a clear reason for moderation actions
3. Don't Block Execution - Keep your analysis fast (<100ms recommended)
4. Log Actions - Log your moderation decisions for debugging
5. Test Thoroughly - Test with various message types before deploying
6. Avoid False Positives - Be conservative with auto-delete actions
7. Use Confidence Scores - For AI-based moderation, require high confidence

⚙️ Configuration Example

Create a config file for your moderation extension:

return [
    'enabled' => true,
    'auto_flag_spam' => true,
    'auto_delete_toxic' => false,  // Safer to flag first
    'spam_keywords' => ['casino', 'viagra', 'lottery'],
    'max_urls_per_message' => 2,
    'min_message_length' => 3,
    'max_message_length' => 2000,
    'rate_limit_messages_per_minute' => 10,
];

🔒 Security Considerations

• Never trust user input - Always sanitize and validate
• Rate limit API calls - If using external AI services
• Cache results - Cache analysis results to avoid repeated processing
• Monitor performance - Track execution time of your filters
• Fail safely - If your filter errors, return ['action' => 'keep']

📈 Example: Complete Moderation Extension

namespace Paymenter\Extensions\Others\MyModerationExtension;

use App\Classes\Extension\Extension;
use Illuminate\Support\Facades\Event;

class MyModerationExtension extends Extension
{
    public function boot()
    {
        Event::listen('socialbase.message.moderation', function ($message) {
            // Skip system messages
            if ($message->is_system) {
                return ['action' => 'keep'];
            }
            
            $content = strtolower($message->content);
            $config = $this->getConfig();
            
            // Check spam keywords
            foreach ($config['spam_keywords'] as $keyword) {
                if (str_contains($content, $keyword)) {
                    return [
                        'action' => 'flag',
                        'reason' => "Spam keyword detected: {$keyword}",
                    ];
                }
            }
            
            // Check URL count
            $urlCount = substr_count($message->content, 'http');
            if ($urlCount > $config['max_urls_per_message']) {
                return [
                    'action' => 'flag',
                    'reason' => "Too many URLs ({$urlCount})",
                ];
            }
            
            // Check message length
            $length = strlen($message->content);
            if ($length > $config['max_message_length']) {
                return [
                    'action' => 'flag',
                    'reason' => "Message too long ({$length} characters)",
                ];
            }
            
            return ['action' => 'keep'];
        });
    }
    
    public function getConfig($values = [])
    {
        return [
            [
                'name' => 'spam_keywords',
                'type' => 'textarea',
                'label' => 'Spam Keywords (one per line)',
                'description' => 'Messages containing these words will be flagged',
            ],
            [
                'name' => 'max_urls_per_message',
                'type' => 'number',
                'label' => 'Maximum URLs per message',
                'default' => 2,
            ],
            [
                'name' => 'max_message_length',
                'type' => 'number',
                'label' => 'Maximum message length',
                'default' => 2000,
            ],
        ];
    }
}

🎉 Summary

The auto-moderation hooks system allows you to:

• ✅ Automatically filter spam and toxic content
• ✅ Implement custom business rules
• ✅ Integrate AI-based content analysis
• ✅ Rate limit users
• ✅ Block malicious URLs
• ✅ Track and handle user reports
• ✅ Build sophisticated content filters without touching core code

Happy moderating! 🛡️