MWX/SpamFerret: Difference between revisions

From Woozle Writes Code
< MWX
Jump to navigation Jump to search
(→‎Code: 6/10 version fights ampersandbots)
No edit summary
 
(48 intermediate revisions by 4 users not shown)
Line 1: Line 1:
==Navigation==
[[category:MediaWiki extension|SpamFerret]]
[[computing]]: [[software]]: [[MediaWiki]]: [[fighting spam posts in MediaWiki|fighting spam]]: [[SpamFerret]]
[[category:unsupported]]
==About==
'''SpamFerret''' was an attempt to improve on the SpamBlacklist extension, as a way for small open-edit MediaWiki sites to minimize spam edits. It is no longer supported.


==Overview==
The version posted here worked with older versions of MediaWiki and PHP, but even when working it was necessary to manually enter patterns (blacklisted items) into the database. Fortunately this isn't hard to do, but if this were to be revived it would be nice to have a more friendly admin interface (especially some way to take a spam-page, break it up into unique URLs, and let you select which ones to add to the database).
[[SpamFerret]] is [[User:Woozle|my]] attempt at an improvement over the SpamBlacklist extension. The version posted here works, but you have to manually enter patterns (blacklisted items) into the database. Fortunately this isn't hard to do, but a more friendly interface would be nice (especially some way to take a spam-page, break it up into unique URLs, and let you select which ones to add to the database).
===Requirements===
SpamFerret requires the mysqli library to be installed/enabled on PHP. ''This would need revision.''
===To-Do===
===To-Do===
* If spam filter db cannot be contacted, fall back to a [[captcha]] rather than letting spam through
* '''Important''': Document the installation process (pretty simple; the only non-obvious bit is the database spec)
* '''Important''': Document the installation process (pretty simple; the only non-obvious bit is the database spec)
* Option to allow banned IP addresses to create account and go to link received by email in order to un-block IP
* Improved pages for reporting automatic IP suspension, [[ampersandbot]]s
* Automated reporting of intercepted spam
* Automated reporting of intercepted spam
* Web-based tool ("Special" page) for:
** Entering new patterns
** Deactivating/modifying disused patterns
** Testing sample spam against existing or candidate filters (Kiki regex tester shows matches for spam which SpamFerret is letting through...)
* Log of changes to "patterns" table
* Management tools for new spam (generate candidate patterns from spam page, allow user to fine-tune and choose which ones to use, and add chosen/tweaked patterns to database)
* Management tools for new spam (generate candidate patterns from spam page, allow user to fine-tune and choose which ones to use, and add chosen/tweaked patterns to database)
* Manual reporting tools
* Easy way to automatically revert to chosen revision while showing all edits below filter-creation form, to make it easier to add new patterns
* Manual reporting tools:
** [BUG] "text tester" utility does not handle diffs properly (at all?) - e.g. pattern #684 isn't matched even though the regex works
** basic data viewing, i.e. received spam grouped by IP address, by triggered filter, or in chronological order
** list patterns least recently used, for possible deactivation
** list patterns least recently used, for possible deactivation
** [[whois]] of all recently promoted domains and create a consolidated list of owners
** [[whois]] of all recently promoted domains and create a consolidated list of owners
* Optional log of complete spam contents, for possible data-mining or filter-training
* <s>Optional log of complete spam contents, for possible data-mining or filter-training (e.g. to answer the question "if I disable this set of filters, how many spams would have gotten through instead of being caught by the other filters?" or "how effective would this proposed new filter have been at catching the spams received thus far?")</s> done
 
===MW Versions===
* [[SpamFerret]] (no version number yet) has been used without modification on MediaWiki versions 1.5.5, 1.7.1, 1.8.2, 1.9.3, 1.11.0, 1.12, 1.14.0, 1.15.0, and 1.15.1.


==Purpose==
==Purpose==
Line 21: Line 38:
* Does not keep a log of failed spam attempts, so there is no way to measure effectiveness
* Does not keep a log of failed spam attempts, so there is no way to measure effectiveness


[[SpamFerret]] will:
[[SpamFerret]], on the other hand:
* be database-driven
* is database-driven
* keep logs and counts of spam attempts by blacklisting and by IP
* keeps logs and counts of spam attempts by blacklisting and by IP
* match domains ("http://*.domain"), URLs ("http://*.domain/path") and catch-phrases ("helo please to forgive my posting but my children are hungary")
* matches domains ("http://*.domain"), URLs ("http://*.domain/path") and catch-phrases ("helo please to forgive my posting but my children are hungary")
** should also be able to match patterns, like long lists of links in a certain format
** can also match patterns, like long lists of links in a certain format


It may also be unsuitable for use on busier wikis, as the checking process (which only happens when an edit is submitted) may take a fair amount of CPU time (checks the entire page once per blacklisted pattern). This shouldn't be a problem for smaller wikis, which are often monitored less frequently than busier wikis and hence are more vulnerable to spam.
It may also be unsuitable for use on busier wikis, as the checking process (which only happens when an edit is submitted) may take a fair amount of CPU time (checks the entire page once per blacklisted pattern). This shouldn't be a problem for smaller wikis, which are often monitored less frequently than busier wikis and hence are more vulnerable to spam.
==Design==
<sql>
CREATE TABLE `patterns` (
  `ID` INT NOT NULL AUTO_INCREMENT,
  `Pattern` varchar(255) COMMENT 'pattern to match (regex)',
  `WhenAdded` DATETIME DEFAULT NULL COMMENT 'when this entry was added',
  `WhenTried` DATETIME DEFAULT NULL COMMENT 'when a spammer last attempted to include this pattern',
  `isActive` BOOL COMMENT 'if FALSE, do not include in checking',
  `isURL` BOOL COMMENT 'TRUE indicates that additional URL-related stats may be collected',
  `isRegex` BOOL COMMENT 'TRUE indicates that the string should not be escaped before feeding to preg_match()',
  `Count` INT DEFAULT 0 COMMENT 'number of attempts',
  PRIMARY KEY(`ID`)
)
ENGINE = MYISAM;
CREATE TABLE `clients` (
  `ID` INT NOT NULL AUTO_INCREMENT,
  `Address` varchar(15) COMMENT 'IP address',
  `WhenFirst` DATETIME COMMENT 'when this IP address first submitted a spam',
  `WhenLast` DATETIME COMMENT 'when this IP address last submitted a spam',
  `Count` INT DEFAULT 0 COMMENT 'number of attempts',
  PRIMARY KEY(`ID`)
)
ENGINE = MYISAM;
CREATE TABLE `attempts` (
  `ID` INT NOT NULL AUTO_INCREMENT,
  `When` DATETIME COMMENT 'timestamp of attempt',
  `ID_Pattern` INT NOT NULL COMMENT '(patterns.ID) matching pattern found',
  `ID_Client` INT NOT NULL COMMENT '(clients.ID) spamming client',
  `PageServer` varchar(63) COMMENT 'identifier of wiki being attacked (usually domain)',
  `PageName` varchar(255) COMMENT 'name of page where the spam would have displayed',
  PRIMARY KEY(`ID`)
)
ENGINE = MYISAM;
</sql>
==Reports==
Eventually, some Specialpages with reports would be nice, but for now you can see what's being blocked, and where the spam attempts are coming from, with this query &ndash; which you can package in a stored procedure, as shown, to make it easier to run (or just use the SELECT statement by itself):
<mysql>CREATE PROCEDURE Attempts()
  BEGIN
    select
      a.ID,
      a.`When`,
      CONCAT('(',a.ID_Pattern,') ',p.Pattern) AS Pattern,
      CONCAT('(',a.ID_Client,') ',c.Address) AS Address,
      PageServer,
      PageName
    from
      (attempts AS a
      LEFT JOIN patterns AS p ON a.ID_Pattern=p.ID)
      LEFT JOIN clients AS c ON a.ID_Client=c.ID
      ORDER BY a.ID DESC;
  END</mysql>
Execute <mysql>CALL Attempts();</mysql> to view the results.


==Code==
==Technical Docs==
This consists of two files. '''SpamFerret.php''' goes in the MediaWiki extensions folder, and '''data.php''' goes in the "includes" folder because PHP seems to want it there. Both files still contain some debugging code, most of which I'll clean up later (some of it calls stubbed debug-printout routines which can come in handy when adding features or fixing the inevitable bugs).
* [[/tables]]
* [[/views]]
==Installation==
Rough installation instructions, to be refined later (and note that code posted here is often not the latest, and there may be incompatibilities due to version non-synchronization between the different files; bug [[User:Woozle]] for the latest code):
* [[SpamFerret.php]] goes in the MediaWiki extensions folder
* [[User:Woozle/data.php|data.php]] goes in the "includes" folder because [[PHP]] seems to want it there.
* Add these lines to your [[MediaWiki/LocalSettings.php|LocalSettings.php]]:
<php>require_once( "$IP/extensions/SpamFerret.php" );
$wgSpamFerretSettings['dbspec'] = /* connection string - see notes below */;
$wgSpamFerretSettings['throttle_retries'] = 5; // 5 strikes and you're out
$wgSpamFerretSettings['throttle_timeout'] = 86400; // 86400 seconds = 24 hours</php>
* '''connection string''' has the following format: mysql:<u>db_user_name</u>:<u>db_user_password</u>@<u>db_server</u>/<u>spamferret_db_name</u>
** '''Example''': mysql:spfbot:b0tpa55@myserver.com/spamferretdb
* SpamFerret.php and data.php still contain some debugging code, most of which I'll clean up later (some of it calls stubbed debug-printout routines which can come in handy when adding features or fixing the inevitable bugs).
===Update Log===
* '''2009-08-05''' Developing [[/Special|SpecialPage]] for administrative purposes
* '''2007-12-27''' see [[SpamFerret.php]] code comments
* '''2007-10-13''' (1) IP throttling, and (2) logging of [[ampersandbot]] attempts
** If an IP address makes more than <u>N</u> spam attempts with no more than <u>T</u> seconds between them, it will not be allowed to post anything until a further <u>T</u> seconds have elapsed without spam.
* '''2007-06-10''' Added code to prevent [[ampersandbot]] edits; need to add logging of those blocks, but don't have time right now. Also don't know if the ampersandbots trim off whitespace or if that's just how MediaWiki is displaying the changes.
* '''2007-06-10''' Added code to prevent [[ampersandbot]] edits; need to add logging of those blocks, but don't have time right now. Also don't know if the ampersandbots trim off whitespace or if that's just how MediaWiki is displaying the changes.
===SpamFerret.php===
* '''2007-08-30''' Current version accommodates some changes to the data.php class library
<php>
<?php
 
# Loader for spam blacklist feature
# Include this from LocalSettings.php
 
if ( defined( 'MEDIAWIKI' ) ) {
require('shared.php');
 
global $wgFilterCallback, $wgPreSpamFilterCallback;
 
if ( $wgFilterCallback ) {
$wgPreSpamFilterCallback = $wgFilterCallback;
} else {
$wgPreSpamFilterCallback = false;
}
 
$wgFilterCallback = 'wfSpamFerretLoader';
$wgExtensionCredits['other'][] = array(
'name' => 'SpamFerret',
        'author' => 'Woozle Staddon',
        'url' => 'http://htyp.org/SpamFerret',
);
 
function wfSpamFerretLoader( &$title, $text, $section ) {
static $spamObj = false;
global $wgSpamFerretSettings, $wgPreSpamFilterCallback;
 
if ( $spamObj === false ) {
$spamObj = new SpamFerret( $wgSpamFerretSettings );
}
 
return $spamObj->filter( $title, $text, $section );
}
class SpamFerret {
var $dbspec;
var $previousFilter = false;
function SpamFerret( $settings = array() ) {
global $IP;
 
foreach ( $settings as $name => $value ) {
$this->$name = $value;
}
}
 
function filter( &$title, $text, $section ) {
global $wgArticle, $wgDBname, $wgMemc, $messageMemc, $wgVersion, $wgOut;
global $wgTitle, $wgServer;
global $debug;
 
$fname = 'wfSpamFerretFilter';
wfProfileIn( $fname );
 
# Call the rest of the hook chain first
if ( $this->previousFilter ) {
$f = $this->previousFilter;
if ( $f( $title, $text, $section ) ) {
wfProfileOut( $fname );
return true;
}
}
 
$retVal = false; // default = assume edit is ok
// Check for ampersandbot
$objArticleCurr = new Article($title);
$objArticleCurr->loadLastEdit();
$txtCurr = $objArticleCurr->getContent();
$lenNew = strlen($text);
if (strpos($txtCurr, $text)===0) {
// new string starts the same as old string; is it a truncated subset?
if ($lenNew < strlen($txtCurr)) {
// new string is a truncation of old string; is the first missing character an ampersand?
$chrDeleted = $txtCurr{$lenNew+1};
if ($chrDeleted == '&') {
// AMPERSANDBOT DETECTED; refuse to save the edit
// The string sent to spamPage() will be shown after "The following text is what triggered our spam filter:"
EditPage::spamPage( 'The removal of everything after the first ampersand. You are an ampersandbot; please go away.' );
// LATER: it would be nice to have a special MediaWiki:Ampersandbot page to display when this happens
$retVal = true;
}
}
}
 
if (!$retval) {
// Open the database
$dbSpam = new clsDatabase($this->dbspec);
$objTblPatterns = new clsDataTable($dbSpam,'patterns');
$objDataPatterns = $objTblPatterns->GetData('isActive');
/*
$debug .= ' objDataPatterns is object:'.is_object($objDataPatterns);
$debug .= ' objDataPatterns.Res is object:'.is_object($objDataPatterns->Res);
$debug .= ' objDataPatterns.Row is array:'.is_array($objDataPatterns->Row);
$debug .= ' objDataPatterns.Res is class '.get_class($objDataPatterns->Res);
$debug .= ' objDataPatterns.Res has '.$objDataPatterns->Res->num_rows.' row(s)';
*/
while(is_array($objDataPatterns->Row)) {
$strPattern = $objDataPatterns->GetValue('Pattern');
$isRegex = $objDataPatterns->GetValue('isRegex');
$idPattern = $objDataPatterns->GetValue('ID');
if ($isRegex) {
$strPattCk = strtolower($strPattern);
$strTextCk = strtolower($text);
$isMatch = preg_match('/'.$strPattCk.'/',$strTextCk,$matches);
if ($isMatch) {
$strMatch = $matches[0];
}
} else {
$strMatch = stristr ($text,$strPattern);
$isMatch = ($strMatch != '');
}
// $debug .= 'ROW: '.DumpArray($objDataPatterns->Row);
if ($isMatch) {
$objDataPatterns->Row = NULL; // stop the search
} else {
$objDataPatterns->NextRow(); // keep looking
}
}
if ( $strMatch != '' ) {
// spam cue found; display the matching text and don't allow the edit to be saved:
wfDebug( "Match!\n" );
// The string sent to spamPage() will be shown after "The following text is what triggered our spam filter:"
EditPage::spamPage( $strMatch );
// Log the spam attempt:
// -- first, get an ID for the IP address:
$strIPAddr = wfGetIP();
$sql = 'SELECT * FROM clients WHERE Address="'.$strIPAddr.'"';
$objDataClients = $dbSpam->Query($sql);
// update or create client record:
if (is_object($objDataClients)) {
if ($objDataClients->RowCount() > 0) {
$isFound = true;
}
}
if ($isFound) {
$idClient = $objDataClients->GetValue('ID');
$sql = 'UPDATE clients SET WhenLast=NOW(),Count=Count+1 WHERE Address="'.$strIPAddr.'"';
$dbSpam->Exec($sql);
} else {
$sql = 'INSERT INTO clients (Address,WhenFirst,Count) VALUES("'.$strIPAddr.'",NOW(),1)';
$dbSpam->Exec($sql);
$idClient = $dbSpam->NewID();
}
$sqlURL = '"'.$dbSpam->SafeParam($wgTitle->getFullURL()).'"';
$sqlSrvr = '"'.$dbSpam->SafeParam($wgServer).'"';
$sqlPage = '"'.$dbSpam->SafeParam($wgTitle->getPrefixedText()).'"';
$sql = 'INSERT INTO attempts (`When`,ID_Pattern,ID_Client,PageServer,PageName) VALUES (NOW(),'.$idPattern.','.$idClient.','.$sqlSrvr.','.$sqlPage.')';
$dbSpam->Exec($sql);
$sql = 'UPDATE patterns SET WhenTried=NOW(), Count=Count+1 WHERE ID='.$idPattern;
$dbSpam->Exec($sql);
$retVal = true;
} else {
// no spam cues found; allow the edit to be saved, if nothing else has tripped the filter
/*
EditPage::spamPage( 'DEBUGGING: '.$debug );
$retVal = true;
/*/
/**/
}
}
 
wfProfileOut( $fname );
return $retVal;
/**/
}
}
 
} // end of 'MEDIAWIKI' check
?>
</php>
 
===data.php===
<php>
<?php
/* ===========================
*** DATA UTILITY CLASSES ***
*/
 
class clsDatabase {
  private $cntOpen; // count of requests to keep db open
  private $strType; // type of db (MySQL etc.)
  private $strUser; // database user
  private $strPass; // password
  private $strHost; // host (database server domain-name or IP address)
  private $strName; // database (schema) name
 
  private $Conn; // connection object
 
  public function __construct($iConn) {
    CallEnter('clsDatabase('.get_class($iConn).')');
    $this->Init($iConn);
    CallExit('clsDatabase()');
  }
  public function Init($iConn) {
// $iConn format: type:user:pass@server/dbname
    CallEnter('clsDatabase.Init('.get_class($iConn).')');
    $this->cntOpen = 0;
    list($part1,$part2) = split('@',$iConn);
    list($this->strType,$this->strUser,$this->strPass) = split(':',$part1);
    list($this->strHost,$this->strName) = explode('/',$part2);
    $this->strType = strtolower($this->strType); // make sure it is lowercased, for comparison
    CallExit('clsDatabase.Init()');
  }
  public function Open() {
    CallEnter('clsDatabase.Open()');
    if ($this->cntOpen == 0) {
// then actually open the db
      if ($this->strType == 'mysql') {
        $this->Conn = new mysqli($this->strHost,$this->strUser,$this->strPass,$this->strName);
      } else {
        $this->Conn = dbx_connect($this->strType,$this->strHost,$this->strName,$this->strUser,$this->strPass);
      }
    }
    $this->cntOpen++;
    CallExit('clsDatabase.Open()');
  }
  public function Shut() {
    CallEnter('clsDatabase.Shut()');
    $this->cntOpen--;
    if ($this->cntOpen == 0) {
      if ($this->strType == 'mysql') {
        $this->Conn->close();
      } else {
        dbx_close($this->Conn);
      }
    }
    CallExit('clsDatabase.Shut()');
  }
  public function Query($iSQL) {
    CallEnter('clsDatabase.Query('.$iSQL.')');
    if ($this->strType == 'mysql') {
      $this->Conn->real_query($iSQL);
      $objQry = $this->Conn->store_result();
    } else {
      $objQry = dbx_query($this->Conn,$iSQL,DBX_RESULT_ASSOC);
    }
    if (is_object($objQry)) {
      $objRec = new clsDataItem($objQry);
      CallExit('clsDatabase.Query('.$iSQL.') -> new clsDataItem');
      return $objRec;
    } else {
      CallExit('clsDatabase.Query('.$iSQL.') -> NULL (ERROR!)');
      return NULL; // empty result set, one way or another
    }
  }
  public function Exec($iSQL) {
// MYSQL only
    $objQry = $this->Conn->prepare($iSQL);
    return $objQry->execute();
  }
  public function NewID() {
    return $this->Conn->insert_id;
  }
  public function SafeParam($iString) {
    return $this->Conn->escape_string($iString);
  }
}
 
class clsDataTable {
  protected $objDB; // clsDatabase
  protected $strName;
 
  public function __construct($iDB,$iName) {
    CallEnter('clsDataTable('.get_class($iDB).','.$iName.')');
    $this->objDB = $iDB;
    $this->strName = $iName;
    $this->objDB->Open();
    if (!isset($this->objDB)) {
      print 'DATABASE NOT OPEN!';
    }
    CallExit('clsDataTable()');
  }
  public function __destruct() {
    $this->objDB->Shut();
  }
  public function GetItem($iID,$iObj=NULL) {
    CallEnter('clsDataTable.GetItem('.$iID.')');
    $sql = 'SELECT * FROM '.$this->strName.' WHERE ID='.$iID;
//    $objQry = dbx_query($this->objDB,$sql,DBX_RESULT_ASSOC);
    $objQry = $this->objDB->Query($sql);
    if (is_object($iObj)) {
      $iObj->Eat($objQry);
      CallExit('clsDataTable.GetItem('.$iID.') -> ate object');
      return $iObj;
    } else {
      CallExit('clsDataTable.GetItem('.$iID.') -> from objDB.Query');
      return $objQry;
    }
  }
  public function GetData($iFilt) {
    CallEnter('clsDataTable.GetData('.$iFilt.')');
    $sql = 'SELECT * FROM '.$this->strName.' WHERE ('.$iFilt.')';
    $objQry = $this->objDB->Query($sql);
    CallExit('clsDataTable.GetData('.$iFilt.')');
    return $objQry;
  }
}
 
class clsDataItem {
  public $Res; // result set
  public $Row; // first (presumably the *only*) row data
  public $Table; // clsDataTable object
  public function __construct($iResults=NULL,$iTable=NULL) {
    CallEnter('clsDataItem('.get_class($iResults).')');
    if (is_object($iResults)) {
      $this->Init($iResults,$iTable);
    }
    CallExit('clsDataItem()');
  }
  public function Init($iResults,$iTable=NULL) {
    CallEnter('clsDataItem.Init('.get_class($iResults).')');
    $this->Res = $iResults;
    $this->Table = $iTable;
// this works for mysql only:
// fetch the first row of the results as an associative array:
    $this->Row = $iResults->fetch_assoc();
    CallExit('clsDataItem.Init()');
  }
  public function Eat($iDataItem) {
    CallEnter('clsDataItem.Eat('.get_class($iDataItem).')');
    $this->Res = $iDataItem->Res;
    $this->Table = $iDataItem->Table;
    $this->Row = $iDataItem->Row;
    CallExit('clsDataItem.Eat()');
  }
  public function GetValue($iName) {
// this works for mysql only
    CallEnter('GetValue('.$iName.')');
    DumpArray($this->Row);
    CallExit('GetValue('.$iName.')');
    return $this->Row[$iName];
  }
  public function RowCount() {
    return $this->Res->num_rows;
  }
  public function NextRow() {
// this works for mysql only:
// fetch the NEXT row of the results as an associative array:
    $this->Row = $this->Res->fetch_assoc();
  }
}
 
/* ========================
*** UTILITY FUNCTIONS ***
*/
function Pluralize($iQty,$iSingular,$iPlural='s') {
if ($iQty == 1) {
return $iSingular;
} else {
return $iPlural;
}
}
/* ========================
*** DEBUGGING FUNCTIONS ***
*/
define(KS_DEBUG_HTML,0);
 
// these could later be expanded to create a call-path for errors, etc.
function CallEnter($iName) {
/*
  global $intCallDepth, $debug;
  $intCallDepth++;
  if (KS_DEBUG_HTML) {
    $debug .= '<br><span style="background: #000000;"><b>'.str_repeat('&gt;&gt; ',$intCallDepth).'</b>'.$iName.'</span>';
  } else {
    $debug .= "\n\n".str_repeat('*',$intCallDepth).$iName;
  }
/**/
}
function CallExit($iName) {
/*
  global $intCallDepth, $debug;
  $intCallDepth--;
  if (KS_DEBUG_HTML) {
    $debug .= '<br><span style="background: #000000; color: yellow;"><b>'.str_repeat('&gt;&gt; ',$intCallDepth).'&lt;&lt; </b>'.$iName.'</span>';
  } else {
    $debug .= "\n\n".str_repeat('*',$intCallDepth).'<';
  }
/**/
}
function DumpArray($iArr) {
  global $intCallDepth, $debug;
/*
  while (list($key, $val) = each($iArr)) {
    if (KS_DEBUG_HTML) {
      $debug .= '<br><span style="background: #000000; color: green;"><b>'.str_repeat('-- ',$intCallDepth+1).'</b>';
      $debug .= " $key => $val";
      $debug .= '</span>';
    } else {
      $debug .= "/ $key => $val /";
    }
  }
/**/
}
?>
</php>

Latest revision as of 20:17, 1 May 2022

About

SpamFerret was an attempt to improve on the SpamBlacklist extension, as a way for small open-edit MediaWiki sites to minimize spam edits. It is no longer supported.

The version posted here worked with older versions of MediaWiki and PHP, but even when working it was necessary to manually enter patterns (blacklisted items) into the database. Fortunately this isn't hard to do, but if this were to be revived it would be nice to have a more friendly admin interface (especially some way to take a spam-page, break it up into unique URLs, and let you select which ones to add to the database).

Requirements

SpamFerret requires the mysqli library to be installed/enabled on PHP. This would need revision.

To-Do

  • If spam filter db cannot be contacted, fall back to a captcha rather than letting spam through
  • Important: Document the installation process (pretty simple; the only non-obvious bit is the database spec)
  • Option to allow banned IP addresses to create account and go to link received by email in order to un-block IP
  • Improved pages for reporting automatic IP suspension, ampersandbots
  • Automated reporting of intercepted spam
  • Web-based tool ("Special" page) for:
    • Entering new patterns
    • Deactivating/modifying disused patterns
    • Testing sample spam against existing or candidate filters (Kiki regex tester shows matches for spam which SpamFerret is letting through...)
  • Log of changes to "patterns" table
  • Management tools for new spam (generate candidate patterns from spam page, allow user to fine-tune and choose which ones to use, and add chosen/tweaked patterns to database)
  • Easy way to automatically revert to chosen revision while showing all edits below filter-creation form, to make it easier to add new patterns
  • Manual reporting tools:
    • [BUG] "text tester" utility does not handle diffs properly (at all?) - e.g. pattern #684 isn't matched even though the regex works
    • basic data viewing, i.e. received spam grouped by IP address, by triggered filter, or in chronological order
    • list patterns least recently used, for possible deactivation
    • whois of all recently promoted domains and create a consolidated list of owners
  • Optional log of complete spam contents, for possible data-mining or filter-training (e.g. to answer the question "if I disable this set of filters, how many spams would have gotten through instead of being caught by the other filters?" or "how effective would this proposed new filter have been at catching the spams received thus far?") done

MW Versions

  • SpamFerret (no version number yet) has been used without modification on MediaWiki versions 1.5.5, 1.7.1, 1.8.2, 1.9.3, 1.11.0, 1.12, 1.14.0, 1.15.0, and 1.15.1.

Purpose

The SpamBlacklist extension has a number of shortcomings:

  • Can only handle a limited number of entries before exceeding the maximum string-length it can process, at which point all spam is allowed through
  • Does not keep track of which entries are still being "tried" (to allow for periodic "cleaning" of the list)
  • Does not keep track of offending IP addresses
  • Handles only domains; cannot blacklist by URL path (for partially compromised servers) or "catch phrases" found in spam and nowhere else
  • Does not keep a log of failed spam attempts, so there is no way to measure effectiveness

SpamFerret, on the other hand:

  • is database-driven
  • keeps logs and counts of spam attempts by blacklisting and by IP
  • matches domains ("http://*.domain"), URLs ("http://*.domain/path") and catch-phrases ("helo please to forgive my posting but my children are hungary")
    • can also match patterns, like long lists of links in a certain format

It may also be unsuitable for use on busier wikis, as the checking process (which only happens when an edit is submitted) may take a fair amount of CPU time (checks the entire page once per blacklisted pattern). This shouldn't be a problem for smaller wikis, which are often monitored less frequently than busier wikis and hence are more vulnerable to spam.

Technical Docs

Installation

Rough installation instructions, to be refined later (and note that code posted here is often not the latest, and there may be incompatibilities due to version non-synchronization between the different files; bug User:Woozle for the latest code):

<php>require_once( "$IP/extensions/SpamFerret.php" ); $wgSpamFerretSettings['dbspec'] = /* connection string - see notes below */; $wgSpamFerretSettings['throttle_retries'] = 5; // 5 strikes and you're out $wgSpamFerretSettings['throttle_timeout'] = 86400; // 86400 seconds = 24 hours</php>

  • connection string has the following format: mysql:db_user_name:db_user_password@db_server/spamferret_db_name
    • Example: mysql:spfbot:b0tpa55@myserver.com/spamferretdb
  • SpamFerret.php and data.php still contain some debugging code, most of which I'll clean up later (some of it calls stubbed debug-printout routines which can come in handy when adding features or fixing the inevitable bugs).

Update Log

  • 2009-08-05 Developing SpecialPage for administrative purposes
  • 2007-12-27 see SpamFerret.php code comments
  • 2007-10-13 (1) IP throttling, and (2) logging of ampersandbot attempts
    • If an IP address makes more than N spam attempts with no more than T seconds between them, it will not be allowed to post anything until a further T seconds have elapsed without spam.
  • 2007-06-10 Added code to prevent ampersandbot edits; need to add logging of those blocks, but don't have time right now. Also don't know if the ampersandbots trim off whitespace or if that's just how MediaWiki is displaying the changes.
  • 2007-08-30 Current version accommodates some changes to the data.php class library
Retrieved from "https://wooz.dev/mw/index.php?title=MWX/SpamFerret&oldid=1352"