Panagiotis (Panos) Garefalakis

Proxying Python Requests

Fri, 30 Oct 2015 00:00:00 -0700

Introduction

Proxying is very useful when conducting intensive web crawling/scrapping or when you just want to hide your identity (anomization).

A convenient way to implement HTTP requests is using Pythons requests library. One of requests most popular features is simple proxying support. HTTP as a protocol has very well-defined semantics for dealing with proxies, and this contributed to the widespread deployment of HTTP proxies.

In this post I am using public proxies to randomise http requests over a number of IP addresses and using a variety of known user agent headers these requests look to have been produced by different applications and operating systems. The whole project is located on github.

TinyProxy Installation

If you wanted to use your own proxy, to avoid high delays from public proxy providers, you could install tinyproxy.

For this purpose we could also use a free EC2 instance. Just remember to enable the needed ports in the security settings.

Installing tinyproxy is pretty straightforward, at least on Ubuntu. To install just type: apt-get install tinyproxy

Then configure the security settings: nano /etc/tinyproxy.conf

In the tinyproxy.conf file configure the port number (default 8888). TinyProxy does not provide authentication, it is using an allow list instead. So before closing the configuration file make sure your IP address is in the Allow list. Visit www.whatismyip.com to find your IP address. Assuming your IP address was 123.123.123.123 the line should be: Allow 123.123.123.123

You can then start to use your proxy server. To change the IP address of the EC2 instance and consequently the proxy server, simply click STOP on the server and then start it again.

Getting Started

The python code below is crawling two different public proxy websites http://proxyfor.eu/geo.php and http://free-proxy-list.net. After collecting the proxy data and filtering the slowest ones it is randomly selecting one of them to query the target url. The request timeout is configured at 30 seconds and if the proxy fails to return a response it is deleted from the application proxy list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
__author__ = 'pgaref'

import requests
from requests.exceptions import ConnectionError
import random
import os
import time
from bs4 import BeautifulSoup
from requests.exceptions import ReadTimeout

class RequestProxy:
    agent_file = 'user_agents.txt'

    def __init__(self, web_proxy_list=["http://54.207.114.172:3333"]):
        self.useragents = self.load_user_agents(RequestProxy.agent_file)
        #####
        # Proxy format:
        # http://<USERNAME>:<PASSWORD>@<IP-ADDR>:<PORT>
        #####
        self.proxy_list = web_proxy_list
        self.proxy_list += self.proxyForEU_url_parser('http://proxyfor.eu/geo.php', 100.0)
        self.proxy_list += self.freeProxy_url_parser('http://free-proxy-list.net')

Generate Random User Agent Header

I have to mention that for each request a different agent header is used. This headers are strong in the /data/user_agents.txt file which contains around 900 different agents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
    # load agents from file
    def load_user_agents(self, useragentsfile):
        """
        useragentfile : string
            path to text file of user agents, one per line
        """
        useragents = []
        with open(useragentsfile, 'rb') as uaf:
            for ua in uaf.readlines():
                if ua:
                    useragents.append(ua.strip()[1:-1-1])
        random.shuffle(useragents)
        return useragents

    def get_random_user_agent(self):
        """
        useragents : string array of different user agents
        :param useragents:
        :return random agent:
        """
        user_agent = random.choice(self.useragents)
        return user_agent

    def generate_random_request_headers(self):
        headers = {
        "Connection" : "close",  # another way to cover tracks
        "User-Agent" : self.get_random_user_agent()}  # select a random user agent
        return headers

Parse & Filter Public Proxies

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
    # parse proxy data from url link
    def proxyForEU_url_parser(self, web_url, speed_in_KBs=100.0):
        curr_proxy_list = []
        content = requests.get(web_url).content
        soup = BeautifulSoup(content, "html.parser")
        table = soup.find("table", attrs={"class": "proxy_list"})

        # The first tr contains the field names.
        headings = [th.get_text() for th in table.find("tr").find_all("th")]

        datasets = []
        for row in table.find_all("tr")[1:]:
            dataset = zip(headings, (td.get_text() for td in row.find_all("td")))
            datasets.append(dataset)

        for dataset in datasets:
            # Check Field[0] for tags and field[1] for values!
            proxy = "http://"
            proxy_straggler  = False
            for field in dataset:
                # Discard slow proxies! Speed is in KB/s
                if field[0] == 'Speed':
                    if float(field[1]) < speed_in_KBs:
                        proxy_straggler = True
                if field[0] == 'IP':
                    proxy = proxy+field[1]+':'
                elif field[0] == 'Port':
                    proxy = proxy+field[1]
            # Avoid Straggler proxies
            if not proxy_straggler:
                curr_proxy_list.append(proxy.__str__())
            #print "{0:<10}: {1}".format(field[0], field[1])
        #print "ALL: ", curr_proxy_list
        return curr_proxy_list

    def freeProxy_url_parser(self, web_url):
        curr_proxy_list = []
        content = requests.get(web_url).content
        soup = BeautifulSoup(content, "html.parser")
        table = soup.find("table", attrs={"class": "display fpltable"})

        # The first tr contains the field names.
        headings = [th.get_text() for th in table.find("tr").find_all("th")]

        datasets = []
        for row in table.find_all("tr")[1:]:
            dataset = zip(headings, (td.get_text() for td in row.find_all("td")))
            datasets.append(dataset)

        for dataset in datasets:
            # Check Field[0] for tags and field[1] for values!
            proxy = "http://"
            for field in dataset:
                if field[0] == 'IP Address':
                    proxy = proxy+field[1]+':'
                elif field[0] == 'Port':
                    proxy = proxy+field[1]
            curr_proxy_list.append(proxy.__str__())
            #print "{0:<10}: {1}".format(field[0], field[1])
        #print "ALL: ", curr_proxy_list
        return curr_proxy_list

Use Request Proxy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
    # generate a randomised request to the specific url given as argument
    def generate_proxied_request(self, url, params={}, req_timeout=30):
        if len(self.proxy_list) < 2:
            self.proxy_list += self.proxyForEU_url_parser('http://proxyfor.eu/geo.php')

        random.shuffle(self.proxy_list)
        req_headers = dict(params.items() + self.generate_random_request_headers().items())

        request = None
        try:
            rand_proxy = random.choice(self.proxy_list)
            request = requests.get(test_url, proxies={"http": rand_proxy},
                                   headers=req_headers, timeout=req_timeout)
        except ConnectionError:
            self.proxy_list.remove(rand_proxy)
            print "Proxy unreachable - Removed Straggling proxy :", rand_proxy, " PL Size = ",len(self.proxy_list)
            pass
        except ReadTimeout:
            self.proxy_list.remove(rand_proxy)
            print "Read timed out - Removed Straggling proxy :", rand_proxy, " PL Size = ", len(self.proxy_list)
            pass
        return request

Connecting the pieces together

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Main method to test functionality - generating request to a local service
if __name__ == '__main__':

    start = time.time()
    req_proxy = RequestProxy()
    print "Initialisation took: ", (time.time()-start)
    print "Size : ", len(req_proxy.get_proxy_list())
    print " ALL = ", req_proxy.get_proxy_list()

    test_url = 'http://localhost:8888'


    while 1:
        print "here"

        start = time.time()
        request = req_proxy.generate_proxied_request(test_url)
        print "Proxied Request Took: ", (time.time()-start), " => Status: ", request.__str__()
        print "Proxy List Size: ", len(req_proxy.get_proxy_list())

        print"-> Going to sleep.."
        time.sleep(10)

##References

TinyProxy documentation and the TinyProxy repository
Will Drevo, “Using a proxy with a randomized user agent in Python requests” (original post no longer online)
Github Project Link

Cluster management with Ansible

Sun, 02 Aug 2015 00:00:00 -0700

Ansible is a configuration management and provisioning tool, similar to Chef, Puppet or Salt.

I’ve found it to be one of the simplest and the easiest to get started with. A lot of this is because it uses SSH to connect to servers and run the configured Tasks. One nice thing about Ansible is that it’s very easy to convert bash scripts, which is still a popular way to accomplish configuration management, into Ansible Tasks. Since it’s primarily SSH based, it’s not hard to see why this might be the case - Ansible ends up running the same commands. We could just script our own provisioners, but Ansible is much cleaner because it automates the process of getting context before running Tasks. With this context, ansible is able to handle most edge cases which is exactly the kind we usually take care of with longer and increasingly complex scripts. Ansible Tasks are idempotent. Without a lot of extra coding, bash scripts are usually not safe to run again and again. Ansible uses “Facts”, which is system and environment information it gathers “context” before running Tasks. Ansible uses these facts to check state and see if it needs to change anything in order to get the desired outcome. This makes it safe to run Ansible Tasks against a server over and over again.

Here I’ll show how easy it is to get started with Anible. We’ll start basic and then add in more features as we improve upon our configurations.

You can find a subset of my personal collection of ansible playbooks and roles for managing a cluster on github.

Installation

Of course we need to start by installing Ansible. Tasks can be run off of any machine Ansible is installed on.

This means there’s usually a “central” server running Ansible commands, although there’s nothing particularly special about what server Ansible is installed on. Ansible is “agentless” - there’s no central agent(s) running. We can even run Ansible from any server; I often run Tasks from my laptop.

Here’s how to install Ansible on Ubuntu 14.04. We’ll use the easy-to-remember ppa:ansible/ansible repository as per the official docs.

sudo apt-add-repository -y ppa:ansible/ansible
sudo apt-get update
sudo apt-get install -y ansible

Host Management

Ansible has a default inventory file used to define which servers it will be managing. After installation, there’s an example one you can reference at /etc/ansible/hosts.

I usually copy and move the default one so I can reference it later: ** sudo cp /etc/ansible/hosts /etc/ansible/hosts.orig **

Now lets create our own hosts inventory.

# A collection of database servers in the 'dbservers' group

[dbservers]
db[01:08].example.com

[localserver]
localhost

Running basic commands

Once we have an inventory configured, we can start running Tasks against the defined servers.

Ansible will assume you have SSH access available to your servers, usually based on SSH-Key. Because Ansible uses SSH, the server it’s on needs to be able to SSH into the inventory servers. It will attempt to connect as the current user it is being run as. If I’m running Ansible as user admin, it will attempt to connect as user admin on the other servers.

If Ansible can directly SSH into the managed servers, we can run commands without too much hastle:

ansible all -m ping -k -u admin

127.0.0.1 | success >> {
    "changed": false,
    "ping": "pong"
}

We can see the output we get from Ansible is some JSON which tells us if the Task made any changes and the result.

If we need to define the user and perhaps some other settings in order to connect to our server, we can. When testing locally, I use the following: ansible localserver -m ping -s -k -u admin

PS: Remember to add all managed host key to the known_hosts file (or just ssh to every single one of them before running ansible).

##About the command options

all - Use all defined servers from the inventory file
-m ping - Use the "ping" module, which simply runs the ping command and returns the results
-s - Use "sudo" to run the commands
-k - Ask for a password rather than use key-based authentication
-u vagrant - Log into servers using user vagrant

##Ansible Modules

Ansible uses “modules” to accomplish most of its Tasks. Modules can do things like install software, copy files, use templates and much more.

Modules are the way to use Ansible, as they can use available context (“Facts”) in order to determine what actions, if any need to be done to accomplish a Task.

If we didn’t have modules, we’d be left running arbitrary shell commands like this: ansible all -s -m shell -a ‘apt-get install nginx’

Here, the sudo apt-get install nginx command will be run using the “shell” module. The -a flag is used to pass any arguments to the module. I use -s to run this command using sudo.

However this isn’t particularly powerful. While it’s handy to be able to run these commands on all of our servers at once, we still only accomplish what any bash script might do.

If we used a more appropriate module instead, we can run commands with an assurance of the result. Ansible modules ensure indempotence - we can run the same Tasks over and over without affecting the final result.

For installing software on Debian/Ubuntu servers, the “apt” module will run the same command, but ensure idempotence.

ansible all -s -m apt -a ‘pkg=nginx state=installed update_cache=true’ 127.0.0.1 | success » { “changed”: false }

##Ansible Playbook

Playbooks can run multiple Tasks and provide some more advanced functionality that we would miss out on using ad-hoc commands. Let’s move the above Task into a playbook.

Playbooks and Roles in Ansible all use Yaml.

nginx.yml

- hosts: localserver
  tasks:
   - name: Install Nginx
     apt: pkg=nginx state=installed update_cache=true

This Task does exactly the same as our ad-hoc command, however I chose to specify my “local” group of servers rather than “all”. We can run it with the ansible-playbook command: ansible-playbook -s -k nginx.yml

#Ansible Store encrypted Pass

ansible-vault create secret

In the file insert: ansible_sudo_pass: mysudopassword

Create a vault.txt file with the password decrypting the pass (make sure to use the right permissions)

You could also use something like that: [whatever] some-host ansible_sudo_pass=’foobar’ in the hosts file which is not secure!

Then our module will look something like that:

- hosts: localserver
  vars_files:
    - secret
#  sudo: true
#  sudo_user: admin
#  remote_user: admin
  tasks:
   - name: update apt cache
     apt: update_cache=yes
#   - name: Install Nginx
#     apt: pkg=nginx state=removed update_cache=true

To run the module use:

ansible-playbook -s --vault-password-file=vault.txt -vvv ngix-test.yml

##Ansible Handlers

A Handler is exactly the same as a Task (it can do anything a Task can), but it will run when called by another Task. You can think of it as part of an Event system; A Handler will take an action when called by an event it listens for.

This is useful for “secondary” actions that might be required after running a Task, such as starting a new service after installation or reloading a service after a configuration change.

---
- hosts: localserver
  tasks:
   - name: Install Nginx
     apt: pkg=nginx state=installed update_cache=true
     notify:
      - Start Nginx

  handlers:
   - name: Start Nginx
     service: name=nginx state=started

We can add a notify directive to the installation Task. This notifies any Handler named “Start Nginx” after the Task is run.

Then we can create the Handler called “Start Nginx”. This Handler is the Task called when “Start Nginx” is notified.

This particular Handler uses the Service module, which can start, stop, restart, reload (and so on) system services. Here we simply tell Ansible that we want Nginx to be started.

Note that Ansible has us define the state you wish the service to be in, rather than defining the change you want. Ansible will decide if a change is needed, we just tell it the desired result.

##Ansible Web interface - Semaphore

Tower is ansible’s commercial interface with really nice functionality but for those looking for an open-source solution semaphore is an ellegent solution.

For installation you could use docker: ### Run redis

    docker run -d \
        --name=redisio \
        redis

### Run mongodb

    docker run -d \
        --name=mongodb \
        -p 127.0.0.1:27017:27017 \
        mongo

### Run semaphore

        docker run -d \
        --name=semaphore \
        --restart=always \
        --link redisio:redis \
        --link mongodb:mongo \
        -e MONGODB_URL="mongodb://mongo/semaphore" \
        -e REDIS_HOST="redis" \
        -p 80:80 \
         castawaylabs/semaphore

### Web interface credentials

        Email:          'admin@semaphore.local'
        Password:       'CastawayLabs'

Customising Semaphore Credentials

    mongo
    show dbs
    use semaphore
    db.getCollection("users").find()
    db.users.update({"_id" : ObjectId("555f52cc5e3ce20100b914ff") } , { $set: { 'email': 'panagiotis@example.com' }, } )

Ansible Ad-hoc Commands

If you want to install a package or run a command just once there is a more convenient way than creating a new playbook. You can use ansible ad-hock commands (also described above) like:

ansible local -m apt -a "name=libssl-dev state=present" -u admin -k --sudo
- Ensure a package is installed but dont update it
anbible databases -s -m apt -a “name=libffi-dev state=present” -k -u admin
- Deploy a package to the database cluster
ansible database -m shell -a 'sudo apt-get update; sudo pip install -U pip virtualenv; echo All Done!' -k -u admin --sudo
- Run multiple shell commands

If you have an error mentioning you dont have enough permissions to run a command at a host:

Make sure in /etc/ansible/ansible.cfg there is a commented out setting called ask_sudo_pass, which needs to be uncommented and set to True.

Create a user

If you want to create a user with ad-hoc coommands you need to generate an encrypted password for your user: mkpasswd --method=SHA-512

Then use the command:

ansible localserver -m user -a 'name=exampleuser shell=/bin/bash comment="Example User" password=XXXX' -k -u admin --sudo

##References

Nginx web server configuration with SSL

Sun, 14 Jun 2015 00:00:00 -0700

About

Nginx is an HTTP server as well as a Proxy (IMAP/POP3 and reverse) written to address the C10K problem and handle more efficiently client requests. Nginx doesn’t rely on threads to handle requests. Instead it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly, predictable amounts of memory under load.

##Installation

sudo apt-get install nginx
ifconfig eth0 | grep inet | awk '{ print $2 }'
sudo service nginx start
update-rc.d nginx defaults

General conf: All Nginx configuration files are located in the /etc/nginx/ directory. The primary configuration file is /etc/nginx/nginx.conf.

##Serving

You can find the default nginx serving directory under:

/srv/www/example.com/public_html/index.html

Using Nginx as a Proxy

    server {
	# Catch all the requests comming to db01.example.com"
    server_name db01.example.com "";
	listen 109.200.XX.XXX:4040;

	gzip             on;
    	gzip_min_length  2000;
    	gzip_proxied     any;
    	gzip_types       text/html application/json;

    	proxy_read_timeout 60s;
    	client_max_body_size 64M;


	
	    ################################
		# Panagiotis Auth Config Rules #
		################################
		allow 109.200.XX.XXX; #example.com
		deny all;

    	# try serving static files directly
    	location ~ /\+f/ {
        	error_page 418 = @proxy_to_app;
        	if ($request_method != GET) {
            	return 418;
        	}
        	expires max;
        	try_files /+files$uri @proxy_to_app;
    	}
    	# try serving docs directly
    	location ~ /\+doc/ {
        	try_files $uri @proxy_to_app;
    	}
    	location / {
        	error_page 418 = @proxy_to_app;
        	return 418;
    	}
        location @proxy_to_app {
		

                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.

		# try_files $uri $uri/ /index.html;
		proxy_pass http://localhost:4040;

                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules

		proxy_set_header X-outside-url $scheme://$host:$server_port;
        	proxy_set_header X-Real-IP $remote_addr;
        	expires -1;  # no-cache
        }

	#location / {
        #	proxy_pass http://localhost:8080/;
    	#}
    }

##Enable directory Listing

Enabling directory listing in a folder in nginx is simple enough with just an autoindex on;directive inside the location directive. You can also enable sitewide directory listing by putting it in the server block or even enable directory access for all sites by putting it in the http block.

server {
	server_name db01.example.com "";
	listen 4141;
	gzip             on;
    	gzip_min_length  2000;
    	gzip_proxied     any;
    	gzip_types       text/html application/json;

    	proxy_read_timeout 60s;
    	client_max_body_size 64M;

	#Path to requirement files
        root   /usr/share/nginx/devpi-requirements/;

	################################
	# panagiotis Auth Config Rules #
	################################
	allow all;

	location / {
               #index  index.php index.html index.htm;
	       try_files $uri $uri/ /index.html index.php;
	       autoindex on;
        }
        #location /somedir {
        #       autoindex on;
        #}
    }

##How To Create a self-signed SSL Certificate on Nginx

TLS, or transport layer security, and its predecessor SSL, which stands for secure sockets layer, are web protocols used to wrap normal traffic in a protected, encrypted wrapper. A self-signed certificate will not validate the identity of your server for your users since it is not signed by one of their web browser’s trusted certificate authorities, but it will allow you to encrypt communications with your web clients. We can start off by creating a directory that will be used to hold all of our SSL information. We should create this under the Nginx configuration directory:

sudo mkdir /etc/nginx/ssl

Now that we have a location to place our files, we can create the SSL key and certificate files in one motion by typing:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

The command output will look something like this:

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:New York City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Bouncy Castles, Inc.
Organizational Unit Name (eg, section) []:Ministry of Water Slides
Common Name (e.g. server FQDN or YOUR name) []:your_domain.com
Email Address []:admin@your_domain.com
Both of the files you created will be placed in the /etc/nginx/ssl directory.

##Adding extra nginx configuration

The only thing we would need to do to get SSL working on this same server block, while still allowing regular HTTP connections, is add a these lines in the nginx configuration file:

listen 443 ssl;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
sudo service nginx restart

This should reload your site configuration, now allowing it to respond to both HTTP and HTTPS (SSL) requests.

Your site should now have SSL functionality. To test both HTTP and HTTPS functionality open a web browser and visit your server’s domain name or IP address:

http://server_domain_or_IP
https://server_domain_or_IP
https://server_domain_or_IP:443

##References

https://www.akamai.com/cloud/guides/how-to-configure-nginx

https://www.digitalocean.com/community/tutorials/how-to-create-an-ssl-certificate-on-nginx-for-ubuntu-14-04

Preventing brute force attacks with fail2ban

Wed, 27 May 2015 00:00:00 -0700

Introduction

Fail2Ban is an intrusion prevention framework written in the Python programming language. It works by reading SSH, ProFTP, Apache etc. logs and uses iptables profiles to block brute-force attempts.

The reason I prefer fail2ban over other tools is because it is one of the simplest solutions, it is surprisingly effective and there is an active community behind it.

How it works

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc. Afterwards it is updating firewall rules to reject malicious IP addresses for a specified amount of time, although any other arbitrary action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc). Fail2Ban is able to reduce the rate of incorrect authentications attempts however it cannot eliminate the risk that weak authentication presents. Configure services to use only two factor or public/private authentication mechanisms if you really want to protect services.

Installation & Configuration

To install fail2ban, type I followed the procedure below:

sudo apt-get update
sudo apt-get install fail2ban

Configuration: To configure fail2ban, make a ‘local’ copy the jail.conf file in /etc/fail2ban

cd /etc/fail2ban
sudo cp jail.conf jail.local

Now edit the file:

sudo nano jail.local

Set the IPs you want fail2ban to ignore, the ban time (in seconds) and maximum number of user attempts to match your needs:

[DEFAULT]
# "ignoreip" can be an IP address, a CIDR mask or a DNS host
ignoreip = 127.0.0.1
bantime  = 3600
maxretry = 3

Setting up an MTA

For an administrator email notifications are essential. For that, you will need sendmail or any other working MTA to forward emails to your address. If you have a working MTA already, you can skip this step. To install an All send-only MTA - replace sendmail with ssmtp:

sudo apt-get install sendmail
sudo apt-get install mailutils ssmtp
sudo nano /etc/ssmtp/ssmtp.conf
ssmtp panagiotis@example.com
hello world!
sudo service sendmail stop
sudo killall sendmail
which ssmtp
sudo rm -rf /usr/sbin/sendmail
sudo ln -s /usr/sbin/ssmtp /usr/sbin/sendmail
sudo service fail2ban restart

If you wish to get bans notifications by email, modify this line with your email address:

destemail = your_email@domain.com

Then find the line:

action = %(action_)s

and change it to:

action = %(action_mw)s

Jail Configuration

Jails are the rules which fail2ban is going to apply to a given application/log:

[ssh]

enabled = true
port    = ssh
filter  = sshd
logpath  = /var/log/auth.log
maxretry = 3

To enable the other profiles, such as [ssh-ddos], make sure the first line beneath the specific profile is:

enabled = true

After that, restart fail2ban to put those settings into effect:

sudo /etc/init.d/fail2ban restart

Advanced: Filters

If you wish to tweak or add log filters, you can find them under /etc/fail2ban/filter.d

Testing

To test fail2ban, look at iptable rules:

sudo iptables -L

Attempt to login to a service that fail2ban is monitoring (preferably from another machine) and look at the iptable rules again to see if that IP source gets added.

Fail2ban is really handy for blocking brute force attacks against your servers but when you have a cluster of servers, it would be quite useful to include the hostname in the subject line of the email you receive when an IP address is blocked. Since fail2ban is not doing that out of the box I followed the simple procedure below to extend the functionality to my needs.

Extending Fail2ban to include the hostname in the mail subject.

There are three steps:

Step 1: Define a variable “host” in you jail.conf, assign the hostname to it and pass it to your action.

Open up /etc/fail2ban/jail.conf and edit the action. In my case the action for dovecot-pop3imap is:

action_mw = %(banaction)s[name=%(__name__)s, 
            port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
              %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", 
              protocol="%(protocol)s", chain="%(chain)s"]

Now add the variable “host”, so the action looks like:

action_mw = %(banaction)s[name=%(__name__)s, 
            port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
              %(mta)s-whois[name=%(__name__)s, 
              dest="%(destemail)s", protocol="%(protocol)s",
              chain="%(chain)s", host=db01.example.com]

Step 2: Use the value in your action.d script.

You can see in my example that the action being called is sendmail-whois. So next open /etc/fail2ban/action.d/sendmail-whois.conf. There are various sections here that send emails. They are “actionstart”, “actionstop” and “actionban”.

My actionstart for example looks like this:

actionstart = printf %%b "Subject: [Fail2Ban] <name>: started on <host>
              From: Fail2Ban <<sender>>
              To: <dest>\n
              Hi,\n
              The jail <name> has been started successfully.\n
              Regards,\n
              Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>

Make similar amendments to the Subject of each section and save the file.

Step 3: Restart fail2ban

For the changes to take effect, you need to restart fail2ban:

service fail2ban restart

Unban an IP

If you are trying to figure out what happens if an IP gets banned by mistake, here is the solution:

$ sudo iptables -L -n  > bans.log
$ sudo fail2ban-client status # to reveal your JAIL name (mine is ssh)
$ sudo fail2ban-client set ssh delignoreip your_ip_address
$ sudo nano /etc/hosts.deny # delete your ip address
$ sudo fail2ban-client reload

Cassandra cluster Rolling upgrade

Sun, 03 May 2015 00:00:00 -0700

The time has come and a new major release of our Cassandra DB is out there, with fancy new features and important patches. The challenge in a running cluster with a number of clients is always the downtime. That’s the reason we always want to spent the minimum time possible with our nodes unavailable and with the correct keyspace replication it is achievable.

Since Datastax documentation is mostly outdated and there is no single document with a complete guide here is the procedure I recently followed to upgrade DSE from version 4.6.2 to 4.6.5.

First, make sure you have an up-to-date daily backup. I prefer OpsCenter incremental backup method but there are other options as well like sstable2json. Then the second step is to upgrade the nodes one-by-one maintaining cluster availability while updating version. Finally, we have to make sure to upgrade all the version nodes. Different cassandra versions in the same cluster can lead to weird incompatibilities.

Step 1

Run node drain which flushes all memtables from the node to SSTables on disk. Cassandra stops listening for connections from the client and other nodes. You typically use this command before upgrading a node to a new version of Cassandra.

nodetool drain -h hostname -u username -pw password

Step 2

Stop the running services, in my case:

dse service
datastax-agent

sudo service dse stop
sudo service datastax-agent stop

Step 3

Keep cassandra, or any other valuable configuration in a safe directory. Remember “Precaution is better than cure”!

cp /etc/dse/cassandra/cassandra.yaml ./cass-upgrade-conf/
cp /etc/dse/cassandra/cassandra-env.sh ./cass-upgrade-conf/

Step 4

Update the package manager repositories and upgrade.

sudo apt-get update
sudo apt-get install dse-full

This can take a while… PS: If you are running Cassandra from source you should download the latest tar.gz instead of using the package manager.

Step 5

If you want to restore any configuration files that you previously backup-up now it’s the appropriate time.

Step 6

Start the services.

sudo service dse restart
sudo service datastax-agent start
dse -v

Always check the logs for any error messages.

tail -f /var/log/cassandra/system.log

If you are changing major version

When you are upgrading from a major version (for example, from 1.2 to 2.0) or a major point release (for example, from Cassandra 2.0 to 2.1), upgrade the sstables on each node.

nodetool upgradesstables

Upgrading OpsCenter

Be sure that OpsCenter is compatible with your version of DataStax Enterprise or Cassandra.

Download and extract the new tarball.

1
curl -L http://downloads.datastax.com/community/opscenter.tar.gz | tar xz

Copy the following files and directories from the old tarball installation directory to the new one.

1
2
3
4
5
6
7
8
conf/clusters/*
conf/event-plugins/*
conf/install_id
conf/log4j.properties
conf/opscenterd.conf
./passwd.db
conf/ssl.conf
ssl/*

Stop the opscenterd instance (if it is running) and start it from the new tarball installation directory.

Extra useful configuration

Time synchronisation between nodes

1
2
3
4
5
date
sudo apt-get install ntp ntpdate
sudo service ntp stop
sudo ntpd -gq
sudo service ntp start

####Recommended Production Settings

1
2
3
4
5
6
7
8
9
10
'echo' 0 > /proc/sys/vm/zone_reclaim_mode
'ulimit' -a
/etc/security/limits.d/cassandra.conf
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited
/etc/sysctl.conf --> vm.max_map_count = 131072 --> sudo sysctl -p
cat /proc/<cassandra pid>/limits
sudo swapoff --all

Getting started with Cgroups

Tue, 07 Apr 2015 00:00:00 -0700

Managing a single process resources running on top of an operating system is not a trivial task. While solutions like Docker or Virtual Machines could completely isolate your process, I was recently looking for a simpler and more straightforward solution at operating system level.

Then I discovered cgroups which is a kernel tool to limit resource usage per user-defined groups. The resources I was mostly interested in are CPU and RAM, but there is also an option to limit the IO usage of a process. To run a new process with limited resources you basically have to remember two things:

Define the cgroup resource limits
Run your process within the croup like:

  sudo cgexec -g cpu:root -g memory:root COMMAND

In my ubuntu server I used the stress tool to generate some load in the machine so I could actually check if the resource limits being applied.

Here are the steps I followed for the cgroups installation:

Install cgoup-bin

sudo apt-get install cgroup-bin

##Create a custom group for your process

mount {
       cpu = /cgroup/cpu;
       cpuacct = /cgroup/cpu;
       memory = /cgroup/memory;
}

group dataparser {
        # Set the cpu and memory limits for this group
        cpu {
                cpu.shares = 500;
        }
        memory {
                memory.limit_in_bytes = 40G;
        }
}

Setting cgroup limits

sudo mkdir /cgroup/cpu/dataparser
sudo mkdir /cgroup/memory/dataparser
#Limit CPU usage to 50%
echo 500 > /cgroup/cpu/dataparser/cpu.shares
# Limit RAM to 40G
echo 40000000000 > /cgroup/memory/dataparser/memory.limit_in_bytes

Mount the cgroups

sudo mount -t cgroup -o cpu,cpuacct cpu /cgroup/cpu/
sudo mount -t cgroup -o memory memory /cgroup/memory/

Create the cgroup with the resource limits

sudo cgcreate -a dataparser:dataparser -t dataparser:dataparser -g cpu:dataparser

Test limits using stress tool

At that point the CPU and RAM usage of you process should never exceed the limits you defined! I found stress tool really useful in this part.

sudo cgexec -g cpu:dataparser -g memory:dataparser stress --cpu 20 --vm 50 --vm-bytes 10G

Extra commands

list mounted cgroups

lscgroup

deleted all cgroups (use with caution)

sudo cgclear

load configuration file

sudo cgconfigparser -l /etc/cgconfig.conf

Of course you could define which users can run processes within each cgroup and also which users can modify resource limits. These options are mostly useful in a multiuser environment with multiple croups defining different resource limits. You can find more information here. I also have to mention here that Docker is using the same trick with croups inside containers to manage the recourses.

Panagiotis (Panos) Garefalakis

Proxying Python Requests

Introduction

TinyProxy Installation

Getting Started

Generate Random User Agent Header

Parse & Filter Public Proxies

Use Request Proxy

Connecting the pieces together

Cluster management with Ansible

Installation

Host Management

Running basic commands

Customising Semaphore Credentials

Ansible Ad-hoc Commands

Ensure a package is installed but dont update it

Deploy a package to the database cluster

Run multiple shell commands

Create a user

Nginx web server configuration with SSL

About

Using Nginx as a Proxy

Preventing brute force attacks with fail2ban

Introduction

How it works

Installation & Configuration

Setting up an MTA

Jail Configuration

Advanced: Filters

Testing

Extending Fail2ban to include the hostname in the mail subject.

Unban an IP

Cassandra cluster Rolling upgrade

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

If you are changing major version

Upgrading OpsCenter

Extra useful configuration

Time synchronisation between nodes

Getting started with Cgroups

Install cgoup-bin

Setting cgroup limits

Mount the cgroups

Create the cgroup with the resource limits

Test limits using stress tool

Extra commands