Linux Server Hardening Guides: IDS and IPS with Fail2ban


Intrusion Detection and Prevention System or IDS & IPS used for hardening security purpose to make our machine more secure and reliable, by implementing this technique we can decrease few risk that can effecting our properties.

Avatar

Zuhri

  |  7 min reads

Introduction #

The define of IDS & IPS is, it just a Guardian! its just protecting our house from the unauthorized but refers to wiki here what they say.

An intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any intrusion activity or violation is typically either reported to an administrator or collected centrally using a security information and event management (SIEM) system. A SIEM system combines outputs from multiple sources and uses alarm filtering techniques to distinguish malicious activity from false alarms.

Some IDS products have the ability to respond to detected intrusions. Systems with response capabilities are typically referred to as an intrusion prevention system (IPS). Intrusion detection systems can also serve specific purposes by augmenting them with custom tools, such as using a honeypot to attract and characterize malicious traffic.

~ Wikipedia

Then for our case the Guardian is called with Fail2ban. Fail2ban just tools with banning technique for who failed to gain access in multiple times, Fail2ban will banned the failed IP address by ruleset using iptables.

Installation #

As usual i will using debian based system, because its easy and old boring stable distro.

apt update
apt install fail2ban rsyslog iptables -y

After done, apt will create by default /etc/fail2ban directory you can create changes on that.

[ root@server: ~ ]
└# ls -l /etc/fail2ban/
total 64
drwxr-xr-x 2 root root  4096 May 24 12:13 action.d
-rw-r--r-- 1 root root  2816 Nov 24  2020 fail2ban.conf
drwxr-xr-x 2 root root  4096 Jul 12  2021 fail2ban.d
drwxr-xr-x 3 root root  4096 May 24 12:13 filter.d
-rw-r--r-- 1 root root 24996 Nov 24  2020 jail.conf
drwxr-xr-x 2 root root  4096 May 24 12:13 jail.d
-rw-r--r-- 1 root root   645 Nov 24  2020 paths-arch.conf
-rw-r--r-- 1 root root  2827 Nov 24  2020 paths-common.conf
-rw-r--r-- 1 root root   573 Nov 24  2020 paths-debian.conf
-rw-r--r-- 1 root root   738 Nov 24  2020 paths-opensuse.conf

Fail2Ban supports brute-force protection for other software, like Apache and ProFTPD. Pre-defined software filters can be found in the /etc/fail2ban/filter.d/ directory. You can enable/disable these by adding additional [software] filter blocks to the /etc/fail2ban/jail.conf file, using the filters file names without file extension. Properties not defined in a specific filter block, are taken from the [DEFAULT] block.

By default debian has make default conf to fail2ban conf in /etc/fail2ban/jail.d/ but am not using openssh or sshd. in our case we will using dropbear. so just put # on that.

[ root@server: ~ ]
└# cat /etc/fail2ban/jail.d/defaults-debian.conf
#[sshd]
#enabled = true

And for example, here i try to enable for ssh services but its dropbear, you can make it in /etc/fail2ban/jail.conf. Search for dropbear then put enabled = true to make it works.

[dropbear]

enabled = true				
port     = ssh
logpath  = %(dropbear_log)s
backend  = %(dropbear_backend)s

Then restart fail2ban services with systemctl restart fail2ban, and see the output below with systemctl status fail2ban!

[ root@server: ~ ]
└# systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2025-05-24 13:14:25 WIB; 8s ago
       Docs: man:fail2ban(1)
    Process: 1097 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
   Main PID: 1100 (fail2ban-server)
      Tasks: 5 (limit: 4060)
     Memory: 10.0M
        CPU: 639ms
     CGroup: /system.slice/fail2ban.service
             └─1100 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

May 24 13:14:25 server fail2ban-server[1100]: fail2ban.jail           [1100]: INFO    Creating new jail 'dropbear'
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.jail           [1100]: INFO    Jail 'dropbear' uses poller {}
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.jail           [1100]: INFO    Initiated 'polling' backend
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO      maxRetry: 5
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO      findtime: 600
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.actions        [1100]: INFO      banTime: 600
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO      encoding: UTF-8
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    Added logfile: '/var/log/auth.log' (pos = 0, hash = ac51cbb52dbf1945a61bd9f56900ce5fa4ce0c98)
May 24 13:14:25 server fail2ban-server[1100]: fail2ban.jail           [1100]: INFO    Jail 'dropbear' started
May 24 13:14:25 server fail2ban-server[1100]: Server ready

If the output show like that fail2ban success to apply configuration changes.

Proof of Concept #

Its time for testing, in this scenario i will act as dumb man who wanna try to access someone house (server). In systemctl status fail2ban output it say max try is 5 and if you see on /etc/fail2ban/jail.conf the bantime = 10m it mean when we failed to own it we will got banned for 10 minute.

[ root@server: ~ ]
└# w
 13:28:43 up 31 min,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/1    192.168.1.210    12:57    2.00s  2.04s  0.02s w

[ root@server: ~ ]
└# fail2ban-client status dropbear
Status for the jail: dropbear
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	0
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	0
   |- Total banned:	0
   `- Banned IP list:	

Right now in our server its just 1 client connected (My PC), then the ban list it say 0.

Then i will using my router with wrong password to login.

root@OpenWrt:~# ssh root@192.168.1.2

Host '192.168.1.2' is not in the trusted hosts file.
(ssh-ed25519 fingerprint SHA256:R63UHN6g/Xih0gI3DNooi+wuoa8YrSiY8OOoJNxXvQk)
Do you want to continue connecting? (y/n) y
root@192.168.1.2's password:
root@192.168.1.2's password:
root@192.168.1.2's password:
root@192.168.1.2's password:
froot@192.168.1.2's password:
root@192.168.1.2's password:
froot@192.168.1.2's password:
froot@192.168.1.2's password:
froot@192.168.1.2's password:
root@192.168.1.2's password:

ssh: Connection to root@192.168.1.2:22 exited: Remote closed the connection

This fail2ban output when i’ll try with wrong password.

[ root@server: ~ ]
└# fail2ban-server status
Status
|- Number of jail:	1
`- Jail list:	dropbear

[ root@server: ~ ]
└# fail2ban-client status dropbear
Status for the jail: dropbear
|- Filter
|  |- Currently failed:	1
|  |- Total failed:	11
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	1
   |- Total banned:	1
   `- Banned IP list:	192.168.1.1

And this systemctl output.

[ root@server: ~ ]
└# systemctl status fail2ban.service
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2025-05-24 13:14:25 WIB; 25min ago
       Docs: man:fail2ban(1)
   Main PID: 1100 (fail2ban-server)
      Tasks: 5 (limit: 4060)
     Memory: 10.5M
        CPU: 4.833s
     CGroup: /system.slice/fail2ban.service
             └─1100 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

May 24 13:36:49 server fail2ban-server[1100]: fail2ban.utils          [1100]: ERROR   7fa1393930 -- returned 127
May 24 13:36:49 server fail2ban-server[1100]: fail2ban.utils          [1100]: INFO    HINT on 127: "Command not found".  Make sure that all commands in 'iptables -w -N f2b-dropbear\niptables -w -A f2b-dropbear -j RETURN\niptables -w -I INPUT -p tcp -m multiport --dports ssh -j f2b-dropbear' are in the PATH of fail2ban-server process (grep -a PATH= /proc/`pidof -x fail2ban-server`/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
May 24 13:36:49 server fail2ban-server[1100]: fail2ban.actions        [1100]: ERROR   Failed to execute ban jail 'dropbear' action 'iptables-multiport' info 'ActionInfo({'ip': '192.168.1.1', 'family': 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x7fa13c9e50>, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x7fa13c8550>})': Error starting action Jail('dropbear')/iptables-multiport: 'Script error'
May 24 13:36:49 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:49
May 24 13:36:50 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:50
May 24 13:36:51 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:50
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:51
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:52
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:52
May 24 13:36:53 server fail2ban-server[1100]: fail2ban.actions        [1100]: NOTICE  [dropbear] 192.168.1.1 already banned

And this journal -xe output.

May 24 13:36:49 server dropbear[2494]: Bad password attempt for 'root' from 192.168.1.1:51630
May 24 13:36:49 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:49
May 24 13:36:50 server dropbear[2494]: Bad password attempt for 'root' from 192.168.1.1:51630
May 24 13:36:50 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:50
May 24 13:36:50 server dropbear[2494]: Bad password attempt for 'root' from 192.168.1.1:51630
May 24 13:36:51 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:50
May 24 13:36:51 server dropbear[2494]: Bad password attempt for 'root' from 192.168.1.1:51630
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:51
May 24 13:36:52 server dropbear[2494]: Bad password attempt for 'root' from 192.168.1.1:51630
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:52
May 24 13:36:52 server dropbear[2494]: Exit before auth from <192.168.1.1:51630>: (user 'root', 10 fails): Max auth tries reached - user 'root'
May 24 13:36:52 server fail2ban-server[1100]: fail2ban.filter         [1100]: INFO    [dropbear] Found 192.168.1.1 - 2025-05-24 13:36:52
May 24 13:36:53 server fail2ban-server[1100]: fail2ban.actions        [1100]: NOTICE  [dropbear] 192.168.1.1 already banned

When i try to access, it say.

root@OpenWrt:~# ssh root@192.168.1.2

ssh: Connection to root@192.168.1.2:22 exited: Connect failed: Connection refused
root@OpenWrt:~# ssh root@192.168.1.2

ssh: Connection to root@192.168.1.2:22 exited: Connect failed: Connection refused
root@OpenWrt:~#

The end #

The conclusion is, dont be dumbass.