Firstly I installed iptables to block them out:
iptables -A INPUT -s %SOME-IP-ADDRESS% -j DROP
After doing this, to save it, you need to run
service iptables save
Later, I want to find some way to automatically block them out, I found fail2ban.
After simply install fail2ban, you need to create a file called jail.local to put settings in:
ignoreip = 127.0.0.1/8 %SOME MORE IP IF YOU LIKE%
bantime = 3600
enabled = true
port = http,https
logpath = %(apache_access_log)s
maxretry = 5
This means if someone tries for 5 times 404 error, there IP address will be banned for 1 hour.
You also need to put a file called apache-script-bots.conf in filter.d folder:
failregex = ^
.* 404 .*$
This setting tell fail2ban to check Apache access log to find all the 404 errors.