Web Server log analysis ( Apache log analyzer )

If you just want to see which url is usually called by which devices or browser then just use goaccess.

For example:

apt-get install goaccess

goaccess  -f /var/log/apache2/vindazo_be_access.log

This way you can discover suspicious requests. Like

[17/Feb/2021:07:01:05 +0100] "GET /job/?q=&l= HTTP/1.0" 200 975813 "-" "ApacheBench/2.3"

In this way you can analyze traffic.

Analyze log with python to process further results in application.

python3 -m pip install apachelogs

 

>>> from apachelogs import LogParser

>>> parser = LogParser("%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"")

>>> # The above log format is also available as the constant `apachelogs.COMBINED`.

>>> entry = parser.parse('209.126.136.4 - - [01/Nov/2017:07:28:29 +0000] "GET / HTTP/1.1" 301 521 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"\n')

>>> entry.remote_host

'209.126.136.4'

>>> entry.request_time

datetime.datetime(2017, 11, 1, 7, 28, 29, tzinfo=datetime.timezone.utc)

>>> entry.request_line

'GET / HTTP/1.1'

>>> entry.final_status

301