logging to logstash from python from scratch in Ubuntu

Step 1: Install few required packages:

Install Nginx

sudo apt-get install nginx

Import GPG key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

This step outputs OK

install transport-https package

sudo apt-get install apt-transport-https

Add elastic repo into system’s repo list

echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list

update repo

sudo apt-get update

Step 2: Install ElasticSearch:

Install elasticsearch package

sudo apt-get install elasticsearch

Enable auto restart of elastic search when the machine reloads

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

start elastic search

sudo systemctl start elasticsearch.service

Note: Do not run this step, this is just to make note of

#sudo systemctl stop elasticsearch.service

restart elastic search ( Whenever any change to /etc/elasticsearch/elasticsearch.yml)

sudo systemctl restart elasticsearch.service

check status of elastic search

service elasticsearch status

Status should show active(running)

Step 3: Elastic Search config

edit elastic search config file

sudo vim /etc/elasticsearch/elasticsearch.yml

change the below values in the config file to allow remote access:

transport.host: localhost
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0

After these changes restart elastic search

Test elastic search:

Either we can test it using the below command or run http://server-ip:9200 in browser

curl localhost:9200

returns something like

{
  "name" : "ip-192-2-1-94",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "UtLYRoRtQjSlMhUcfNno2A",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Step 4: Install Kibana

Install kibana package

sudo apt-get install kibana

configure kibana

sudo vi /etc/kibana/kibana.yml

change the below keys in the config

server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: [“http://localhost:9200”]

start kibana

sudo systemctl start kibana

enable kibana start on server restart

sudo systemctl enable kibana

test kibana:

open http://server-ip:5601 in browser

Step 5: Install Logstash

install logstash

sudo apt-get install logstash

configure logstash to get logs via tcp ( which is sent by logging python )

We can configure the flow within logstash i.e. to accept logs, filter logs and output logs

This is not to be confused with logstash.yml file which has logstash configuration

create a flow config file

sudo nano /etc/logstash/logstash.conf

Paste the below config

input { 
    tcp{
        port => 5959
    }
 }
output{
 elasticsearch{
 hosts => ["localhost:9200"]
 index => "ctc"
 }
stdout{codec => rubydebug}
}

Run the below command to set the flow config to logstash

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.conf

enable auto restart and start service

sudo systemctl enable logstash.service
sudo systemctl start logstash

Step 6: Enable Authentication

elastic search plugin called x-pack is by default installed with elastic search.

enable security in elasticsearch config

xpack.security.enabled: true

restart elasticsearch after the change using

sudo systemctl restart elasticsearch

There will be built in users by default which has no password set yet

We must set the passwords to use authentication

set passwords for the modules in ELK stack using

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

set elasticsearch username and password in kibana config for it to access

change kibana config

sudo vi /etc/kibana/kibana.yml

uncomment and change the below lines

elasticsearch.username: "kibana_system"
elasticsearch.password: "Password"

restart kibana using

sudo systemctl restart kibana

Now elasticsearch, kibana console asks for authentication

Now we make logstash accessible to users without auth, But in the backend, logstash has to access elasticsearch using credentials

So First we stop the running instance of logstash using

sudo systemctl stop logstash

Then we make the changes to our flow config as below (where we add user and password parameters)

sudo vi /etc/logstash/logstash.conf

Replace the current config with the following config

input { 
    tcp{
        port => 5959
    }
 }
output{
 elasticsearch{
 hosts => ["localhost:9200"]
 index => "ctc"
 user => "elastic"
 password => "Password"
 }
stdout{codec => rubydebug}
}

Now run

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.conf

and start the logstash service again using

sudo systemctl start logstash

Step 7: Access logstash from python:

import logging
import logstash
import random
test_logger = logging.getLogger('service_name')
test_logger.setLevel(logging.DEBUG)
test_logger.addHandler(logstash.TCPLogstashHandler('logstash-server-ip', 5959 , version=1))
extra = {
    'app_name':"service_name"
}
#This code logs 10 times
test_logger.debug('There is some error in the request', extra=extra)
test_logger.info('INFO', extra=extra)
test_logger.warning('WARNING', extra=extra)
test_logger.critical('CRITICAL', extra=extra)
test_logger.error('ERROR', extra=extra)