logging to logstash from python from scratch in Ubuntu
Step 1: Install few required packages:
Install Nginx
sudo apt-get install nginx
Import GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
This step outputs OK
install transport-https package
sudo apt-get install apt-transport-https
Add elastic repo into system’s repo list
echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list
update repo
sudo apt-get update
Step 2: Install ElasticSearch:
Install elasticsearch package
sudo apt-get install elasticsearch
Enable auto restart of elastic search when the machine reloads
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
start elastic search
sudo systemctl start elasticsearch.service
Note: Do not run this step, this is just to make note of
#sudo systemctl stop elasticsearch.service
restart elastic search ( Whenever any change to /etc/elasticsearch/elasticsearch.yml)
sudo systemctl restart elasticsearch.service
check status of elastic search
service elasticsearch status
Status should show active(running)
Step 3: Elastic Search config
edit elastic search config file
sudo vim /etc/elasticsearch/elasticsearch.yml
change the below values in the config file to allow remote access:
transport.host: localhost
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0
After these changes restart elastic search
Test elastic search:
Either we can test it using the below command or run http://server-ip:9200 in browser
curl localhost:9200
returns something like
{
"name" : "ip-192-2-1-94",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "UtLYRoRtQjSlMhUcfNno2A",
"version" : {
"number" : "7.10.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
"build_date" : "2020-12-05T01:00:33.671820Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Step 4: Install Kibana
Install kibana package
sudo apt-get install kibana
configure kibana
sudo vi /etc/kibana/kibana.yml
change the below keys in the config
server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: [“http://localhost:9200”]
start kibana
sudo systemctl start kibana
enable kibana start on server restart
sudo systemctl enable kibana
test kibana:
open http://server-ip:5601 in browser
Step 5: Install Logstash
install logstash
sudo apt-get install logstash
configure logstash to get logs via tcp ( which is sent by logging python )
We can configure the flow within logstash i.e. to accept logs, filter logs and output logs
This is not to be confused with logstash.yml file which has logstash configuration
create a flow config file
sudo nano /etc/logstash/logstash.conf
Paste the below config
input {
tcp{
port => 5959
}
}output{
elasticsearch{
hosts => ["localhost:9200"]
index => "ctc"
}
stdout{codec => rubydebug}
}
Run the below command to set the flow config to logstash
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.conf
enable auto restart and start service
sudo systemctl enable logstash.service
sudo systemctl start logstash
Step 6: Enable Authentication
elastic search plugin called x-pack is by default installed with elastic search.
enable security in elasticsearch config
xpack.security.enabled: true
restart elasticsearch after the change using
sudo systemctl restart elasticsearch
There will be built in users by default which has no password set yet
We must set the passwords to use authentication
set passwords for the modules in ELK stack using
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
set elasticsearch username and password in kibana config for it to access
change kibana config
sudo vi /etc/kibana/kibana.yml
uncomment and change the below lines
elasticsearch.username: "kibana_system"
elasticsearch.password: "Password"
restart kibana using
sudo systemctl restart kibana
Now elasticsearch, kibana console asks for authentication
Now we make logstash accessible to users without auth, But in the backend, logstash has to access elasticsearch using credentials
So First we stop the running instance of logstash using
sudo systemctl stop logstash
Then we make the changes to our flow config as below (where we add user and password parameters)
sudo vi /etc/logstash/logstash.conf
Replace the current config with the following config
input {
tcp{
port => 5959
}
}output{
elasticsearch{
hosts => ["localhost:9200"]
index => "ctc"
user => "elastic"
password => "Password"
}
stdout{codec => rubydebug}
}
Now run
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/logstash.conf
and start the logstash service again using
sudo systemctl start logstash
Step 7: Access logstash from python:
import logging
import logstash
import randomtest_logger = logging.getLogger('service_name')
test_logger.setLevel(logging.DEBUG)test_logger.addHandler(logstash.TCPLogstashHandler('logstash-server-ip', 5959 , version=1))extra = {
'app_name':"service_name"
}#This code logs 10 times
test_logger.debug('There is some error in the request', extra=extra)
test_logger.info('INFO', extra=extra)
test_logger.warning('WARNING', extra=extra)
test_logger.critical('CRITICAL', extra=extra)
test_logger.error('ERROR', extra=extra)