Logging to logstash in python from scratch for Windows
This quick tutorial is done in Windows OS. Only the installing and starting the services part changes in other OSes. You could find many resources online to install ELK stack on your OS.
Step 1. Download ELK stack one by one
https://www.elastic.co/downloads/elasticsearch
Step 2. After unzipping, copy the path of the bin of all these services into path env var
Step 3. Start Elastic Search and Kibana using elasticsearch.bat and kibana.bat inside bin folder
verify using http://localhost:9200/ and http://localhost:5601/ for Logstash and Kibana
Step 4. Create logstash.conf file . It can be as simple as below
input {
tcp{
port => 5959
}
}output{
elasticsearch{
hosts => [“localhost:9200”]
index => “ctc”
}
stdout{codec => rubydebug}
}
Step 5. open cmd and run => logstash -f path_to_logstash.conf
this starts logstash
verify it using http://localhost:9600/
Now the flow goes like this
Logstash server is running at 9200 listening to logs at 5959
The logs it gets is sent to elastic search running at 9200 and the data is indexed using the name “ctc”
Then Kibana picks it up using the index name
Step 6. Run the following program
import logging
import logstash
import randomtest_logger = logging.getLogger('Service Name')
test_logger.setLevel(logging.DEBUG)test_logger.addHandler(logstash.TCPLogstashHandler('0.0.0.0', 5959 , version=1))extra = {
'app_name':"Service_Name"
}#This code logs 10 times
test_logger.debug('DEBUG', extra=extra)
test_logger.info('INFO', extra=extra)
test_logger.warning('WARNING', extra=extra)
test_logger.critical('CRITICAL', extra=extra)
test_logger.error('ERROR', extra=extra)
After sending the logs, it could be seen in the command prompt of logstash like below
and you could also verify the new index is created(if logs sent for the first time) in elastic search also
check the indices in elastic search using http://localhost:9200/_cat/indices
Step 7. Check logs in Kibana
If you are checking the logs in kibana for the first time, else skip the below steps
1. Go to kibana http://localhost:5601/
2. Open the menu, then go to Stack Management > Kibana > Index Patterns
3. Click on create index pattern
4. Give the index name (which was given in logstash.conf file ) . In this case it is “ctc”
5. Click Next step
6. Select the Time Filter field name, then click Create index pattern
Then Go to kibana http://localhost:5601/ and click discover
Then select the index pattern
Click on ctc* index pattern
You could make use of the filters to visualize the logs based on time, app_name, pattern etc
