4 Best practices and solutions for managing Django Logs in production server.

Published on May 18,2020 by Maulik

I am sure you all understand Django logs. And use them during development. This article talks about 4 best practices we can use in our Django web application production server. Imagine if you don’t see an error log on the Django development server console. To solve any issue, it becomes a guessing game where Django developer is on losing side. Django developer can not do Django development if logs are not visible. I am sure I don’t need to explain the importance of logs on the production Django web application server. Let us start.

Django framework uses python’s default logging module. Django framework has four logging terminologies which need to be understood before proceeding further.

1. 1. Logger – Consider it as a named bucket where all logs are sent for processing.
   2. Handler – It defines logs processing logic.
   3. Filters – It decides which handler process the logs.
   4. Formatters – It customizes the log format.

We must organize logs in a format where we can analyze logs and derive meaning out of them. For example, by default, we do not get the file name, file path, and line number from log. That is the necessary information required to debug bugs or to know where the information is coming in logs. Django developers must customize the logs format as per their requirement specific to the Django development project while giving Django development services.

It takes time to set up logging formats and logging tools in the Django development process. Still, it pays off to all Django developers when they deploy Django web applications in a production environment. So this is one of the most important steps of all before deploying the Django web application project to the production server and launching the application for users.

1. Formatting Django logs with custom log formatter.

What is formatting Django logs?

Defining the structure of Django logs is called the formatting of logs.

Points to be kept in mind while setting up formats of logs.

• The Source of log emission should be visible.
• Eliminate information which is unnecessary in all log levels.
• Must set appropriate log formats for each log level like info, debug, warning, and error.

For example,

Below is the Info logs format which we want to have with our Django web application

<Log Level>: <Message> - <File Location>:<Line No>

Below is the Warning logs format which we want to have with our Django web application

<Log Level>: <Message>

Below is the Error logs format which we want to have with our Django web application

<Log Level>: <Datetime> <Filename> <Http Method> <Message> <File Location> <Line No.>

Please check step by step implementation on How to create different custom logs formatter for Info, Warning, and Error logs in the Django web application?

2. Integrating Google Stackdriver logging in Django web application

In the real world Django web application running in the production, we can not open logs files and search log which needs attention. Google Stackdriver, now it is known as Google Operations, is a perfect tool for monitoring, searching, and managing the Django logs.

It does not matter if you are running your application on your dedicated server, Amazon Web Services, Azure, Digital Ocean, or Google Cloud, Google StackDriver, or Google Operations integration is possible.

By using Google Stackdriver or Google Operations, we can have a web console to search and analyze logs in Google Cloud Console. It is highly scalable solutions and performs real fast.

And importantly, till 50GB of logs storage, Google Operations does not charge any cost for it. It is free till 50 GB of storage.

Logs console looks like:

3. Nesting sub logs under single request log in Google Stackdriver with Django web application

What is the nesting of logs?

Parent and child relationship between all sub logs with its original request log. All these sub logs generate during single request execution. It is called the nesting of logs.

Example of nesting of logs:

.
└── Main Request log < log Message > 
    │   ├── Sub Log 1 < Info Log> < log Message > 
    │   ├── Sub Log 2 < Info Log> < log Message > 
    │   ├── Sub Log 3 < Info Log> < log Message > 
    │   ├── Sub Log 4 < Info Log> < log Message > 
    │   ├── Sub Log 5 < ERROR Log> < log Message > 
└── Second Main Request log < log Message > 
└── Third Main Request log < log Message > 

For example, if request created four Info logs and one error log, and all logs appear as a separate log, it will become difficult for Django developers who are debugging an issue. Such log information consumes more time to detect the root request, which caused this error.

So nesting of logs is a good practice that helps you to bundle or club all related logs into one master log.

View of nested logs on Google Cloud Logging Console.:

4. Storing Django logs in MongoDB for logs database

If you are not using Google Stackdriver or Google Operations. And want to have your support tool for your Django web application support team. We need to store all logs in the database. MongoDB is a NoSQL document-based database; it can store no structured data.

Once we have the logs MongoDB database. With this database support tool can be created.

Django developers, please check the step by step explanation on How to store Django logs in MongoDB database?

Conclusion:

1. Do customized Format logs as required by the project.
2. Use Google Stackdriver or Google Operations tool to manage logging.
3. Implement nesting of logs on Google Stackdriver or Google Operations tool.
4. Storing Django logs in MongoDB for logs Database.