How to store Django logs in MongoDB database?

Published on May 18,2020 by Maulik

Info, Warning, or Error logs are vital information that helps to detect issues and solving them. When developing our support tools for our product, we would need to store error logs in the database. It helps to generate full meaning information out of error events occurring in the Django web application.

To know what the problem is, we already solved half problem.

What is the need to store logs in MongoDB?

Django web application server logs need to be monitored and managed systematically to keep Django web application servers running continuously and encounter the least errors in the server. Logs management is essential to achieve 100% uptime and serve all user requests.

If we use organized tools like google stackdriver for searching, viewing, and managing logs, we do not need to store logs in MongoDB.

If we want our customized tool to identify and track bugs in the Django web application, we need a logs storage. Of course, you can not keep querying the traditional log files. We need a query mechanism to query specific terms from Django web application logs.

What is MongoDB?

MongoDB is a NoSQL database. It means it is a non-relational database—definition of MongoDB on its official website.

MongoDB is a document database, which means it stores data in JSON-like documents. We believe this is the most natural way to think about data and is much more expressive and powerful than the traditional row/column model.

Can we use MongoDB with Django ORM?

Yes, we can use MongoDB with Django web application ORM. We need to use a library named Djongo, which is MongoDB connector.

Why choose MongoDB for storing logs?

MongoDB is a document-based database, and we do not need to have complicated relationships in the records of logs. We can not have fix structure of error logs or any other logs. So MongoDB is the complete NoSQL database for storing logs.

Steps to store Django logs in MongoDB database

  1. Create logging middleware and handler.
  2. Register logging handler in Django settings.py file.
  3. Create a custom exception handler to store the error stack trace in MongoDB.
  4. View logs in MongoDB.

Please find the source code repository for storing Django logs in MongoDB.

Create logging middleware.

  • Following is the directory structure of the project:
  • Create the logging middleware file logging_middleware.py in your app. The directory should look like this:
  • .
    ├── .gitignore.py    
    ├── README.md  
    ├── manage.py  
    ├── requirements.txt
    └── logging_formatter #<your main app>
        │   ├── __init__.py
        │   ├── asgi.py
        │   ├── settings.py
        │   ├── urls.py
        │   ├── wsgi.py
        │   ├── views.py
        │   ├── log_middleware.py

  • Let’s override the logging handler class to store logs in the database.
  • Here is the example of storing the Django web app logs in MongoDB.
  • Update your log_middleware.py file as follow:
  • import logging
    import os
    import time
    from time import gmtime, strftime
    from pymongo import MongoClient
     
    class DatabaseLoggingHandler(logging.Handler):
     
       def __init__(self, database, collection="logs"):
           logging.Handler.__init__(self)
           self.client = MongoClient("localhost")
           self.db = self.client[database]
           self.collection = self.db[collection]
     
       def emit(self, record):
           """save log record in file or database"""
           formatted_message = self.format(record)
     
           database_record = {
             "level": record.levelname,
             "module": record.module,
             "line": record.lineno,
             "asctime": record.asctime if getattr(record, "asctime", None) else strftime("%Y-%m-%d %H:%M", gmtime()),
             "message": record.message # use `formatted_message` for store formatted log
           }
     
           try:
               self.collection.insert_one(database_record)
           except Exception as e:
               print(e)
          
          
    class FilterLevels(logging.Filter):
       def __init__(self, filter_levels=None):
           super(FilterLevels, self).__init__()
           self._filter_levels = filter_levels
     
       def filter(self, record):
           if record.levelname in self._filter_levels:
             return True
           return False

     

Register logging handler in Django settings.py file.

  • LOGGING = {
       'version': 1,
       'disable_existing_loggers': True,
       'filters': {
           'filter_info_level': {
               '()': 'logging_formatter.log_middleware.FilterLevels',
               'filter_levels' : [
                   "INFO"
               ]
           },
           'filter_error_level': {
               '()': 'logging_formatter.log_middleware.FilterLevels',
               'filter_levels' : [
                   "ERROR"
               ]
           },
           'filter_warning_level': {
               '()': 'logging_formatter.log_middleware.FilterLevels',
               'filter_levels' : [
                   "WARNING"
               ]
           }
       },
       'formatters': {
           'info-formatter': {
               'format': '%(levelname)s : %(message)s - [in %(pathname)s:%(lineno)d]'
           },
           'error-formatter': {
               'format': '%(levelname)s : %(asctime)s {%(module)s} [%(funcName)s] %(message)s- [in %(pathname)s:%(lineno)d]',
               'datefmt': '%Y-%m-%d %H:%M'
           },
           'short': {
               'format': '%(levelname)s : %(message)s'
           }
       },
       'handlers': {
           'customHandler_1': {
               'formatter': 'info-formatter',
               'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
               'database': 'logging_formatter',
               'collection': 'logs',
               'filters': ['filter_info_level'],
           },
           'customHandler_2': {
               'formatter': 'error-formatter',
               'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
               'database': 'logging_formatter',
               'collection': 'logs',
               'filters': ['filter_error_level'],
           },
           'customHandler_3': {
               'formatter': 'short',
               'class': 'logging.StreamHandler',
               'filters': ['filter_warning_level'],
           },
       },
       'loggers': {
           'customLogger': {
               'handlers': [
                   'customHandler_1',
                   'customHandler_2',
                   'customHandler_3'
               ],
               'level': 'DEBUG',
           },
       },
    }

     

  • After implementing the above changes, you can see the logs with INFO and ERROR level in the database and logs with WARNING levels in the console.

Create a custom exception handler to store the error stack trace in MongoDB.

  • We can also store the unexpected exception ( 500 Internal Server Error ) logs in the database. It is beneficial for debugging any exception on the production server.
  • For that, create the exception handler as follow:
  • logging_formater/exception_handler.py
  • from rest_framework.views import exception_handler
    import traceback
    import logging
     
    logger = logging.getLogger('customLogger')
     
    def handle_exception(exc, context):
       error_response = exception_handler(exc, context)
       logger.error(traceback.format_exc())
       return error_response

     

  • Register the middleware class in settings.py file:
  • REST_FRAMEWORK = {
      'DEFAULT_PERMISSION_CLASSES': (
          'rest_framework.permissions.AllowAny',
      ),
      'DEFAULT_AUTHENTICATION_CLASSES': (
          'rest_framework.authentication.SessionAuthentication',
          'rest_framework.authentication.BasicAuthentication',
      ),
      'EXCEPTION_HANDLER': 'logging_formatter.exception_handler.handle_exception'
    }

     

  • With these configurations, all unexpected Django web application error logs with HTTP status code 500, save all its stack trace in MongoDB.

View logs in MongoDB.

We have the GUI tool Studio 3t for view, searching, and updating MongoDB documents. We can view the specific logs by executing the search query using GUI. So all Django web application logs are now viewed on GUI Studio 3t.

Conclusion:

  • We just learned to store all Django logs in MongoDB. Our error database is now ready. We can create a support tool for a product support team.

0 Comments

Related Articles

Debugging a Django web application in VS Code

Published on May 12,2020 by Maulik

Debugging a Django web application in VS Code

Web applications, when built in their entirety, are a complex piece of software passing the execution of the code from one file to another within …

Read full article

How to setup Django, Gunicorn, Nginx and PostgreSQL service using docker compose?

Published on October 09,2020 by Maulik

How to setup Django, Gunicorn, Nginx and PostgreSQL service using docker compose?

Docker helps to simplify and set up a uniform platform for development, staging, and production environments. DevOps efforts are reduced by using docker technology. This …

Read full article

How to create a common response format for 200, 400, 500 responses by creating custom exception handler in Django Rest Framework?

Published on May 28,2020 by Maulik

How to create a common response format for 200, 400, 500 responses by creating custom exception handler in Django Rest Framework?

In micro-services architecture, multiple client applications are consuming the backend API. The backend server does the core business logic and all heavy lifting. The client …

Read full article

copied to clipboard

Sign up for our newsletter

Please join our news letter which we share every month, you would love interesting python and django news letters.

We understand no one like spamming, your emails are safe with us.

Copyright © Django Circle All Rights Reserved.