Info, Warning, or Error logs are vital information that helps to detect issues and solving them. When developing our support tools for our product, we would need to store error logs in the database. It helps to generate full meaning information out of error events occurring in the Django web application.

To know what the problem is, we already solved half problem.

What is the need to store logs in MongoDB?

Django web application server logs need to be monitored and managed systematically to keep Django web application servers running continuously and encounter the least errors in the server. Logs management is essential to achieve 100% uptime and serve all user requests.

If we use organized tools like google stackdriver for searching, viewing, and managing logs, we do not need to store logs in MongoDB.

If we want our customized tool to identify and track bugs in the Django web application, we need a logs storage. Of course, you can not keep querying the traditional log files. We need a query mechanism to query specific terms from Django web application logs.

What is MongoDB?

MongoDB is a NoSQL database. It means it is a non-relational database—definition of MongoDB on its official website.

MongoDB is a document database, which means it stores data in JSON-like documents. We believe this is the most natural way to think about data and is much more expressive and powerful than the traditional row/column model.

Can we use MongoDB with Django ORM?

Yes, we can use MongoDB with Django web application ORM. We need to use a library named Djongo, which is MongoDB connector.

Why choose MongoDB for storing logs?

MongoDB is a document-based database, and we do not need to have complicated relationships in the records of logs. We can not have fix structure of error logs or any other logs. So MongoDB is the complete NoSQL database for storing logs.

Steps to store Django logs in MongoDB database

1. Create logging middleware and handler.
2. Register logging handler in Django settings.py file.
3. Create a custom exception handler to store the error stack trace in MongoDB.
4. View logs in MongoDB.

Please find the source code repository for storing Django logs in MongoDB.

Create logging middleware.

• Following is the directory structure of the project:
• Create the logging middleware file logging_middleware.py in your app. The directory should look like this:

• .
  ├── .gitignore.py    
  ├── README.md  
  ├── manage.py  
  ├── requirements.txt
  └── logging_formatter #<your main app>
      │   ├── __init__.py
      │   ├── asgi.py
      │   ├── settings.py
      │   ├── urls.py
      │   ├── wsgi.py
      │   ├── views.py
      │   ├── log_middleware.py
  

   

• Let’s override the logging handler class to store logs in the database.
• Here is the example of storing the Django web app logs in MongoDB.
• Update your log_middleware.py file as follow:

• import logging
  import os
  import time
  from time import gmtime, strftime
  from pymongo import MongoClient
   
  class DatabaseLoggingHandler(logging.Handler):
   
     def __init__(self, database, collection="logs"):
         logging.Handler.__init__(self)
         self.client = MongoClient("localhost")
         self.db = self.client[database]
         self.collection = self.db[collection]
   
     def emit(self, record):
         """save log record in file or database"""
         formatted_message = self.format(record)
   
         database_record = {
           "level": record.levelname,
           "module": record.module,
           "line": record.lineno,
           "asctime": record.asctime if getattr(record, "asctime", None) else strftime("%Y-%m-%d %H:%M", gmtime()),
           "message": record.message # use `formatted_message` for store formatted log
         }
   
         try:
             self.collection.insert_one(database_record)
         except Exception as e:
             print(e)
        
        
  class FilterLevels(logging.Filter):
     def __init__(self, filter_levels=None):
         super(FilterLevels, self).__init__()
         self._filter_levels = filter_levels
   
     def filter(self, record):
         if record.levelname in self._filter_levels:
           return True
         return False
  

   

Register logging handler in Django settings.py file.

• LOGGING = {
     'version': 1,
     'disable_existing_loggers': True,
     'filters': {
         'filter_info_level': {
             '()': 'logging_formatter.log_middleware.FilterLevels',
             'filter_levels' : [
                 "INFO"
             ]
         },
         'filter_error_level': {
             '()': 'logging_formatter.log_middleware.FilterLevels',
             'filter_levels' : [
                 "ERROR"
             ]
         },
         'filter_warning_level': {
             '()': 'logging_formatter.log_middleware.FilterLevels',
             'filter_levels' : [
                 "WARNING"
             ]
         }
     },
     'formatters': {
         'info-formatter': {
             'format': '%(levelname)s : %(message)s - [in %(pathname)s:%(lineno)d]'
         },
         'error-formatter': {
             'format': '%(levelname)s : %(asctime)s {%(module)s} [%(funcName)s] %(message)s- [in %(pathname)s:%(lineno)d]',
             'datefmt': '%Y-%m-%d %H:%M'
         },
         'short': {
             'format': '%(levelname)s : %(message)s'
         }
     },
     'handlers': {
         'customHandler_1': {
             'formatter': 'info-formatter',
             'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
             'database': 'logging_formatter',
             'collection': 'logs',
             'filters': ['filter_info_level'],
         },
         'customHandler_2': {
             'formatter': 'error-formatter',
             'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
             'database': 'logging_formatter',
             'collection': 'logs',
             'filters': ['filter_error_level'],
         },
         'customHandler_3': {
             'formatter': 'short',
             'class': 'logging.StreamHandler',
             'filters': ['filter_warning_level'],
         },
     },
     'loggers': {
         'customLogger': {
             'handlers': [
                 'customHandler_1',
                 'customHandler_2',
                 'customHandler_3'
             ],
             'level': 'DEBUG',
         },
     },
  }
  

   

• After implementing the above changes, you can see the logs with INFO and ERROR level in the database and logs with WARNING levels in the console.

Create a custom exception handler to store the error stack trace in MongoDB.

• We can also store the unexpected exception ( 500 Internal Server Error ) logs in the database. It is beneficial for debugging any exception on the production server.
• For that, create the exception handler as follow:
• logging_formater/exception_handler.py

• from rest_framework.views import exception_handler
  import traceback
  import logging
   
  logger = logging.getLogger('customLogger')
   
  def handle_exception(exc, context):
     error_response = exception_handler(exc, context)
     logger.error(traceback.format_exc())
     return error_response
  

   

• Register the middleware class in settings.py file:

• REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.AllowAny',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.BasicAuthentication',
    ),
    'EXCEPTION_HANDLER': 'logging_formatter.exception_handler.handle_exception'
  }
  

   

• With these configurations, all unexpected Django web application error logs with HTTP status code 500, save all its stack trace in MongoDB.

View logs in MongoDB.

We have the GUI tool Studio 3t for view, searching, and updating MongoDB documents. We can view the specific logs by executing the search query using GUI. So all Django web application logs are now viewed on GUI Studio 3t.

Read more:

4 Best practices and solutions for managing Django Logs in production server.

How to create different custom logs formatter for Info, Warning and Error logs in Django web application?

How to manage nesting of logs in Google Stackdriver with Django Web Apps?

Conclusion:

• We just learned to store all Django logs in MongoDB. Our error database is now ready. We can create a support tool for a product support team.