How to store Django logs in MongoDB database?

Published on May 18,2020 by Maulik

Info, Warning, or Error logs are vital information that helps to detect issues and solving them. When developing our support tools for our product, we would need to store error logs in the database. It helps to generate full meaning information out of error events occurring in the Django web application.

To know what the problem is, we already solved half problem.

What is the need to store logs in MongoDB?

Django web application server logs need to be monitored and managed systematically to keep Django web application servers running continuously and encounter the least errors in the server. Logs management is essential to achieve 100% uptime and serve all user requests.

If we use organized tools like google stackdriver for searching, viewing, and managing logs, we do not need to store logs in MongoDB.

If we want our customized tool to identify and track bugs in the Django web application, we need a logs storage. Of course, you can not keep querying the traditional log files. We need a query mechanism to query specific terms from Django web application logs.

What is MongoDB?

MongoDB is a NoSQL database. It means it is a non-relational database—definition of MongoDB on its official website.

MongoDB is a document database, which means it stores data in JSON-like documents. We believe this is the most natural way to think about data and is much more expressive and powerful than the traditional row/column model.

Can we use MongoDB with Django ORM?

Yes, we can use MongoDB with Django web application ORM. We need to use a library named Djongo, which is MongoDB connector.

Why choose MongoDB for storing logs?

MongoDB is a document-based database, and we do not need to have complicated relationships in the records of logs. We can not have fix structure of error logs or any other logs. So MongoDB is the complete NoSQL database for storing logs.

Steps to store Django logs in MongoDB database

  1. Create logging middleware and handler.
  2. Register logging handler in Django settings.py file.
  3. Create a custom exception handler to store the error stack trace in MongoDB.
  4. View logs in MongoDB.

Please find the source code repository for storing Django logs in MongoDB.

Create logging middleware.

  • Following is the directory structure of the project:
  • Create the logging middleware file logging_middleware.py in your app. The directory should look like this:

  • .
    ├── .gitignore.py    
    ├── README.md  
    ├── manage.py  
    ├── requirements.txt
    └── logging_formatter #<your main app>
        │   ├── __init__.py
        │   ├── asgi.py
        │   ├── settings.py
        │   ├── urls.py
        │   ├── wsgi.py
        │   ├── views.py
        │   ├── log_middleware.py

  • Let’s override the logging handler class to store logs in the database.
  • Here is the example of storing the Django web app logs in MongoDB.
  • Update your log_middleware.py file as follow:
  • import logging
import os
import time
from time import gmtime, strftime
from pymongo import MongoClient
 
class DatabaseLoggingHandler(logging.Handler):
 
   def __init__(self, database, collection="logs"):
       logging.Handler.__init__(self)
       self.client = MongoClient("localhost")
       self.db = self.client[database]
       self.collection = self.db[collection]
 
   def emit(self, record):
       """save log record in file or database"""
       formatted_message = self.format(record)
 
       database_record = {
         "level": record.levelname,
         "module": record.module,
         "line": record.lineno,
         "asctime": record.asctime if getattr(record, "asctime", None) else strftime("%Y-%m-%d %H:%M", gmtime()),
         "message": record.message # use `formatted_message` for store formatted log
       }
 
       try:
           self.collection.insert_one(database_record)
       except Exception as e:
           print(e)
      
      
class FilterLevels(logging.Filter):
   def __init__(self, filter_levels=None):
       super(FilterLevels, self).__init__()
       self._filter_levels = filter_levels
 
   def filter(self, record):
       if record.levelname in self._filter_levels:
         return True
       return False

     

Register logging handler in Django settings.py file.

  • LOGGING = {
   'version': 1,
   'disable_existing_loggers': True,
   'filters': {
       'filter_info_level': {
           '()': 'logging_formatter.log_middleware.FilterLevels',
           'filter_levels' : [
               "INFO"
           ]
       },
       'filter_error_level': {
           '()': 'logging_formatter.log_middleware.FilterLevels',
           'filter_levels' : [
               "ERROR"
           ]
       },
       'filter_warning_level': {
           '()': 'logging_formatter.log_middleware.FilterLevels',
           'filter_levels' : [
               "WARNING"
           ]
       }
   },
   'formatters': {
       'info-formatter': {
           'format': '%(levelname)s : %(message)s - [in %(pathname)s:%(lineno)d]'
       },
       'error-formatter': {
           'format': '%(levelname)s : %(asctime)s {%(module)s} [%(funcName)s] %(message)s- [in %(pathname)s:%(lineno)d]',
           'datefmt': '%Y-%m-%d %H:%M'
       },
       'short': {
           'format': '%(levelname)s : %(message)s'
       }
   },
   'handlers': {
       'customHandler_1': {
           'formatter': 'info-formatter',
           'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
           'database': 'logging_formatter',
           'collection': 'logs',
           'filters': ['filter_info_level'],
       },
       'customHandler_2': {
           'formatter': 'error-formatter',
           'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler',
           'database': 'logging_formatter',
           'collection': 'logs',
           'filters': ['filter_error_level'],
       },
       'customHandler_3': {
           'formatter': 'short',
           'class': 'logging.StreamHandler',
           'filters': ['filter_warning_level'],
       },
   },
   'loggers': {
       'customLogger': {
           'handlers': [
               'customHandler_1',
               'customHandler_2',
               'customHandler_3'
           ],
           'level': 'DEBUG',
       },
   },
}

     

  • After implementing the above changes, you can see the logs with INFO and ERROR level in the database and logs with WARNING levels in the console.

Create a custom exception handler to store the error stack trace in MongoDB.

  • We can also store the unexpected exception ( 500 Internal Server Error ) logs in the database. It is beneficial for debugging any exception on the production server.
  • For that, create the exception handler as follow:
  • logging_formater/exception_handler.py
  • from rest_framework.views import exception_handler
import traceback
import logging
 
logger = logging.getLogger('customLogger')
 
def handle_exception(exc, context):
   error_response = exception_handler(exc, context)
   logger.error(traceback.format_exc())
   return error_response

     

  • Register the middleware class in settings.py file:
  • REST_FRAMEWORK = {
  'DEFAULT_PERMISSION_CLASSES': (
      'rest_framework.permissions.AllowAny',
  ),
  'DEFAULT_AUTHENTICATION_CLASSES': (
      'rest_framework.authentication.SessionAuthentication',
      'rest_framework.authentication.BasicAuthentication',
  ),
  'EXCEPTION_HANDLER': 'logging_formatter.exception_handler.handle_exception'
}

     

  • With these configurations, all unexpected Django web application error logs with HTTP status code 500, save all its stack trace in MongoDB.

View logs in MongoDB.

We have the GUI tool Studio 3t for view, searching, and updating MongoDB documents. We can view the specific logs by executing the search query using GUI. So all Django web application logs are now viewed on GUI Studio 3t.

Conclusion:

  • We just learned to store all Django logs in MongoDB. Our error database is now ready. We can create a support tool for a product support team.

Share this
LinkedIn
Copy URL

48 Comments

JohnnyEnaks

10 hours, 47 minutes

price of cialis 20 mg <a href=" https://cialistlf.com/# ">cialis coupon</a> tadalafil tablets 20 mg india

JohnnyEnaks

1 day, 1 hour

cialis pharmacy <a href=" https://cialistlf.com/# ">cialis 20 mg price</a> tadalafil best price 20 mg

JohnnyEnaks

2 days, 8 hours

cialis coupon <a href=" https://cialistlf.com/# ">where to buy liquid cialis</a> cialis coupon

NormanMuh

2 days, 18 hours

stromectol for sale <a href=" https://stromectoldrs.online/# ">stromectol without a doctor prescription</a> stromectol 3 mg tablets price

NormanMuh

3 days, 9 hours

ed pills for sale <a href=" https://edpilldrs.com/# ">ed dysfunction treatment</a> ed meds online

NormanMuh

4 days

best treatment for ed <a href=" https://edpilldrs.online/# ">top erection pills</a> buy erection pills

NormanMuh

4 days, 14 hours

buying from canadian online pharmacies <a href=" https://canadadrs.com/# ">highest rated canadian pharmacies</a> meds online without doctor prescription

NormanMuh

5 days, 6 hours

stromectol 12 mg tablets <a href=" https://stromectoldrs.com/# ">stromectol for humans for sale</a> stromectol price usa

Rogerontob

5 days, 23 hours

buy propecia without prescription <a href=" https://finasteridemen.com/# ">purchase finasteride without a prescription</a> cheap propecia pills

Rogerontob

6 days, 12 hours

generic propecia for cheap without precscription <a href=" https://finasteridemen.com/# ">where to buy finasteride</a> buy propecia without a prescription

Rogerontob

1 week

fincar <a href=" https://finasteridemen.com/# ">propecia prices</a> finasteride tablets

Rogerontob

1 week, 1 day

propecia for hair loss <a href=" https://finasteridemen.com/# ">finasteride online</a> propecia cost

Rogerontob

1 week, 2 days

online prescription for ed meds <a href=" https://canadapillsshop.com/# ">prescription drugs</a> ed meds online without doctor prescription

Jeffreybed

1 week, 4 days

erection pills viagra online <a href=" https://cheapdrugsmrt.com/# ">online canadian drugstore</a> ed meds online canada

Jeffreybed

1 week, 5 days

scabies ivermectin <a href=" https://stromectolmrt.com/# ">ivermectin for dogs dosage in mg</a> ivermectin 200mg

Jeffreybed

1 week, 6 days

ed drugs online <a href=" https://cheapdrugsmrt.com/# ">erectile dysfunction pills</a> ed devices

Jeffreybed

1 week, 6 days

aspirin and ed <a href=" https://cheapdrugsmrt.com/# ">top ed pills</a> ed aids

Jeffreybed

2 weeks

buy prednisone 20mg <a href=" https://prednisonemrt.com/# ">prednisone online paypal</a> prednisone 60 mg price

Jeffreybed

2 weeks, 1 day

prednisone 5 mg tablet <a href=" https://prednisonemrt.com/# ">prednisone 30 mg</a> prednisone price

iqetmhgobn

2 weeks, 1 day

Django Circle | How to store Django logs in MongoDB database? <a href="http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/">aiqetmhgobn</a> iqetmhgobn http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/ [url=http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/]uiqetmhgobn[/url]

Jeffreybed

2 weeks, 1 day

cheap drugs <a href=" https://cheapdrugsmrt.com/# ">diabetes and ed</a> anti fungal pills without prescription

Jeffreybed

2 weeks, 2 days

ed meds online without doctor prescription <a href=" https://cheapdrugsmrt.com/# ">best pill for ed</a> ed pills that really work

Charleswar

2 weeks, 4 days

<a href=" https://stromectol.company/# ">ivermectin for gapeworm</a> ivermax vs ivermectin

Stacyliz

2 weeks, 5 days

prednisone nz <a href=" https://deltasone.shop/# ">brand prednisone</a> order prednisone online canada

ovexyfef

2 weeks, 5 days

sildenafil for pah <a href="https://viagrahom.com/">sildenafil over the counter cvs</a> female viagra over the counter

Stacyliz

2 weeks, 6 days

ivermectin fda label <a href=" http://stromectol.best/# ">stromectol dosage for humans</a> fenbendazole and ivermectin combination

Stacyliz

3 weeks, 1 day

40 mg cialis too much <a href=" http://cialistadalafil.store/# ">where to get cialis cheap</a> original cialis pills

vffqdzhnw

3 weeks, 2 days

Django Circle | How to store Django logs in MongoDB database? vffqdzhnw http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/ <a href="http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/">avffqdzhnw</a> [url=http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/]uvffqdzhnw[/url]

Stacyliz

3 weeks, 2 days

top ed drugs <a href=" http://cheapdrugs.best/# ">buy prescription drugs from india</a> prescription drugs canada buy online

HaroldArbib

3 weeks, 4 days

cheap erectile dysfunction pills online <a href=" https://drugsus.shop/# ">generic ed pills</a> buy medications online

HaroldArbib

3 weeks, 6 days

how to overcome ed <a href=" http://drugsfast.store/# ">vacuum therapy for ed</a> cheap ed medication

HaroldArbib

4 weeks

ed cures that actually work <a href=" http://drugsfast.store/# ">medications for ed</a> medication for ed dysfunction

HaroldArbib

4 weeks, 1 day

supplements for ed <a href=" https://drugsus.shop/# ">cheap drugs</a> best pharmacy online

FrankNurse

1 month

ed drugs <a href=" https://drugsmst.com/# ">prices of viagra at walmart</a> ed meds online

FrankNurse

1 month

ed medicines <a href=" https://drugsmst.com/# ">erectal disfunction</a> home remedies for ed

FrankNurse

1 month

buy clomid 50mg online uk <a href=" https://clomidmst.com/# ">clomid 2017</a> how to get clomid online

FrankNurse

1 month

clomid prescription cost <a href=" http://clomidus.store/# ">get clomid online</a> buy clomid online with paypal

FrankNurse

1 month

buy amoxicillin 250mg <a href=" https://amoxilus.fun/# ">amoxicillin 500</a> buy amoxicillin 250mg

FrankNurse

1 month, 1 week

can i buy clomid over the counter in uk <a href=" https://clomiden.fun/# ">buy clomid online paypal</a> how to buy clomid online uk

FrankNurse

1 month, 1 week

buy doxycycline <a href=" http://doxycyclinefast.store/# ">doxycycline tablets online india</a> doxycycline 20 mg capsules

EdwardAbimb

1 month, 1 week

cialisnonprecription <a href=" http://cialiscnd.com/# ">generic cialis 20mg</a> cialis and paypal

EdwardAbimb

1 month, 1 week

curved penis and cialis <a href=" http://cialiscnd.com/# ">cialis marke 20mg</a> cheapest cialis tadalafil 20 mg

Strojoism

1 month, 2 weeks

how to inject ivermectin in a dog <a href=" https://stromectolns.com/# ">ivermectin pills for humans for sale</a> ivermectin 250ml

Artrcriff

1 month, 2 weeks

does cialis have a generic <a href=" https://cls20.com/# ">best price for cialis 20mg</a> cialis 36 hour dosage

irretty

1 month, 2 weeks

Nicholas assured him that this was a common side effect of the procedure that would resolve spontaneously. [url=https://oscialipop.com]buy cialis 5mg daily use[/url] <a href=https://oscialipop.com>cialis online</a> It is a bridge connecting various parts of the brain. Mwiccl https://oscialipop.com - best place to buy generic cialis online Amoxil Amoxil Vs Wiki Vaxbay

Travismindy

1 month, 3 weeks

how much does generic viagra cost <a href=" https://edviagralove.com/# ">buy viagra online without prescription</a> how long does 25mg viagra last

AbrtNew

2 months, 1 week

<a href="https://t.me/filmfilmfilmes/9">Филомена</a>

AbertNew

2 months, 2 weeks

коли почалась друга світова війна <a href="http://bitly.com/skilky-shche-bude-tryvaty-viyna-v-ukrayini">коли закінчиться війна в україні 2022 екстрасенси</a> коли закінчиться війна в україні 2022

Related Articles

Debugging a Django web application in VS Code

Published on May 12,2020 by Maulik

Debugging a Django web application in VS Code

Web applications, when built in their entirety, are a complex piece of software passing the execution of the code from one file to another within …

Read full article

How to setup Django, Gunicorn, Nginx and PostgreSQL service using docker compose?

Published on October 09,2020 by Maulik

How to setup Django, Gunicorn, Nginx and PostgreSQL service using docker compose?

Docker helps to simplify and set up a uniform platform for development, staging, and production environments. DevOps efforts are reduced by using docker technology. This …

Read full article

How to create a common response format for 200, 400, 500 responses by creating custom exception handler in Django Rest Framework?

Published on May 28,2020 by Maulik

How to create a common response format for 200, 400, 500 responses by creating custom exception handler in Django Rest Framework?

In micro-services architecture, multiple client applications are consuming the backend API. The backend server does the core business logic and all heavy lifting. The client …

Read full article

copied to clipboard

Sign up for our newsletter

Please join our news letter which we share every month, you would love interesting python and django news letters.

We understand no one like spamming, your emails are safe with us.

Copyright © Django Circle All Rights Reserved.