How to store Django logs in MongoDB database?
Published on May 18,2020 by Maulik
Info, Warning, or Error logs are vital information that helps to detect issues and solving them. When developing our support tools for our product, we would need to store error logs in the database. It helps to generate full meaning information out of error events occurring in the Django web application.
To know what the problem is, we already solved half problem.
What is the need to store logs in MongoDB?
Django web application server logs need to be monitored and managed systematically to keep Django web application servers running continuously and encounter the least errors in the server. Logs management is essential to achieve 100% uptime and serve all user requests.
If we use organized tools like google stackdriver for searching, viewing, and managing logs, we do not need to store logs in MongoDB.
If we want our customized tool to identify and track bugs in the Django web application, we need a logs storage. Of course, you can not keep querying the traditional log files. We need a query mechanism to query specific terms from Django web application logs.
What is MongoDB?
MongoDB is a NoSQL database. It means it is a non-relational database—definition of MongoDB on its official website.
MongoDB is a document database, which means it stores data in JSON-like documents. We believe this is the most natural way to think about data and is much more expressive and powerful than the traditional row/column model.
Can we use MongoDB with Django ORM?
Yes, we can use MongoDB with Django web application ORM. We need to use a library named Djongo, which is MongoDB connector.
Why choose MongoDB for storing logs?
MongoDB is a document-based database, and we do not need to have complicated relationships in the records of logs. We can not have fix structure of error logs or any other logs. So MongoDB is the complete NoSQL database for storing logs.
Steps to store Django logs in MongoDB database
- Create logging middleware and handler.
- Register logging handler in Django settings.py file.
- Create a custom exception handler to store the error stack trace in MongoDB.
- View logs in MongoDB.
Please find the source code repository for storing Django logs in MongoDB.
Create logging middleware.
- Following is the directory structure of the project:
- Create the logging middleware file logging_middleware.py in your app. The directory should look like this:
-
.
├── .gitignore.py
├── README.md
├── manage.py
├── requirements.txt
└── logging_formatter #<your main app>
│ ├── __init__.py
│ ├── asgi.py
│ ├── settings.py
│ ├── urls.py
│ ├── wsgi.py
│ ├── views.py
│ ├── log_middleware.py - Let’s override the logging handler class to store logs in the database.
- Here is the example of storing the Django web app logs in MongoDB.
- Update your log_middleware.py file as follow:
-
import logging import os import time from time import gmtime, strftime from pymongo import MongoClient class DatabaseLoggingHandler(logging.Handler): def __init__(self, database, collection="logs"): logging.Handler.__init__(self) self.client = MongoClient("localhost") self.db = self.client[database] self.collection = self.db[collection] def emit(self, record): """save log record in file or database""" formatted_message = self.format(record) database_record = { "level": record.levelname, "module": record.module, "line": record.lineno, "asctime": record.asctime if getattr(record, "asctime", None) else strftime("%Y-%m-%d %H:%M", gmtime()), "message": record.message # use `formatted_message` for store formatted log } try: self.collection.insert_one(database_record) except Exception as e: print(e) class FilterLevels(logging.Filter): def __init__(self, filter_levels=None): super(FilterLevels, self).__init__() self._filter_levels = filter_levels def filter(self, record): if record.levelname in self._filter_levels: return True return False
Register logging handler in Django settings.py file.
-
LOGGING = { 'version': 1, 'disable_existing_loggers': True, 'filters': { 'filter_info_level': { '()': 'logging_formatter.log_middleware.FilterLevels', 'filter_levels' : [ "INFO" ] }, 'filter_error_level': { '()': 'logging_formatter.log_middleware.FilterLevels', 'filter_levels' : [ "ERROR" ] }, 'filter_warning_level': { '()': 'logging_formatter.log_middleware.FilterLevels', 'filter_levels' : [ "WARNING" ] } }, 'formatters': { 'info-formatter': { 'format': '%(levelname)s : %(message)s - [in %(pathname)s:%(lineno)d]' }, 'error-formatter': { 'format': '%(levelname)s : %(asctime)s {%(module)s} [%(funcName)s] %(message)s- [in %(pathname)s:%(lineno)d]', 'datefmt': '%Y-%m-%d %H:%M' }, 'short': { 'format': '%(levelname)s : %(message)s' } }, 'handlers': { 'customHandler_1': { 'formatter': 'info-formatter', 'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler', 'database': 'logging_formatter', 'collection': 'logs', 'filters': ['filter_info_level'], }, 'customHandler_2': { 'formatter': 'error-formatter', 'class': 'logging_formatter.log_middleware.DatabaseLoggingHandler', 'database': 'logging_formatter', 'collection': 'logs', 'filters': ['filter_error_level'], }, 'customHandler_3': { 'formatter': 'short', 'class': 'logging.StreamHandler', 'filters': ['filter_warning_level'], }, }, 'loggers': { 'customLogger': { 'handlers': [ 'customHandler_1', 'customHandler_2', 'customHandler_3' ], 'level': 'DEBUG', }, }, }
- After implementing the above changes, you can see the logs with INFO and ERROR level in the database and logs with WARNING levels in the console.
Create a custom exception handler to store the error stack trace in MongoDB.
- We can also store the unexpected exception ( 500 Internal Server Error ) logs in the database. It is beneficial for debugging any exception on the production server.
- For that, create the exception handler as follow:
logging_formater/exception_handler.py
-
from rest_framework.views import exception_handler import traceback import logging logger = logging.getLogger('customLogger') def handle_exception(exc, context): error_response = exception_handler(exc, context) logger.error(traceback.format_exc()) return error_response
- Register the middleware class in settings.py file:
-
REST_FRAMEWORK = { 'DEFAULT_PERMISSION_CLASSES': ( 'rest_framework.permissions.AllowAny', ), 'DEFAULT_AUTHENTICATION_CLASSES': ( 'rest_framework.authentication.SessionAuthentication', 'rest_framework.authentication.BasicAuthentication', ), 'EXCEPTION_HANDLER': 'logging_formatter.exception_handler.handle_exception' }
- With these configurations, all unexpected Django web application error logs with HTTP status code 500, save all its stack trace in MongoDB.
View logs in MongoDB.
We have the GUI tool Studio 3t for view, searching, and updating MongoDB documents. We can view the specific logs by executing the search query using GUI. So all Django web application logs are now viewed on GUI Studio 3t.
Conclusion:
- We just learned to store all Django logs in MongoDB. Our error database is now ready. We can create a support tool for a product support team.
48 Comments
JohnnyEnaks
10 hours, 47 minutes
price of cialis 20 mg <a href=" https://cialistlf.com/# ">cialis coupon</a> tadalafil tablets 20 mg india
JohnnyEnaks
1 day, 1 hour
cialis pharmacy <a href=" https://cialistlf.com/# ">cialis 20 mg price</a> tadalafil best price 20 mg
JohnnyEnaks
2 days, 8 hours
cialis coupon <a href=" https://cialistlf.com/# ">where to buy liquid cialis</a> cialis coupon
NormanMuh
2 days, 18 hours
stromectol for sale <a href=" https://stromectoldrs.online/# ">stromectol without a doctor prescription</a> stromectol 3 mg tablets price
NormanMuh
3 days, 9 hours
ed pills for sale <a href=" https://edpilldrs.com/# ">ed dysfunction treatment</a> ed meds online
NormanMuh
4 days
best treatment for ed <a href=" https://edpilldrs.online/# ">top erection pills</a> buy erection pills
NormanMuh
4 days, 14 hours
buying from canadian online pharmacies <a href=" https://canadadrs.com/# ">highest rated canadian pharmacies</a> meds online without doctor prescription
NormanMuh
5 days, 6 hours
stromectol 12 mg tablets <a href=" https://stromectoldrs.com/# ">stromectol for humans for sale</a> stromectol price usa
Rogerontob
5 days, 23 hours
buy propecia without prescription <a href=" https://finasteridemen.com/# ">purchase finasteride without a prescription</a> cheap propecia pills
Rogerontob
6 days, 12 hours
generic propecia for cheap without precscription <a href=" https://finasteridemen.com/# ">where to buy finasteride</a> buy propecia without a prescription
Rogerontob
1 week
fincar <a href=" https://finasteridemen.com/# ">propecia prices</a> finasteride tablets
Rogerontob
1 week, 1 day
propecia for hair loss <a href=" https://finasteridemen.com/# ">finasteride online</a> propecia cost
Rogerontob
1 week, 2 days
online prescription for ed meds <a href=" https://canadapillsshop.com/# ">prescription drugs</a> ed meds online without doctor prescription
Jeffreybed
1 week, 4 days
erection pills viagra online <a href=" https://cheapdrugsmrt.com/# ">online canadian drugstore</a> ed meds online canada
Jeffreybed
1 week, 5 days
scabies ivermectin <a href=" https://stromectolmrt.com/# ">ivermectin for dogs dosage in mg</a> ivermectin 200mg
Jeffreybed
1 week, 6 days
ed drugs online <a href=" https://cheapdrugsmrt.com/# ">erectile dysfunction pills</a> ed devices
Jeffreybed
1 week, 6 days
aspirin and ed <a href=" https://cheapdrugsmrt.com/# ">top ed pills</a> ed aids
Jeffreybed
2 weeks
buy prednisone 20mg <a href=" https://prednisonemrt.com/# ">prednisone online paypal</a> prednisone 60 mg price
Jeffreybed
2 weeks, 1 day
prednisone 5 mg tablet <a href=" https://prednisonemrt.com/# ">prednisone 30 mg</a> prednisone price
iqetmhgobn
2 weeks, 1 day
Django Circle | How to store Django logs in MongoDB database? <a href="http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/">aiqetmhgobn</a> iqetmhgobn http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/ [url=http://www.gig1p92eo58kjl80og9v6y9483e18e5ss.org/]uiqetmhgobn[/url]
Jeffreybed
2 weeks, 1 day
cheap drugs <a href=" https://cheapdrugsmrt.com/# ">diabetes and ed</a> anti fungal pills without prescription
Jeffreybed
2 weeks, 2 days
ed meds online without doctor prescription <a href=" https://cheapdrugsmrt.com/# ">best pill for ed</a> ed pills that really work
Charleswar
2 weeks, 4 days
<a href=" https://stromectol.company/# ">ivermectin for gapeworm</a> ivermax vs ivermectin
Stacyliz
2 weeks, 5 days
prednisone nz <a href=" https://deltasone.shop/# ">brand prednisone</a> order prednisone online canada
ovexyfef
2 weeks, 5 days
sildenafil for pah <a href="https://viagrahom.com/">sildenafil over the counter cvs</a> female viagra over the counter
Stacyliz
2 weeks, 6 days
ivermectin fda label <a href=" http://stromectol.best/# ">stromectol dosage for humans</a> fenbendazole and ivermectin combination
Stacyliz
3 weeks, 1 day
40 mg cialis too much <a href=" http://cialistadalafil.store/# ">where to get cialis cheap</a> original cialis pills
vffqdzhnw
3 weeks, 2 days
Django Circle | How to store Django logs in MongoDB database? vffqdzhnw http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/ <a href="http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/">avffqdzhnw</a> [url=http://www.g2jdcdowvv6720679907i7hl406vcuk1s.org/]uvffqdzhnw[/url]
Stacyliz
3 weeks, 2 days
top ed drugs <a href=" http://cheapdrugs.best/# ">buy prescription drugs from india</a> prescription drugs canada buy online
HaroldArbib
3 weeks, 4 days
cheap erectile dysfunction pills online <a href=" https://drugsus.shop/# ">generic ed pills</a> buy medications online
HaroldArbib
3 weeks, 6 days
how to overcome ed <a href=" http://drugsfast.store/# ">vacuum therapy for ed</a> cheap ed medication
HaroldArbib
4 weeks
ed cures that actually work <a href=" http://drugsfast.store/# ">medications for ed</a> medication for ed dysfunction
HaroldArbib
4 weeks, 1 day
supplements for ed <a href=" https://drugsus.shop/# ">cheap drugs</a> best pharmacy online
FrankNurse
1 month
ed drugs <a href=" https://drugsmst.com/# ">prices of viagra at walmart</a> ed meds online
FrankNurse
1 month
ed medicines <a href=" https://drugsmst.com/# ">erectal disfunction</a> home remedies for ed
FrankNurse
1 month
buy clomid 50mg online uk <a href=" https://clomidmst.com/# ">clomid 2017</a> how to get clomid online
FrankNurse
1 month
clomid prescription cost <a href=" http://clomidus.store/# ">get clomid online</a> buy clomid online with paypal
FrankNurse
1 month
buy amoxicillin 250mg <a href=" https://amoxilus.fun/# ">amoxicillin 500</a> buy amoxicillin 250mg
FrankNurse
1 month, 1 week
can i buy clomid over the counter in uk <a href=" https://clomiden.fun/# ">buy clomid online paypal</a> how to buy clomid online uk
FrankNurse
1 month, 1 week
buy doxycycline <a href=" http://doxycyclinefast.store/# ">doxycycline tablets online india</a> doxycycline 20 mg capsules
EdwardAbimb
1 month, 1 week
cialisnonprecription <a href=" http://cialiscnd.com/# ">generic cialis 20mg</a> cialis and paypal
EdwardAbimb
1 month, 1 week
curved penis and cialis <a href=" http://cialiscnd.com/# ">cialis marke 20mg</a> cheapest cialis tadalafil 20 mg
Strojoism
1 month, 2 weeks
how to inject ivermectin in a dog <a href=" https://stromectolns.com/# ">ivermectin pills for humans for sale</a> ivermectin 250ml
Artrcriff
1 month, 2 weeks
does cialis have a generic <a href=" https://cls20.com/# ">best price for cialis 20mg</a> cialis 36 hour dosage
irretty
1 month, 2 weeks
Nicholas assured him that this was a common side effect of the procedure that would resolve spontaneously. [url=https://oscialipop.com]buy cialis 5mg daily use[/url] <a href=https://oscialipop.com>cialis online</a> It is a bridge connecting various parts of the brain. Mwiccl https://oscialipop.com - best place to buy generic cialis online Amoxil Amoxil Vs Wiki Vaxbay
Travismindy
1 month, 3 weeks
how much does generic viagra cost <a href=" https://edviagralove.com/# ">buy viagra online without prescription</a> how long does 25mg viagra last
AbrtNew
2 months, 1 week
<a href="https://t.me/filmfilmfilmes/9">Филомена</a>
AbertNew
2 months, 2 weeks
коли почалась друга світова війна <a href="http://bitly.com/skilky-shche-bude-tryvaty-viyna-v-ukrayini">коли закінчиться війна в україні 2022 екстрасенси</a> коли закінчиться війна в україні 2022