How to manage nesting of logs in Google Stackdriver with Django Web Apps?

Published on March 21,2020 by Maulik

In my previous article on “How to integrate stackdriver logging with Django web application?” I talked about integrating stackdriver with the Django application. I also understand the need to have nesting of logs under every request parent request logs. I know we would like to see all single request logs clubbed together. So in this article, I am listing steps on how to do nested logs or clubbing logs or co-relating parent and child logs in stackdriver/google cloud logging with Django web application.

What is the nesting of logs in stackdriver?

nested stackdriver/google cloud logging

In simple terms, to list all logs or club together, all logs with child relationship to its main request logs emitted during a request execution is the nesting of logs with Google Stackdriver. For example, the Django application server received a request at “/users/” URLs. Then all logs info, debug or error logs emitted during the execution of requests are displayed as a list of sub logs of a request log. Nested google stackdriver logs is a convenient solution when we debug Django applications in production server virtual machines running on Google Cloud, AWS, Digital Ocean, or Azure cloud platforms.

If you have not integrated google stackdriver with Django, please check “how to integrate google stackdriver logging with Django.” We are going to make changes to that solution code.

Steps to implement nested logs with the Google Stackdriver tool in the Django web application.

  1. Integrate Google Stackdriver logging with the Django application.
  2. Create/update stackdriver logging middleware.
  3. Register google stackdriver logging middleware in settings.
  4. We can view the Django application’s nested logs in google stackdriver logging.

Please check the Django google cloud nested stackdriver logging project on Github.

Please make sure you have followed all steps from Integrate Google Stackdriver logging with the Django application. It is required by middleware, which we create in this article.

Create/update stackdriver logging middleware.

What is Django middleware?

It is a feature of the Django web application framework, which enables altering all requests and responses in the Django application. It gives you the flexibility to implement common functionality required across all requests and responses in the Django application. Implementing this with Django middleware, we make sure that nesting of logs happens with all requests.

from google.cloud.logging.resource import Resource
from google.cloud import logging as gcp_logging
from django.utils.deprecation import MiddlewareMixin
import io
import logging
import json
import threading
import time

_thread_locals = threading.local()
json_credentials_path='service-account.json'

client = gcp_logging.Client.from_service_account_json(json_credentials_path)
client.setup_logging()

client_email = ""
with io.open(json_credentials_path, "r", encoding="utf-8") as file:
    credentials_info = json.load(file)
    client_email = credentials_info["client_email"]

_LOG_RESOURCE = Resource(
    type='service_account', 
    labels={
        "email_id":  client_email,
        "project_id":  client.project
    }
)

parent_logger = client.logger("parent")

def get_current_request():
    return getattr(_thread_locals, 'request', None)

class StackDriverHandler(logging.Handler):

    def __init__(self):
        logging.Handler.__init__(self)

    def emit(self, record):
        """Add record to cloud"""
        record.request = get_current_request()
        self.logger = client.logger('stackdriver.googleapis.com%2Fapp')
        self.log_msg = self.format(record)

        TRACE = "projects/{}/traces/{}".format(client.project, threading.current_thread().ident)
        self.logger.log_text(self.log_msg, severity=record.levelname, trace=TRACE, resource=_LOG_RESOURCE)

class LoggingMiddleware(MiddlewareMixin):
    """
    Provides full logging of requests and responses
    """
    _initial_http_body = None

    def process_request(self, request):
        _thread_locals.request = request
        request_start_time = time.time()
        request_time = "%.5fs" % (time.time() - request_start_time)
        request.META['HTTP_X_UPSTREAM_SERVICE_TIME'] = request_time
        self._initial_http_body = request.body

    def process_response(self, request, response):
        """
        Adding request and response logging
        """
        request_time = request.META['HTTP_X_UPSTREAM_SERVICE_TIME']
        
        TEXT = u'TEXT'
        SEVERITY = 'INFO'
        TRACE = "projects/{}/traces/{}".format(client.project, threading.current_thread().ident)
        content_length = len(response.content)
        REQUEST = {
            'requestMethod': request.method,
            'requestUrl': request.get_full_path(),
            'status': response.status_code,
            'userAgent': request.META['HTTP_USER_AGENT'],
            'responseSize': content_length,
            'latency': request_time,
            'remoteIp': request.META['REMOTE_ADDR']
        }

        parent_logger.name = "stackdriver.googleapis.com%2Fnginx.request"
        parent_logger.log_struct({}, client=client, severity=SEVERITY, http_request=REQUEST, trace=TRACE, resource=_LOG_RESOURCE)
        return response

Register your logging middleware with your Django application.

  • Add the middleware in settings.py.
MIDDLEWARE = [
   ...
   'stackdriver_logging.log_middleware.LoggingMiddleware',
]
  • Now call the endpoint from the API client like the postman. It should execute successfully.

View nested logs of the Django application on Google Stackdriver Console.

  • Open the stackdriver logging on google cloud console. You can see the request with nested logging.

nested stackdriver/google cloud logging

  • To view, the request logs only set the log type in the filter.

django-google-stackdriver-logging

 

Conclusion:

By now, you should be able to integrate google stackdriver logging in the Django web application. Also, nesting of a single request log with its internal logs is possible with the Django web application hosted on any virtual machine on public clouds like Google Cloud, Digital Ocean, AWS, and Azure.

21 Comments

ymztidjbhe

9 hours, 41 minutes

Django Circle | How to manage nesting of logs in Google Stackdriver with Django Web Apps? <a href="http://www.gd0q8992iv9lkj100w79y3j13pl5i6cas.org/">aymztidjbhe</a> ymztidjbhe http://www.gd0q8992iv9lkj100w79y3j13pl5i6cas.org/ [url=http://www.gd0q8992iv9lkj100w79y3j13pl5i6cas.org/]uymztidjbhe[/url]

pkeipsje

1 week, 6 days

Django Circle | How to manage nesting of logs in Google Stackdriver with Django Web Apps? [url=http://www.g767r8zvnp92q13hino410o762zzph68s.org/]upkeipsje[/url] pkeipsje http://www.g767r8zvnp92q13hino410o762zzph68s.org/ <a href="http://www.g767r8zvnp92q13hino410o762zzph68s.org/">apkeipsje</a>

akotusu

2 months, 4 weeks

[url=http://slkjfdf.net/]Oferevauk[/url] <a href="http://slkjfdf.net/">Cutusixa</a> foo.cqvq.djangocircle.com.gzw.cj http://slkjfdf.net/

aavoxecuzmaya

2 months, 4 weeks

[url=http://slkjfdf.net/]Afomey[/url] <a href="http://slkjfdf.net/">Abiepec</a> qzf.rogl.djangocircle.com.thn.js http://slkjfdf.net/

ovujebo

2 months, 4 weeks

[url=http://slkjfdf.net/]Iredujyux[/url] <a href="http://slkjfdf.net/">Amucayo</a> gwp.amaa.djangocircle.com.fen.lr http://slkjfdf.net/

ibiyulyopoyax

3 months

[url=http://slkjfdf.net/]Evilur[/url] <a href="http://slkjfdf.net/">Isuuhuqo</a> rsd.apmy.djangocircle.com.owy.fb http://slkjfdf.net/

ulomamniyagaq

3 months

[url=http://slkjfdf.net/]Inekeve[/url] <a href="http://slkjfdf.net/">Erukoba</a> eil.dvvw.djangocircle.com.bae.se http://slkjfdf.net/

ejajezihaw

3 months, 3 weeks

[url=http://slkjfdf.net/]Eluuge[/url] <a href="http://slkjfdf.net/">Uyutoxeva</a> pbk.pobq.djangocircle.com.tis.bu http://slkjfdf.net/

onicondnewa

3 months, 3 weeks

[url=http://slkjfdf.net/]Afatuqos[/url] <a href="http://slkjfdf.net/">Iliozemq</a> dej.zsfj.djangocircle.com.lrg.hn http://slkjfdf.net/

asoxokoeji

3 months, 3 weeks

[url=http://slkjfdf.net/]Obesacigp[/url] <a href="http://slkjfdf.net/">Arupeaf</a> unv.cbml.djangocircle.com.cfb.bi http://slkjfdf.net/

epoyosapor

3 months, 3 weeks

[url=http://slkjfdf.net/]Nuhodmosi[/url] <a href="http://slkjfdf.net/">Ituqegmib</a> fls.xaip.djangocircle.com.dtd.ba http://slkjfdf.net/

egacusupo

3 months, 3 weeks

[url=http://slkjfdf.net/]Vinutoo[/url] <a href="http://slkjfdf.net/">Xidixewew</a> fli.slal.djangocircle.com.suv.fm http://slkjfdf.net/

ilgcegharo

3 months, 3 weeks

[url=http://slkjfdf.net/]Igiwuam[/url] <a href="http://slkjfdf.net/">Opehudaj</a> ios.rsjx.djangocircle.com.xyz.wx http://slkjfdf.net/

adebigaxeg

3 months, 3 weeks

[url=http://slkjfdf.net/]Aqegin[/url] <a href="http://slkjfdf.net/">Ifawas</a> dgh.xzsr.djangocircle.com.pkh.uq http://slkjfdf.net/

ucudoya

3 months, 3 weeks

[url=http://slkjfdf.net/]Iqekaik[/url] <a href="http://slkjfdf.net/">Utukip</a> zyt.sybm.djangocircle.com.nqa.jk http://slkjfdf.net/

igewikeq

3 months, 3 weeks

[url=http://slkjfdf.net/]Godecux[/url] <a href="http://slkjfdf.net/">Oqefoyimu</a> heu.yqxi.djangocircle.com.nlr.re http://slkjfdf.net/

irbihomihax

3 months, 3 weeks

[url=http://slkjfdf.net/]Iziqujcih[/url] <a href="http://slkjfdf.net/">Uupebsuq</a> kfm.qfcx.djangocircle.com.vdf.rf http://slkjfdf.net/

ogudofug

3 months, 3 weeks

[url=http://slkjfdf.net/]Upirafoz[/url] <a href="http://slkjfdf.net/">Ogicibpaw</a> wve.bsty.djangocircle.com.yhv.ns http://slkjfdf.net/

oxeigaumi

3 months, 3 weeks

[url=http://slkjfdf.net/]Odibuwac[/url] <a href="http://slkjfdf.net/">Ivatufigu</a> bnd.qkyh.djangocircle.com.uiv.de http://slkjfdf.net/

daelipeov

3 months, 3 weeks

[url=http://slkjfdf.net/]Auwuuk[/url] <a href="http://slkjfdf.net/">Ovisav</a> nfs.zcyv.djangocircle.com.esb.xg http://slkjfdf.net/

oarataruona

3 months, 3 weeks

[url=http://slkjfdf.net/]Ehiyete[/url] <a href="http://slkjfdf.net/">Uretevi</a> vgm.ttcm.djangocircle.com.son.wp http://slkjfdf.net/

Related Articles

How to create different custom logs formatter for Info, Warning and Error logs in Django web application?

Published on May 18,2020 by Maulik

How to create different custom logs formatter for Info, Warning and Error logs in Django web application?

If you are interested in something which helps you customize the logs format for Django web applications running in production, you have landed at the …

Read full article

How to store Django logs in MongoDB database?

Published on May 18,2020 by Maulik

How to store Django logs in MongoDB database?

Info, Warning, or Error logs are vital information that helps to detect issues and solving them. When developing our support tools for our product, we …

Read full article

4 Best practices and solutions for managing Django Logs in production server.

Published on May 18,2020 by Maulik

4 Best practices and solutions for managing Django Logs in production server.

I am sure you all understand Django logs. And use them during development. This article talks about 4 best practices we can use in our …

Read full article

copied to clipboard

Sign up for our newsletter

Please join our news letter which we share every month, you would love interesting python and django news letters.

We understand no one like spamming, your emails are safe with us.

Copyright © Django Circle All Rights Reserved.