Sunday, 8 April 2018

// // Leave a Comment

Run Elasticsearch 6.2 Cluster On AWS Spot Fleet

How To Run Elasticsearch 6.2 Cluster In Docker Using Spot Instances In AWS
Dockerising Elasticsearch 6.2

Elasticsearch is a very useful databases capable of handling enormous amount of data. Its capacity and performance is great but comes with a cost. If you are holding TBs of data in an Elasticsearch cluster then you might end up having around 10 data nodes, 3 master nodes and 1-3 client nodes. These numbers will cost you roughly around $7K per month if you have config similar to 30GB RAM on data nodes.

What if you can reduce this cost to less than $2K per month? Yes, you can save a lot if you use AWS spot instances.

Here we will setup a cluster having 3 data nodes of 16 GB RAM each and a master node having 8GB of RAM. We are keeping number of nodes limited just for the demo but you can grow this as per your requirement or use-case. So, let's get started...

Steps to be followed:

  • Create security group elasticsearch-sg
  • Create Elasticsearch Config (elasticsearch.yml and jvm.options)
  • Create S3 bucket to hold Elasticsearch config files
  • Create IAM Policy to access bucket
  • Create IAM Role and assign bucket access policy to it
  • Create Base Image (AMI) from which we will be spawning master node and data node
  • Create On-demand Master Node
  • Create Data nodes on Spot Fleet 

#1 Create security group "elasticsearch-sg"

#2 Elasticsearch Configuration files

elasticsearch_master.yml for master node:

cluster.name: spotes

path.data: /usr/share/elasticsearch/data

network.host: 0.0.0.0

http.port: 9200

node.master: true

node.data: false

node.name: "nodename"

transport.tcp.compress: true

bootstrap.memory_lock: true

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.unicast.hosts: ['es1.xyz.vpc']

thread_pool.bulk.queue_size: 500

elasticsearch_data.yml file for data node:

cluster.name: spotes

path.data: /usr/share/elasticsearch/data

network.host: 0.0.0.0

http.port: 9200

node.master: false

node.data: true

node.name: "nodename"

bootstrap.memory_lock: true

transport.tcp.compress: true

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.unicast.hosts: ['es1.xyz.vpc']

thread_pool.bulk.queue_size: 500

master.jvm.options file for master node:

-Xms4g
-Xmx4g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+DisableExplicitGC
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError

data.jvm.options file for data node

-Xms8g
-Xmx8g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+DisableExplicitGC
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError

#3 Create S3 bucket to hold Elasticsearch config files

Create a bucket name es-configurations and upload all configuration files we created above



#4 Create IAM Policy to access bucket

Create following IAM policy (es-configurations-bucket-access)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::es-configurations/*"
        }
    ]
}


#5 Create IAM Role and assign bucket access policy to it

Create an IAM role "elasticsearch-role" and attach above policy to it.

#6 Create Base Image (AMI) from which we will be spawning master node and data node

First we will launch an instance in which we will install docker, aws-cli and download elasticsearch docker image. After installing these basic stuff we will create an AMI from it which will be used to launch master and data node.

Now go ahead launching an instance by providing the following userdata to it:

#!/bin/bash

# output log of userdata to /var/log/user-data.log
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# Install awscli
apt-get update
apt install awscli -y

# Set max_map_count
echo 262144 | sudo tee /proc/sys/vm/max_map_count

# Install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-cache policy docker-ce
apt-get install -y docker-ce
service docker restart

# Get official elasticsearch docker image
docker pull docker.elastic.co/elasticsearch/elasticsearch:6.2.3

# Create /etc/elasticsearch directory to hold elasticsearch config files like elasticsearch.yml and jvm.options
mkdir -p /etc/elasticsearch


When you are done running the above script, create an AMI from the current instance.

#7 Create On-demand Master Node

Create an on-demand instance of type having 8GB of memory as we are giving 4GB of HEAP and provide the following userdata to it:

#!/bin/bash
 
set -x
# output log of userdata to /var/log/user-data.log
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
 
aws s3 cp s3://es-configurations/elasticsearch_master.yml /etc/elasticsearch/elasticsearch.yml --region ap-south-1
aws s3 cp s3://es-configurations/master.jvm.options /etc/elasticsearch/jvm.options --region ap-south-1
 
sed -i -e "s/nodename/${HOSTNAME}/g" /etc/elasticsearch/elasticsearch.yml
 
mkdir -p /vol/es
 
chown -R 1000:1000 /vol
chown -R 1000:1000 /etc/elasticsearch
 
sysctl -w vm.max_map_count=262144
 
#start docker container
docker run --net=host -d -p 9200:9200 -e "xpack.security.enabled=false" --restart unless-stopped -v /vol/es:/usr/share/elasticsearch/data -v /etc/elasticsearch/jvm.options:/usr/share/elasticsearch/config/jvm.options -v /etc/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml  --ulimit nofile=65536:65536 --ulimit memlock=-1:-1 docker.elastic.co/elasticsearch/elasticsearch:6.2.3


After launching master node. Make a route53 entry for es1.xyz.vpc with private IP or any domain you want to use for your master node.

#8 Create Data nodes on Spot Fleet 

Now we will, create spot fleet request to launch data nodes as spot instance. Go to "Spot Requests" in AWS ec2 dashboard and click on "Request Spot Instance" button:



  • Select "Request and Maintain", set "total target capacity to 3" as we will be launching 3 data nodes.
  • Select the AMI we created above. Choose any instance type having 16GB of RAM (as we are setting HEAP to 8GB).
  • Select required VPC, AZ.
  • Add additional disk of size 50GB (This could differ as per your requirement)
  • You can provide health check, monitoring and other options.
  • Provide a security group (elasticsearch-sg in our case)
  • Give a key-pair name which can be used to SSH
  • Select "elasticsearch-role" in "IAM Instance Profile"
  • Provide the following userdata:
#!/bin/bash
 
set -x
# output log of userdata to /var/log/user-data.log
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
 
aws s3 cp s3://es-configurations/elasticsearch_data.yml /etc/elasticsearch/elasticsearch.yml --region ap-south-1
aws s3 cp s3://es-configurations/data.jvm.options /etc/elasticsearch/jvm.options --region ap-south-1

 
sed -i -e "s/nodename/${HOSTNAME}/g" /etc/elasticsearch/elasticsearch.yml
 
mkfs.xfs /dev/xvdba
mkdir -p /vol/es
mount /dev/xvdba /vol/es
 
chown -R 1000:1000 /vol
chown -R 1000:1000 /etc/elasticsearch
 
sysctl -w vm.max_map_count=262144
 
#start docker container
docker run --net=host -d -p 9200:9200 -e "xpack.security.enabled=false" --restart unless-stopped -v /vol/es:/usr/share/elasticsearch/data -v /etc/elasticsearch/jvm.options:/usr/share/elasticsearch/config/jvm.options -v /etc/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml  --ulimit nofile=65536:65536 --ulimit memlock=-1:-1 docker.elastic.co/elasticsearch/elasticsearch:6.2.3
 

You can leave other settings to default. Click on "Launch", this will create a spot request and will launch three nodes which will eventually join the cluster.
After the nodes are ready, go to master node and make a curl request to check if nodes are in the cluster:


curl localhost:9200/_cat/nodes?v

This will show the list of all nodes.
Read More

Saturday, 7 April 2018

// // Leave a Comment

Run Elasticsearch 6.2 In Docker

How To Run Elasticsearch 6.2 In Docker
Dockerising Elasticsearch 6.2


Elasticsearch is a very useful databases being used by almost every company. one of the major use-case is observed in ELK or EFK stack for centralised logging. Setting up elasticsearch from scratch with traditional method could be tedious.

First create a bucket "es-configurations" to store elasticsearch configuration file

upload the file elasticearch.yml with the following content:

cluster.name: mycluster

path.data: /usr/share/elasticsearch/data

network.host: 0.0.0.0

http.port: 9200

node.master: true

node.data: true

node.name: "nodename"

bootstrap.memory_lock: true

transport.tcp.compress: true

Also upload jvm.options file with the following content. Make sure to change heap parameter as per your available memory.

-Xms4g
-Xmx4g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+DisableExplicitGC
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError

Create following IAM policy, say "elasticsearch-bucket-access" to access config file in "elasticsearch" bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::es-configurations/*"
        }
    ]
}

Create an IAM role "elasticsearch-role" and attach above policy to it.

Now go ahead launching an instance, attach above role to it and provide the following userdata to it:

#!/bin/bash

###
# This is the first part which can be used to prepare base-image
###

# output log of userdata to /var/log/user-data.log
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# Install awscli
apt-get update
apt install awscli -y

# Set max_map_count
echo 262144 | sudo tee /proc/sys/vm/max_map_count

# Install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-cache policy docker-ce
apt-get install -y docker-ce
service docker restart

# Get official elasticsearch docker image
docker pull docker.elastic.co/elasticsearch/elasticsearch:6.2.3

# Create /etc/elasticsearch directory to hold elasticsearch config files like elasticsearch.yml and jvm.options
mkdir -p /etc/elasticsearch

###
# Second part of script downloads elasticsearch configuration files from S3 and run container
###

# Get elasticsearch config files from S3
aws s3 cp s3://es-configurations/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml --region ap-south-1
aws s3 cp s3://es-configurations/jvm.options /etc/elasticsearch/jvm.options --region ap-south-1

# Replace nodename in elasticsearch.yml file with hostname
sed -i -e "s/nodename/${HOSTNAME}/g" /etc/elasticsearch/elasticsearch.yml

# Mount a secondary Volume for elasticsearch data directory
mkfs.xfs /dev/xvdb
mkdir -p /vol/es
mount /dev/xvdba /vol/es

# change ownership of data directory and config directory to user with 1000 id as in container elasticsearch runs with user 1000
chown -R 1000:1000 /vol
chown -R 1000:1000 /etc/elasticsearch

# Make sure vm.max_map_count is 262144
sysctl -w vm.max_map_count=262144

#start docker container
docker run --net=host -d -p 9200:9200 -e "xpack.security.enabled=false" --restart unless-stopped -v /vol/es:/usr/share/elasticsearch/data -v /etc/elasticsearch/jvm.options:/usr/share/elasticsearch/config/jvm.options -v /etc/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml  --ulimit nofile=65536:65536 --ulimit memlock=-1:-1 docker.elastic.co/elasticsearch/elasticsearch:6.2.3

Read More

Friday, 6 April 2018

// // Leave a Comment

How To Check - Connected IPs To Server | IPv4 And IPv6 Information | Kernel IP Routing | Network Interface Transactions

How To List Connected IPs To Server?

List Kernel IP Routing

List Network Interface Transactions

Check IPv4 & IPv6 Information

Usage of netstat Command


netstat is a command-line utility to list network connections, routing tables, interface statistics multicast memberships and masquerade connections.

# List all ports and IPs (tcp and upd)

netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0     52 192.168.0.2:ssh             192.168.0.1:egs             ESTABLISHED
tcp        1      0 192.168.0.2:59292           www.gov.com:http            CLOSE_WAIT
tcp        0      0 localhost:smtp              *:*                         LISTEN
tcp        0      0 *:59482                     *:*                         LISTEN
udp        0      0 *:35036                     *:*
udp        0      0 *:npmp-local                *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     16972  /tmp/orbit-root/linc-76b-0-6fa08790553d6
unix  2      [ ACC ]     STREAM     LISTENING     17149  /tmp/orbit-root/linc-794-0-7058d584166d2

# List Only IPs


netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
  • Here -t is for tcp connections
  • 2>/dev/null redirects all unwanted output to /dev/null
  • grep :80 displays results only for IPs connected on port 80
  • awk '{print $5}' displays fifth column only
  • cut -d : -f1 is used to extract only IPs excluding port with delimeter ":"
  • sort command sort the result
  • uniq command groups the result
  • sort -nr again sort it in reverse order
  • head will display top 10 results

# Display kernel IP routing

netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.0.0     *               255.255.255.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default         192.168.0.1     0.0.0.0         UG        0 0          0 eth0

# List Network Interface Transactions

netstat -i
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0     4459      0      0      0     4057      0      0      0 BMRU
lo        16436   0        8      0      0      0        8      0      0      0 LRU

# Check IPv4 & IPv6 Information

netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            1      224.0.0.251
eth0            1      all-systems.mcast.net
lo              1      ff02::1
eth0            1      ff02::202
eth0            1      ff02::1:ffb4:da21
eth0            1      ff02::1


Read More

Saturday, 24 March 2018

// // Leave a Comment

How To Connect DynamoDB With AWS Lambda

Connect To DynamoDB With AWS Lambda


AWS lambda can be used to save your data to databases like MySQL, DynamoDB. If you don't want to manage your own application instance and MongoDB then you can opt for AWS DynamoDB which is AWS managed service for NoSQL database. AWS lambda will replace your application instance and you will be only charged for the number of execution whereas if you run your own application ec2 instance then you would be charged for every second the instance is up. So if you have a small application which can fit in lambda-dynamodb use-case then this could save your cost.
Let's see how to setup Lambda with DynamoDB:

Create DynamoDB Table

  • Go to DynamoDB Dashboard. 
  • Create a table, give it any name say, "myinfo"
  • Keep the "type" to string
  • Keep other settings as default
  • Click "create" button to create the table

  • After creating table, you will see the following table details


Create Lambda Function

  • Open Lambda Dashboard, click on "create function" button
  • Do not select any blueprint, click on "Author From Scratch" button


  • Give the function name "myinfo"
  • In Role section select "choose from existing role"
  • Then select "dynamo_access" role (This role is being already created by us. You need to create this explicitly by attaching dynamodb full access policy to this role)
  • Now click on "create function" button
  • Now in the function code, paste the code from the below gist
  • Select runtime as "Node.js 6.10" and leave handler as default
  • Now click on "save and test"
  • This will open a modal to create a test-event, give the event name myinfo.
  • Put the below data in the text-pane    -    {"name": "appychip"}
  • Now click on "create"

  • Click on "Test" button and you will see the lambda is getting executed and prints out "SUCCESS" if executed successfully.

  • You can also check DynamoDB table "myinfo", the entry for {"name": "appychip"} got created when you executed this function.


This is how you can play around and create your app using lambda and dynamodb. If you want to put an API endpoint for this Lambda function then you can watch our video tutorial and go through our blog-post.
Read More

Friday, 2 March 2018

// // Leave a Comment

Few Post Giving 404 Not Found | Wordpress

Wordpress Post Giving 404 Not Found Because of (.) Dot In Permalink


We have a website https://npminstall.org which is a search engine to search npm packages and get the steps to install them. Website was running fine. After submission of sitemap on Goggle WebMaster  Tool, it gave lots of errors with error message that some posts are giving 404 errors.
Google Webmaster:







On checking, It got observed that all permalinks having a dot "." in the title are giving 404. To get rid of this issue add the following snippet to function.php of the theme being used:

remove_filter('sanitize_title', 'sanitize_title_with_dashes');
add_filter('sanitize_fitler', 'sanitize_filter_se_119069');
function sanitize_filter_se_119069($title, $raw_title = '', $context = 'display'){
    $title = strip_tags($title);
    // Preserve escaped octets.
    $title = preg_replace('|%([a-fA-F0-9][a-fA-F0-9])|', '---$1---', $title);
    // Remove percent signs that are not part of an octet.
    $title = str_replace('%', '', $title);
    // Restore octets.
    $title = preg_replace('|---([a-fA-F0-9][a-fA-F0-9])---|', '%$1', $title);

    if (seems_utf8($title)) {
        if (function_exists('mb_strtolower')) {
            $title = mb_strtolower($title, 'UTF-8');
        }
        $title = utf8_uri_encode($title, 200);
    }

    $title = strtolower($title);
    $title = preg_replace('/&.+?;/', '', $title); // kill entities

//  $title = str_replace('.', '-', $title);

    if ( 'save' == $context ) {
        // Convert nbsp, ndash and mdash to hyphens
        $title = str_replace( array( '%c2%a0', '%e2%80%93', '%e2%80%94' ), '-', $title );

        // Strip these characters entirely
        $title = str_replace( array(
            // iexcl and iquest
            '%c2%a1', '%c2%bf',
            // angle quotes
            '%c2%ab', '%c2%bb', '%e2%80%b9', '%e2%80%ba',
            // curly quotes
            '%e2%80%98', '%e2%80%99', '%e2%80%9c', '%e2%80%9d',
            '%e2%80%9a', '%e2%80%9b', '%e2%80%9e', '%e2%80%9f',
            // copy, reg, deg, hellip and trade
            '%c2%a9', '%c2%ae', '%c2%b0', '%e2%80%a6', '%e2%84%a2',
            // grave accent, acute accent, macron, caron
            '%cc%80', '%cc%81', '%cc%84', '%cc%8c',
        ), '', $title );

        // Convert times to x
        $title = str_replace( '%c3%97', 'x', $title );
    }

    $title = preg_replace('/[^%a-z0-9 _-]/', '', $title);
    $title = preg_replace('/\s+/', '-', $title);
    $title = preg_replace('|-+|', '-', $title);
    $title = trim($title, '-');

    return $title;
}

After adding this, just restart the php-fpm process with command sudo service php-fpm restart and the pages which were giving 404 will start giving 200 response code. 
Read More

Saturday, 17 February 2018

// // Leave a Comment

How to Make REST API in Django

Django API

Create REST API Using Core Django

Django is a popular web framework in the Python world for rapid development. Here we will learn, how to quickly make REST API (Representational State Transfer Application Programming Interface) using Django.

# Prerequisites:

Make sure, you have following things installed

# End Result:

We will make a small application that will keep track of all my friends with their respective details.
  • API to list all friends. 
  • API to get detail of a particular friend. 
  • API to add a friend in the list. 
  • API to delete a friend from the list. 

# Video Tutorial



After installing prerequisites, follow the below steps:

# Create Project

django-admin startproject my_project

Here the project name is my_project. This command will create a directory called my_project in your current working directory, with some pre populated directory and files. This directory is root directory for you project.

# Create App

cd my_project
python manage.py startapp my_app

The app name is my_app. You can create as many app you want inside one project. This command has created my_app directory inside the project's root directory.

# Database Creation and Setup

Let's use Sqlite as our database in this tutorial. If you are planning to use database other than Sqlite, you need to do appropriate settings in settings.py file of project.

To create tables, edit the models.py file of the app.

vim my_app/models.py

Here, we will create a table called, MyFriendList. To do this paste the following code in the opened file.
from django.db import models

class MyFriendList(models.Model):
    friend_name = models.CharField(max_length=200)
    mobile_no = models.IntegerField()


mobile_no = models.IntegerField() 

Table MyFriendList has two columns. First column is friend_name of char type and of max_length 200. Second column is mobile_no of integer type.

To actually create this Table in our database system, we need to run migrate commands. But before that, we need to add this app in our project. For this, open our project's config file.

vim my_project/settings.py 

Now, search for INSTALLED_APPS setting and edit it. It should look like this.

INSTALLED_APPS = [
    'my_app.apps.MyAppConfig',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Now, we will run migrate commands.

python manage.py makemigrations my_app
python manage.py migrate

First command, will make the migration files (a file that contains the equivalent SQL statements for Database) and second command will execute it. After executing migrate command, you should see migration of files that are not created by us. These are some required tables by Django.

# Create Views

Views are used to define business logic. Here we will make three views, to list, create and delete the friend list. First open the views.py file of my_app application.

vim my_app/views.py

Now paste the following code in the file:


from django.http import JsonResponse
from django.views import View
from django.views.decorators.csrf import csrf_exempt
from django.utils.decorators import method_decorator
import json

from .models import MyFriendList


class MyFriend(View):
    def get(self, request):
        friend_list = list(MyFriendList.objects.values())
        return JsonResponse(friend_list, safe=False) 

    # To turn off CSRF validation (not recommended in production)
    @method_decorator(csrf_exempt)
    def dispatch(self, request, *args, **kwargs):
        return super(MyFriend, self).dispatch(request, *args, **kwargs)

    def post(self, request):
        data = request.body.decode('utf8')
        data = json.loads(data)
        try:
            new_friend = MyFriendList(friend_name=data["friend_name"], mobile_no=data["mobile_no"])
            new_friend.save()
            return JsonResponse({"created": data}, safe=False)
        except:
            return JsonResponse({"error": "not a valid data"}, safe=False)

class MyFriendDetail(View):

    @method_decorator(csrf_exempt)
    def dispatch(self, request, *args, **kwargs):
        return super(MyFriendDetail, self).dispatch(request, *args, **kwargs)

    def get(self, request, pk):
        friend_list = {"friend": list(MyFriendList.objects.filter(pk=pk).values())}
        return JsonResponse(friend_list, safe=False)

    def put(self, request, pk):
        data = request.body.decode('utf8')
        data = json.loads(data)
        try:
            new_friend = MyFriendList.objects.get(pk=pk)
            data_key = list(data.keys())
            for key in data_key:
                if key == "friend_name":
                    new_friend.friend_name = data[key]
                if key == "mobile_no":
                    new_friend.mobile_no = data[key]
            new_friend.save()
            return JsonResponse({"updated": data}, safe=False)
        except MyFriendList.DoesNotExist:
            return JsonResponse({"error": "Your friend having provided primary key does not exist"}, safe=False)
        except:
            return JsonResponse({"error": "not a valid data"}, safe=False)

    def delete(self, request, pk):
        try:
            new_friend = MyFriendList.objects.get(pk=pk)
            new_friend.delete()
            return JsonResponse({"deleted": True}, safe=False)
        except:
            return JsonResponse({"error": "not a valid primary key"}, safe=False)

Let's understand the above code:

  • First, we imported the necessary modules and models for our application.
  • We have class MyFriend, containing get and post method that serves for GET and POST HTTP method respectively.
  • We have override the dispatch method of View class to turn off CSRF validation. However, its not recommended for production. We just turned it off for sake of simplicity.
  • get method of MyFriend class, lists all the friends.
  • post method of MyFriend class, allows to add new friend in the existing list.
  • After that, we have MyFriendDetail class, containing get, put and delete method that serves for GET, PUT and DELETE HTTP method respectively.
  • get method of MyFriendDetail class, takes one argument called "pk" (primary key) and provides the complete detail of the friend having that primary key.
  • put method of MyFriendDetail class, takes the "pk" to update the particular friend's details.
  • delete method of MyFriendDetail class, also takes the "pk" argument, and deletes the friend from the friend list having that pk as primary key. 

# Route Setup

Now, we need to configure endpoint or URL to access the app. First define route for my_app by editing project's urls.py file.

vim my_project/urls.py

Now, edit this file as follow:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('friends/', include('my_app.urls')),
    path('admin/', admin.site.urls),
]

Here, we have defined the route friends/ for our app my_app. You can define different routes here for multiple apps.

Now define routes for the API of my_app. To do this create urls.py file inside my_app.

vim my_app/urls.py 

Now, paste following code inside the file:

from django.urls import path

from my_app.views import MyFriend, MyFriendDetail

urlpatterns = [
    path('', MyFriend.as_view()),
    path('<int:pk>/', MyFriendDetail.as_view()),
]

Let's understand the URL patterns

We have defined empty pattern for MyFreind class, so that our API will have structure like <server_name:port>/friends/. (Note: friends/ came from my_app application route that we defined in project wide urls.py)


For class MyFriendList, we have defined integer pattern for the key pk. Here API will have structure like <server_name:port>/friends/1/.

So, we are done with all the coding. We have model, views and urls in place. Now start the development server.

python manage.py runserver 

Output:
Performing system checks...

System check identified no issues (0 silenced).
February 17, 2018 - 18:57:06
Django version 2.0.2, using settings 'my_project.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Hurray! Our Django REST APIs are ready on http://127.0.0.1:8000/. Let's try it now.

I am using Postman app to hit the API.

# Add a friend in friend list.

API: /friends/
Method: POST
Data: {"friend_name": "abc", "mobile_no": 8968678990}



# Get the friend list.

API: /friends/
Method: GET





# Edit name of a friend

API: /friends/6/
Method: PUT
Data: {"friend_name": "xyz"}





# Fetch detail of a particular friend

API: /friends/6/
Method: GET



# Delete a friend from friend list:

API: /friends/6/
Method: DELETE






Whoa! We successfully tested the CRUD for my_app.


Next step is to learn deployment of Django application. For this checkout this tutorial.
Read More
// // 5 comments

Setup EFK Stack | Elasticsearch, Fluentd & Kibana With High Availability

How To Setup Fluentd With High Availability
Setup Fluentd With Forwarder-Aggregator Architecture

Fluentd is one of the most popular alternative to logstash because of its features which are missing in logstash. So before setting up fluentd let's have a look and compare both:
  • Fluentd has builtin architecture for high availability (There could be more than one aggregator)
  • Fluentd consumes less memory as compare to logstash
  • Log parsing, tagging is more easy
  • Tag based log routing is possible
Let's start building our centralized logging system using Elasticsearch, Fluentd and Kibana (EFK).
We will be following the architecture which has fluentd-forwarder(td-agent), fluentd-aggregator, Elasticsearch and Kibana. Fluentd-forwarder (the agent) reads the logs from a file and forward the logs to aggregator. Aggregator decides what should be the index_name and which host of Elasticsearch to send logs. The Elasticsearch is on a separate instance to receive logs and it also have kibana setup to visualise elasticsearch data.

# Architecture:

Following is the architecture for High Availability. Here there are multiple log-forwarder on each application node which are forwarding logs to log-aggregators. There are two aggregators shown in below architecture, if one fails then forwarders start sending logs to the second one.

# Video Tutorial


# Setup Elasticsearch & Kibana

We have already covered the setup of Elasticsearch and Kibana in one of our tutorial. Please follow this post to install Elasticsearch and Kibana.

# Log Pattern

We are considering the log format shown below:

INFO  [2018-02-17 17:14:55,827 +0530] [pool-5-thread-4] [] com.amazon.sqs.javamessaging.AmazonSQSExtendedClient: S3 object deleted, Bucket name: sqs-bucket, Object key: 63c1a5b8-4ddc-4136-b086-df6a8486414a.
INFO  [2018-02-17 17:14:56,124 +0530] [pool-5-thread-9] [] com.amazon.sqs.javamessaging.AmazonSQSExtendedClient: S3 object read, Bucket name: sqs-bucket, Object key: 2cc06f96-283f-4da7-9402-f08aab2df999.

# Log Regex

This regex is based on the logs above and needs to be specified in the source section of td-agent.conf file of forwarder.

/^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/

# Setup fluentd-aggregator

We will setup only one aggregator for this tutorial. However, you may setup two aggregators for high availability. On aggregator instance run following command:

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
sudo apt-get install make libcurl4-gnutls-dev --yes
sudo apt-get install build-essential
sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch

# After setup edit conf file and customise configuration
sudo vi /etc/td-agent/td-agent.conf

Content of /etc/td-agent/td-agent.conf. Replace host IP with your elasticsearch instance IP.

<source>
  @type forward
   port 24224
</source>

<match myorg.**>
  @type copy
    <store>
    @type file
    path /var/log/td-agent/forward.log
  </store>

  <store>
    @type elasticsearch_dynamic
    #elasticsearch host IP/domain
    host 192.168.1.4
    port 9200
    index_name fluentd-${tag_parts[1]+ "-" + Time.at(time).getlocal("+05:30").strftime(@logstash_dateformat)}

    #logstash_format true
    #logstash_prefix fluentd

    time_format %Y-%m-%dT%H:%M:%S
    #timezone +0530
    include_timestamp true

    flush_interval 10s
  </store>
</match>

Restart fluentd-aggregator process and check the logs with the following command:

sudo service td-agent restart

# check logs
tail -f /var/log/td-agent/td-agent.log

# Setup fluentd-forwarder

To setup forwarder, run following command on application instance.

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
# customise config in file td-agent.conf
sudo vi /etc/td-agent/td-agent.conf

Content of /etc/td-agent/td-agent.conf. Replace path with the path of your application log and aggregator IP with the IP of your aggregator Instance. You may use domain instead of IPs.

<match td.*.*>
  @type tdlog
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td

  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
</match>

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward

<source>
  @type forward
  port 24224
</source>

<source>
  @type http
  port 8888
</source>

## live debugging agent

<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

<source>
  @type tail
  path /var/log/myapp.log
  pos_file /var/log/td-agent/myorg.log.pos
  tag myorg.myapp
  format /^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/

  time_format %Y-%m-%d %H:%M:%S,%L %z
  timezone +0530
  time_key time
  keep_time_key true
  types time:time
</source>

<match myorg.**>
   @type copy
   <store>
    @type file
    path /var/log/td-agent/forward.log
  </store>

  <store>
    @type forward
    heartbeat_type tcp

    #aggregator IP
    host 192.168.1.86
    flush_interval 30s
  </store>

  # secondary host is optional
  # <secondary>
  #    host 192.168.0.12
  # </secondary>
</match>

Restart fluentd-forwarder process and check logs with the following command:

sudo service td-agent restart

# check logs
tail -f /var/log/td-agent/td-agent.log

Now after restarting td-agent on both forwarder and aggregator, you can see data being stored to elasticsearch.  When elasticsearch start receiving data from aggregator, you can make index pattern in kibana and start visualising the logs.

# Create Index Pattern In Kibana

Once you start getting logs in Elasticsearch, You can create an index pattern in kibana to visualise the logs. We have specified the index_name in fluentd to be of format fluentd-myapp-2018.02.12, so we will create an index pattern fluentd-* Follow the below steps shown in pictures to create an index pattern.


Finally after creating index pattern, logs will start appearing in Discover tab of dashboard


Hurray!!! You have successfully setup EFK stack to centralise your logging. 
Read More