Thursday, 13 September 2018

How to avoid receiving same message twice in RabbitMQ

In this post, we will see how to avoid receiving / processing duplicate message more than once in RabbitMQ.

Background:
RabbitMQ is a messaging queue software (also called message broker) that implemented the Advanced Message Queuing Protocol (AMQP). The producer submits message on queue and consumer processes it. But sometimes we use to see the issue with duplicate processing of same message in case where consumer decide to acknowledge message after processing it.

Problem Statement and Solution:
Suppose there is a message that is published into message broker. There can be two scenario in case of consumers for this message type:

  • Single Consumer: 
    • Consumer receives the message and start processing it and dies (or hung) in between: In this case if the acknowledgement has not been send back to RabbitMQ message broker, the message will remain in queue and when the consumer starts again, it will receive the same message. 
      • Solution: In the above case, developer needs to know if the business logic is processed or not. So developer needs to implement status of the message in either database or flat file with the message id. If same message is received after consumer restarts, developer can take a call to process again or not
    • Consumer receives message and processed it but still it receives same message again:
      • Solution: If you are facing this issue, then the problem lies in the heart-beat  timeout between consumer and RabbitMQ. So here consumer has received the message and its thread is processing it but due to missed heart-beat (due to network latency or whatever reason) RabbitMQ assumes the consumer has died and it closes the connection with consumer. Meanwhile as the consumer thread is active, it is still processing the message and completed it successfully but it was unable to acknowledge it back to RabbitMQ. The solution lies in increasing the heart-beat timeout so that it doesn't misses the heart-beat.
  • Multiple Consumer:
    • Consumer receives the message and start processing it and dies (or hung) in between: In this case if the acknowledgement has not been send back to RabbitMQ message broker, the message will remain in queue and it has been send back to another consumer waiting for message
      • Solution: In the above case, developer needs to know if the business logic is processed or not. So developer needs to implement status of the message in either database or flat file with the message id. If same message is received after consumer restarts, developer can take a call to process again or not
    • Consumer receives message and processed it but other consumer receives same message again:
      • Solution: If you are facing this issue, then the problem lies in the heart-beat  timeout between consumer and RabbitMQ. So here consumer has received the message and its thread is processing it but due to missed heart-beat (due to network latency or whatever reason) RabbitMQ assumes the consumer has died and it closes the connection with consumer. Meanwhile as the consumer thread is active, it is still processing the message and completed it successfully but it was unable to acknowledge it back to RabbitMQ. The solution lies in increasing the heart-beat timeout so that it doesn't misses the heart-beat. 
Thats all about problems related to consumer receiving duplicate messages and its possible solutions. If you have query / suggestion, please feel free to post in comment section. Thanks.

Wednesday, 5 September 2018

Max Priority Queue Implementation in Java

In this post, we will see how to implement max priority queue in Java.

Background:
Priority Queue is an non-blocking unbounded queue, the elements of which are arranged in either natural order (if no comparator is provided) or comparator provided as part of constructor. In both scenario, the queue is arranged in ascending order and the head of queue will have least value. Honestly, this does not make sense for Priority Queue to arrange elements in ascending order which can be called as Min Priority Queue

Have the elements arranged in reversed order will make it Max Priority Queue

Implementation:

Now the question arises, how can we make the Priority Queue as Max Priority Queue with the api provided by Java.
Here is the code example for Max Priority Queue in Java


package com.blogspot.tech693;

import java.util.Collections;
import java.util.PriorityQueue;

public class MaxPriorityQueue{

 public static void main(String[] args){
  // Generic Priority Queue -- Natural order
  PriorityQueue pq = new PriorityQueue<>(6);
  pq.add(4);
  pq.add(2);
  pq.add(1);
  pq.add(6);
  pq.add(5);
  
  System.out.println("Generic / Min Priority Queue::" + pq.poll());
  System.out.println("Generic / Min Priority Queue::" + pq.poll());
  
  // Max Priority Queue Example
  PriorityQueue pq1 = new PriorityQueue<>(6, Collections.reverseOrder());
  pq1.add(4);
  pq1.add(2);
  pq1.add(1);
  pq1.add(6);
  pq1.add(5);
  
  System.out.println("Max Priority Queue::" + pq1.poll());
  System.out.println("Max Priority Queue::" + pq1.poll());
 }
 
}



Output:

Generic / Min Priority Queue::1

Generic / Min Priority Queue::2

Max Priority Queue::6

Max Priority Queue::5


Explanation:
In the above code example, we tried to achieve Priority Queue in descending order. I strongly believe the head of the queue should be the highest element. By using Collections.reverseOrder() method we can conver the Priority Queue into Max Priority Queue.

That's all for Max Priority Queue Implementation in Java. If you have questions / suggestions, please mention in comments sections. Thanks. 

Related Articles:

You may also like:

Sunday, 5 August 2018

Java when another new thread comes to synchronized block does it goes to wait state

In this post, I will discuss what happens when another new thread comes to synchronized block and what happens to its state.

Brief Introduction:

Lets start with understanding what is synchronized block. In Java, synchronized block is the set of code under curly brackets which can only be accessed by a thread holding lock of the object on which it is defined. For example

public class TestSynchronizedBlock {
 public void method1(){
  synchronized(this) {
   System.out.println("Inside synchronized block");
   // Do your stuff here
   System.out.println("Exiting synchronized block");
  }
 }
}


Consider the code above, in class TestSynchronizedBlock, inside method1 the synchronized block can be accessed by the thread who is holding lock for current object of TestSynchronizedBlock (as it is holding lock for this i.e. object of TestSynchronizedBlock class. Every Object (instance of class) in Java has one monitor lock (it is the part of Object class, thus is part of any class object) and the thread which first try to enter into synchronized block gets the monitor lock.

Conclusion:

As I said above, the thread which first try to enter into synchronized block has the exclusive monitor lock and it keeps the locks till the thread is executing inside the block. So if Thread A is executing inside synchronized block, what happens when Thread B try to execute. Thread B will go to wait state (waiting for lock) and it will release any other monitor lock it holds, if any. After Thread A will complete executing the synchronized block, it will release the lock and Thread B will get notified and will hold the monitor lock and then start executing (means it will come out of wait state and goes to Execution state). This holds true incase of multiple threads trying to acquire lock. It is worth to mention that If there is more than one thread waiting for lock and in wait state, after current thread completes the execution, only one thread out of waiting n thread will get notified and there is no such sequence as which one with get notified. You can also refer Spin lock for deep dive on locking mechanism.

That's all for synchronized block in context of multiple threads and its status. If you have any query / doubt / suggestion, please drop a comment.

Java Spin lock implementation

In this article we will see how to implement Spin Lock in Java.

Introduction:

Lets first understand what is Spin Lock. Spin Lock is a type of lock where the threads instead of getting blocked keeps looking for locks. In Spin Lock, the thread does not go into wait state. The thread remains in Runnable / Running state and continuously trying to get lock (say in while loop) and this is the reason it is named as Spin Lock.

Usecase / Advantage:

So the next question that pop ups in mind is that where to use the Spin Lock over Wait Lock. Here are the following use-cases:

  • Avoiding Deadlock : 

    • Spin Lock is another way to avoid Deadlock. Since the thread trying for lock does not go into wait state instead spinning for lock, using Spin Lock avoids deadlock.

  • When there is very small functionality between lock and unlock : 

    • Suppose I want to just write small functionality bounded by locks. If you use mechanism where thread goes to wait state (eg. Reentrant Lock ) and then back to Runnable state (after acquiring lock), this means you are making call cpu expensive for a thread going into wait and runnable state. Perhaps you could choose Spin Lock which is a kind of busy waiting for a very short period of time before acquiring lock.

Disadvantage:

Do-nothing loop : As Spin Locks constantly checking if locks are available, it keeps wasting CPU time which could be allocated to other productive thread. Thats why longer duration of spin lock is not recommended.

Code Example:

Here is the Code Example in Java for Spin Lock Implementation:


SpinLock.java
package com.blogspot.tech693;

import java.util.concurrent.locks.ReentrantLock;

public class SpinLock extends ReentrantLock{

public SpinLock() {
  super();
 }


 public void lock() {
  while(!super.tryLock()) {
    // Do Nothing
  }

 }

 public void unlock() {
  super.unlock();
 }

}

Thats all for Spin Lock Example in Java. If you have questions / doubts, please write it on comment sections Thanks.

Related Articles:

You may also like:

Tuesday, 17 July 2018

Introduction to SonarQube - Continuous code quality

In this post, we will see the uses and benefits of SonarQube.

Background
One of the important process in CI/CD is code scan wherein it scans the code for code smells, vulnerabilities, non-standard code etc.

Introduction
To maintain good code quality, it requires continuous scanning of code after each code-checkin. One of the important tool for code scanning is SonarQube which doesn't only scans the code but provide details about its standard and list out where all the issue with its quality. SonarQube job is to provide you with Blockers (i.e. code that is poorly written that can potentially cause issues with the application running in production), critical, major and minor. It also presents you with a value of how much standard your code is. SonarQube currently supports 21 language. Even you can scan your code directly without writing any linting configuration. 

Features
SonarQube scans the code by using methodology called as Quality Gates. This means each scan that we do on our code is by comparing our code with the quality gates. SonarQube has a default quality gate called “SonarQube way”. Whenever we scan the code, it is scanned against default quality gate “SonarQube way” and results are displayed. SonarQube also gives us the ability to configure our own Quality Gate.

SonarQube Quality gates are the best way to ensure that standards are met and regulated across all the projects and every developer oblige it in the organization.  Quality gates are defined with set of threshold measures set on your project like code coverage, technical debt measure (which can increase software entropy), number of blockers/critical issues, security rating, unit test pass rate etc.
 
In order to pass a Quality gate, the project should pass through each of the thresholds defined and it is just a set of conditions the project must meet before it can qualify for production release. When SonarQube runs it will identify if the code meets all the quality thresholds we have defined – else it will fail the quality gate and will not allow you to check in code to the Source code system. 

It also acts as a gatekeeper for bad code when it is integrated with repository tool like GitHub. GitHub provides a way to not allow merging of code (i.e. Pull request) if SonarQube scan fails. Using Quality Gates, we can also scan for new issues that may arise with our new code which is being written.

We can define a quality gate in SonarQube by using an option "create a Quality gate". This is the place where we configure the thresholds for the project / code to meet. After configuring quality gate, it need to be assigned to the project. After assigning, the subsequent scan on the project will be scanned against the quality gates.


That's all the brief introduction to SonarQube. If you have a question / comment, please mention in comment section. Thanks!

Saturday, 14 July 2018

[solved] local variable defined in an enclosing scope must be final or effectively final

In this article we will see how to resolve local variable defined in an enclosing scope must be final or effectively final error for lambda expression.


Background


Java 8 introduces lambda expression which makes an easy way to implement interface which has only one abstract method (also known as functional interfaces). It is a requirement for lambda expression that if it is using local variable, it must be effective final. 


Meaning of Effectively final:


Effectively final are those variable which value doesn't change after it has been set. The sudden question that came in our mind that if this is the meaning of effectively final, why doesn't we declare the variable with final keyword. The difference between effectively final (its a terminology and not a java keyword) and keyword final is that effectively final behaves like final variable even it has not been declared final. 
If you declare a variable as final, its value cannot be changed (will throw compilation error) while those variable which value has been set once and never changed (without declaring as final) is called effectively final. This has to be taken care by developer if he is considering any variable as effectively final because compiler will not throw exception in this case, if the coder changes the value (not in context of lambda expression).


Code Example and Solution:


Consider the below code example which will give the following compilation error:
local variable defined in an enclosing scope must be final or effectively final


Code with Compilation error:



@FunctionalInterface

interface  SingleMethodInterface{

    void value();

}



public class Test {  

    public static void main(String[] args) {

        int i = 10; // This should be effectively final because it is being used by lambda expression.

        // If the value of i changes, it will throw the exception.

        // lambda expression implementing the value method. This is just implementation of interface

        // Here intentionally changing the value of i (increasing by one)

        SingleMethodInterface singleMethodInterface = ()-> System.out.println("Value of i is " + ++i); 

        // Calling the implemented interface method. Since the value is changed, so it will give compilation error

        singleMethodInterface.value();

    }

}


Code with no error


@FunctionalInterface

interface  SingleMethodInterface{

    void value();

}



public class Test {  

    public static void main(String[] args) {

        int i = 10; // This should be effectively final because it is being used by lambda expression.

        // If the value of i changes, it will throw the exception.

        // lambda expression implementing the value method. This is just implementation of interface

        // Here no changing the value of i

        SingleMethodInterface singleMethodInterface = ()-> System.out.println("Value of i is " + i); 

        // Calling the implemented interface method. Since the value is not changed so no error here

        singleMethodInterface.value();

    }

}


That's all for resolution of error for lambda expression with respect to local variable. If you have any further query, feel free to write out in comment section.

Friday, 13 July 2018

How to encrypt string using ansible vault with example

In this article we will see how to encrypt string in Ansible using vault.

Introduction

Encrypting password, keys and variable is required to make system roboust. Using Ansible vault we can store the sensitive data and use the vault when running playbooks on remote machines. Using vault, the sensitive data remains safe.

How to encrypt string using vault

It is required to pass sensitive information if we are using Ansible as a configuration management system or a orchestration engine. For eg. we might need to pass password or ssh key while running playbook. By using vault, we can write sensitive data to a file which Ansible can read and utilize the data from within.

For these ansible provides a way to protect your data at rest. This feature is called vault, which allows user to encrypt text files, strings etc so that they are stored "at rest" in encrypted format. Without the key or a significant amount of computing power, the data is indecipherable.ansible-vault command is provided by Ansible in securing stuff.

Code Example


[root@server1 vault]# ansible-vault encrypt_string --vault-id a_password_file 'pass123' --name 'user1'


Result:

user1: !vault |

      $ANSIBLE_VAULT;1.1;AES256

      64576845869345693586856858858685086748057683457845788957865897856045876085707851

      20364856840580208230475024378508247502483650489658498568436856486584685684658445

      34085620345824852034850824598622865984065893460598263846598406895248658094856249

      23452345
 

If you want to use vault-id label then:

[root@server1 vault]# ansible-vault encrypt_string --vault-id myuser@password 'pass123' --name 'user1'
  

Result:

user1: !vault |

      $ANSIBLE_VAULT;1.1;AES256;myuser

      64576845869345693586856858858685086748057683457845788957865897856045876085707851

      20364856840580208230475024378508247502483650489658498568436856486584685684658445

      34085620345824852034850824598622865984065893460598263846598406895248658094856249

      23452345
 

This is how we can encrypt string in Ansible using vault. If you have any query related to Ansible vault, feel free to ask in comment section. Thanks.