Web security: Understanding different type of injection attacks

Introduction:

In this article, I will try to explain different type of injection attacks that can be used to exploit a web application. Injection is one of the most dangerous attacks that targets web applications. The current OWASP Top 10 list still lists the injection in top three web application security risks. All injection attacks are based on using user supplied untrusted data in an interpreter as part of a command or query.

If you want to see hands-on examples with these injection attacks and want to hack the vulnerable applications you have developed, you can check out my new course: Web security: Injection Attacks with Java & Spring Boot

Let’s summarise the 5 widely used injection attacks in web applications: SQL , NoSQL, LDAP, LOG and CSV Injections.

SQL Injection:

SQL injection occurs when user supplied untrusted data send to a database engine as part of a command or query.

As a result of SQL injection attacker can bypass authentication, disclosure data on database, cause alter or delete data which will break data integrity, run system commands on database server.

An example SQL Injection attack using an always true condition;

Let’s say an application uses below sql query as a string variable;

sql = select username, password from users where username=’%s’;

For above query if an attacker provides a username as xxx’ or ‘1’=’1;

The resulting query will be as below;

select username, password from users where username=’xxx’ or ‘1’=’1’

Here, or ‘1’=’1 part is an always true condition, and it will result a true where condition which will return all username/password pairs from the database.

To defend against SQL injection at least the following protections should be used;

• Using a strong validation and sanitisation mechanism
• Using parametrised queries with prepared statements

NoSQL Injection:

In NoSQL injection attacker inject the code into commands to execute in the NoSQL engine.

NoSQL query syntax is specific to the NoSQL engine. Usually, the queries are written in the programming language of the application like JavaScript or Java. For example in MongoDB, Injecting JavaScript code using $where query operator is possible.

To defend against NoSQL injection at least the following protections should be used;

• Using a strong validation and sanitisation mechanism
• Using parametrised queries
• Using built-in features of the NoSQL engine to create secure queries

LDAP Injection:

In LDAP injection attacker inject the code into commands to execute in the LDAP engine.

LDAP require queries with some special characters to be used in construction of predicates. If an attacker can send an input directly as a control character the query can be changed to do unwanted operations.

To defend against LDAP injection at least the following protections should be used;

• Using a strong validation and sanitisation mechanism
• Using parametrised queries
• Using built-in features and trusted libraries to create LDAP queries

LOG Injection:

In LOG injection the attacker inject code into logs which can lead to different vulnerabilities by polluting or compromising the system logs.

Through LOG injection an attacker can perform;

• Log forging by using CRLF(\n\r) characters
• XSS attacks
• Code execution

To defend against LOG injection at least the following protections should be used;

• Using a strong validation and sanitisation mechanism
• At least filtering out or replacing CRLF characters before printing in log
• Limiting the size of the log messages
• Using available protections in the log framework and from trusted libraries
• Preventing to save or view the Javascript content through logs

CSV Injection:

CSV Injection, also known as Formula Injection, occurs when websites embed untrusted input inside CSV files.

CSV injection can be used to steal data from CSV or Excel files and running formulas to run commands on the user’s machine that is viewing the document.

To defend against CSV injection at least the following protections should be used;

• Using a strong validation and sanitisation mechanism
• Never directly input user provided data into files that includes formula definition
• Wrap each cell field in double quotes
• Prepend each cell field with a single quote
• Escape every double quote using an additional double quote

Conclusion:

An insecure software can easily cause serious consequences for businesses and individuals. Those consequences can include financial abuse, theft and loss of reputation. Vulnerable software will be especially cause huge damages on important industries such as finance and healthcare. Someone can try to hack your application on purpose, or a user can compromise it unintentionally. Therefore it is always needed to program defensively to prevent compromising your application in either case.

To secure web application and prevent injection vulnerability;

• Separate user provided data from the executable content
• Validate and escape user provided data before using it
• Run the web application with the least privilege that is needed
• Always follow defense-in-depth methodology to prevent the attacks in multiple layers.

If you want to see hands-on examples with these injection attacks and want to hack the vulnerable applications you have developed, you can check out my new course: Web security: Injection Attacks with Java & Spring Boot

In this course you will learn web security and perform different type of injection attacks using the following steps in each section.

• Development of the vulnerable web application using Java, Spring boot and Spring security
• Hacking of the application with various attack payloads and with Ethical hacking examples
• Protection steps and the implementations to prevent injection attacks

Event-Driven Architecture: Why and How to use EDA?

Introduction:

Microservices architecture leads to build highly scalable, available and resilient systems while developing and deploying each service independently, possibly using different technologies. While microservices have a lot of advantages it also brings some level of difficulty when it comes to tracing, testing, logging and especially with data sharing, transactions.

 

When working with microservices each service should own its data. This will prevent single point of failure with other services, like using a common data store. However it will also come with three problems;

• Sharing data between services
• Running transactions across services. 
• Scalability requires to replicate data store in each instance of a service, otherwise it will lead to another single point of failure; the single data store shared by instances of a service

If you want to see an event-driven microservices architecture in live, you can check my course on udemy: Event-Driven Microservices: Spring Boot, Kafka and Elasticsearch

To share data Rest calls can be used. However a Rest call will be a synchronous operation, because Rest uses Http to communicate, and Http is a synchronous protocol which requires to send a request and wait for a response immediately, blocking the caller. As you can imagine with multiple services communicating over the network this would cause significant problems. There can be delays causing higher response times, and if there is no timeout applied properly on caller service, that can block indefinitely. 

In addition to that, to query the data on a specific service, we will be restricted with the abilities of the data store’s query capability on that specific service. For example, the data store could be Cassandra or Elasticsearch, or a Relational database, and the data you want to query will be stored differently in each case. This may create differences in the returned data format and on the performance of a query.

 

For transactions across services, distributed transactions (two-phase commit - 2PC) can be used. However this will require the data stores in each service to support 2PC, which may not always be the option. Besides, according to the CAP theorem we cannot choose consistency, availability and partition tolerance at the same time. In most cases choosing availability and partition tolerance is a must, so we need to find a solution other than a distributed transaction that requires traditional consistency. This solution will lead to an eventual consistency.

 

The main problem resides in above mentioned points is the strong coupling of different services. Could there be a better way to decouple services truly?

 

And for the third problem above, scalability, could there be a way to remove the state from the service and scale much more easily? 

EDA comes to rescue:

An event-driven architecture (EDA), when implemented correctly, can be used to resolve the issues mentioned above. It leads to accomplish true decoupling of services, and to build to a better microservices architecture with more resilient services. In essence EDA helps to;

• Create resilient services, as a service has no direct communication with other services. When a target service is down, it will not affect the caller service because of the decoupling.
• Use asynchronous communication between services. A target service will produce/create an event at a time. Then the caller service can consume/read that event at a later time. It also allows to adjust the consumption according to caller service processing power and leads to replay the events if needed.
• Favour to use a state store for all services, removing the state from service and leading to better scalability.

To better explain how EDA helps to create more resilient and asynchronous communicated services, let’s go into the details of EDA a bit.

 

First of all, let’s discuss what is an event. An event is basically a change of state on the system that can be recognized, reacted and processed.

Since the events are the core components of EDA, we can ask how events help EDA to be asynchronous and non-blocking? An event is always created by a producer, and can be consumed by one or more consumers. The important thing is that those producers and consumers are completely decoupled and don't need to know each other. An event can directly go to a subscribed consumer or go to an event store if used, such as Kafka, which is the preferred way most of the time. In any case, events will be one-way, and never expect a response. Consider a communication that uses Http protocol, such as Rest. Such an Http call needs to include a request and a corresponding response which be a synchronous operation. Since an event never needs a response, an architecture that uses it such as EDA will naturally be a non-blocking/asynchronous system.

 

Using an event store in EDA will make the things easier, as all producers and consumers only need to know about the event store. Besides, by keeping all events in the event store we can make the services truly stateless. That will then lead to truly scalable services as without a state creating a new instance of a service will be much easier. When we say an event store that is used by all producers and consumers, you may think that we are creating another single point of failure. That might be true if the event store solution is not an easily scalable and resilient solution.

 

I would favour Kafka at this point as an event store, being an event-log, and a fault-tolerant and naturally scaling solution thanks to built-in partioning. From the Kafka documentation;

 

Event sourcingis a style of application design where state changes are logged as a time ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.

 

Note that, any new type of event will be a corresponding topic in kafka terms.

Let's leave the details of Kafka, as general and as an event store, for another post and continue with the details of EDA.

With a scalable and resilient event store, consumers will be able to replay an event, favouring availability of the system, when there is an error or simply when they are offline. 

 

One important aspect of an event is that it must be immutable. Therefore the event store should also support immutability, which can be achieved by the concept of event logs. In an event log you cannot change and delete an event after it occurred. Any update to the state of an application will be a new event. The latest state of an application can be obtained by replaying the events in all history. 

Now, let’s remember the issues we listed at the beginning of this post. We asked;

• Could there be a better way to decouple services truly?
• Could there be a way to remove the state from the service and scale much more easily? 

And pointed the problems about;

• Sharing data
• Transactions across services
• Scaling

 

Here are the solutions to these problems by EDA;

• Truly decouple the services. They don’t need to know about each other. Being producers and consumers that are creating and consuming events, the services only need to know about the event store.
• As the states will be logged in the event store, the stateless services can easily scale by just creating new instances.
• Can share data asynchronously using events and an event store. Note that a materialized view is always an option for pre-loaded joined data, across services. This materialized views can be constructed as a local database. To keep this view up to date a consumer needs to be defined by subscribing to the corresponding event in the event store.
• Can handle transactions that span multiple services in an eventual consistency manner using events. Such as a producer creates an event for a change and the corresponding consumer will react to this event to complete a transaction asynchronously.

Additionally, EDA has also below benefits for a microservices architecture if implemented with a sophisticated event broker.

• Easy asynchronous communication of services
• No Api change issue, as no direct call between services
• Independent development of components, i.e; no API dependency to other services
• Better resiliency and fault-tolerant
• Better scaling with stateless services
• Reduced complexity by giving some responsibilities to event broker

Conclusion:

 

Having the ability to create truly decoupled services, EDA is a perfect match for reliable, fault-tolerant and better scalable microservices architecture by solving general issues raised like communication of services and api changes. Events can be any real time changed state, such as signals created by sensors, dropped packets on networks, security violations on file systems, or creating tweets on twitter. EDA can be used in all types of applications that can be pictured using events.

As you can imagine not all systems can completely fit into an event driven, asynchronous and eventual consistent architecture. In that case EDA can be combined with a synchronous solution like REST in case there is a need to do an operation with traditional consistency manner, following ACID transactions.

 

If you want to see an event-driven microservices architecture in live, you can check my course on udemy: Event-Driven Microservices: Spring Boot, Kafka and Elasticsearch

Understanding Type of References in Java

Introduction:

       There are four types of references in Java and for each of them garbage collector behaves differently.

- Strong References

- Weak References

- Soft References

- Phantom References

Below I will try to explain each type of reference's characteristics and possible use cases.

  

Strong References:

     The default type of reference is the strong reference. When you define an object regularly it will have strong reference and as long as an object has a strong reference the object will not be eligible for garbage collector. When we set the strong variable to null, that object will become eligible to garbage collection and as long as GC runs the space that object uses will be reclaimed. 

Let's see a strong reference behaviour with an example.

Below StrongReference class has a nested class named A. It creates an instance of A and set it to null. At that time that object should be eligible for GC.

package references;

import java.io.File;
import java.util.Objects;

public class StrongReference {
    private static final int RETRY = 3;
    public static void main(String[] args) throws InterruptedException {
        deleteOldDumps();
        A a = new A("Strong Reference");
        references.HeapDump.dumpHeap("java/heap-dumps/strongRefBeforeGCEligible.hprof", false);
        a = null; // Make the strong reference eligible for GC
        references.HeapDump.dumpHeap("java/heap-dumps/strongRefBeforeGC.hprof", false);
        runGC();
        references.HeapDump.dumpHeap("java/heap-dumps/strongRefAfterGC.hprof", false);
    }

    static class A {
        A(String s) {
            this.s = s;
        }
        String s;
    }

    static void deleteOldDumps() throws InterruptedException {
        boolean isDeleted = deleteFiles();
        int attempt = 0;
        while (!isDeleted && attempt++ < RETRY) {
            isDeleted = deleteFiles();
            Thread.sleep(1000 * attempt);
        }
    }

    private static boolean deleteFiles() {
        File dir = new File("java/heap-dumps");
        File[] files = dir.listFiles();
        for (File file : Objects.requireNonNull(files)) {
            file.delete();
        }
        return files.length != 0;
    }

    static void runGC() throws InterruptedException {
        System.out.println("Running GC..");
        System.gc(); // Hint to run gc
        Thread.sleep(2000L); // sleep hoping to let GC thread run
        System.out.println("Finished running GC..");
    }
} 

Here we use "com.sun.management:type=HotSpotDiagnostic" mbean to dump memory to later inspect it with jhat command line utility. Remember to delete the previous dump files with deleteOldDumps method otherwise heap dump will give "File exists" error.

We use this class to use HotSpotDiagnostic mbean and dump memory to a hprof file.

Note that the second parameter of dumpHeap method is a boolean and if we set it true it will only dump live objects which will cause running GC before taking heap dump so that we do not need to hint with System.gc(). In this example we set it the second parameter to false and try to hint GC to run with System.gc() instead.

Note that to monitor GC runs with System.gc() call, run the class with "-verbose:gc" JVM parameter so that we can see the GC running in logs.

We take 3 dumps file before setting the object to null, after setting to null and after hinting to run GC.

Now let's run the class and follow the steps below using jhat command to see our memory snapshot through web interface.

- Run "jhat -J-Xmx1g -port 8000 strongRefBeforeGCEligible.hprof"  with different ports for all 3 hprof files.

- Open localhost:8000/histo (we will use the heap histogram to examine the references)

The output of the program is:

Running GC..
[GC (System.gc())  21053K->1256K(502784K), 0.0010238 secs]
[Full GC (System.gc())  1256K->979K(502784K), 0.0061717 secs]
Finished running GC..

And the web interface of jhat we will see the below 3 results for 3 different hprof files.

Class class references.StrongReference$A	Instance Count 1	Total Size 8

                             strongRefBeforeGCEligible.hprof(localhost:8000/histo/)

Class class references.StrongReference$A	Instance Count 1	Total Size 8

                             strongRefBeforeGC.hprof(localhost:8001/histo/)

Class class references.StrongReference$A	Instance Count 0	Total Size 0

                             strongRefAfterGC.hprof(localhost:8002/histo/)

As we expected after GC runs the non-referenced object of class A reclaimed by GC.

  

Weak References:

         A weak reference can be created with java.lang.ref.WeakReference class. If JVM sees an object has only a weak reference during GC it will be collected.

There is a get() method of WeakReference which returns object itself if it is not garbage collected yet or returns null if it is removed already.

Weak references can be used to prevent memory leaks. For example for a temporary object which will be used in the application for a while, if you want to keep some additional information, you generally use a global map with temporary object as key and additional information as value. This can be seen as a in-memory cache. With a strong referenced map, even if the temporary object lifecycle ends, since we a reference from the map, the temporary object and the additional info will remain in heap for more time than the expected lifecycle of the object. This causes memory leak and it can be prevented using Weak references. Fortunately we do not need to implement this as java has a built-in WeakHashMap class and if you use this map GC will collect the temporary object if the only reference to this object is from WeakHashMap.

Note that WeakHashMap can hold a ReferenceQueue which is defined during creation of Weak reference. This queue will hold the reference object (not the referent) after GC runs. We can poll the reference after the referent garbage collected. Before GC collect the referent queue poll will return null.

Below is an example of using WeakHashMap and WeakReference.

package references;

import java.lang.ref.Reference;

import java.lang.ref.ReferenceQueue;

import java.lang.ref.WeakReference;

import java.util.ArrayList;

import java.util.List;

import java.util.Map;

import java.util.WeakHashMap;

import static references.StrongReference.runGC;

import static references.StrongReference.deleteOldDumps;

public class WeakReferenceTest {

    public static void main(String[] args) throws InterruptedException {

        deleteOldDumps();

        //1.) WeakHashMap example

        System.out.println("1.) WeakHashMap example:");

        A a = new A("a");

        List<Object> objects = new ArrayList<>();

        B b = new B(objects);

        Map<A, B> map = new WeakHashMap<>();

        map.put(a, b);

        a = null;

        b = null;

        //Note that B will be GC'ed after the second GC because expungeStaleEntries method

        //call needed to remove entry first as it holds a ref to b

        System.out.println("Map size before gc: " + map.size());

        runGC();

        //expungeStaleEntries method cleaned up the entry with null key. 

        //We expect map size as 0 

        System.out.println("Map size after gc: " + map.size());

        System.out.println("--------------------------------------------------------");

        System.out.println("2.) WeakReference example:");

        //2.) WeakReference example

        //Do not use String a referent of weak ref.

        //Since it is pooled you cannot observe weakreference.

        //WeakReference's referent property is already null after GC

        //as opposed to PhantomReference(prior to java 9)

        A a2 = new A("a");

        ReferenceQueue<A> referenceQueue = new ReferenceQueue<>();

        Reference<A> weakReference = new WeakReference<>(a2, referenceQueue); 

        HeapDump.dumpHeap("java/heap-dumps/weakRefBeforeGCEligible.hprof", false);

        a2 = null;

        System.out.println("Reference.get() before gc: " + weakReference.get());

        System.out.println("Referencequeue.poll() before gc: " + referenceQueue.poll());

        System.out.println("is enqueued: " + weakReference.isEnqueued());

        HeapDump.dumpHeap("java/heap-dumps/weakRefBeforeGC.hprof", false);

        runGC();

        HeapDump.dumpHeap("java/heap-dumps/weakRefAfterGC.hprof", false);

        System.out.println("Reference.get() after gc: " + weakReference.get());

        System.out.println("is enqueued: " + weakReference.isEnqueued());

        Reference polledRef = referenceQueue.poll(); 

        System.out.println("Refs are equal: " + (weakReference == polledRef));

        System.out.println("ReferenceQueue.poll() after gc: " + polledRef); 

       //referent is already cleared and we expect null here

        System.out.println("Referent inside reference after gc: " + polledRef.get());

        System.out.println("--------------------------------------------------------");

        System.out.println("3.) Getting inner information from weak reference" +

                " after the referent garbage collected");

        //3.) Reaching some inner information through weakreference after

        //referent the object garbage collected

        ReferenceQueue<A> referenceQueue2 = new ReferenceQueue<>();

        A a3 = new A("a3");

        String innerInfo = "Inner info";

        WeakA weakA = new WeakA(a3, innerInfo, referenceQueue2);

        a3 = null; 

        runGC();

        //should not be null but the referent inside it automatically

        //removed and polledRef.get() returns null as it is garbage collected 

        polledRef = referenceQueue2.poll();

        System.out.println("Referent inside reference after gc: " + polledRef.get());

        weakA = (WeakA) polledRef;

        System.out.println("Get inner information from the polled reference: "

                 + weakA.innerInfo);

    }

    static class A {

        public A(String s) {

            this.s = s;

        }

        String s;

    }

    static class WeakA extends WeakReference<A> {

        String innerInfo;

        public WeakA(A a, String innerInfo, ReferenceQueue<A> queue) {

            super(a, queue); 

            this.innerInfo = innerInfo;

        }

    }

    static class B {

        List<Object> largeObject;

        public B(List<Object> largeObject) {

            this.largeObject = largeObject;

        }

    }

The output of the program is:

1.) WeakHashMap example:
Map size before gc: 1
Running GC..
[GC (System.gc())  10526K->584K(502784K), 0.0008749 secs]
[Full GC (System.gc())  584K->375K(502784K), 0.0034859 secs]
Finished running GC..
Map size after gc: 0
--------------------------------------------------------
2.) WeakReference example:
Reference.get() before gc: references.WeakReferenceTest$A@6e5e91e4
Referencequeue.poll() before gc: null
is enqueued: false
Running GC..
[GC (System.gc())  23202K->1551K(502784K), 0.0012596 secs]
[Full GC (System.gc())  1551K->1284K(502784K), 0.0066106 secs]
Finished running GC..
Reference.get() after gc: null
is enqueued: true
Refs are equal: true
ReferenceQueue.poll() after gc: java.lang.ref.WeakReference@30946e09
Referent inside reference after gc: null
--------------------------------------------------------
3.) Getting inner information from weak reference after the referent garbage collected
Running GC..
[GC (System.gc())  9179K->1420K(502784K), 0.0007208 secs]
[Full GC (System.gc())  1420K->1284K(502784K), 0.0074525 secs]
Finished running GC..
Referent inside reference after gc: null
Get inner information from the polled reference: Inner info

The first part shows an example usage of WeakHashMap. We see that the map size is one before GC but it is zero after GC as the key set to null before GC and it is collected because the only reference to it is the Weak reference from the WeakHashMap. Note that normally collecting the key will not remove the Entry from the map but we see that the size of map is zero. This is because there is a expungeStaleEntries method in WeakHashMap implementation which is called during ordinary map operations like size/put. This method iterates through ReferenceQueue of WeakReference and if it finds a reference get the hash of the reference from it and use that has to find and remove the Entry object from the map itself which in turn let the value of the entry to be garbage collected in the next GC cycle (In above example it is a B object). It is more effective than running through all map and remove an entry if key is null. It uses a technique similar to the third part of the above example. By holding a reference object which extends WeakReference we can add some additional information to the reference itself, and even though the inner referent inside the reference objects is cleared (after referent garbage collected) we can still reach that inner additional information to use it later, like using the hash to remove the Entry in the map.

Lastly the second part of the above example shows a basic WeakReference example where the object is garbage collected after the only reference to it is a weak reference as we see from the below jhat results.

And again the web interface of jhat we will see the below 3 results for 3 different hprof files. 

Class class references.WeakReference$A	Instance Count 1	Total Size 8

                              weakRefBeforeGCEligible.hprof(localhost:8000/histo/)

Class class references.WeakReference$A	Instance Count 1	Total Size 8

                              weakRefBeforeGC.hprof(localhost:8001/histo/)

Class class references.WeakReference$A	Instance Count 0	Total Size 0

                              weakRefAfterGC.hprof(localhost:8002/histo/)

Soft References:

       A soft reference can be created with java.lang.ref.SoftReference class. A soft reference does garbage collected only if there is not enough memory left in heap. That means it can be used for memory sensitive caches where we do not want it to be removed from memory because reading from the source is more expensive. 

Note that if an application uses almost all allowed heap memory, we possibly not get that much advantage from using a cache as an almost full memory will make our application already slow in processing and response times as it will require a lot of swap between memory and disk. However soft references can still be used instead of strong references in some circumstances where we want to prevent out of memory errors.

package references;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import static references.StrongReference.deleteOldDumps;
import static references.StrongReference.runGC;
public class SoftReferenceTest {
    public static void main(String[] args) throws InterruptedException {
        deleteOldDumps();
        A a = new A("a");
      ReferenceQueue<A> referenceQueue = new ReferenceQueue<>();
        Reference<A> softReference = new SoftReference<>(a, referenceQueue);
        HeapDump.dumpHeap("java/heap-dumps/softRefBeforeGCEligible.hprof", false);
        a = null;
        System.out.println("Reference.get() before gc: " + softReference.get());
        System.out.println("Referencequeue.poll() before gc: " + referenceQueue.poll());
        System.out.println("is enqueued: " + softReference.isEnqueued());
        HeapDump.dumpHeap("java/heap-dumps/softRefBeforeGC.hprof", false);
        runGC();
        HeapDump.dumpHeap("java/heap-dumps/softRefAfterGC.hprof", false);
        System.out.println("Reference.get() after gc: " + softReference.get());
        System.out.println("is enqueued: " + softReference.isEnqueued());
        Reference polledRef = referenceQueue.poll();
        System.out.println("ReferenceQueue.poll() after gc: " + polledRef);
    }

    static class A {
        String s; 

        public A(String s) {
            this.s = s; 
       }
   }
}

The output of the program is:

Reference.get() before gc: references.SoftReferenceTest$A@5387f9e0
Referencequeue.poll() before gc: null
is enqueued: false
Running GC..
[GC (System.gc())  26317K->1544K(502784K), 0.0016028 secs]
[Full GC (System.gc())  1544K->1300K(502784K), 0.0070209 secs]
Finished running GC..
Reference.get() after gc: references.SoftReferenceTest$A@5387f9e0
is enqueued: false
ReferenceQueue.poll() after gc: null

If we check the jhat result after GC, we will see soft reference is not cleared.

Class class references.SoftReference$A	Instance Count 1	Total Size 8

                               softRefAfterGC.hprof(localhost:8002/histo/)

Phantom References:

        A phantom reference can be created with java.lang.ref.PhantomReference class. Unlike weak and soft references whereby we can control how objects garbage collected, a phantom reference is used for pre-mortem cleanup actions before GC remove the object. 

It mainly used for a replacement for finalize method which is unreliable and can slow down the application as JVM uses separate thread pool for finalization and an object with a finalize method consumes more resources than a normal object. 

Another problem with finalize method is that it allows the objects to be resurrected during finalization and for this reason at least two GC cycles need to be run as first GC makes the object finalizable and the second GC reclaimed object if it is not resurrected. And between this two GC cycles finalizer thread must have been run to actually reclaim the object. If the finalizer thread does not run, more than two number of GC can run and this means the object will wait to be reclaimed for a long time although it is not used anymore which can cause an out of memory exception even though most the objects in heap are garbage.

And as per doc;

        You should also use finalization only when it is absolutely necessary. Finalization is a nondeterministic -- and sometimes unpredictable -- process. The less you rely on it, the smaller the impact it will have on the JVM and your application.

  

A phantom reference is enqueud to the reference queue by garbage collector after it determines that the referent object is phantom reachable that is has no reference left.

Note that the time the reference is enqueued is not certain and it can be any time after GC determines the referent is to reclaimable.

Note that to prevent resurrection, phantom reference get method always return null. Also prior to java 9, unlike weak and soft references, phantom referenced objects (referent) are not automatically cleared and the referent will stay in memory either until calling clear method to clear it or the reference itself being garbage collected. With java 9 the referent is automatically cleared when the reference is enqueued.

Note that creating a phantom reference without a reference queue is useless as the get method returns null and since there is no queue it will never be enqueued.

As an example of using phantom reference to clean up objects you can check the FileCleaningTracker class from apache commons which uses PhantomReference to delete the file physically if the object representing the file is cleaned up.

Another use case apart from doing a pre-mortem cleanup action could be determining the time to load a heavy object after another one cleaned up. Since a phantom reference queued on a reference queue tells us the referent itself happens to be garbage collected we can use that information to load another object. However especially before java 9 where the referent still referenced from the phantom reference itself even after you see the reference in queue we cannot be sure the object really claimed in terms of memory. Even after java 9 seeing the reference in reference queue does not tell us certainly that the space that object is using really claimed. Instead it means the space hold by the referent object will happen to be reclaimed and there is no way to resurrect it so we can do whatever cleanup operation we want.

Let's see an example usage of PhantomReference below where we do a cleanup operation.

Note that we used MyFinalizer class which extends PhantomReference and holds some additional information to be used in clean up after the object phantom reachable.

Also note that we used a shutdown hook as a separate thread and force the phantom reference monitoring thread to finish before the application exit. For that purpose we used a CountDownLatch in shutdown hook which is a barrier allowing us to wait another thread to finish.

Finally, in the monitoring thread we read the reference from reference queue, do the cleanup operation with the information we set during the creation of reference. Note that we need to call the clear method of reference after we finish clean up so that the referent will be set to null and ready to clean up by GC. If we do not call clean method on reference we will see the referent is still there. See the shutdown hook thread code which uses reflection to get the referent. (Do not use reflection to get the referent when using phantom reference, because it can break the promise of not resurrection of phantom reference).

If you run the same code with a jdk version 9 or greater you will see that the referent is automatically set to null and you do not need to call reference.clear() method anymore.

package references;

import java.lang.ref.PhantomReference;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import static references.StrongReference.deleteOldDumps;
import static references.StrongReference.runGC;
public class PhantomRefTest {
    private volatile static boolean finishing = false;
    private static final long WAIT_TIME_FOR_REMOVING_PHANTOM_REFERENCE_MS = 5000L;

    public static void main(String[] args) throws InterruptedException {
      deleteOldDumps();
      PhantomRefTest phantomRefTest = new PhantomRefTest();
      A a = phantomRefTest.new A("a");
      ReferenceQueue<A> referenceQueue = new ReferenceQueue<>();
      String someInfoToUseInCleanUp = "some info";
      Reference<A> reference = phantomRefTest.new MyFinalizer(a, someInfoToUseInCleanUp, referenceQueue);
      HeapDump.dumpHeap("java/heap-dumps/phantomRefBeforeGCEligible.hprof", false);
      a = null;
      System.out.println("Phantom reference always return null: " + reference.get());
      System.out.println("Phantom reference queue return null before GC: " + referenceQueue.poll());
      HeapDump.dumpHeap("java/heap-dumps/phantomRefBeforeGC.hprof", false);
      runGC();
      HeapDump.dumpHeap("java/heap-dumps/phantomRefAfterGC.hprof", false);
      CountDownLatch countDownLatch = new CountDownLatch(1);
      Runtime.getRuntime().addShutdownHook(new Thread(() -> {
         System.out.println("Caught shutdown hook, finishing phantom reference monitoring!");
         finishing = true;
         try {
          countDownLatch.await();
          HeapDump.dumpHeap("java/heap-dumps/phantomRefAfterCleanUp.hprof", false);
          //if you check internal referent here (by looking with debugger), i
          //it will be null as we cleared it.
          //If we would not call finalizer.clear() in monitoring thread here we will see the object value.
          //Note that after java 9 the referent cleared internally not waiting the 
          //phantomref object to be cleared.
          System.out.println("Getting internal referent by reflection as reference.get() always null"); 
          Field referentField = Reference.class.getDeclaredField("referent");
          referentField.setAccessible(true);
          A referent = (A) referentField.get(reference);
          System.out.println("Check referent after clean up: " + (referent == null ? null : referent.s));
          } catch (InterruptedException | NoSuchFieldException | IllegalAccessException e) {
              e.printStackTrace();
         }
        System.out.println("Finished all running threads. Exiting!");
        }));
        phantomRefTest.monitorPhantomReference(referenceQueue, reference, countDownLatch);
    }
private void monitorPhantomReference(ReferenceQueue<A> referenceQueue, Reference<A> reference,
 CountDownLatch latch) {
        ExecutorService executorService = null;
        try {
            executorService = Executors.newSingleThreadExecutor();
            Runnable runnable = () -> {
                try {
                    while (!finishing) {
                        try {
                            System.out.println("Start reading phantom reference from queue!");
                            MyFinalizer finalizer = (MyFinalizer) referenceQueue.
                                    remove(WAIT_TIME_FOR_REMOVING_PHANTOM_REFERENCE_MS);
                            if (finalizer != null) {
                                System.out.println("We got the phantom referenced object in the queue");   
                                System.out.println("Reference queue return the same ref object: " +
                                        (finalizer == reference));
                                System.out.println("Phantom reference always return null: " +
                                        finalizer.get());
                                finalizer.cleanUp();
                                //if you do not call this, it will be cleared after phantomref 
                                //object itself cleared. It is cleared automatically after java9
                     finalizer.clear();
                           } catch(IllegalArgumentException e){
                                e.printStackTrace();
                                continue;
                            } catch(InterruptedException e){
                                continue;
                            }
                        }
                        System.out.println("Exiting phantom reference monitoring thread!");
                    } catch(Exception e){
                        e.printStackTrace();
                    } finally{
                        System.out.println("Counting down the latch in phantom
 reference monitoring thread!");
                        latch.countDown();
                    }
                } ;
                executorService.submit(runnable);
            } catch(Exception e){
                e.printStackTrace();
            } finally{
                if (executorService != null) {
                    executorService.shutdown();
                }
            }
        }

        private class A {
            public A(String s) {
                this.s = s;
            }
            String s;
        }
        /** The object will be phantomly reachable only when you have
 no reference from your app. */
        private class MyFinalizer extends PhantomReference<A> {
            String someInfoToUseInCleanUp;
            public MyFinalizer(A referent, String someInfo, ReferenceQueue<? super A> q) {
                super(referent, q);
                this.someInfoToUseInCleanUp = someInfo;
            }
            public void cleanUp() {
                System.out.println("Clean up resources with info " + someInfoToUseInCleanUp);
            }
        }
    }

The output of the program is;

Phantom reference always return null: null

Phantom reference queue return null before GC: null

Running GC..

Finished running GC..

Start reading phantom reference from queue!

We got the phantom referenced object in the queue

Reference queue return the same ref object: true

Phantom reference always return null: null

Clean up resources with info some info

Start reading phantom reference from queue!

Start reading phantom reference from queue!

Caught shutdown hook, finishing phantom reference monitoring!

Exiting phantom reference monitoring thread!

Counting down the latch in phantom reference monitoring thread!

Getting internal referent by reflection as reference.get() always return null

Check referent after clean up: null

Finished all running threads. Exiting!

Note that we run the program from command line and exit with ctrl+c when the monitoring thread is running inside the loop. And we see that the shutdown thread set finishing to true and waits the monitoring thread to exit so that no code interrupted but finished as it should finish.

The web interface of jhat we will see the below 5 results for 5 different hprof files.

Class class references.PhantomReference$A	Instance Count 1	Total Size 16

                            phantomRefBeforeGCEligible.hprof(localhost:8000/histo/)

Class class references.PhantomReference$A	Instance Count 1	Total Size 16

                            phantomRefBeforeGC.hprof(localhost:8001/histo/)

Class class references.PhantomReference$A	Instance Count 1	Total Size 16

                            phantomRefAfterGC.hprof(localhost:8002/histo/)

Class class references.PhantomReference$A	Instance Count 0	Total Size 0

                            phantomRefAfterCleanUp.hprof(localhost:8003/histo/)

Class class references.PhantomReference$A	Instance Count 1	Total Size 16

                            phantomRefAfterCleanUp.hprof(localhost:8004/histo/)
                               (for the case reference.clear() method not called)

Note that we have a total size greater than the previous examples (16 bytes whils it was 8). This is because we used a non-static inner class which has reference to the outer class. For more information about object retained spaces you can check this post.

From jhat result we see that instance is still occupy space even after GC run. Because the first GC makes the referent phantom reachable and let you do cleanup operation by getting the reference from the queue. After cleaning up the reference by reference.clear method, if we run GC we will see that the object is really cleaned up from memory.

If we run the same class with jdk11, we will see that the object is already cleared when we get the reference from reference queue (without calling the reference.clear() method). We can now see the contents of hprof file using visualVM as jhat is removed starting with jdk9. If we look at the result of phantomRefAfterGC.hprof, wee see that the referent is cleared even though clear method not called explicitly unlike the results of jdk8.

This tells us that with jdk version greater than 8 we might now use the phantom reference even for loading a large object when we can get the previous object reference as at that time it is removed from memory. But be sure checking the memory snapshot for your environment to be sure the object is really removed.

phantomRefAfterGC.hprof(with jdk11)

Note that with jdk 11 our reflection code will get the below warning. This implies in coming releases reflection might not work anymore. 

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by references.PhantomRefTest (file:/myblog/out/production/java/) to field java.lang.ref.Reference.referent

WARNING: Please consider reporting this to the maintainers of references.PhantomRefTest

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Conclusion:

       We see different types of references in java. An ordinary reference is a strong reference which is the default type for all java objects. Weak and soft references help to control how objects are garbage collected while phantom reference helps to do some clean up operation when GC decides to remove and object from memory. You can find the source code for this post here.

References:

• https://alvinalexander.com/java/jwarehouse/commons-io-2.0/src/main/java/org/apache/commons/io/FileCleaningTracker.java.shtml
• https://www.ibm.com/developerworks/java/library/j-jtp11225/
• https://dzone.com/articles/weak-soft-and-phantom-references-in-java-and-why-they-matter
• https://blogs.oracle.com/sundararajan/programmatically-dumping-heap-from-java-applications