Tree Locking

Rather than locking every involved document, as in the previous option, we could lock just part of the directory tree. We will need exclusive access to the file or directory that we want to rename, which can be achieved with an exclusive lock document:

{ "lock_type": "exclusive" }

And we need shared locks on any parent directories, with a shared lock document:

{
  "lock_type":  "shared",
  "lock_count": 1 
}
The lock_count records the number of processes that hold a shared lock.

A process that wants to rename /clinton/projects/elasticsearch/README.txt needs an exclusive lock on that file, and a shared lock on /clinton, /clinton/projects, and /clinton/projects/elasticsearch.

A simple create request will suffice for the exclusive lock, but the shared lock needs a scripted update to implement some extra logic:

if (ctx._source.lock_type == 'exclusive') {
  assert false; 
}
ctx._source.lock_count++ If the lock_type is exclusive, the assert statement will throw an exception, causing the update request to fail.
Otherwise, we increment the lock_count.

This script handles the case where the lock document already exists, but we will also need an upsert document to handle the case where it doesn’t exist yet. The full update request is as follows:

POST /fs/lock/%2Fclinton/_update 
{
  "upsert": { 
    "lock_type":  "shared",
    "lock_count": 1
  },
  "script": "if (ctx._source.lock_type == 'exclusive')
  { assert false }; ctx._source.lock_count++"
}
The ID of the document is /clinton, which is URL-encoded to %2fclinton.
The upsert document will be inserted if the document does not already exist.

Once we succeed in gaining a shared lock on all of the parent directories, we try to create an exclusive lock on the file itself:

PUT /fs/lock/%2Fclinton%2fprojects%2felasticsearch%2fREADME.txt/_create
{ "lock_type": "exclusive" }

Now, if somebody else wants to rename the /clinton directory, they would have to gain an exclusive lock on that path:

PUT /fs/lock/%2Fclinton/_create
{ "lock_type": "exclusive" }

This request would fail because a lock document with the same ID already exists. The other user would have to wait until our operation is done and we have released our locks. The exclusive lock can just be deleted:

DELETE /fs/lock/%2Fclinton%2fprojects%2felasticsearch%2fREADME.txt

The shared locks need another script that decrements the lock_count and, if the count drops to zero, deletes the lock document:

if (--ctx._source.lock_count == 0) {
  ctx.op = 'delete' 
}
Once the lock_count reaches 0, the ctx.op is changed from update to delete.

This update request would need to be run for each parent directory in reverse order, from longest to shortest:

POST /fs/lock/%2Fclinton%2fprojects%2felasticsearch/_update
{
  "script": "if (--ctx._source.lock_count == 0) { ctx.op = 'delete' } "
}

Tree locking gives us fine-grained concurrency control with the minimum of effort. Of course, it is not applicable to every situation—the data model must have some sort of access path like the directory tree for it to work.

Note

None of the three options—global, document, or tree locking—deals with the thorniest problem associated with locking: what happens if the process holding the lock dies?

The unexpected death of a process leaves us with two problems:

  • How do we know that we can release the locks held by the dead process?
  • How do we clean up the change that the dead process did not manage to complete?

These topics are beyond the scope of this book, but you will need to give them some thought if you decide to use locking.

While denormalization is a good choice for many projects, the need for locking schemes can make for complicated implementations. Instead, Elasticsearch provides two models that help us deal with related entities: nested objects and parent-child relationships.

 

[mks_progress

In the real world, relationships matter: blog posts have comments, bank accounts have transactions, customers have bank accounts, orders have order lines, and directories have files and subdirectories.

Relational databases are specifically designed—and this will not come as a surprise to you—to manage relationships:

  • Each entity (or row, in the relational world) can be uniquely identified by a primary key.
  • Entities are normalized. The data for a unique entity is stored only once, and related entities store just its primary key. Changing the data of an entity has to happen in only one place.
  • Entities can be joined at query time, allowing for cross-entity search.
  • Changes to a single entity are atomic, consistent, isolated, and durable. (See ACID Transactions for more on this subject.)
  • Most relational databases support ACID transactions across multiple entities.

But relational databases do have their limitations, besides their poor support for full-text search. Joining entities at query time is expensive—more joins that are required, the more expensive the query. Performing joins between entities that live on different hardware is so expensive that it is just not practical. This places a limit on the amount of data that can be stored on a single server.

Elasticsearch, like most NoSQL databases, treats the world as though it were flat. An index is a flat collection of independent documents. A single document should contain all of the information that is required to decide whether it matches a search request.

While changing the data of a single document in Elasticsearch is ACIDic, transactions involving multiple documents are not. There is no way to roll back the index to its previous state if part of a transaction fails.

This FlatWorld has its advantages:

  • Indexing is fast and lock-free.
  • Searching is fast and lock-free.
  • Massive amounts of data can be spread across multiple nodes, because each document is independent of the others.

But relationships matter. Somehow, we need to bridge the gap between FlatWorld and the real world. Four common techniques are used to manage relational data in Elasticsearch:

  • Application-side joins
  • Data denormalization
  • Nested objects
  • Parent/child relationships

Often the final solution will require a mixture of a few of these techniques