Global Locking

We can avoid concurrency issues completely by allowing only one process to make changes at any time. Most changes will involve only a few files and will complete very quickly. A rename of a top-level directory may block all other changes for longer, but these are likely to be much less frequent.

Because document-level changes in Elasticsearch are ACIDic, we can use the existence or absence of a document as a global lock. To request a lock, we try to create the global-lock document:

PUT/fs/lock/global/_create{}

If this create request fails with a conflict exception, another process has already been granted the global lock and we will have to try again later. If it succeeds, we are now the proud owners of the global lock and we can continue with our changes. Once we are finished, we must release the lock by deleting the global lock document:

DELETE/fs/lock/global

Depending on how frequent changes are, and how long they take, a global lock could restrict the performance of a system significantly. We can increase parallelism by making our locking more fine-grained.

Document Locking

Instead of locking the whole filesystem, we could lock individual documents by using the same technique as previously described. A process could use a scan-and-scroll request to retrieve the IDs of all documents that would be affected by the change, and would need to create a lock file for each of them:

PUT/fs/lock/_bulk{"create":{"_id":1}}{"process_id":123}{"create":{"_id":2}}{"process_id":123}...
The ID of the lock document would be the same as the ID of the file that should be locked.
The process_id is a unique ID that represents the process that wants to perform the changes.

If some files are already locked, parts of the bulk request will fail and we will have to try again.

Of course, if we try to lock all of the files again, the create statements that we used previously will fail for any file that is already locked by us! Instead of a simple create statement, we need an update request with an upsert parameter and this script:

if(ctx._source.process_id!=process_id){assertfalse;}ctx.op='noop';
process_id is a parameter that we pass into the script.

assert false will throw an exception, causing the update to fail.

Changing the op from update to noop prevents the update request from making any changes, but still returns success.

The full update request looks like this:

POST/fs/lock/1/_update{"upsert":{"process_id":123},"script":"if ( ctx._source.process_id != process_id )  { assert false }; ctx.op = 'noop';""params":{"process_id":123}}

If the document doesn’t already exist, the upsert document will be inserted—much the same as the create request we used previously. However, if the document does exist, the script will look at the process_id stored in the document. If it is the same as ours, it aborts the update (noop) and returns success. If it is different, the assert false throws an exception and we know that the lock has failed.

Once all locks have been successfully created, the rename operation can begin. Afterward, we must release all of the locks, which we can do with a delete-by-query request:

POST/fs/_refreshDELETE/fs/lock/_query{"query":{"term":{"process_id":123}}}
The refresh call ensures that all lock documents are visible to the delete-by-query request.

Document-level locking enables fine-grained access control, but creating lock files for millions of documents can be expensive. In certain scenarios, such as this example with directory trees, it is possible to achieve fine-grained locking with much less work