The ElasticSearch Saga Continues – InApps Technology 2022

First off, if you haven’t kept up with the saga of Amazon Web Services and Elastic, here’s the briefest of recaps. A few years ago, AWS basically forked ElasticSearch to offer it as a service, much to the open source community’s dismay. In response, and after some time, Elastic decided to change the licensing on ElasticSearch earlier this year to restrict its downstream use, again, much to the open source community’s dismay. AWS then announced it would fork the project to keep it fully open source, suddenly becoming the apparent good guy in the scenario. Finally, just a few months ago, AWS released OpenSearch under the Apache License, Version 2.0 (ALv2), essentially completing the circle.

Well, until now.

The back and forth between ElasticSearch and AWS continues this week, this time with Elastic making further attempts at closing off access to ElasticSearch and shutting out AWS. AWS, in response, has said that it is working on keeping clients of OpenSearch and Elasticsearch compatible with open source.

AWS says that “OpenSearch aims to provide wire compatibility with open source distributions of Elasticsearch 7.10.2, the software from which it was derived,” making it easy to migrate to OpenSearch. While Elastic can’t do anything about that, they can make changes to some open source client libraries that are commonly used.

“Over the past few weeks, Elastic added new logic to several of these clients that rejects connections to OpenSearch clusters or to clusters running open source distributions of Elasticsearch 7, even those provided by Elastic themselves. While the client libraries remain open source, they now only let applications connect to Elastic’s commercial offerings,” AWS writes.

If Elastic were looking to get back into the good graces of the open source community, this surely does not seem like the way.

Kudos to @elastic for making us all collateral damage in its war with @awscloud. It’s my bad for pinning dependencies as >=7.0.0,<8.0.0 and getting this update automatically on a deploy. But still, pretty crappy to break the ES python package for anyone using AWS. #elasticsearch pic.twitter.com/Vb5VatOXdl

— Brad Root (@amiantos) August 4, 2021

Instead, AWS is again coming out as the savior of open source in this scenario, it would seem, this time promising to offer “a set of new open source clients that make it easy to connect applications to any OpenSearch or Elasticsearch cluster” that “will be derived from the last compatible versions of corresponding Elastic-maintained clients before product checks were added.”

“In the spirit of openness and interoperability, we will make reasonable efforts to maintain compatibility with all Elasticsearch distributions, even those produced by Elastic,” they write.

In the meantime, while the OpenSearch community works on creating the replacement libraries, AWS recommends that users do not update to the latest version of any Elastic-maintained clients, lest their applications potentially cease functioning.

Legitimately killing them with kindness. This is the type of moves that elasticsearch should be making because it’s in the best interest of the community. They have made their stance clear though. It’s about the money.

— David Tippett (@dtaivpp) August 5, 2021

This Week in Programming

• Facebook Open Sources Computational Integrity Tool: You Game of Thrones fans should be ultimately pleased with Facebook’s latest open source project Winterfell, a STARK prover and verifier. Beyond the cultural reference, Winterfell is an implementation of the Scalable Transparent Arguments of Knowledge (STARK) prover and verifier, and more specifically makes it possible for the average developer to “benefit from proofs of computational integrity (CI) that would normally require an in-depth knowledge of cryptography to implement.” CI proofs allow users to run a computation, get a result, and then “convince anyone that you did the computation correctly without their having to rerun the computation themselves.” A subset of this is the zero-knowledge proof (ZKP), which allows the same functionality, while also obscuring the inputs. All of this becomes increasingly pertinent with the recent trend of blockchain, but Facebook writes that “ZKPs have numerous potential applications outside of the blockchain space as well” but they haven’t really taken off because of the expertise and computation required. “We developed Winterfell to bridge these gaps and to bring ZKPs within reach of regular developers,” Facebook writes in its blog post. Written in Rust, Winterfell has been released to Crates.io and comes with an end-to-end tutorial as well as an examples crate.

Should you get into software engineering only for the money?!? #CodeTok https://t.co/Mgbp26YMsK pic.twitter.com/bcluGWGdZa

— Scott Hanselman (@shanselman) August 5, 2021

• Rust Pushes for GATs Stabilization: In a blog post this week about the push for GATs stabilization, Jack Huey, a member of the Traits Working Group, assures their readers again and again that, whether they know it, understand it, or not, the move to add generic associated types (GATs) is “very exciting” and a “big deal,” indeed. Apparently, Rust has been trying to add GATs for quite some time now — the RFC was first opened in April of 2016, predating even the push for const generics. And if you still doubt its importance, he points to the tracking issue on GitHub, noting it is the “most upvoted issue on the Rust repository.” The main news here is that the generic_associated_types feature is no longer “incomplete,” which means you will no longer get a warning if you’re trying to use it on the nightly build. For the full reasoning as to why this is important, and what exactly GATs are, head on over to the blog post to read about all the changes made to the compiler to get GATs to work, but beyond that, the team is looking to you to help stabilize the new feature. “We need you to test this feature, to file issues for any bugs you find or for potential diagnostic improvements. Also, we’d love for you to just tell us about some interesting patterns that GATs enable over on Zulip,” they write.
• FSF Wants Your Thoughts on GitHub Copilot: While some may feel that GitHub Copilot, the new “AI pair programmer” from GitHub trained on publicly available source code, is generally not infringing copyright, the Free Software Foundation (FSF) is not so sure about the new “Service as a Software Substitute.” In its call for white papers on philosophical and legal questions around Copilot, the FSF writes that “Copilot raises many other questions which require deeper examination,” such as whether a neural network trained in this manner can be considered fair use and if the code created by the tool can be considered to be infringing on copyrights. “Even if everything might be legally copacetic,” they write, “activists wonder if there isn’t something fundamentally unfair about a proprietary software company building a service off their work.” As such, the FSF is calling for white papers on the topic — see the blog post for a bulleted list of specific areas of interest — and will pay out $500 for published papers.

What’s the Big O Notation of deleting all of your code and starting over again?

— Carla Notarobot ‍ (@CarlaNotarobot) August 5, 2021

• The Stats Are In: If digging through the numbers excites you, we have two recent releases to satisfy your statistical desires. First, the 2021 Stack Overflow Developer Survey is here, with answers from more than 80,000 respondents worldwide, offering insights into everything from how developers learn, to what languages and frameworks they use the most, to which ones offer the best pay. Spoiler alert: Rust once again takes the “most loved” language spot, for the sixth year in a row. Click on through to the full results to find out more. And while we’re at it, the RedMonk Programming Language Rankings also came out this week, showing a mostly stable field when it comes to their calculations, with JavaScript remaining number one, and Java moving back up to number two, alongside Python. More notably, according to RedMonk, is the relative stagnation of Go, Kotlin, and Rust, which it says “may reflect a new emerging reality of systems languages.” The three are grouped together as “would-be challengers for the title of enterprise application language of choice,” and RedMonk notes that Java does not seem to be going anywhere. “It seems plausible, therefore, that Java is retaining — through a combination of adaptability on its part and inertia on the enterprise’s — a large share of the enterprise applications market, meaning that its would-be challengers — languages like Go, Rust and to a lesser extent Kotlin because of the shared JVM platform — are competing less with Java than with each other,” they write. “If that hypothesis is correct, we should expect Java to sustain its performance and future gains from Go, Kotlin and Rust — if any — will be harder to come by as they compete for shares of a smaller pool of workloads.”

The best part of working from home is you can scream at your code as loud as you want.

— Monica Lent (@monicalent) August 5, 2021

Photo by Martin Adams on Unsplash.