🆕 ReactiveSearch Scripts: JavaScript functions for when milliseconds matter 🚀

Introducing: JavaScript user-defined functions to enhance the search experience

ReactiveSearch scripts is the outcome of when a JavaScript developer walks into an Elasticsearch bar 🍺

On a more serious note: When building a production grade search experience, one of the key challenges one runs into is defining a search relevancy strategy that works at scale. Lucene based search engines like ElasticSearch get you pretty far with a good out of the box algorithm along with tuning options. But when working at scale, there are hundreds of edge-cases, cascading side effects, long tail searches, synonyms to consider that you need to rely on an external system. At appbase.io, we have built Query Rules to meet this challenge — a set of “If this, then that” style rules that are configurable to meet a business’s unique search relevance needs.

However, at scale — using rules can be cumbersome and hard to maintain. To meet this need, we’re announcing a beta release of ReactiveSearch Scripts, a feature we’ve been hacking 🛠 on for the past several months. Scripts are user defined JavaScript functions that are executed in a latency optimized manner (powered by Google’s V8 engine). These are typically used during the request or response cycle of a search query, but can also be invoked with additional trigger conditions, or at a periodic schedule as a cron job.

Scripts are currently available as an action with appbase.io query rules (more ways to utilize scripts coming 🔜)

Why and When To Use ReactiveSearch Scripts

Scripts are architecturally similar to Cloudflare Workers (in that they utilize the V8 engine for execution) and serve similar use-cases to popular FaaS services such as AWS Lambda, Azure Cloud Functions, et al. Hence, an important question to address is when ReactiveSearch scripts are useful and when other options might be better.

If you’re powering search experiences where milliseconds matter, you should be using scripts. Besides utilizing the V8 engine and not suffering the cold start problem (since we don’t use containers), scripts are also run on the same server environment where the search engine and the appbase.io service is run. Scripts typically run in under 5ms [1] (~10x-20x faster than other services) and can perform IO operations against the search engine faster than an external service running in a different region or edge can.

[1] A request / response transformation script is this quick 🚀. Additional behaviors (e.g. performing NLP or using crypto utils may incur some more latencies). We will be benchmarking this more thoroughly in a future post.

ReactiveSearch Scripts are purposefully designed to serve search use-cases. They come with global packages like fetch, compromise (nlp), crypto.js, lodash to help with common request and response manipulations.

When using another FaaS service is better

If you’re performing an operation e.g. clustering, creating dense vectors, indexing data that’s not user facing or time-sensitive, ReactiveSearch scripts don’t offer an edge to using other FaaS services.

Use-cases for ReactiveSearch Scripts

Modify a request before querying Elasticsearch

• Use script to pre-process the user search query using NLP techniques
• Add dynamic fields to an Elasticsearch indexing request, for e.g. calculate age field from date of birth and add it to the document being indexed

Transform the search response

• Promote or hide search results based on a dynamic criterion
• Use an external API call to return additional context with the search response (e.g. implement a did you mean functionality)

Create side effects

• Implement a saved search functionality by posting search query to a 3rd party API (asynchronously)
• Log indexing requests or search queries that match a set criterion
• Send e-mail alerts based on a certain set of search queries

Run periodically

A script can also be run in a cron mode, where it is invoked at specific intervals.

• Regularly update the search index with data from a primary data source (e.g. Sync updated data from Postgres or MongoDB),
• Enrich data in the search index, e.g. perform named entity recognition,
• Perform classification of data, e.g. unsupervised clustering

Because scripts can call external APIs in both sync and async modes, there are no limits to what is possible.

Show me the code 🧑‍💻

The code snippet below uses the appbase.io dashboard’s editor functionality to validate a script. It uses compromise.js (a built-in module available in ReactiveSearch scripts) to interpret a user query such as: “smart phones under 10K” to extract the price information and then set a filter for the price before querying Elasticsearch.

Image: appbase.io dashboard’s validate script editor: You can

The validate script dashboard feature uses the Monaco editor (the code editor that powers VSCode IDE) and lets you:

• Interactively write a script and catch any compile time errors
• Catch any runtime errors by simulating an entire request / response cycle
• Get a real response by hitting the search engine
• View any user set console logs
• Set environment variables that are available during the runtime.

The _script/validate endpoint can also be used as an API outside of the dashboard, so here’s a Repl that you can try out live:

Script - Request modification with compromise

Scripts are available today as part of Query Rules as a script action — a set of configurable rules to extend the search engine’s default relevance strategy.

Read more about scripts in the docs, and play with code examples on using additional authorization schemes with crypto.js, using sync and async fetch calls to interact with external APIs, modify an indexing or bulk request, using cron script to add a document into the search index periodically.

Coming Soon ➡️

If you’re excited about the potential of ⚡️ fast JavaScript functions, stay tuned for more posts! We’re working on improving the scripts Dx and making them more powerful 🦾