Open-sourcing ABC: Import MongoDB, SQL, JSON, CSV, Firestore, Redis Data Into Elasticsearch
We’ve open-sourced the ABC import project with our latest 1.0.0 Beta Release 3. The ABC import project has been based on the transporter project from the Compose team.
We’re excited to open-source ABC 🎉:
- 4 years since the initial release, ABC continues to be used by several of our customers / community users.
- We’ve struggled to keep up with regular releases for ABC, the last release took a year! Our core focus at appbase.io going forward has on providing the best app search experience, and open-sourcing the project now allows the community to continue relying on it.
Over the past years, we’ve added several new import adaptors as well as maintained support for the existing adaptors.
ABC allows importing data from these sources today:
With this release, we have also added support for importing data to an OpenSearch cluster.
Key Benefits
- Whether your data resides in Postgres or a JSON file or MongoDB or in all three places, abc can index the data into Elasticsearch. It is the only tool that allows working with all these sources at once or individually:
csv,json,postgres,mysql,sqlserver,mongodb,elasticsearch. - It can keep the Elasticsearch index synced in realtime with the data source.
Note:This is currently only supported for MongoDB and Postgres. abc importis a single line command that allows doing all of the above. It doesn’t require any external dependencies, takes zero lines of code configuration, and runs as an isolated process with a minimal resource footprint.- abc also supports configurable user-defined JS data transformations (think better ETL) for advanced uses to map data types, columns or transform the data itself before the data gets indexed into Elasticsearch.
Data Transformation Examples
Using ABC, you can use data transformations to enrich, transform, or collapse data prior to importing it into Elasticsearch index.
- Here’s an example showing how to import data from specific tables in a database.
- Here’s an example that shows how to dynamically import data to different indices.
- Importing data is a process that needs to be often repeated. Here’s an example showing how to index data with document IDs set from the data itself to avoid duplication of documents.
- Here’s an example that shows how to set an Elasticsearch index mapping prior to starting the data import.
Data transformation is primarily supported using the goja project — which allows running a JavaScript VM in Go, allowing users to define transformations in JavaScript. Some more example usages of Goja can be found over here.
Get ABC
Download the binary of v1.0.0-beta.3 release for MacOS (both ARM and AMD builds are supported), Linux or Windows. After that, put the binary in a folder of your choice. You can put it in PATH so that it can be called from anywhere. Then test the installation as
> abc versionabc is also available as a docker image.
docker pull appbaseio/abcRead the following importer specific guides for ABC:
