Hotrod is a general-purpose, distributed runtime for data engineering workloads. As a platform, Hotrod handles the ingestion, transformation and movement of data from sources to destinations.
The key features of Hotrod is all about making data engineering easier and faster.
A Simple Mental Model
Data processing workloads and jobs are defined as Hotrod Pipes, which expresses inputs, transformations and outputs.
A Language For Data Engeering
Hotrod’s powerful Pipe Language provides a consistent, easy to learn method of defining data engineering workloads in a standardized manner, with advanced features like templating, contextual variables and central configuration.
Built with Rust, Hotrod let’s you get more done with less, especially when your data flows are measured in megabits per second.
With a variety of inputs & outputs, Hotrod supports reading and writing data to and from files, network sockets, API’s, databases, queues, and more.
Hotrod can be leveraged in a variety of data processing scenarios.
Hotrod can act as an ingestion layer for incoming data streams to perform filtering, sampling, pre-processing and data volume reduction, and then egress data into your existing systems, storage (on-disk, AWS S3, Azure Blob Storage), or your favourite analytics solution like Elastic Search, Scuba or Splunk.
Connect & Bridge Data Flows
Hotrod can be deployed as a relay or a bridge that connects different systems, protocols and technologies. Use Hotrod as a bridge between HTTP services and Queues (AMQP, Kafka, NSQ), or turn static data sources (e.g. SQL sources) into data streams. Hotrod can even consume UDP and TCP data streams directly.
By co-locating Hotrod agents with other applications running on servers, Hotrod can collect and route metrics, logs and other data to different locations. Use Hotrod for log shipping, or to instrument systems that lack observability and telemetry mechanisms.
Hotrod can compliment your existing monitoring systems and infrastructure. Use Hotrod to generate, collect and process data and monitoring events centrally, or at the edge.
Reduce Data Volume
Hotrod can be leveraged to reduce the volume of data flows. Hotrod’s runtime can perform sampling, aggregation, filtering and down-sizing on and within data streams, to enable data volume reduction.