turn event streams into (micro) services
Pyroclast is a Platform as a Service for building real-time event streaming applications. Send events to Pyroclast from your favorite programming language, then build and launch horizontally scalable microservices to perform complex streaming tasks. Pyroclast makes real-time streaming easy.
Event Streaming as a Service
Pyroclast bridges the gap between event data and the rest of your architecture. Fire off events from any data source into our hosted storage. Spin up services to enrich and aggregate those events. Query those services using HTTP from your existing applications.
What's the big deal?
Through a combination of 4 major advancements, Pyroclast helps you deliver real-time applications to production at a lower cost and on a faster timeline.
Streaming services are defined by a flexible, data-driven model that makes it easy to perform enrichment, filtering, queries, view materialization, streaming joins, deduplication, and more. Pyroclast uses a specialized compiler to transform these models into distributed, high performance programs and executes them on a backing stream processor. This approach to modeling relieves you from the burden of implementing distributed systems complexities like consistent checkpointing, failure recovery, performance tuning, and autoscaling.
Authoring and verifying streaming applications has traditionally been an activity with a slow feedback loop. Pyroclast overcomes this challenge by employing a technique called simulation. Pyroclast's Simulator let's you work with your model definition and sample data side-by-side, cross-compiling your model on the fly as you work. Its visual display of how mutual changes to your model and sample data affect one another dramatically reduces the time it takes to find and fix bugs. The Simulator operates faster than human reaction time, delivering an unparalleled experience of total control.
Services operate non-destructively over event streams, meaning that two or more services can operate over the same sequence of events in a conflict-free manner. Furthermore, a service can "rewind the clock" at any time of its choosing and replay previously seen events. Time traveling is the foundation for implementing speculative queries, auditing, and retroactive code correction. Pyroclast achieves this by storing all events verbatim in the order they were received.
As event pass through each service, any queries it defines will have its results incrementally updated to reflect the latest information seen. When query results are requested, results are available for immediate access. No extra work happens at read time because of the inversion of query control. This style of data flow eliminates the torturous application design problem of slow analytics queries.
How do I use it?
- You: send events to Pyroclast via programming language SDKs
- Pyroclast: receives your events and places them into hosted storage
- You: define service models to transform, enrich, and query your events
- Pyroclast: runs your services in a high performance, scalable, fault-tolerant environment
- Pyroclast: maintains and serves materialized views of your events through service HTTP endpoints
- You: build applications against those HTTP endpoints
What's it good for?
- streaming queries: incrementally materialize query results over data streams
- streaming ETL: extract, transform, and load data into external services in real time
- streaming joins: join multiple data streams together based off arbitrary criteria
- time-series analytics: maintain rolling aggregated views of timestamped events
- data enrichment: reshape and supplement events with a powerful suite of collection operations
- deduplication: filter out duplicate messages and compact a stream of records
Get started now
Let's talk details
Everything is an event
An event is a single unit of data which represents something that happened. Every system has events - they’re everywhere! Events are usually represented by maps with keys denoting all the attributes of what happened. They’re also typically timestamped by capturing when the action occurred. Beyond being structured as a map, events tend do not have rigid schemas.
Pyroclast is a Platform as a Service for storing and processing events. Fire off events to Pyroclast with our client libraries, data extraction tools, or HTTP API.
Events are stored in topics
Pyroclast stores collections of events in topics. A topic is named stream of events, typically indicative of a feed or a category of information. When you send Pyroclast an event, you tell it which topic to store it under. A topic uniquely numbers every event it receives in order, attaching some metadata to each entry. Topics are durably stored and geographically replicated.
Services consume topics
While topics store events, services are responsible for processing those events to produce meaningful information. Services are definitions of how to transform, filter, and aggregate the events that it receives. Each service automatically and inivisibly pairs itself with a backing high performance stream processor, executing service definitions at blazing speeds. Services can both read from and write to topics.
Services materialize views of events
Services can also execute queries about events. A service will create a materialized view of each query in real time. This means that as events are processed from the topic, a service creates an HTTP-accessible value for that query. Whenever your application would like to know the answer for that query, Pyroclast will respond instantly with the current value.
Travel through time
Pyroclast makes it easy to do things that ordinarily require complex designs, like time travel. Made a mistake? Business rules changed? Services can be rewound to resume reading events from any point in time in a topic, including the beginning. Time travel let’s you play back historical information and process it in a different way - without disrupting existing services consuming from the same topic.
Flight Test before Launching
Pyroclast's hallmark feature is called the Simulator. The biggest economical drawback of developing event streaming applications is the turnaround time between finding a problem, fixing it, and deploying. Pyroclast shows you how your sample data changes on every keystroke. Go from minutes of turnaround time to milliseconds.
- Build interactively with millisecond-scale turnaround time
- Drag-and-drop entire sample data sets
- Debug runtime problems with the same tools used for development