A curated list of awesome big data frameworks, ressources and other awesomeness.

Open-Source Web UI for Apache Kafka Management

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Fancy stream processing made operationally mundane

Real-time Data Integration and Transformation: use SQL to transform, deliver, and act on fast-changing data.

🌊 Online machine learning in Python

Readyset is a MySQL and Postgres wire-compatible caching layer that sits in front of existing databases to speed up queries and horizontally scale read throughput. Under the hood, ReadySet caches the results of cached select statements and incrementally updates these results over time as the underlying data changes.

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

Utils for streaming large files (S3, HDFS, gzip, bz2...)

Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

A lightweight stream processing library for Go

Pravega - Streaming as a new software defined storage primitive

Python Stream Processing

Python Streaming DataFrames for Kafka

Real-time stream processing for python

Trill is a single-node query processor for temporal or streaming data.

Open-Source Web UI for managing Apache Kafka clusters

📐 Pushing the boundaries of simplicity

Superdiff provides a complete and readable diff for both arrays and objects. Plus, it supports stream and file inputs for handling large datasets efficiently, is battle-tested, has zero dependencies, and is super fast.

⚡ Single-pass algorithms for statistics