dorkhub

reductstore

High-performance, time-indexed object storage for robotics and industrial IoT

reductstore
Rust36545 forksApache-2.0updated 2 days ago
visit the demogit clone https://github.com/reductstore/reductstore.gitreductstore/reductstore

ReductStore banner

GitHub release (latest SemVer) GitHub Workflow Status Docker Pulls GitHub all releases codecov Community

Make robotics and industrial data queryable

ReductStore is high-performance, time-indexed object storage for robotics and industrial IoT.

Store terabytes of images, video, sensor readings, logs, files, and ROS bags in their native binary formats. Every record is organized by timestamp and can be enriched with labels, so you can retrieve exactly the data you need by time range and operational context.

Instead of combining a time-series database, generic object storage, metadata indexes, and custom retention jobs, ReductStore provides one system for ingesting, organizing, retaining, and querying multimodal time-series data.

Why Teams Pick ReductStore

  • Make binary-first robotics and industrial data queryable by time range and labels
  • Built for workloads such as camera frames, sensor payloads, logs, files, and ROS bags
  • Replicate only selected records to the cloud to reduce bandwidth and storage cost
  • Apply quotas and lifecycle policies in the same system that stores the data

Proof Points

🚀 60k+ downloads   •   🏭 100+ production deployments   •   📦 1 PB+ managed data

⭐ 350+ GitHub stars   •   🛠️ 4+ years of active development

When You Should Use It

ReductStore is ideal for scenarios where you have a continuous stream of binary data and your application has metadata that can be attached to the data as labels. If you ingest binary data with labels into a ReductStore instance, you can use it to:

  1. Query objects by time range and labels, e.g. "give me all the images from camera-1 over the last hour where the device status is 'error'".
  2. Replicate only selected data to another ReductStore instance, e.g. "replicate all the data from camera-1 where the device status is 'error' to the cloud instance for further analysis".
  3. Manage the data lifecycle with policies, e.g. "keep all the data from camera-1 for 30 days, then compress it and keep it for another 60 days before deleting it".

In such scenarios, ReductStore can help you manage and transfer your data efficiently based on its metadata, without combining a TSDB and blob storage. This keeps your architecture simple and efficient.

Who It Is For

  • Robotics platforms that need to replay camera frames, telemetry, and logs around incidents or model failures
  • Industrial IoT pipelines that collect binary payloads at the edge and forward only filtered data upstream
  • Edge AI systems that need retention, labeling, and historical retrieval without building a custom storage stack

Get Started

The quickest way to get up and running is with Docker:

mkdir -p ./data
sudo chown -R 10001:10001 ./data
docker run -p 8383:8383 -v ${PWD}/data:/data \
  -e RS_API_TOKEN=my-secret-token reduct/store:latest

Or download a Linux binary directly from the latest release:

curl -LO https://github.com/reductstore/reductstore/releases/latest/download/reductstore.x86_64-unknown-linux-gnu.tar.gz
tar -xzf reductstore.x86_64-unknown-linux-gnu.tar.gz
mkdir -p ./data
RS_DATA_PATH=./data RS_API_TOKEN=my-secret-token ./reductstore

For a more in-depth guide, visit the Getting Started and Download sections.

After initializing the instance, you can start writing and querying data immediately. Here's a Python sample:

import asyncio

from reduct import Client

async def main():
    async with Client("http://localhost:8383", api_token="my-secret-token") as client:
        bucket = await client.create_bucket("my-bucket", exist_ok=True)

        await bucket.write(
            "camera-1",
            b"hello, reductstore",
            timestamp="2024-01-01T10:00:00Z",
            labels={"device": "camera-1"},
        )

        async for record in bucket.query(
            "camera-1",
            start="2024-01-01T10:00:00Z",
            stop="2024-01-01T10:00:01Z",
            when={"&device": {"$eq": "camera-1"}},
        ):
            print((await record.read_all()).decode())

asyncio.run(main())

Next Steps

Learn more and pick the next piece you need:

Build From Source

ReductStore uses a vendored protoc binary through Cargo, so contributors do not need to install Protocol Buffers manually. To check the workspace locally, run:

cargo check --workspace

When You Should Not Use It

ReductStore is not the best fit for every data storage problem. You should consider another solution when:

  1. You have only numerical data that can be easily ingested into a time-series database.
  2. You have only blob data to store and do not need to access it as historical data. In this case, object storage or a file system may be a better fit.
  3. You need a message broker. Although ReductStore provides subscription and publishing functionality, it is designed for data storage and streaming, not message queueing.

Community & Contribution

If you've found a bug, have ideas, built something with ReductStore, or want to contribute directly, here are the best places to jump in:

  • Questions and Ideas: Join our Discourse community to ask questions, share ideas, and collaborate with fellow ReductStore users.
  • Bug Reports: Open an issue on our GitHub repository. Please provide as much detail as possible so we can address it effectively.
  • First Contributions: Pick a task from good first issues or help wanted.
  • Pull Requests: Open PRs only for an existing issue. Comment on the issue and get assigned before you start work. Use the repository pull request template when you open the PR, and update CHANGELOG.md for user-visible changes. If you want to propose a new feature or workflow, open an issue first or start the discussion on the community forum before writing code.
  • Show Your Work: Share your projects, benchmarks, and lessons learned on our Discourse community.
  • Support the Project: If ReductStore is useful to you, give us a ⭐ on GitHub.

Contributors

Thanks to everyone who has contributed to ReductStore.

@atimin @mother-6000 @AnthonyCvn @DibbayajyotiRoy @rohankumardubey @rtadepalli @FirasCh3 @vbmade2000 @LordAizen1 @tuanhungngyn @upuddu @gitcommit90 @lntutor @mambaz @victor1234 @aschenbecherwespe @renghen

Your support fuels our passion and drives us to keep improving.

Together, let's build better storage for robotics and industrial data. 🚀

more like this

perplexity-cli

🧠 A simple command-line client for the Perplexity API. Ask questions and receive answers directly from the terminal! 🚀🚀🚀

Python176

ML-MCU

Code for IoT Journal paper 'ML-MCU: A Framework to Train ML Classifiers on MCU-based IoT Edge Devices'

C51

AeroPath

:hugs: AeroPath: An airway segmentation benchmark dataset with challenging pathology

Jupyter Notebook53

search

search projects, people, and tags