Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.

Note 2: This guide will constantly be updated with new info as becomes available and please feel to make an issue if you think something should be added.

Getting Started with Self-Hosting

Back to the Top

Self-Hosting is the practice of locally hosting(on premises & private web servers) and managing software applications by a person or organization instead of monthly subscriptions from Software as a service (SaaS) providers.

Most self-hosted software can be installed using Docker, a packaging system which allows software to bundle their configuration and dependencies and isolate them from your operating system. Software using docker can be installed using the command line or via graphical interfaces such as Portainer. Software is installed with Docker by downloading an image file containing the application, then creating a copy that sets up its own dependencies and configuration within what is called a container. Without containers you would often need to install different versions of the same programming languages or tools to satisfy the dependencies for the software you want to use which can get complicated.

Tools for Self-Hosting

Back to the Top

Containers

Back to the Top

Container is a standard unit of software that packages up code and all its dependencies(including CPU, memory, file storage, and network connections) so the application runs quickly and reliably from one computing environment to another.

Application Container Security Guide | NIST (PDF)

Container Image is a lightweight, standalone, executable package of software that includes everything needed to run an application such as the code, runtime, system tools, system libraries, and settings.

Best places to get Container Images:

Docker Compose is a tool that was developed to help define and share multi-container applications. With Compose, we can create a YAML file to define the services and with a single command, can spin everything up or tear it all down.

Docker Include is a Compose application can declare dependency on another Compose application. This is useful if you want to reuse other Compose files. Also, if you need to factor out parts of your application model into separate Compose files so they can be managed separately or shared with others.

Kompose is a conversion tool for Docker Compose to container orchestrators such as Kubernetes or OpenShift.

SwarmKit is a toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

Containerd is a daemon that manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond. It is available for Linux and Windows.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images. Podman provides a command line interface (CLI) familiar to anyone who has used the Docker Container Engine.

Lima is a tool that launches Linux virtual machines with automatic file sharing and port forwarding (similar to WSL2), and containerd. It's a great free and open-source alternative for Docker Desktop.

Colima is a container runtimes on macOS (and Linux) with minimal setup.

Portainer Community Edition is a lightweight service delivery platform for containerized applications that can be used to manage Docker, Swarm, Kubernetes and ACI environments. It is designed to be as simple to deploy as it is to use.

Yacht is a container management UI with a focus on templates and 1-click deployments.

Kitematic is a simple application for managing Docker containers on Mac, Linux and Windows letting you control your app containers from a graphical user interface (GUI).

HashiCorp Nomad is a simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-premises and clouds at scale.

Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtimes.

OpenNebula is an open source platform delivering a simple but feature-rich and flexible solution to build and manage enterprise clouds for virtualized services, containerized applications and serverless computing.

Buildah is a command line tool to build Open Container Initiative (OCI) images. It can be used with Docker, Podman, Kubernetes.

Red Hat Universal Base Images (UBI) is a tool that offers a way to build your container images on a foundation of Red Hat Enterprise Linux software. They are OCI-compliant, container-based, operating system images with complementary runtime languages and packages that are freely redistributable. Easily find UBI images in the Red Hat container catalog, and they are buildable and deployable anywhere.

Red Hat Quay is a project that Builds, Stores, and Distributes your Applications and Containers.

ctop is a tool that provides a concise and condensed overview of real-time metrics for multiple containers as well as a single container view for inspecting a specific container. It comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

container-images is a collection of container images used in CI across various opencontainers projects.

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including OCI and Docker).

Shipwright is a WebUI to generate templates for Yacht, Portainer, Docker-Compose, and Unraid.

Alnoda Workspaces is an open-source portable containerized browser-based development environments in Docker containers. You can create your own custom workspace or customize any of the workspaces with your preferred stack of applications without knowing much of the Docker.

Autoheal monitors and restarts unhealthy docker containers.

Dozzle is a small lightweight application with a web based interface to monitor Docker logs. It doesn’t store any log files. It is for live monitoring of your container logs only.

Diun is a tool that receive notifications when a Docker image is updated on a Docker registry.

WatchTower is a process for automating Docker container base image updates.

Kasm Workspaces is a a highly configurable container streaming platform that enables you to stream and deliver containerized applications over the web. It offers tools that you can use to create desktop workspaces and provide access to virtual desktops to end users. It also ensures data loss prevention as well as secure and private web browsing.

Nginx Proxy is an automation tool that sets up a container running nginx and docker-gen. Docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped.

Visual Studio Code Dev Containers is an extension that lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of Visual Studio Code's full feature set. A devcontainer.json file in your project tells VS Code how to access (or create) a development container with a well-defined tool and runtime stack.

CI/CD

Back to the Top

CI/CD: Continuous Integration and Continuous Delivery

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers.

Woodpecker is a CI service, a community fork of Drone.

Travis CI is a hosted continuous integration service used to build and test software projects hosted at GitHub.

Circle CI is a continuous integration and continuous delivery platform that helps software teams work smarter, faster.

Buddy is a fully-featured DevOps platform with no learning curve that packs everything you need from a CI/CD tool.

Buildbot is a continuous integration tool which automates the compile or test cycle required to validate changes to the project code base. It queues jobs, executes the jobs when the required resources are available, and reports the results.

Development

Back to the Top

Proxmox VE(Virtual Environment) is an open-source platform for enterprise virtualization. It has a built-in web interface that you can use to easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

Terraform provider plugin for Proxmox is a Terraform provider for the Proxmox virtualization platform and exposes Terraform resources to provision QEMU VMs and LXC Containers.

OTF is an open source alternative to Terraform Enterprise. Includes SSO, team management, agents, and as many applies as you can throw hardware at.

Semaphore UI is a modern UI for Ansible. It lets you easily run Ansible playbooks, get notifications about fails, control access to deployment system.

APITable is an API-oriented low-code platform for building collaborative apps and better than all other Airtable open-source alternatives.

Chisel Kubernetes Operator is a Kubernetes operator for Chisel. It allows you to use Chisel as a LoadBalancer provider for your Kubernetes cluster, similar to inlets-operator.

Docker-pgautoupgrade is a PostgreSQL Docker container that automatically upgrades your database. It's whole purpose in life is to automatically detect the version of PostgreSQL used in the existing PostgreSQL data directory, and automatically upgrade it (if needed) to the required version of PostgreSQL.

IT-Tools is a collection of handy online tools for developers, with great UX.

Lazygit is a simple terminal UI for git commands, written in Go with the gocui library.

LazyDocker is a simple terminal UI for both docker and docker-compose, written in Go with the gocui library.

Code-Server is Visual Studio Code running on a remote server, accessible through the browser.

Turbopilot is an open source large-language-model based code completion engine that runs locally on your CPU.

Self-Hosted Sentry nightly is an official bootstrap for running your own Sentry with Docker. Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept.

Visual Studio Live Share is a service/extension that enables you to collaboratively edit and debug with others in real time, regardless of the programming languages you're using or app types you're building. You can instantly and securely share your current project, start a joint debugging session, share terminal instances, forward localhost web apps, have voice calls, and more.

GistPad is a Visual Studio Code extension that allows you to edit GitHub Gists and repositories from the comfort of your favorite editor. You can open, create, delete, fork and star gists and repositories, and then seamlessly begin editing files as if they were local, without ever cloning, pushing or pulling anything.

Live Server is an extension for Visual Studio Code that launches a development local Server with live reload feature for static & dynamic pages.

Gitea is a community managed painless self-hosted Git service.

Act is a a tool to run your GitHub Actions locally.

Act runner is a runner for Gitea based on act.

GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. Self-host GitLab on your own servers, in a container, or on a cloud provider.

Bonobo Git Server - Set up your own self hosted git server on IIS for Windows. Manage users and have full control over your repositories with a nice user friendly graphical interface.

Fossil - Distributed version control system featuring wiki and bug tracker.

Gerrit - A code review and project management tool for Git based projects.

Gitblit - Pure Java stack for managing, viewing, and serving Git repositories.

gitbucket - Easily installable GitHub clone powered by Scala.

Gitea - Community managed fork of Gogs, lightweight code hosting solution.

Gitlist - Web-based git repository browser - GitList allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history and diffs.

Gitolite - Gitolite allows you to setup git hosting on a central server, with fine-grained access control and many more powerful features.

GitPrep - Portable Github clone.

Gogs - Painless self-hosted Git Service written in Go.

Kallithea - Source code management system that supports two leading version control systems, Mercurial and Git, with a web interface.

Klaus - Simple, easy-to-set-up Git web viewer that Just Works.

Lavagna - Lavagna is an open-source issue/project management tool designed for small teams. Lightweight, pure Java, easy to install, easy to use.

Leantime - Leantime is a lean project management system for small teams and startups helping to manage projects from ideation through delivery.

Taiga is an open-source project management software for cross-functional teams that work agile across both scrum and kanban frameworks.

Planka is a realtime kanban board for workgroups built with React and Redux.

Microgit - Git hosting service made in Crystal and Lucky. MIT Crystal

OneDev - All-In-One DevOps Platform. With Git Management, Issue Tracking, and CI/CD. Simple yet Powerful.

OpenProject - OpenProject is a web-based project management system.

Pagure - A lightweight, powerful, and flexible git-centric forge with features laying the foundation for federated and decentralized development.

Phorge - Phorge is an open source, community driven platform for collaborating, managing, organizing and reviewing software development projects.

Redmine - Redmine is a flexible project management web application. (Demo,

RhodeCode - RhodeCode is an open source platform for software development teams. It unifies and simplifies repository management for Git, Subversion, and Mercurial.

SCM Manager - The easiest way to share and manage your Git, Mercurial and Subversion repositories over http.

Taiga - Agile Project Management Tool based on the Kanban and Scrum methods.

Titra - Time-tracking solution for freelancers and small teams.

Traq - Project management and issue tracking system written in PHP.

Tuleap - Tuleap is a libre suite to plan, track, code and collaborate on software projects.

UVDesk - UVDesk community is a service oriented, event driven extensible opensource helpdesk system that can be used by your organization to provide efficient support to your clients effortlessly whichever way you imagine.

ZenTao - An agile(scrum) project management system/tool.

k3s-ansible is the easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more.

Soft Serve is a tasty, self-hostable Git server for the command line.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

Corosync Cluster Engine is a Group Communication System with additional features for implementing high availability within applications.

Glow is a terminal based markdown reader designed from the ground up to bring out the beauty—and power—of the CLI. It's used to discover markdown files, read documentation directly on the command line and stash markdown files to your own private collection, so you can read them anywhere.

Deep Lake is a data lake for deep learning applications. Our open-source dataset format is optimized for rapid streaming and querying of data while training models at scale, and it includes a simple API for creating, storing, and collaborating on AI datasets of any size. It can be deployed locally or in the cloud, and it enables you to store all of your data in one place, ranging from simple annotations to large videos.

Node-Red is a low-code programming for event-driven applications.

krunvm is a CLI-based utility for creating microVMs from OCI images, using libkrun and buildah.

Zeal is an offline documentation browser for software developers inspired by Dash.

Web servers

Back to The Top

Web servers

Apache - Most popular web server.

OpenResty Manager - The easiest using, powerful and beautiful OpenResty Manager(Nginx Enhanced Version), open source alternative to OpenResty Edge.

Beakon - A self-host location sharing webserver. Beakon aims to leak as little data as possible and uses mostly self-contained libraries and local database files. Where possible, it will reference local files and not reach out over any network.

Caddy - The HTTP/2 Web Server with Fully Managed TLS.

Cherokee - Lightweight, high-performance web server/reverse proxy.

Lighttpd - Web server more optimized for speed-critical environments.

Nginx - Reverse proxy, load balancer, HTTP cache, and web server.

uWSGI - The uWSGI project aims at developing a full stack for building hosting services.

Web Performance

HAProxy - Software based load Balancing, SSL offloading and performance optimization, compression, and general web routing.

Squid - Caching proxy for the web supporting HTTP, HTTPS, FTP, and more.

Traefik - Taefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease.

Varnish - HTTP based web application accelerator focusing on optimizing caching and compression.

LLMs

Back to the Top

Large Language Models (LLMs) is a language model that uses artificial neural networks to generate text (AI chatbots/search engines). Some notable ones are GPT-3, GPT-4, BLOOM, and LLaMA.

llama.cpp is a Port of Facebook's LLaMA model in C/C++.

ollama is a tool to get up and running with Llama 2 and other large language models locally.

LocalAI is a self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware with no GPU required. It's an API to run ggml compatible models: llama, gpt4all, rwkv, whisper, vicuna, koala, gpt4all-j, cerebras, falcon, dolly, starcoder, and many others.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenLLM is an open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease.

Llama-gpt is a self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device.

Llama2 webui is a tool to run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use llama2-wrapper as your local llama2 backend for Generative Agents/Apps.

Llama2.c is a tool to Train the Llama 2 LLM architecture in PyTorch then inference it with one simple 700-line C file (run.c).

Alpaca.cpp is a fast ChatGPT-like model locally on your device. It combines the LLaMA foundation model with an open reproduction of Stanford Alpaca a fine-tuning of the base model to obey instructions (akin to the RLHF used to train ChatGPT) and a set of modifications to llama.cpp to add a chat interface.

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

LoLLMS WebUI is a the hub for LLM (Large Language Model) models. It aims to provide a user-friendly interface to access and utilize various LLM models for a wide range of tasks. Whether you need help with writing, coding, organizing data, generating images, or seeking answers to your questions.

LM Studio is a tool to Discover, download, and run local LLMs.

Ava PLS small, all-in-one desktop app to run LLMs locally.

Gradio Web UI is a tool for Large Language Models. Supports transformers, GPTQ, llama.cpp (ggml/gguf), Llama models.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

KoboldCpp is an easy-to-use AI text-generation software for GGML models. It's a single self contained distributable from Concedo, that builds off llama.cpp, and adds a versatile Kobold API endpoint, additional format support, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info, author's note, characters, and scenarios.

Minima is a configurable conversational RAG system that runs LLM locally and on-premises using containers.

ChatGPT

Back to the Top

Chatbot UI for ChatGPT

Chatbot UI by mckaywrigley is an advanced chatbot kit for OpenAI's chat models built on top of Chatbot UI Lite using Next.js, TypeScript, and Tailwind CSS. This version of ChatBot UI supports both GPT-3.5 and GPT-4 models. Conversations are stored locally within your browser. You can export and import conversations to safeguard against data loss. See a demo.

Chatbot UI Lite by mckaywrigley is a simple chatbot starter kit for OpenAI's chat model using Next.js, TypeScript, and Tailwind CSS. See a demo.

Build locally with Docker:

docker build -t chatgpt-ui .
docker run -e OPENAI_API_KEY=xxxxxxxx -p 3000:3000 chatgpt-ui

Running Locally on Windows, MacOS, and Linux:

1. Clone Project Repo

git clone https://github.com/mckaywrigley/chatbot-ui.git

2. Install Dependencies

npm i

3. Provide OpenAI API Key

Create a .env.local file in the root of the repo with your OpenAI API Key:

OPENAI_API_KEY=YOUR_KEY

You can set OPENAI_API_HOST where access to the official OpenAI host is restricted or unavailable, allowing users to configure an alternative host for their specific needs.
Additionally, if you have multiple OpenAI Organizations, you can set OPENAI_ORGANIZATION to specify one.

4. Run App

npm run dev

You done you should be able to start chatting with ChatGPT!

Chatbot UI

MiniGPT-4 is an enhancing Vision-language Understanding with Advanced Large Language Models

Launching Demo Locally

Try out the demo demo.py on your local machine by running

python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0

Here, the demo loads Vicuna as 8 bit by default to save some GPU memory usage. Besides, the default beam search width is 1. Under this setting, the demo cost about 23G GPU memory. If you have a more powerful GPU with larger GPU memory, you can run the model in 16 bit by setting low_resource to False in the config file minigpt4_eval.yaml and use a larger beam search width.

MiniGPT-4 Demo

GPT4All is an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue based on LLaMa.

GPT4All UI is a Flask web application that provides a chat UI for interacting with the GPT4All chatbot.

llama.cpp is a Port of Facebook's LLaMA model in C/C++.

Serge is a web interface for chatting with Alpaca through llama.cpp. Fully self-hosted & dockerized, with an easy to use API.

OpenPlayground is a playfround for running ChatGPT-like models locally on your device.

Vicuna is an open source chatbot trained by fine tuning LLaMA. It apparently achieves more than 90% quality of chatgpt and costs $300 to train.

Yeagar ai is a Langchain Agent creator designed to help you build, prototype, and deploy AI-powered agents with ease.

DoctorGPT is a lightweight self-contained binary that monitors your application logs for problems and diagnoses them.

HttpGPT is an Unreal Engine 5 plugin that facilitates integration with OpenAI's GPT based services (ChatGPT and DALL-E) through asynchronous REST requests, making it easy for developers to communicate with these services. It also includes Editor Tools to integrate Chat GPT and DALL-E image generation directly in the Engine.

Automation

Back to the Top

Accelerated Text - Automatically generate multiple natural language descriptions of your data varying in wording and structure.

Activepieces - No-code business automation tool like Zapier or Tray. For example, you can send a Slack notification for each new Trello card.

ActiveWorkflow - An intelligent process and workflow automation platform based on software agents.

Alltube - Web GUI for youtube-dl, a program to download videos and audio from more than 100 websites.

AmIUnique - Learn how identifiable you are on the Internet (browser fingerprinting tool).

Automatisch - Business automation tool that lets you connect different services like Twitter, Slack, and more to automate your business processes (Open source Zapier alternative).

Baserow - Open source online database tool and Airtable alternative. Create your own database without technical experience.

betanin - Music organization man-in-the-middle of your torrent client and music player. Based on beets.io, similar to Sonarr and Radarr.

ChiefOnboarding - Employee onboarding platform that allows you to provision user accounts and create sequences with todo items, resources, text/email/Slack messages, and more! Available as a web portal and Slack bot.

Datasette - An open source multi-tool for exploring and publishing data, easy import and export and database management.

Eonza - Eonza is used to create scripts and automate tasks on servers or VPS hosting. Manage your servers from any browser on any device.

Exadel CompreFace - Face recognition system that provides REST API for face recognition, face detection, and other face services, and is easily deployed with docker. There are SDKs for Python and JavaScript languages. Can be used without prior machine learning skills.

feed2toot - Feed2toot parses a RSS feed, extracts the last entries and sends them to Mastodon.

feedmixer - FeedMixer is a WSGI (Python3) micro web service which takes a list of feed URLs and returns a new feed consisting of the most recent n entries from each given feed(Returns Atom, RSS, or JSON).

Headphones - Automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent, Deluge and Blackhole.

Healthchecks - Django app which listens for pings and sends alerts when pings are late.

HRConvert2 - Drag-and-drop file conversion server with session based authentication, automatic temporary file maintenance, and logging capability.

Huginn - Allows you to build agents that monitor and act on your behalf.

Kibitzr - Lightweight personal web assistant with powerful integrations.

Krayin - Free and Opensource Laravel CRM Application.

Leon - Open-source personal assistant who can live on your server.

Lidarr - Lidarr is a music collection manager for Usenet and BitTorrent users.

Matchering - A containerized web app for automated music mastering. An open-source alternative to LANDR, eMastered, and MajorDecibel.

Medusa - Automatic Video Library Manager for TV Shows. It watches for new episodes of your favorite shows, and when they are posted it does its magic. (Source Code) GPL-3.0 Python

MeTube - Web GUI for youtube-dl, with playlist support. Allows downloading videos from dozens of websites. AGPL-3.0 Python/Nodejs/Docker

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

nefarious - Web application that automates downloading Movies and TV Shows.

NocoDB - No-code platform that turns any database into a smart spreadsheet. It can be considered as an Airtable or Smartsheet alternative.

OliveTin - OliveTin is a web interface for running Linux shell commands.

Patrowl - Open Source, Smart and Scalable Security Operations Orchestration Platform.

Podgrab - Lightweight podcast manager and automatic podcast episode downloader. It will monitor podcasts for your and download them automatically whenever a new episode goes live.

pyLoad - Lightweight, customizable and remotely manageable downloader for 1-click-hosting sites like rapidshare.com or uploaded.to.

Radarr - Radarr is an independent fork of Sonarr reworked for automatically downloading movies via Usenet and BitTorrent, à la Couchpotato.

SickRage - SickRage is an automatic Video Library Manager for TV Shows. Automatic torrent/nzb searching, downloading, and processing at the qualities you want.

SiteInspector - Web-based tool for catching spelling errors, grammatical errors, broken links, and other errors on websites.

Sonarr - Automatic TV Shows downloader and manager for Usenet and BitTorrent. It can grab, sort and rename new episodes and automatically upgrade the quality of files already downloaded when a better quality format becomes available.

StackStorm - StackStorm (aka IFTTT for Ops) is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions and ChatOps.

µTask - Automation engine that models and executes business processes declared in yaml.

Configuration Management

Back to The Top

Ansible - is a tool is a powerful, agentless tool that works everywhere and with everything. When you add in proven enterprise engineering and support from Red Hat that's written in Python.

Ansible.Ai is an AI for Ansible Content Development tool to automate in your IT infrastructure and it will generate syntactically correct playbook to help you get there.

CFEngine - is a Lightweight agent system where the configuration state is specified via a declarative language.

mgmt - is a next generation config management written in Go.

Pallet - is a Infrastructure definition, configuration and management via a Clojure DSL.

Puppet - is an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

Chef - is a powerful automation platform that transforms infrastructure into code automating how infrastructure is configured, deployed and managed across any environment.

(R)?ex - is a friendly automation framework to any combinations of local and remote execution, push and pull style of management, or imperative and declarative approach.

Salt - is an event-driven automation tool and framework to deploy, configure, and manage complex IT systems. It automates common infrastructure administration tasks and ensure that all the components of your infrastructure are operating in a consistent desired state.

Fleek is an all-in-one management system for everything you need to be productive on your computer.

Cloud Storage

Back to The Top

Swift - A highly available, distributed, eventually consistent object/blob store.

Syncthing - Open Source system for private, encrypted and authenticated distribution of data.

git-annex assistant - A synchronized folder on each of your MacOS and Linux computers, Android devices, removable drives, NAS appliances, and cloud services.

NextCloud - Provides access to your files via the web.

ownCloud - Provides universal access to your files via the web, your computer or your mobile devices.

Seafile - Another Open Source Cloud Storage solution.

SparkleShare - Provides cloud storage and file synchronization services. By default, it uses Git as a storage backend.

Cloud

Back to the Top

Linode

Linode is a cloud hosting company that provides virtual private servers and variety of other cloud services.

Linode Tools

Linode Cloud Manager is a user- and mobile-friendly interface to deploy and manage virtual machines, configure networking, and control user accounts.

Linode API is a tool that makes easy to configure, manage, and deploy user management, billing, support tickets, and more with programmatic access to Linode products and services.

Linode CLI is a tool to deploy and manage Linux servers from Linode without leaving the command line.

Linode Images is a service to capture, store, and deploy your custom images across Linodes or data centers. Easily create your own raw disk image and upload a compressed .gz image file (up to 5 GB) using the Cloud Manager or API to easily deploy to the Linode size and data center you need.

Linode Integrations is a collection of integrations lets you connect infrastructure and dev tools to the Linode platform. That let's you manage your Linode resources using the tools you know and love.

StackScripts is a tool to automatically configure new Linode instances using simple scripts. Create your own StackScript or browse the community StackScript library.

Linode Bare Metal is the single-tenant solution for applications and organizations with security, compliance, and performance needs. Bare Metal combines direct hardware access and the flexibility of a virtual machine.

Nextcloud

Back to the Top

Nextcloud is an industry-leading, on-premises content collaboration platform for file sync & share and communication server. It is fully open source and you can host it yourself or pay a company to do it for you. Also checkout the following links below:

Nexcloud login screen

Nextcloud Hub is a tool that allows you to share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks. As fully on-premises solution, Nextcloud Hub provides the benefits of online collaboration without the compliance and security risks.

Nexcloud Hub

Nextcloud AIO (All In One) is a tool that provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Nextcloud Desktop Client is a tool to synchronize files from Nextcloud Server with your computer.

Nextcloud Deck is a kanban style organization tool aimed at personal planning and project organization for teams integrated with Nextcloud.

Nextcloud Files is a tool tool that allows your employees have easy access to their files, photos and documents to work and can share and collaborate with team members, customers and partners. So IT knows nobody besides those they shared with has access to those files.

Nextcloud Talk is a tool that protects your communication better than other team collaboration platforms like Microsoft Teams or Slack, making sure your data stays on your servers. It also goes further than other encrypted communication technologies by keeping even metadata from leaking.

Nextcloud Home is a tool that allows you store your documents, calendar, contacts and photos on your server at home, at one of at one Nextcloud's providers or in a data center you trust.

Nextcloud Enterprise is a service that gives professional organizations software optimized and tested for mission critical environments.

Nextcloud Outlook Integration is a tool that automatically upload files to replace large attachments or integrate Calendars and Contacts in Microsoft Outlook.

Collabora Online in Nextcloud is a powerful LibreOffice-based online office suite with collaborative editing, which supports all major document, spreadsheet and presentation file formats and works in all modern browsers.

ONLYOFFICE integration in Nextcloud is a service that empowers your users to collaborate on office documents with team members in real time. It has compatibility with Microsoft Office formats means perfect documents, every time.

Nextcloud VM(virtual machine appliance) is a set of carefully crafted family of *nix scripts, which interactively guide you through a quality-controlled installation of a Nextcloud instance for Home/SME Server and scripts for Raspberry Pi 4. It is Community developed and maintained.

LibreSign is a Libre digital signature app for Nextcloud.

DigitalOcean

Back to the Top

DigitalOcean is a cloud infrastructure provider that provides developers cloud services that help to deploy and scale applications that run simultaneously on multiple computers with data centers worldwide.

DigitalOcean Tools

DigitalOcean API is a service that manages your DigitalOcean infrastructure with our RESTful API.

DigitalOcean Client libraries is a collection of libraries lets you use the DigitalOcean API in a variety of programming languages.

DigitalOcean CLI is a service that manages your DigitalOcean infrastructure through your terminal with our open source Command Line Interface (CLI).

Terraform provider is a service that allows the user treat their DigitalOcean infrastructure like code with Terraform.

DigitalOcean Custom images is a service that quickly builds your environment in the cloud by provisioning servers with your own custom image, or choose from various Linux distributions.

Container Registry is a service that easily stores, manages, and protects private container images.

Back4app Web Deployment

Back4app Web Deployment is a Container as a Service (CaaS) provider platform that allows the dev teams to build and deploy containerized applications with no downtime. You can simply connect it to a GitHub repository and publish the code within seconds.

MinIO Object Storage

Back to the Top

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads. It's one of the fastest object storage platforms globally, with a read/write speed of 183GB/s-171GB/s if you use standard hardware. It can function as the main storage tier for many workloads like Spark, TensorFlow, Presto, Hadoop HDFS, and H2O.

MinIO UI

Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:

Podman

podman run -p 9000:9000 -p 9001:9001 \
  quay.io/minio/minio server /data --console-address ":9001"

Docker

#docker run -p 9000:9000 --name minio -d minio/minio server /export

If you're using an SSD mounted at /mnt/sdd, then we can run the following to use it instead:

# docker run -v /mnt/ssd:/export -p 9000:9000 --name minio -d minio/minio server /export

MacOS

brew install minio/stable/minio
minio server /data

Binary Download for MacOS

wget https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio
./minio server /data

Linux

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data

Architecture	URL
64-bit Intel/AMD	https://dl.min.io/server/minio/release/linux-amd64/minio
64-bit ARM	https://dl.min.io/server/minio/release/linux-arm64/minio
64-bit PowerPC LE (ppc64le)	https://dl.min.io/server/minio/release/linux-ppc64le/minio
IBM Z-Series (S390X)	https://dl.min.io/server/minio/release/linux-s390x/minio

Windows

To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:

https://dl.min.io/server/minio/release/windows-amd64/minio.exe

Use the following command to run a standalone MinIO server on the Windows host. Replace D:\ with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the minio.exe executable, or add the path to that directory to the system $PATH:

minio.exe server D:\

Install from Source

Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow How to install Golang. The minimum version required is go1.19.

go install github.com/minio/minio@latest

After you install MinIO:

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

When you run Minio you will be issued a key and a secret. These are used by the client or the web front-end to connect securely. I found my codes by typing in docker logs minio.

Created minio configuration file at /root/.minio

Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000
AccessKey: accessCode
SecretKey: secretCode
Region:    us-west-1
SQS ARNs:  <none>

Browser Access:
   http://172.17.0.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://172.17.0.2:9000 accessCode secretCode

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide

Drive Capacity: 50 GiB Free, 70 GiB Total

If you'd like to learn more then most of the Minio client commands support a help flag or give info on the command line:

NAME:
  mc - Minio Client for cloud storage and filesystems.

USAGE:
  mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]

COMMANDS:
  ls       List files and folders.
  mb       Make a bucket or a folder.
  cat      Display file and object contents.
  pipe     Redirect STDIN to an object or file or STDOUT.
  share    Generate URL for sharing.
  cp       Copy files and objects.
  mirror   Mirror buckets and folders.
  diff     Show differences between two folders or buckets.
  rm       Remove files and objects.
  events   Manage object notifications.
  watch    Watch for files and objects events.
  policy   Manage anonymous access to objects.
  session  Manage saved sessions for cp and mirror commands.
  config   Manage mc configuration file.
  update   Check for new mc update.
  version  Print version info.
  help, h  Shows a list of commands or help for one command

Advanced options

You can have your client point to multiple Minio servers, which is really neat especially if you're working on a distributed team.

Minio's test-server called "play" is already configured in the default client, you can see all the servers you have configured with mc config host list.

To upload the photo to Minio's "play" S3 server just type in:

# mc mb play/somebucketname

# mc cp ~/Downloads/IMG_2016120-25.jpg play/somebucketname

Recursive uploads:

If you want to test something larger out you could try uploading your entire Downloads photo, and then you should use the --recursive flag to make sure nothing's missed:

# mc cp --recursive ~/Downloads/IMG_2016120-25.jpg myminio/photos

Databases

Back to the Top

SQL

SQL is a standard language for storing, manipulating and retrieving data in relational databases.

Coolify is an open-source & self-hostable Heroku/Netlify alternative.

MySQL is a fully managed database service to deploy cloud-native applications using the world's most popular open source database.

PostgreSQL is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

PostgREST is a tool that serves a fully RESTful API from any existing PostgreSQL database. It provides a cleaner, more standards-compliant, faster API than you are likely to write from scratch.

NocoDB is an open source #NoCode platform that turns any database into a smart spreadsheet. It turns any MySQL, PostgreSQL, SQL Server, SQLite & MariaDB into a smart-spreadsheet.

DBeaver is a free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.

OmniDB is a web-based tool for database management.

Navicat is a series of graphical database management and development software produced by CyberTech Ltd. for MySQL, MariaDB, MongoDB, Oracle, SQLite, PostgreSQL and Microsoft SQL Server.

HeidiSQL is free software, and has the aim to be easy to learn. It lets you see and edit data and structures from computers running one of the database systems MariaDB, MySQL, Microsoft SQL, PostgreSQL and SQLite.

Beekeeper Studio is a cross-platform SQL editor and database manager(MySQL, Postgres, SQLite, SQL Server, and more.) available for Linux, Mac, and Windows.

UI Bakery is a web-based low-code internal tool builder. It can visualize the data pulled from PostgreSQL, MongoDB, MySQL, MicrosoftSQL, Redis.

IBM DB2 is a collection of hybrid data management products offering a complete suite of AI-empowered capabilities designed to help you manage both structured and unstructured data on premises as well as in private and public cloud environments. Db2 is built on an intelligent common SQL engine designed for scalability and flexibility.

OracleDB is a powerful fully managed database helps developers manage business-critical data with the highest availability, reliability, and security.

MariaDB is an enterprise open source database solution for modern, mission-critical applications.

EventQL is a distributed, analytical database. It allows you to store massive amounts of structured data and explore it using SQL and other programmatic query facilities.

CockroachDB is the SQL database for building global, scalable cloud services that survive disasters.

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day.

SQLite Database Browser is an open source SQL tool that allows users to create, design and edits SQLite database files. It lets users show a log of all the SQL commands that have been issued by them and by the application itself.

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.

InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or monitoring and alerting purposes, user dashboards, Internet of Things sensor data, and visualizing and exploring the data and more. It also has support for processing data from Graphite.

Atlas is an in-memory dimensional time series database.

dbWatch is a complete database monitoring/management solution for SQL Server, Oracle, PostgreSQL, Sybase, MySQL and Azure. Designed for proactive management and automation of routine maintenance in large scale on-premise, hybrid/cloud database environments.

Adminer is an SQL management client tool for managing databases, tables, relations, indexes, users. Adminer has support for all the popular database management systems such as MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch and MongoDB.

Knex is a query builder for PostgreSQL, MySQL, CockroachDB, SQL Server, SQLite3 and Oracle, designed to be flexible, portable, and fun to use.

rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine.

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.

SQLModel is a library for interacting with SQL databases from Python code, with Python objects. It is designed to be intuitive, easy to use, highly compatible, and robust.

Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.

DbVisualizer is a SQL management tool that allows users to manage a wide range of databases such as Oracle, Sybase, SQL Server, MySQL, H3, and SQLite.

AppDynamics Database is a management product for Microsoft SQL Server. With AppDynamics you can monitor and trend key performance metrics such as resource consumption, database objects, schema statistics and more, allowing you to proactively tune and fix issues in a High-Volume Production Environment.

Toad is a SQL Server DBMS toolset developed by Quest. It increases productivity by using extensive automation, intuitive workflows, and built-in expertise. This SQL management tool resolve issues, manage change and promote the highest levels of code quality for both relational and non-relational databases.

Lepide SQL Server is an open source storage manager utility to analyse the performance of SQL Servers. It provides a complete overview of all configuration and permission changes being made to your SQL Server environment through an easy-to-use, graphical user interface.

Sequel Pro is a fast MacOS database management tool for working with MySQL. This SQL management tool helpful for interacting with your database by easily to adding new databases, new tables, and new rows.

ElasticSearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

Trino is a Distributed SQL query engine for big data. It is able to tremendously speed up ETL processes, allow them all to use standard SQL statement, and work with numerous data sources and targets all in the same system.

Tableau is a Data Visualization software used in relational databases, cloud databases, and spreadsheets. Tableau was acquired by Salesforce in August 2019.

DataGrip is a professional DataBase IDE developed by Jet Brains that provides context-sensitive code completion, helping you to write SQL code faster. Completion is aware of the tables structure, foreign keys, and even database objects created in code you're editing.

RStudio is an integrated development environment for R and Python, with a console, syntax-highlighting editor that supports direct code execution, and tools for plotting, history, debugging and workspace management.

NoSQL

Back to the Top

NoSQL is a database that is interchangeably referred to as "nonrelational, or "non-SQL" to highlight that the database can handle huge volumes of rapidly changing, unstructured data in different ways than a relational (SQL-based) database with rows and tables.

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB.

Apache Cassandra™ is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Cassandra provides linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

Apache HBase™ is an open-source, NoSQL, distributed big data store. It enables random, strictly consistent, real-time access to petabytes of data. HBase is very effective for handling large, sparse datasets. HBase serves as a direct input and output to the Apache MapReduce framework for Hadoop, and works with Apache Phoenix to enable SQL-like queries over HBase tables.

Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.

Redis(REmote DIctionary Server) is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

FoundationDB is an open source distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. FoundationDB was acquired by Apple in 2015.

CouchbaseDB is an open source distributed multi-model NoSQL document-oriented database. It creates a key-value store with managed cache for sub-millisecond data operations, with purpose-built indexers for efficient queries and a powerful query engine for executing SQL queries.

MongoDB is a document database meaning it stores data in JSON-like documents.

NoSQLBooster is a cross-platform IDE for MongoDB v2.6-5.0, which provides a build-in MongoDB script debugger, SQL query, server monitoring tools, chaining fluent query, query code generator, task scheduling, ES2020 support, and advanced IntelliSense experience.

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time.

Neo4j is a graph database management system that provides an array of tools, libraries, and frameworks to make development faster and easier.

Remote Access

Back to the Top

FreeRDP is a free remote desktop protocol library and clients.

Rustdesk is an open source virtual/remote desktop infrastructure for everyone. Display and control your PC (Windows, macOS, and Linux) and Android devices.

TinyPilot is a tool that enables KVM over IP letting you control any computer remotely.

X2Go is open source remote desktop software for Linux that uses a modified NX 3 protocol. It gives remote access to a Linux system's GUI.

Apache Guacamole is a clientless remote desktop gateway. It supports standard protocols like VNC, RDP, and SSH.

Remmina is a Remote access screen and file sharing to your desktop. It has Remote Access Protocol Plugins for RDP, SSH, SPICE, VNC, X2Go, HTTP/HTTPS.

Remotely is a remote control and remote scripting solution, built with .NET 6, Blazor, SignalR Core, and WebRTC.

P2P Remote Desktop is a portable, no configuration or installation needed remote desktop tool.

Cloudflare Tunnel is a tunneling daemon that proxies traffic from the Cloudflare network to your origins. This daemon sits between Cloudflare network and your origin (a webserver). This attracts client requests and sends them to you via this daemon, without requiring you to poke holes on your firewall and your origin(webserver) can remain as closed as possible.

WireGuard® is a straight-forward, fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec while avoiding the massive headache. WireGuard is designed as a general-purpose VPN for running on embedded interfaces and super computers alike, fit for many circumstances. It's cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Tailscale is a WireGuard-based app that makes secure, private networks easy for teams of any scale. It works like an overlay network between the computers of your networks using all kinds of NAT traversal sorcery.

Headscale is an open source, self-hosted implementation of the Tailscale coordination server.

MeshCentral is a full computer management web site. It can run your own web server to remotely manage and control computers on a local network or anywhere on the internet. Once you get the server started, create device group and download and install an agent on each computer you want to manage.

VNC Viewer is a free remote desktop application that use can use on your iPhone, iPad, Mac, Windows and Linux computers from anywhere in the world.

TightVNC is a free remote desktop application. It can see the desktop of a remote machine and control it with your local mouse and keyboard, just like you would do it sitting in the front of that computer.

KRDC is a client application that allows you to view or even control the desktop session on another machine that is running a compatible server. VNC and RDP is supported.

Krfb Desktop Sharing is a server application that allows you to share your current session with a user on another machine, who can use a VNC client to view or even control the desktop.

wayvnc is a VNC server for wlroots-based Wayland compositors (no_entry Gnome, KDE and Weston are not supported). It attaches to a running Wayland session, creates virtual input devices, and exposes a single display via the RFB protocol.

Waypipe is a proxy for Wayland clients. It forwards Wayland messages and serializes changes to shared memory buffers over a single socket.

Virtualization

Back to the Top

HVM (Hardware Virtual Machine) is a virtualization type that provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.

PV(ParaVirtualization) is an efficient and lightweight virtualization technique introduced by the Xen Project team, later adopted by other virtualization solutions. PV does not require virtualization extensions from the host CPU and thus enables virtualization on hardware architectures that do not support Hardware-assisted virtualization.

Network functions virtualization (NFV) is the replacement of network appliance hardware with virtual machines. The virtual machines use a hypervisor to run networking software and processes such as routing and load balancing. NFV allows for the separation of communication services from dedicated hardware, such as routers and firewalls. This separation means network operations can provide new services dynamically and without installing new hardware. Deploying network components with network functions virtualization only takes hours compared to months like with traditional networking solutions.

Software Defined Networking (SDN) is an approach to networking that uses software-based controllers or application programming interfaces (APIs) to communicate with underlying hardware infrastructure and direct traffic on a network. This model differs from that of traditional networks, which use dedicated hardware devices (routers and switches) to control network traffic.

Virtualized Infrastructure Manager (VIM) is a service delivery and reduce costs with high performance lifecycle management Manage the full lifecycle of the software and hardware comprising your NFV infrastructure (NFVI), and maintaining a live inventory and allocation plan of both physical and virtual resources.

Management and Orchestration(MANO) is an ETSI-hosted initiative to develop an Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV. Two of the key components of the ETSI NFV architectural framework are the NFV Orchestrator and VNF Manager, known as NFV MANO.

Magma is an open source software platform that gives network operators an open, flexible and extendable mobile core network solution. Their mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. Magma is 3GPP generation (2G, 3G, 4G or upcoming 5G networks) and access network agnostic (cellular or WiFi). It can flexibly support a radio access network with minimal development and deployment effort.

OpenRAN is an intelligent Radio Access Network(RAN) integrated on general purpose platforms with open interface between software defined functions. Open RANecosystem enables enormous flexibility and interoperability with a complete openess to multi-vendor deployments.

Open vSwitch(OVS)is an open source production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).

Edge is a distributed computing framework that brings enterprise applications closer to data sources such as IoT devices or local edge servers. This proximity to data at its source can deliver strong business benefits, including faster insights, improved response times and better bandwidth availability.

Multi-access edge computing (MEC) is an Industry Specification Group (ISG) within ETSI to create a standardized, open environment which will allow the efficient and seamless integration of applications from vendors, service providers, and third-parties across multi-vendor Multi-access Edge Computing platforms.

Virtualized network functions(VNFs) is a software application used in a Network Functions Virtualization (NFV) implementation that has well defined interfaces, and provides one or more component networking functions in a defined way. For example, a security VNF provides Network Address Translation (NAT) and firewall component functions.

Cloud-Native Network Functions(CNF) is a network function designed and implemented to run inside containers. CNFs inherit all the cloud native architectural and operational principles including Kubernetes(K8s) lifecycle management, agility, resilience, and observability.

Physical Network Function(PNF) is a physical network node which has not undergone virtualization. Both PNFs and VNFs (Virtualized Network Functions) can be used to form an overall Network Service.

Network functions virtualization infrastructure(NFVI) is the foundation of the overall NFV architecture. It provides the physical compute, storage, and networking hardware that hosts the VNFs. Each NFVI block can be thought of as an NFVI node and many nodes can be deployed and controlled geographically.

Virtualization-based Security (VBS) is a hardware virtualization feature to create and isolate a secure region of memory from the normal operating system.

Hypervisor-Enforced Code Integrity (HVCI) is a mechanism whereby a hypervisor, such as Hyper-V, uses hardware virtualization to protect kernel-mode processes against the injection and execution of malicious or unverified code. Code integrity validation is performed in a secure environment that is resistant to attack from malicious software, and page permissions for kernel mode are set and maintained by the hypervisor.

NVIDIA virtual GPU (vGPU) is a software enables powerful GPU performance for workloads ranging from graphics-rich virtual workstations to data science and AI, enabling IT to leverage the management and security benefits of virtualization as well as the performance of NVIDIA GPUs required for modern workloads.

AMD MxGPU is a hardware-based virtualized GPU solution, is built on industry standard SR-IOV (Single-Root I/O Virtualization) technology and allows multiple virtualized users per physical GPU to work remotely.

Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.

KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.

QEMU is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.

Quickemu is a program that quickly create and run optimised Windows, macOS and Linux desktop virtual machines.

Hyper-V enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.

Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM. The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.

VirtManager is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.

oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible. Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services that provide serverless operational models. It runs workloads in lightweight virtual machines, called microVMs, which combine the security and isolation properties provided by hardware virtualization technology with the speed and flexibility of containers.

Foreman is a free open source project that gives you the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage your servers life cycle, on-premises or in the cloud.

Harvester is an open source hyper-converged infrastructure (HCI) software built on Kubernetes.

Anthos is a modern application management platform that provides a consistent development and operations experience for cloud and on-premises environments.

HyperKit is a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor, based on xhyve/bhyve, which is optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit. HyperKit currently only supports macOS using the Hypervisor.framework making it a core component of Docker Desktop for Mac.

Intel® Graphics Virtualization Technology (Intel® GVT) is a full GPU virtualization solution with mediated pass-through, starting from 4th generation Intel Core (TM) processors with Intel processor graphics(Broadwell and newer). It can be used to virtualize the GPU for multiple guest virtual machines, effectively providing near-native graphics performance in the virtual machine and still letting your host use the virtualized GPU normally.

Apple Hypervisor is a frameowrk that builds virtualization solutions on top of a lightweight hypervisor, without third-party kernel extensions. Hypervisor provides C APIs so you can interact with virtualization technologies in user space, without writing kernel extensions (KEXTs). As a result, the apps you create using this framework are suitable for distribution on the Mac App Store.

Apple Virtualization Framework is a framework that provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. This framework is used to boot and run a Linux-based operating system in a custom environment that you define. It also supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.

Apple Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

Cilicon is a macOS App that leverages Apple's Virtualization Framework to create, provision and run ephemeral virtual machines with minimal setup or maintenance effort. You should be able to get up and running with your self-hosted CI in less than an hour.

Xen is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.

Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).

Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

Password Management

Back to the Top

Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault.

Bitwarden Server is a project contains the APIs, database, and other core infrastructure items needed for the "backend" of all bitwarden client applications. Checkout Bitwarden's self-hosted release repository.

Vaultwarden is an unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs.

Passbolt is an open-source/self-hosted password manager for teams. It allows you to securely share and store credentials. For instance, the wifi password of your office, the administrator password of a router or your organization's social media account passwords, all of them can be secured using passbolt.

KeePassXC is a modern, secure, and open-source password manager that stores and manages your most sensitive information. You can run KeePassXC on Windows, macOS, and Linux systems. It saves many different types of information, such as usernames, passwords, URLs, attachments, and notes in an offline, encrypted file that can be stored in any location, including private and public cloud solutions.

AuthPass.app is an Open-Source Password Manager for mobile and desktop that is Keepass 2.x (kdbx 3.x) compatible.

pass is an open-source unix-based password utilitiy with various gui clients

SSH

Back to The Top

Secure Shell Protocol (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network.

Advanced SSH config is a tool that enhances ssh_config file capabilities, completely transparent.

AutoSSH is a tool that automatically respawn ssh session after network interruption.

ContainersSSH is an SSH Server that Launches Containers in Kubernetes and Docker on demand.

Cluster SSH is a tool that controls a number of xterm windows via a single graphical console.

DSH is a Dancer's shell / distributed shell - Wrapper for executing multiple remote shell commands from one command line.

Flightplan is a node.js library for streamlining application deployment or systems administration tasks (local and remote hosts).

Mosh is a command-line program, like SSH. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux.

Parallel SSH is an asynchronous parallel SSH library designed for large scale automation. It differentiates ifself from alternatives, other libraries and higher level frameworks like Ansible or Chef.

SSH Audit is a tool for SSH server & client configuration auditing (banner, key exchange, encryption, mac, compression, compatibility, security, etc).

Sshwifty is a SSH and Telnet connector made for the Web. It can be deployed on your computer or server to provide SSH and Telnet access interface for any compatible (standard) web browser.

SSHrc is a tool that sources ~/.sshrc on your local computer after logging in remotely.

StormSSH is a command line tool to manage SSH connections.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

VPN

Back to The Top

VPN (Virtual Private Network) is a service that encrypts your internet traffic on unsecured networks to protect your online identity, hide your IP address, and shield your online data from third parties.

Wireguard - New minimal VPN Solution that is very fast.

OpenVPN - Uses a custom security protocol that utilizes SSL/TLS for key exchange.

Pritunl - OpenVPN based solution that's easy to set up.

SoftEther - Multi-protocol software VPN with advanced features.

sshuttle - Poor man's VPN.

strongSwan - Complete IPsec implementation for Linux.

tinc - Distributed p2p VPN.

LDAP

Back to The Top

LDAP Servers

389 Directory Server - Developed by Red Hat.

Apache Directory Server - Apache Software Foundation project written in Java.

Fusion Directory - Improve the Management of the services and the company directory based on OpenLDAP. OpenDJ - Fork of OpenDS.

OpenDS - Another directory server written in Java.

OpenLDAP - Developed by the OpenLDAP Project.

LDAP management

Apache Directory Studio - The Eclipse-based LDAP browser and directory client

Log Management

Back to The Top

Echofish - A web based real-time event log aggregation, analysis, monitoring and management system.

Fluentd - Log Collector and Shipper.

Flume - Distributed log collection and aggregation system.

Graylog2 - Pluggable Log and Event Analysis Server with Alerting options.

Heka - Stream processing system which may be used for log aggregation.

Elasticsearch - A Lucene Based Document store mainly used for log indexing, storage and analysis.

Kibana - Visualize logs and time-stamped data.

Logstash - Tool for managing events and logs.

Octopussy - Log Management Solution (Visualize/Alert/Report).

DNS

Back to The Top

Duckdns - A free service which will point a DNS (sub domains of duckdns.org) to an IP of your choice. The service is completely free, and doesn't require reactivation or forum posts to maintain its existence.

dnsmasq - A lightweight service providing DNS, DHCP and TFTP services to small-scale networks.

MagicDNS is a tool that automatically registers DNS names for devices in your network.

Bind - The most widely used name server software.

djbdns - A collection of DNS applications, including tinydns.

Designate - DNS REST API that support several DNS servers as its backend.

Knot - High performance authoritative-only DNS server.

Lexicon is a tool that provides a way to manipulate DNS records on multiple DNS providers in a standardized way.

NSD - Authoritative only, high performance, simple name server.

PowerDNS - DNS server with a variety of data storage back-ends and load balancing features.

CoreDNS is a DNS server/forwarder, written in Go, that chains plugins. Each plugin performs a (DNS) function.

Unbound - Validating, recursive, and caching DNS resolver.

Yadifa - Lightweight authoritative Name Server with DNSSEC capabilities powering the .eu top-level domain.

Network Tools

Back to the Top

MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth.

Mongoose is a networking library for C/C++. It implements event-driven non-blocking APIs for TCP, UDP, HTTP, WebSocket, MQTT. It is designed for connecting devices and bringing them online.

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

Eclipse Mosquitto is an open source implementation of a server for version 5.0, 3.1.1, and 3.1 of the MQTT protocol.

Ejabberd is an open-source, robust, scalable and extensible realtime platform built using Erlang/OTP, that includes XMPP Server, MQTT Broker and SIP Service.

Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.

LibreSpeed is a network speed test tool that can be run on your LAN or hosted in the cloud.

SmokePing is a deluxe latency measurement tool. It can measure, store and display latency, latency distribution and packet loss. It uses RRDtool to maintain a longterm data-store and to draw pretty graphs, giving up to the minute information on the state of each network connection.

Tailnet is your private network. When you log in for the first time to Tailscale on your phone, laptop, desktop, or cloud VM, a tailnet is created. For personal users, you are a tailnet of many devices and one person. Each device gets a private Tailscale IP address in the CGNAT range and every device can talk directly to every other device, wherever they are on the internet.

Tailscale SSH is a service that allows Tailscale to manage the authentication and authorization of SSH connections on your tailnet.

Tailscale Funnel is a feature that allows you to route traffic from the wider internet to one or more of your Tailscale nodes. You can think of this as publicly sharing a node for anyone to access, even if they don’t have Tailscale themselves.

Cockpit is a web-based graphical interface for servers, intended for everyone. It uses your system’s normal user logins and privileges. Network-wide logins are also supported through single-sign-on and other authentication techniques.

NetBox is a leading solution for modeling and documenting modern networks. It combines the traditional disciplines of IP address management (IPAM) and datacenter infrastructure management (DCIM) with powerful APIs and extensions, NetBox provides the ideal "source of truth" to power network automation.

Network UPS Tools (NUT) is a project that provides support for Power Devices, such as Uninterruptible Power Supplies, Power Distribution Units, Automatic Transfer Switches, Power Supply Units and Solar Controllers. NUT provides a common protocol and set of tools to monitor and manage such devices, and to consistently name equivalent features and data points, across a vast range of vendor-specific protocols and connection media types.

Dnsmasq is a tool that provides network infrastructure for small networks: DNS, DHCP, router advertisement and network boot. It is designed to be lightweight and have a small footprint, suitable for resource constrained routers and firewalls. It has also been widely used for tethering on smartphones and portable hotspots, and to support virtual networking in virtualisation frameworks. Supported platforms include Linux (with glibc and uclibc), Android, BSD, and MacOS.

Nginx proxy manager (NPM) is a reverse proxy management system running on Docker. It's easy to set up and does not require users to know how to work with Nginx servers or SSL certificates.

Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.

Pi-hole is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software, intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations.

OWASP Amass is a tool that performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques.

Smap is a port scanner built with shodan.io's free API. It takes same command line arguments as Nmap and produces the same output which makes it a drop-in replacament for Nmap.

ORY Oathkeeper is an Identity & Access Proxy (IAP) and Access Control Decision API that authorizes HTTP requests based on sets of Access Rules.

Ory Kratos is a developer-friendly, security-hardened and battle-test Identity, User Management and Authentication system for the Cloud. The Kratos identity server (similiar to Auth0, Okta, Firebase) with Ory-hardened authentication, MFA, FIDO2, TOTP, WebAuthn, profile management, identity schemas, social sign in, registration, account recovery, passwordless.

Ory Hydra is a hardened, OpenID Certified OAuth 2.0 Server and OpenID Connect Provider optimized for low-latency, high throughput, and low resource consumption. Ory Hydra is not an identity provider (user sign up, user login, password reset flow), but connects to your existing identity provider through a login and consent app.

Ory Keto is an Open Source (Go) implementation of Zanzibar: Google's Consistent, Global Authorization System. It ships gRPC, REST APIs, newSQL, and an easy and granular permission language. Supports ACL, RBAC, and other access models.

AdGuard Home is a DNS relay station with ad/tracker/other blocking, IP address redirections, and DNS-over-HTTPS.

NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.

Supabase is an open source Firebase alternative. It is building the features of Firebase using enterprise-grade open source tools.

Plik is a scalable & friendly temporary file upload system (Wetransfer like) in golang.

Restify is a framework, utilizing connect style middleware for building REST APIs.

Traefik is an open source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. What sets Traefik apart, besides its many features, is that it automatically discovers the right configuration for your services.

Traefik Mesh is a simple, yet full-featured service mesh. It is container-native and fits as your de-facto service mesh in your Kubernetes cluster. It supports the latest Service Mesh Interface specification SMI that facilitates integration with pre-existing solution.

DuckDNS is a free service that allows you to bind your own favorite subdomain under duckdns.org to the public IP address in use from your router, even though such address is dynamically allocated by your internet service provider and therefore changes over time.

Trust-DNS is a Rust based DNS client, server, and Resolver, built to be safe and secure from the ground up.

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, ease of use, and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

sshuttle is a transparent proxy server that works as a poor man's VPN that forwards connection over ssh. It works with Linux and MacOS and supports DNS tunneling.

NetHopper is a Multi-Cloud Application Network as a Service. The easiest way to visualize, connect, secure, protect, and monitor microservices across any cluster, site, cloud, or network.

Cypress is a tool that makes it fast, easy and reliable testing for anything that runs in a browser.

Kimchi is an HTML5 based management tool for KVM. It is designed to make it as easy as possible to get started with KVM and create your first guest.

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere.

FreeRDP is a free remote desktop protocol library and clients.

Pimox is a port of Proxmox to the Raspberry Pi allowing you to build a Proxmox cluster of Rapberry Pi's or even a hybrid cluster of Pis and x86 hardware.

PiKVM is a very simple and fully functional Raspberry Pi-based KVM over IP.

Firezone is a self-hosted WireGuard®-based VPN server and Linux firewall.

Monoid is an open-source suite of tools for automating data privacy.

Pinecone is an experimental overlay routing protocol suite which is the foundation of the current P2P Matrix demos. It is designed to provide end-to-end encrypted connectivity between devices at a global scale over any compatible medium (currently TCP, WebSockets, Bluetooth Low Energy etc), allowing multi-hop peer-to-peer connectivity between devices even in places where there is no Internet connectivity.

Service Discovery

Back to The Top

Consul is a tool for service discovery, monitoring and configuration. Install Consul on Self-Hosted Kubernetes Clusters.

Linkerd is an ultralight, security-first service mesh for Kubernetes. Linkerd adds critical security, observability, and reliability features to your Kubernetes stack with no code change required.

Doozerd is a highly-available, completely consistent store for small amounts of extremely important data.

Admiral is a tool for for service discovery that provides automatic configuration and service discovery for multicluster Istio service mesh.

ScaleCube is a library that simplifies the development of reactive and distributed applications by providing an embeddable microservices library. It connects distributed microservices in a way that resembles a fabric when viewed collectively. It greatly simplifies and streamlines asynchronous programming and provides a tool-set for managing microservices architecture.

DPS(dns-proxy-server) is a lightweight end user (Developers, Server Administrators) DNS server tool for service discovery, which make it easy to develop in systems where one hostname can solve to different IPs based on the configured environment, so you can:

Solve hostnames from local configuration database.
Solve hostnames from docker containers using docker hostname option or HOSTNAMES env.
Solve hostnames from a list of configured remote DNS servers(as a proxy) if no answer of two above .
Graphic interface to Create/List/Update/Delete A/CNAME records.
Solve host machine IP using host.docker hostname.
Access container by its container name / service name.
Specify from which network solve container IP.

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Self-Hosting-Guide

Self Hosting Guide

A guide for getting started with Self Hosting devices including software and hardware that will make you a better and more efficient Self Hosting.

Table of Contents

Getting Started with Self-Hosting

Tools for Self-Hosting

Containers

CI/CD

Development

Web servers

LLMs

ChatGPT

Running Locally on Windows, MacOS, and Linux:

Automation

Configuration Management

Cloud Storage

Cloud

Linode

Nextcloud

DigitalOcean

Back4app Web Deployment

MinIO Object Storage

Podman

Docker

MacOS

Binary Download for MacOS

Linux

Windows

Install from Source

Advanced options

Databases

SQL

NoSQL

Remote Access

Virtualization

Password Management

SSH

VPN

LDAP

Log Management

DNS

Network Tools

Service Discovery

Security

mriya

Cypher-Notepad

gns3theme

hal-core

search