dorkhub
show your thing

awesome-data-analysis

🚀 500+ curated resources for Data Analysis & Data Science: Python, SQL, Statistics, ML, AI, Visualization, Cheatsheets…

PavelGrigoryevDS
★ 1.7k⑂ 252 forksCC0-1.0updated 1 month ago
aianalyticsawesome-listbig-databusiness-intelligencedashboarddata-analysisdata-science
visit the demogit clone https://github.com/PavelGrigoryevDS/awesome-data-analysis.gitPavelGrigoryevDS/awesome-data-analysis
README.mdfork it — it’s yours

Awesome Data Analysis Awesome

Web Page PRs Welcome CC0

500+ curated resources for data analysis and data science: tools, libraries, roadmaps, cheatsheets, interview guides and more.

📖 For comfortable reading: Web version

🌱 Want to improve? Suggest here or Welcome to Discussions

🌟 Join us in making data analysis more efficient! GitHub stars

Maintained with ❤️

📑 Contents

🏆 Awesome Data Science Repositories

Curated collections of high-quality GitHub repos for inspiration and learning.

⬆ back to contents

🗺️ Roadmaps

Step-by-step guides and skill trees to master data science and analytics.

⬆ back to contents

🐍 Python

Resources

A collection of resources for learning and mastering Python programming.

⬆ back to contents

Data Manipulation with Pandas and Numpy

Tutorials and best practices for working with Pandas and Numpy.

⬆ back to contents

Useful Python Tools for Data Analysis

A collection of Python libraries for efficient data manipulation, cleaning, visualization, validation, and analysis.

Data Processing & Transformation

  • Pandas - Powerful Python library for data analysis and manipulation with flexible data structures.
  • NumPy - Fundamental package for scientific computing in Python with multidimensional array support.
  • Pandas DQ - Data type correction and automatic DataFrame cleaning.
  • Vaex - High-performance Python library for lazy Out-of-Core DataFrames.
  • Polars - Multithreaded, vectorized query engine for DataFrames.
  • Fugue - Unified interface for Pandas, Spark, and Dask.
  • TheFuzz - Fuzzy string matching (Levenshtein distance).
  • DateUtil - Extensions for standard Python datetime features.
  • Arrow - Enhanced work with dates and times.
  • Pendulum - Alternative to datetime with timezone support.
  • Dask - Parallel computing for arrays and DataFrames.
  • Modin - Speeds up Pandas by distributing computations.
  • Pandarallel - Parallel operations for pandas DataFrames.
  • DataCleaner - Python tool for automatically cleaning and preparing datasets.
  • Pandas Flavor - Add custom methods to Pandas.
  • Pandas DataReader - Reads data from various online sources into pandas DataFrames.
  • Sklearn Pandas - Bridge between Pandas and Scikit-learn.
  • CuPy - A NumPy-compatible array library accelerated by NVIDIA CUDA for high-performance computing.
  • Numba - A JIT compiler that translates a subset of Python and NumPy code into fast machine code.
  • Pandas Stubs - Type stubs for pandas, improves IDE autocompletion.
  • Petl - ETL tool for data cleaning and transformation.

⬆ back to contents

Automated EDA and Visualization Tools

  • AutoViz - Automatic data visualization in 1 line of code.
  • Sweetviz - Automatic EDA with dataset comparison.
  • Lux - Automatic DataFrame visualization in Jupyter.
  • YData Profiling - Data quality profiling & exploratory data analysis.
  • Missingno - Visualize missing data patterns.
  • Vizro - Low-code toolkit for building data visualization apps.
  • Yellowbrick - Visual diagnostic tools for machine learning.
  • Great Tables - Create awesome display tables using Python.
  • DataMapPlot - Create beautiful plots of data maps.
  • Datashader - Quickly and accurately render even the largest data.
  • PandasAI - Conversational data analysis using LLMs and RAG.
  • Mito - Jupyter extensions for faster code writing.
  • D-Tale - Interactive GUI for data analysis in a browser.
  • Pandasgui - GUI for viewing and filtering DataFrames.
  • PyGWalker - Interactive UIs for visual analysis of DataFrames.
  • QGrid - Interactive grid for DataFrames in Jupyter.
  • Pivottablejs - Interactive PivotTable.js tables in Jupyter.

⬆ back to contents

Data Quality & Validation

  • PyOD - Outlier and anomaly detection.
  • Alibi Detect - Outlier, adversarial and drift detection.
  • Pandera - Data validation through declarative schemas.
  • Cerberus - Data validation through schemas.
  • Pydantic - Data validation using Python type annotations.
  • Dora - Automate EDA: preprocessing, feature engineering, visualization.
  • Great Expectations - Data validation and testing.

⬆ back to contents

Feature Engineering & Selection

  • FeatureTools - Automated feature engineering.
  • Feature Engine - Feature engineering with Scikit-Learn compatibility.
  • Prince - Multivariate exploratory data analysis (PCA, CA, MCA).
  • Fitter - Figures out the distribution your data comes from.
  • Feature Selector - Tool for dimensionality reduction of machine learning datasets.
  • Category Encoders - Extensive collection of categorical variable encoders.
  • Imbalanced Learn - Handling imbalanced datasets.

⬆ back to contents

Specialized Data Tools

  • cuDF - A GPU DataFrame library for loading, joining, and aggregating data.
  • Faker - Generates fake data for testing.
  • Mimesis - Generates realistic test data.
  • Geopy - Geocoding addresses and calculating distances.
  • PySAL - Spatial analysis functions.
  • Scattertext - Beautiful visualizations of language differences among document types.
  • IGraph - A library for creating and manipulating graphs and networks, with bindings for multiple languages.
  • Joblib - A lightweight pipelining library for Python, particularly useful for saving and loading large NumPy arrays.
  • ImageIO - A library that provides an easy interface to read and write a wide range of image data.
  • Texthero - Text preprocessing, representation and visualization.
  • Geopandas - Geographic data operations with pandas.
  • NetworkX - Network analysis and graph theory.
  • Chardet - Python library to detect the character encoding of text and files.

⬆ back to contents

🗃️ SQL & Databases

Resources

SQL tutorials and database design principles.

⬆ back to contents

Databases

Popular open-source database systems for a variety of use cases and data models.

  • SQLite – Embedded, file‑based SQL engine, the most used database in the world.
  • PostgreSQL – Advanced open‑source relational database with extensibility and SQL compliance.
  • MySQL – Widely used relational database management system (Oracle’s community version).
  • MariaDB – Community developed fork of MySQL with enhanced features.
  • DuckDB – In‑process analytical database designed for fast OLAP queries.
  • ClickHouse – Columnar database for real‑time analytics on large datasets.
  • Apache Cloudberry - A mature open-source MPP database evolved from Greenplum with a newer PostgreSQL kernel.
  • TimescaleDB – Time‑series SQL database built on PostgreSQL.
  • TDengine – Time‑series database for IoT, vehicles, and industrial monitoring.
  • MongoDB – Document database with a flexible, JSON‑like data model.
  • Cassandra – Highly scalable, distributed NoSQL database for large‑scale deployments.
  • Redis – In‑memory data structure store used as database, cache, and message broker.
  • CockroachDB – Distributed SQL database with strong consistency and horizontal scaling.
  • Neo4j – Native graph database for connected data and relationships.
  • InfluxDB – Purpose‑built time‑series database for metrics and events.
  • ScyllaDB – Drop‑in Cassandra alternative written in C++ for higher performance.
  • Apache HBase – Distributed, scalable, big data store modeled after Google's Bigtable, running on top of HDFS.

⬆ back to contents

Tools

A collection of libraries and drivers for seamless database access and interaction.

  • PyODBC - Python library for ODBC database access.
  • SQLAlchemy - SQL toolkit and ORM for Python.
  • Psycopg2 - PostgreSQL database adapter.
  • MySQL Connector/Python - MySQL driver for Python.
  • PonyORM - ORM for Python with dynamic query generation.
  • PyMongo - Official MongoDB driver for Python.
  • SQLiteviz - A tool for exploring SQLite databases and visualizing the results of your queries.
  • DB Browser for SQLite - A high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.
  • DBeaver - A free universal database tool and SQL client for developers, SQL programmers, and administrators.
  • Beekeeper Studio - A modern, easy-to-use SQL client and database manager with a clean, cross-platform interface.
  • DBConvert Streams - Database IDE for exploring, migrating, and syncing PostgreSQL, MySQL, files, and S3.
  • SQLFluff - A modular SQL linter and auto-formatter designed to enforce consistent style and catch errors in SQL code.
  • PyMySQL - A pure-Python MySQL client library for interacting with MySQL databases from Python applications.
  • Vanna.AI - An AI-powered tool for generating SQL queries from natural language questions.
  • SQLChat - A chat-based SQL client that allows you to query databases using natural language conversations.
  • Records - SQL queries to databases via Python syntax.
  • Dataset - JSON-like interface for working with SQL databases.
  • SQLGlot - A no-dependency SQL parser, transpiler, and optimizer for Python.

⬆ back to contents

📊 Data Visualization

Resources

Color theory, chart selection guides, and storytelling tips.

⬆ back to contents

Tools

Libraries for static, interactive, and 3D visualizations.

  • Matplotlib - A comprehensive library for creating static, animated, and interactive visualizations in Python.
  • Seaborn - A statistical data visualization library based on Matplotlib.
  • Plotly - A library for creating interactive plots and dashboards.
  • Altair - A declarative statistical visualization library for Python.
  • Bokeh - A library for creating interactive visualizations for modern web browsers.
  • HoloViews - A tool for building complex visualizations easily.
  • Geopandas - An extension of Pandas for geospatial data.
  • Folium - A library for visualizing data on interactive maps.
  • Pygal - A Python SVG charting library.
  • Plotnine - A grammar of graphics for Python.
  • Bqplot - A plotting library for IPython/Jupyter notebooks.
  • PyPalettes - A large (+2500) collection of color maps for Python.
  • Deck.gl - A WebGL-powered framework for visual exploratory data analysis of large datasets.
  • Python for Geo - Contextily: add background basemaps to your plots in GeoPandas.
  • OSMnx - A package to easily download, model, analyze, and visualize street networks from OpenStreetMap.
  • Apache ECharts - A powerful, interactive charting and visualization library for browser-based applications.
  • VisPy - A high-performance interactive 2D/3D data visualization library leveraging the power of OpenGL.
  • Glumpy - A Python library for scientific visualization that is fast, scalable and beautiful, based on OpenGL.
  • Pandas-bokeh - Bokeh plotting backend for Pandas.
  • QGIS - Free, open source, cross-platform geographic information system (GIS).
  • Flourish - Platform for creating interactive data visualizations and stories without coding.

⬆ back to contents

📈 Dashboards & BI

Resources

Ttutorials for building and enhancing dashboards and visualizations using various tools and frameworks.

⬆ back to contents

Tools

Frameworks for building custom dashboard solutions.

  • Dash - Framework for creating interactive web applications.
  • Streamlit - Simplified framework for building data applications.
  • Panel - Python library for creating custom interactive web apps and dashboards.
  • Gradio - Tool for creating and sharing machine learning applications.
  • OpenSearch Dashboards - A powerful data visualization and dashboarding tool for OpenSearch data, forked from Kibana.
  • GridStack.js - A library for building draggable, resizable responsive dashboard layouts.
  • Tremor - A React library to build dashboards fast with pre-built components for charts, KPIs, and more.
  • Appsmith - An open-source platform to build and deploy internal tools, admin panels, and CRUD apps quickly.
  • Grafanalib - A Python library for generating Grafana dashboards configuration as code.
  • H2O Wave - A Python framework for rapidly building and deploying realtime web apps and dashboards for AI and analytics.
  • Shiny for Python - Python version of the popular R Shiny framework.
  • Voilà - Turn Jupyter notebooks into standalone web applications.
  • Reflex - Full-stack Python framework for building web apps.
  • Taipy - Python library for building web applications and interactive dashboards.
  • Evidence - Business intelligence platform that uses SQL and Markdown for reports.

⬆ back to contents

Software

A list of leading tools and platforms for data visualization and dashboard creation.

  • Tableau - Leading data visualization software.
  • Microsoft Power BI - Business analytics tool for visualizing data.
  • QlikView - Tool for data visualization and business intelligence.
  • Metabase - User-friendly open-source BI tool.
  • Apache Superset - Open-source data exploration and visualization platform.
  • Preset - A platform for modern business intelligence, providing a hosted version of Apache Superset.
  • Metabase - The simplest way to get analytics and business intelligence for everyone in your company.
  • Redash - Tool for visualizing and sharing data insights.
  • Grafana - Dashboarding and monitoring tool.
  • Datawrapper - User-friendly chart and map creation tool.
  • ChartBlocks - Online chart creation platform.
  • Infogram - Tool for creating infographics and visual content.
  • Google Data Studio - Free tool for creating interactive dashboards and reports.
  • Rath - Next-generation automated data exploratory analysis and visualization platform.
  • Kibana - The official visualization and dashboarding tool for the Elastic Stack (Elasticsearch, Logstash, Beats).

⬆ back to contents

🕸️ Web Scraping & Crawling

Resources

A collection of valuable resources, tutorials, and libraries for web scraping with Python.

⬆ back to contents

Tools

A list of libraries and tools for web scraping.

  • Requests - A simple, yet elegant, HTTP library for Python.
  • BeautifulSoup - A library for parsing HTML and XML documents.
  • Selenium - A tool for automating web applications for testing purposes.
  • Scrapy - An open-source and collaborative web crawling framework for Python.
  • Browser Use - A library for browser automation and web scraping.
  • Gerapy - Distributed Crawler Management Framework based on Scrapy, Scrapyd, Django, and Vue.js.
  • AutoScraper - A smart, automatic, fast, and lightweight web scraper for Python.
  • Feedparser - A library to parse feeds in Python.
  • Trafilatura - A Python & command-line tool to gather text and metadata on the web.
  • You-Get - A tiny command-line utility to download media contents (videos, audios, images) from the web.
  • MechanicalSoup - A Python library for automating interaction with websites.
  • ScrapeGraph AI - A Python scraper based on AI.
  • Snscrape - A social networking service scraper in Python.
  • Ferret - A web scraping system that lets you declaratively describe what data to extract using a simple query language.
  • Grab - A Python framework for building web scraping apps, providing a high-level API for asynchronous requests.
  • Playwright - Python version of the Playwright browser automation library.
  • PyQuery - A jQuery-like library for parsing HTML documents in Python.
  • Helium - High-level Selenium wrapper for easier web automation.
  • Scrapling - A framework for building web scrapers and crawlers.
  • Crawl4AI - Advanced web crawling framework designed for AI and data extraction tasks.

⬆ back to contents

🔢 Mathematics

A collection of resources for learning mathematics, particularly in the context of data science and machine learning.

⬆ back to contents

🎲 Statistics & Probability

Resources

A selection of resources focused on statistics and probability, including tutorials and comprehensive guides.

⬆ back to contents

Tools

A collection of tools focused on statistics and probability.

  • SciPy - Fundamental library for scientific computing and statistics.
  • Statsmodels - Statistical modeling, testing, and data exploration.
  • PyMC - A probabilistic programming library for Python that allows for flexible Bayesian modeling.
  • Pingouin - Statistical package with improved usability over SciPy.
  • scikit-posthocs - Post-hoc tests for statistical analysis of data.
  • Lifelines - Survival analysis and event history analysis in Python.
  • scikit-survival - Survival analysis built on scikit-learn for time-to-event prediction.
  • Bootstrap - Bootstrap confidence interval estimation methods.
  • PyStan - Python interface to Stan for Bayesian statistical modeling.
  • ArviZ - Exploratory analysis of Bayesian models with visual diagnostics.
  • PyGAM - A Python library for generalized additive models with built-in smoothing and regularization.
  • NumPyro - A probabilistic programming library built on JAX for high-performance Bayesian modeling.
  • Causal Impact - A Python implementation of the R package for causal inference using Bayesian structural time-series models.
  • DoWhy - A Python library for causal inference that supports explicit modeling and testing of causal assumptions.
  • Patsy - A Python library for describing statistical models and building design matrices.
  • Pomegranate - Fast and flexible probabilistic modeling library for Python with GPU support.
  • Pgmpy - Python library for probabilistic and causal inference using graphical models.

⬆ back to contents

🧪 A/B Testing

A collection of resources focused on A/B testing.

⬆ back to contents

⏳ Time Series Analysis

Resources

A collection of resources for understanding time series fundamentals and analytical techniques.

⬆ back to contents

Tools

A collection of tools for working with temporal data.

  • Facebook Prophet - A procedure for forecasting time series data based on an additive model.
  • Uber Orbit - A Python package for Bayesian time series forecasting and inference.
  • sktime - A unified Python framework for machine learning with time series, compatible with scikit-learn.
  • GluonTS - A Python toolkit for probabilistic time series modeling, built on MXNet.
  • Time-Series-Library - A library for deep learning-based time series analysis and forecasting.
  • TimesFM - A pretrained time series foundation model from Google Research for zero-shot forecasting.
  • PyTorch Forecasting - A PyTorch-based library for time series forecasting with neural networks.
  • Time-series-prediction - A collection of time series prediction methods and implementations.
  • PlotJuggler - A tool to visualize and analyze time series data logs in real-time.
  • TSFresh - Automatically extracting features from time series data.
  • pmdarima - Python library for ARIMA modeling and time series analysis.
  • Kats - Toolkit for analyzing time series data from Facebook Research.

⬆ back to contents

⚙️ Data Engineering

Resources

A collection of resources to help you build and manage robust data pipelines and infrastructure.

⬆ back to contents

Tools

A collection of tools for building, deploying, and managing data pipelines and infrastructure.

  • dbt-core - A framework for transforming data in your warehouse using SQL and Jinja.
  • Apache Spark - A unified engine for large-scale data processing and analytics.
  • Apache Kafka - A distributed event streaming platform for building real-time data pipelines.
  • Dagster - A data orchestrator for machine learning, analytics, and ETL.
  • Apache Airflow - A platform to programmatically author, schedule, and monitor workflows.
  • Apache Hive - A data warehouse software for reading, writing, and managing large datasets in distributed storage using SQL.
  • Apache Hadoop - A framework that allows for the distributed processing of large data sets across clusters of computers.
  • Luigi - A Python module for building complex and batch-oriented data pipelines.
  • Apache Iceberg - A high-performance table format for huge analytic datasets.
  • Apache Cassandra - A highly scalable distributed NoSQL database designed for handling large amounts of data across many commodity servers.
  • Apache Flink - A framework for stateful computations over unbounded and bounded data streams (real-time stream processing).
  • Apache Beam - A unified model for defining both batch and streaming data-parallel processing pipelines.
  • Apache Pulsar - A cloud-native, distributed messaging and streaming platform.
  • Delta Lake - A storage layer that brings ACID transactions to Apache Spark and big data workloads.
  • Apache Hudi - An open data lakehouse platform, built on a high-performance open table format.
  • Trino - A distributed SQL query engine designed for fast analytic queries against large datasets.
  • DataHub - A metadata platform for the modern data stack.
  • OpenLineage - An open framework for collection and analysis of data lineage.
  • Kedro - A framework for creating reproducible, maintainable and modular data science code.
  • Apache Calcite - A dynamic data management framework that allows for SQL parsing, optimization, and federation.
  • Prefect - Workflow orchestration for building resilient data pipelines.
  • Apache Arrow - Universal columnar format and multi-language toolbox for fast data interchange.
  • Kestra - An open-source, event-driven orchestrator that simplifies data workflow management.
  • Conductor - Orchestration engine for running complex, multi-step workflows and business processes.

⬆ back to contents

📖 Natural Language Processing (NLP)

Resources

A selection of resources for learning and applying natural language processing in Python.

⬆ back to contents

Tools

A collection of powerful libraries and frameworks for natural language processing.

  • Natural Language Toolkit (NLTK) - A leading platform for building Python programs to work with human language data.
  • TextBlob - A simple library for processing textual data.
  • SpaCy - An open-source software library for advanced NLP in Python.
  • BERT - A transformer-based model for NLP tasks.
  • Flair - A simple framework for state-of-the-art NLP.
  • OpenHands - A library and framework for building applications with large language models.
  • Stanford CoreNLP - A Java suite of core NLP tools providing fundamental linguistic analysis capabilities.
  • John Snow Labs Spark-NLP - A state-of-the-art Natural Language Processing library built on Apache Spark.
  • TextAttack - A Python framework for adversarial attacks, data augmentation, and model training in NLP.
  • Gensim - Topic modeling and natural language processing library for Python.
  • Stanza - Python NLP library for many human languages, from the Stanford NLP Group.
  • SentenceTransformers - Framework for state-of-the-art sentence and text embeddings.
  • LangExtract - Google's library for structured information extraction from text using language models.
  • Rasa - Open-source framework for building contextual AI assistants and chatbots.

⬆ back to contents

🤖 Machine Learning & AI

Resources

A collection of resources to help you learn and apply machine learning concepts and techniques.

⬆ back to contents

Tools

A collection of tools for developing and deploying machine learning models.

Machine Learning

  • Scikit-learn - Machine learning library for classical algorithms and model building.
  • XGBoost - Optimized distributed gradient boosting library for tree-based models.
  • LightGBM - Fast, distributed, high-performance gradient boosting framework.
  • CatBoost - High-performance gradient boosting on decision trees with categorical features support.
  • H2O-3 - Open-source distributed machine learning platform.
  • cuML - GPU-accelerated machine learning algorithms from RAPIDS.
  • dlib - Modern C++ toolkit containing machine learning algorithms and tools.
  • SHAP - Game theoretic approach to explain the output of any machine learning model.
  • InterpretML - Fit interpretable models and explain blackbox machine learning.
  • Optuna - Hyperparameter optimization framework.

Deep Learning

  • TensorFlow - End-to-end open source platform for machine learning and deep learning.
  • PyTorch - Deep learning framework with strong support for research and production.
  • PyTorch Lightning - PyTorch wrapper for high-performance AI research.
  • PyTorch Ignite - High-level library to help with training and evaluating neural networks.
  • Keras - High-level neural networks API, running on top of TensorFlow.
  • Fast.ai - Deep learning library simplifying training fast and accurate neural nets.
  • HuggingFace Transformers - Model-definition framework for state-of-the-art machine learning models.
  • HuggingFace Diffusers - Library for state-of-the-art pretrained diffusion models.
  • PEFT - Library for efficiently adapting large pretrained models.
  • YOLOv5 - Real-time object detection system.
  • Ultralytics - YOLOv8 and other computer vision models.
  • ONNX - Open standard for machine learning interoperability.
  • PyTorch Geometric - Geometric deep learning extension library for PyTorch.
  • Pyro - Deep universal probabilistic programming with Python and PyTorch.
  • Skorch - Scikit-learn compatible neural network library.
  • Sonnet - DeepMind's library for building complex neural networks.
  • JAX - Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more.
  • TensorFlow Models - Official TensorFlow repository with models and examples.
  • Fenn - A simple framework that automates ML/DL workflows by providing prebuilt trainers, templates, logging, configuration management, and much more.

⬆ back to contents

🚀 MLOps

Resources

Materials and curated lists for machine learning operations.

⬆ back to contents

Tools

Platforms and utilities for deploying, monitoring, and maintaining ML systems.

  • ColossalAI - High-performance distributed training framework.
  • DVC - Version control system for machine learning projects.
  • Evidently - Tool for analyzing and monitoring data and model drift.
  • Deepchecks - Validation for ML models and data.
  • Sematic - Tool to build, debug, and execute ML pipelines with native Python.
  • netdata - Real-time performance monitoring.
  • meilisearch - Fast, open-source search engine.
  • vLLM - High-throughput and memory-efficient inference library for LLMs.
  • haystack - LLM framework for building search and question answering systems.
  • Kubeflow - Machine learning toolkit for Kubernetes.
  • Seldon Core - Open source platform for deploying and monitoring machine learning models in production.
  • Feast - A feature store for machine learning that manages and serves ML features to models.
  • BentoML - Framework for building, shipping, and scaling ML applications.
  • MLflow - Open-source platform for the complete machine learning lifecycle.
  • Wandb - Tool for experiment tracking, dataset versioning, and model management.
  • Comet ML - ML platform for tracking, comparing and optimizing experiments.
  • Netflix Metaflow - A human-friendly Python library for helping scientists and engineers build and manage real-life data science projects.
  • mindsdb - Platform for integrating AI into databases and applications.
  • KServe - Standardized serverless inference platform for deploying and serving machine learning models on Kubernetes.
  • SQLFlow - Brings machine learning capabilities to SQL, enabling model training and prediction using SQL syntax.
  • Jina AI Serve - Framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets.
  • LiteLLM - Unified interface to call all LLM APIs (OpenAI, Anthropic, Cohere, etc.) with consistent output formatting.

⬆ back to contents

🧠 AI Applications & Platforms

Resources

A collection of resources focused on AI applications and platforms.

  • Awesome LLM Apps - Collection of awesome LLM apps with AI Agents and RAG using OpenAI, Anthropic, Gemini and opensource models.
  • Awesome Generative AI - A curated list of modern Generative Artificial Intelligence projects and services.
  • AI Agents for Beginners - Microsoft's course on designing and building AI agents.
  • Generative AI for Beginners - Course on generative AI for beginners from Microsoft.
  • Ai Dev Tools Zoomcamp - Free hands-on course on modern tools for building and deploying AI applications.
  • LLM Course - Practical course to master large language models from start to finish.
  • Awesome AI Agents - A curated list of AI autonomous agents, environments, and frameworks.
  • AI Collection - The Generative AI Landscape - A Collection of Awesome Generative AI Applications.
  • Awesome AI Apps - A collection of projects showcasing RAG, agents, workflows, and other AI use cases.
  • System Prompts and Models - System Prompts, Internal Tools & AI Models from various AI applications and coding tools.
  • RAG Techniques - Collection of advanced techniques for Retrieval-Augmented Generation.
  • Awesome LangChain - Awesome list of tools and projects with the awesome LangChain framework.
  • Awesome AI Tools - A curated list of Artificial Intelligence Top Tools.
  • Awesome LLM Security - A curation of awesome tools, documents and projects about LLM Security.
  • Claude Cookbooks - Official Anthropic examples and recipes for working with Claude AI.
  • Hands On Large Language Models - Covers LLM fundamentals, prompt engineering, and fine-tuning.
  • AI Engineering Hub - Resources for building, deploying, and maintaining AI systems.
  • Agents Towards Production - Code-first tutorials for building production-grade GenAI agents.
  • LLM Engineer Toolkit - Curated list of 120+ LLM libraries across various categories.
  • GenAI Agents - Repository of AI agent implementations and tutorials.
  • AI Notes - Personal notes and essays on AI and software development.
  • Open LLMs - Comprehensive list of open-source large language models and their capabilities.
  • Prompt Engineering Guide - Guides, papers, and resources for prompt engineering with LLMs.
  • Prompt Engineering - Collection of prompt engineering techniques and strategies.
  • 500 AI Agents Projects - 500+ AI agent projects with code for learning and inspiration.
  • Generative AI - Roadmap and resources for mastering generative AI technologies.
  • Awesome N8N - Collection of templates, integrations, and resources for the n8n automation platform.
  • Free Llm Api Resources - Up-to-date list of free APIs for accessing large language models (LLMs).

⬆ back to contents

Tools

A collection of frameworks, platforms, and end-user applications for building and deploying AI-powered solutions.

AI Agents & Automation

  • n8n - Workflow automation platform for connecting APIs and services.
  • crewAI - Framework for orchestrating role-playing AI agents.
  • autogen - Framework for building multi-agent conversational systems.
  • AutoGPT - Autonomous AI agent that can complete complex tasks.
  • LangGraph - Framework for building stateful, multi-actor applications with LLMs, with cycles and control flow.
  • Agents.md - Open source framework for building agentic AI systems. ogrammatically.
  • OpenManus - Open-source platform for building and deploying AI agents.
  • youtu-agent - Multi-modal intelligent agent framework by Tencent Cloud.
  • trae-agent - Tool-using reasoning agent with execution-augmented reasoning.
  • deepagents - LangChain framework for building sophisticated multi-agent systems.
  • mem0 - AI memory system for long-term context and personalized interactions.
  • web-ui - AI-powered browser automation framework for web interaction.
  • Agent-S - Open agentic framework that autonomously interacts with computer GUIs like a human.
  • Mastra - Open-source AI agent platform for building and scaling production-grade autonomous agents.
  • Langflow - Powerful visual platform for building and deploying AI-powered agents and workflows.
  • agenticSeek - Framework for building and deploying AI agents with advanced reasoning and tool use.
  • Flowise - Open-source UI visual tool for building custom LLM orchestration flows and AI agents.
  • MetaGPT - Multi-agent framework that simulates roles in a software company to build projects.
  • Local Deep Research - Local AI research assistant that searches web, papers, and documents.
  • Gptme - AI agent CLI that writes code, uses terminal, browses web, and runs locally.
  • Rowboat - Open-source AI coworker that learns from your emails/meetings to automate drafting, prep, and tasks.
  • Everyrow - AI-powered data operations SDK. Semantic deduplication, fuzzy merging, and intelligent ranking for data analysis workflows.
  • Personal Ai Infrastructure - Framework for building a personal AI assistant with memory, skills, and learning ability.
  • N8N Workflows - Collection of ready-to-use workflow templates for the n8n automation platform.
  • Skyvern - AI browser automation using LLMs & computer vision. Playwright-compatible SDK + no-code workflows.
  • OpenWork - Open-source desktop alternative to Claude Cowork for running agents, skills, and MCP locally with team collaboration features.
  • DeepAnalyze - Agentic LLM for autonomous data science, which can autonomously complete a wide range of data-centric tasks without human intervention.

Development Frameworks & Tools

  • LangChain - Framework for developing applications powered by language models.
  • LlamaIndex - Data framework for LLM-based applications with RAG capabilities.
  • openai-python - Official Python library for OpenAI API.
  • openai-agents-python - Official OpenAI framework for building AI agents.
  • ragflow - Open-source RAG (Retrieval-Augmented Generation) workflow platform.
  • firecrawl - Web crawling and data extraction service for AI applications.
  • Fabric - Framework for augmenting humans using AI.
  • Dyad - Open-source platform for building AI applications with custom API keys.
  • Langflow - Powerful visual platform for building and deploying AI-powered agents and workflows.
  • NeMo - Scalable generative AI framework from NVIDIA for LLMs, Multimodal, and Speech AI.
  • Deepcode - AI-powered agent framework for automatic code generation from research papers and text.

Code Generation & Assistance

  • gpt-engineer - AI-powered code generation tool.
  • gpt-pilot - AI pair programmer that writes entire applications.
  • tabby - Self-hosted AI coding assistant.

Model Deployment & Platforms

  • Ollama - Tool for running large language models locally.
  • OpenLLM - Open platform for operating large language models in production.
  • LocalAI - Self-hosted, local-first AI model deployment platform.
  • dify - Visual LLM application development platform.
  • LLaMA-Factory - Easy-to-use LLM fine-tuning framework.
  • unsloth - Library for faster and more memory-efficient LLM fine-tuning.
  • LocalGPT - Fully private, on-premise document intelligence platform for chatting with your documents using local LLMs.

AI Reliability & Debugging

  • DeepEval - Pytest-style unit testing framework for LLMs. Metrics for RAG, agents, hallucination, summarization, and custom criteria.
  • RAGAS - Evaluation toolkit for LLM apps. Metrics, test generation, and insights for optimizing RAG pipelines and agents.
  • Phoenix - AI observability platform. Tracing, datasets, experiments, and playground for troubleshooting and evaluating LLM apps.
  • WFGY - Open-source debugging infrastructure for RAG and AI agents. Includes 16-problem RAG failure map and TXT stress-test engine.

End-User Applications

  • open-webui - Web interface for interacting with various LLMs.
  • ComfyUI - Visual node-based interface for Stable Diffusion.
  • lobe-chat - Modern AI conversation interface.
  • LibreChat - Open-source ChatGPT alternative.
  • quivr - Personal second brain and AI assistant.
  • upscayl - AI-powered image upscaling tool.
  • facefusion - AI face swapping and enhancement tool.
  • DocsGPT - Documentation-based question answering system.
  • Deep Research - AI-powered research assistant for iterative, deep research on any topic.
  • Screenpipe - Local AI that records, searches, and automates tasks based on your screen and audio.
  • Jaaz - Open-source multimodal creative assistant and privacy-focused alternative to Canva/Manus for local image/video generation.
  • DeepTutor - AI-powered personalized learning assistant with document Q&A, exercise generation, and deep research capabilities.

Additional Tools

  • Bagel - Open-source unified multimodal model for understanding and generating images.
  • Whisper - Robust speech recognition model for transcription and translation.
  • ChatTTS - Generative TTS model optimized for natural, expressive daily dialogue with fine-grained prosody control.
  • NeuTTS - On-device TTS model with instant voice cloning from audio samples..
  • Everything Claude Code - Collection of resources, guides, and tools for effective Claude Code AI assistant use.

⬆ back to contents

☁️ Cloud Platforms & Infrastructure

Resources

A collection of resources for mastering cloud-native technologies, containerization, and infrastructure management.

⬆ back to contents

Tools

Tools for containerization, orchestration, infrastructure as code, and cloud-native development.

Containerization & Orchestration

  • Docker - Open platform for developing, shipping, and running applications in containers.
  • Docker Compose - A tool for defining and running multi-container Docker applications.
  • Kubernetes - Production-grade container orchestration system.
  • Kompose - Conversion tool from Docker Compose to Kubernetes.

Infrastructure as Code

  • Terraform - Infrastructure as Code tool.
  • OpenTofu - Open source fork of Terraform.
  • Pulumi - Modern IaC platform using familiar programming languages.
  • CDK8s - Define Kubernetes apps using familiar languages.

CI/CD & GitOps

  • Jenkins - Open source automation server.
  • Argo CD - Declarative GitOps continuous delivery.
  • Argo Workflows - Container-native workflow engine.
  • Tekton - Kubernetes-native CI/CD framework.
  • Spinnaker - Multi-cloud continuous delivery.
  • Dagger - Portable devkit for CI/CD pipelines.

Service Mesh & API Gateways

  • Traefik - Modern HTTP reverse proxy and load balancer.
  • Kong - Cloud-native API Gateway.
  • Apache APISIX - Dynamic API gateway.
  • Envoy Gateway - Manages Envoy Proxy as gateway.
  • Higress - Cloud-native API gateway based on Istio.
  • Meshery - Service mesh management.

Kubernetes Ecosystem

  • Helm - Package manager for Kubernetes.
  • Kustomize - Configuration customization for Kubernetes.
  • Kubernetes Dashboard - Web-based UI for Kubernetes.
  • Skaffold - Continuous development for Kubernetes.
  • Tilt - Local development for Kubernetes.
  • Flagger - Progressive delivery operator.
  • KubeVela - Application delivery platform.
  • KubeSphere - Kubernetes multi-cloud management.

Developer Platforms & Control Planes

Additional Tools

  • Vagrant - Tool for building and managing portable virtual development environments as code.

⬆ back to contents

⚡ Productivity

Resources

A collection of resources to enhance productivity.

  • Positron - A next-generation data science IDE.
  • Nanobrowser - An open-source AI web automation tool with multi-agent system that runs directly in your browser.
  • Best of Jupyter - Ranked list of notable Jupyter Notebook, Hub, and Lab projects.
  • Deepnote - AI native data science notebook platform compatible with Jupyter, featuring real-time collaboration, environment management, and integrations.
  • AFFiNE - All-in-one workspace for notes, docs, and data visualization.
  • Marimo - Reactive Python notebook for reproducible and interactive data science.
  • ChatGPT Data Science Prompts - A collection of useful prompts for data scientists using ChatGPT.
  • Gamma.app - AI-powered platform for creating and sharing presentations and documents.
  • Cookiecutter Data Science - A standardized project structure for data science projects.
  • Learn Regex - Comprehensive guide to learning regular expressions with examples and exercises.
  • Awesome Regex - Curated collection of regex tools, libraries, and learning resources.
  • The Markdown Guide - Comprehensive guide to learning Markdown.
  • Readme-AI - A tool to automatically generate README.md files for your projects.
  • Markdown Here - Extension for writing emails in Markdown and rendering them before sending.
  • MarkText - Simple and elegant markdown editor for documentation.
  • QuarkDown - Lightweight markdown processor for fast document rendering.
  • screenshot-to-code - AI tool that converts screenshots into code for various frontend stacks.
  • Codebeautify - All-in-one online code formatter and beautifier for Python, SQL, JSON, and more.
  • Notion - An all-in-one workspace for note-taking and task management.
  • Trello - A visual project management tool.
  • Habitica - A habit-building and productivity app that treats your life like a role-playing game.
  • Bujo - Tools to help transform the way you work and live.
  • Parabola - An AI-powered workflow builder for organizing data.
  • Asana - A project management platform for tracking work and projects.
  • Puter - An open-source, browser-based computing environment and cloud OS.
  • Milkdown - Plugin-driven, WYSIWYG markdown editor framework inspired by Typora.
  • PDFMathTranslate - AI tool for bilingual scientific PDF translation preserving formulas, charts, and layout.

⬆ back to contents

Useful Linux Tools

A selection of tools to enhance productivity and functionality in Linux environments.

  • tldr-pages - Simplified and community-driven man pages with practical examples.
  • Bat - Cat clone with syntax highlighting.
  • Exa - Modern replacement for ls.
  • Ripgrep - Faster grep alternative.
  • Zoxide - Smarter cd command.
  • Peek - Simple animated GIF screen recorder with an easy to use interface.
  • CopyQ - Clipboard manager with advanced features.
  • Translate Shell - Command-line translator using Google Translate, Bing Translator, Yandex.Translate, etc.
  • Espanso - Cross-platform Text Expander written in Rust.
  • Flameshot - Powerful yet simple to use screenshot software.
  • DrawIO Desktop - An open-source diagramming software for making flowcharts, process diagrams, and more.
  • Inkscape - A powerful, free, and open-source vector graphics editor for creating and editing visualizations.
  • Rclone - A command-line program to manage files on cloud storage.
  • Rsync - A fast and versatile file copying tool that can synchronize files and directories between two locations over a network or locally.
  • Timeshift - System restore tool for Linux that creates filesystem snapshots using rsync+hardlinks or BTRFS snapshots.
  • Backintime - A comfortable and well-configurable graphical frontend for incremental backups.
  • Fzf - A command-line fuzzy finder.
  • Osquery - SQL powered operating system instrumentation, monitoring, and analytics.
  • GNU Parallel - A tool to run jobs in parallel.
  • HTop - An interactive process viewer.
  • Ncdu - A disk usage analyzer with an ncurses interface.
  • Thefuck - A command line tool to correct your previous console command.
  • Miller - A tool for querying, processing, and formatting data in various file formats (CSV, JSON, etc.), like awk/sed/cut for data.
  • jq - Command-line JSON processor for parsing and manipulating JSON data.
  • yq - Portable command-line YAML processor (like jq for YAML and XML).
  • q - Run SQL directly on CSV or TSV files from the command line.
  • VisiData - Interactive multitool for tabular data exploration in the terminal.
  • csvkit - Suite of command-line tools for working with CSV data.
  • httpie - Modern command-line HTTP client for API testing and debugging.
  • glances - Cross-platform system monitoring tool for resource usage analysis.
  • hyperfine - Command-line benchmarking tool for performance testing.
  • termgraph - Draw basic graphs in the terminal for quick data visualization.
  • fd - Simple, fast and user-friendly alternative to 'find'.
  • dust - More intuitive version of du written in rust.
  • bottom - Cross-platform graphical process/system monitor.
  • Keychain - Tool for managing and securely storing passwords and secrets.

⬆ back to contents

Useful VS Code Extensions

A collection of extensions to enhance functionality and productivity in Visual Studio Code.

⬆ back to contents

📚 Skill Development & Career

Practice Resources

A collection of resources to enhance skills and advance your career in data analysis and related fields.

⬆ back to contents

Curated Jupyter Notebooks

A selection of curated Jupyter notebooks to support learning and exploration in data science and analysis.

⬆ back to contents

Data Sources & Datasets

A collection of resources for accessing datasets and data sources for analysis and projects.

more like this

perplexity-cli

🧠 A simple command-line client for the Perplexity API. Ask questions and receive answers directly from the terminal! 🚀🚀🚀

Python★ 176
@dawid-szewc

agentlytics

Comprehensive analytics dashboard for AI coding agents — Cursor, Windsurf, Claude Code, VS Code Copilot, Zed, Antigravi…

JavaScript★ 555
@f

AeroPath

:hugs: AeroPath: An airway segmentation benchmark dataset with challenging pathology

Jupyter Notebook★ 53
@raidionics

search

search projects, people, and tags