Awesome Data Science with Python

A curated list of awesome resources for practicing data science using Python, including not only libraries, but also links to tutorials, code snippets, blog posts and talks.

Core

pandas - Data structures built on top of numpy.
scikit-learn - Core ML library, intelex.
matplotlib - Plotting library.
seaborn - Data visualization library based on matplotlib.
ydata-profiling - Descriptive statistics using ProfileReport.
sklearn_pandas - Helpful DataFrameMapper class.
missingno - Missing data visualization.
rainbow-csv - VSCode plugin to display .csv files with nice colors.

General Python Programming

Advanced Python Features - Generics, Protocols, Structural Pattern Matching and more.
uv - Dependency management.
pdm - For large binary distributions, works with uv.
just - Command runner. Replacement for make.
python-dotenv - Manage environment variables.
structlog - Python logging.
more_itertools - Extension of itertools.
tqdm - Progress bars for for-loops. Also supports pandas apply().
hydra - Configuration management.
ruff - Extremely fast Python linter and code formatter, replaces flake8, black, and isort.

Pandas Tricks, Alternatives and Additions

duckdb - Efficiently run SQL queries on pandas DataFrame, duckplyr for R, Great Intro.
ducklake - Duckdb extention for storing data in a datalake.
fireducks - Speedier alternative to pandas with similar API.
pandasvault - Large collection of pandas tricks.
polars - Multi-threaded alternative to pandas.
xarray - Extends pandas to n-dimensional arrays.
mlx - An array framework for Apple silicon.
pandas_flavor - Write custom accessors like .str and .dt.
daft - Distributed DataFrame.
vaex - Out-of-Core DataFrames.
modin - Parallelization library for faster pandas DataFrame.
swifter - Apply any function to a pandas DataFrame faster (works with modin).
narwhals - Write dataframe-agnostic code compatible with pandas, polars, cuDF, and more.

Tables

great-tables - Display tabular data nicely.

Interactive Dataframe Visualization

pygwalker - Interactive dataframe.
marimo - Visualization and reproducible environment.
lux - DataFrame visualization within Jupyter.
dtale - View and analyze Pandas data structures, integrating with Jupyter.
pandasgui - GUI for viewing, plotting and analyzing Pandas DataFrames.
quak - Scalable, interactive data table, twitter.
data-formulator - Data visualization tool.

Environment and Jupyter

Jupyter Tricks
nteract - Open Jupyter Notebooks with doubleclick.
papermill - Parameterize and execute Jupyter notebooks, tutorial.
nbdime - Diff two notebook files, Alternative GitHub App: ReviewNB.
RISE - Turn Jupyter notebooks into presentations.
handcalcs - More convenient way of writing mathematical equations in Jupyter.
notebooker - Productionize and schedule Jupyter Notebooks.
voila - Turn Jupyter notebooks into standalone web applications. Voila grid layout.

Jupyter Alternatives

positron - Data Science IDE.
Deepnote - Data Science platform with real-time collaboration, environment management.

Extraction + OCR

textract - Extract text from any document.
docling - Text extraction.
DeepSeek-OCR - OCR.
chandra - OCR.

Big Data

spark - DataFrame for big data, cheatsheet, tutorial.
dask, dask-ml - Pandas DataFrame for big data and machine learning library, resources, talk1, talk2, notebooks, videos.
h2o - Helpful H2OFrame class for out-of-memory dataframes.
cuDF - GPU DataFrame Library, Intro.
cupy - NumPy-like API accelerated with CUDA.
ray - Flexible, high-performance distributed execution framework.
bottleneck - Fast NumPy array functions written in C.
petastorm - Data access library for parquet files by Uber.
zarr - Distributed NumPy arrays.
NVTabular - Feature engineering and preprocessing library for tabular data by Nvidia.
tensorstore - Reading and writing large multi-dimensional arrays (Google).

Command line tools, CSV

csvkit - Command line tool for CSV files.
csvsort - Sort large csv files.

Classical Statistics

Books

Lakens - Improving Your Statistical Inferences - Testing, Effect Sizes, Confidence Intervals, Sample Size, Equivalence Testing, Sequential Analysis, Github
Models Demystified - From Linear Regression to Deep Learning. Github.
The Math Behind Artificial Intelligence - Engineering-focused book covering linear algebra, calculus, probability & statistics, and optimization theory with Python examples.

Datasets

Rdatasets - Collection of more than 2000 datasets, stored as csv files (R package).
crimedatasets - Datasets focused on crimes, criminal activities (R package).
educationr - Datasets related to education (performance, learning methods, test scores, absenteeism) (R package).
MedDataSets - Datasets related to medicine, diseases, treatments, drugs, and public health (R package).
oncodatasets - Datasets focused on cancer research, survival rates, genetic studies, biomarkers, epidemiology (R package).
timeseriesdatasets_R - Time series datasets (R package).
usdatasets - US-exclusive datasets (crime, economics, education, finance, energy, healthcare) (R package).
economic datasets - Economic datasets.

p-values

The ASA Statement on p-Values: Context, Process, and Purpose
Greenland - Statistical tests, P-values, confidence intervals, and power: a guide to misinterpretations
Rubin - Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses
Gigerenzer - Mindless Statistics
Rubin - That's not a two-sided test! It's two one-sided tests! (TOST)
Lakens - How were we supposed to move beyond p < .05, and why didn’t we?
McShane et al. - Abandon Statistical Significance
Ho et al. - Moving beyond P values data analysis with estimation graphics
Lakens - The probability of p-values as a function of the statistical power of a test - p-value distribution is right-skewed and becomes even more skewed the higher the power of the test.

Correlation

Guess the Correlation - Correlation guessing game.
phik - Correlation between categorical, ordinal and interval variables.
hoeffd - Hoeffding's D Statistics, measure of dependence (R package).

Confidence Intervals

Morey - The fallacy of placing confidence in confidence intervals

Packages

statsmodels - Statistical tests.
linearmodels - Instrumental variable and panel data models.
nomograms - Visualization for linear models, explanation (Part of rms R package)
pingouin - Statistical tests. Pairwise correlation between columns of pandas DataFrame
scipy.stats - Statistical tests.
scikit-posthocs - Statistical post-hoc tests for pairwise multiple comparisons.
Bland-Altman Plot 1, 2 - Plot for agreement between two methods of measurement.
ANOVA
StatCheck - Extract statistics from articles and recompute p-values (R package).
tost - Two One-Sided Test (TOST) for equivalence.
DABEST-python - Mean difference plots.
Durga - Mean difference plots (R package).

Effect Size

MOTE Effect Size Calculator - Shiny App, R package
Estimating Effect Sizes From Pretest-Posttest-Control Group Designs - Scott B. Morris, Twitter

Statistical Tests

test_proportions_2indep - Proportion test.
G-Test - Alternative to chi-square test, power_divergence.

Comparing Two Populations

torch-two-sample - Friedman-Rafsky Test: Compare two population based on a multivariate generalization of the Runstest. Explanation, Application

Power and Sample Size Calculations

pwrss - Statistical Power and Sample Size Calculation Tools (R package), Tutorial with t-test

Interim Analyses / Sequential Analysis / Stopping

Stop Early Stopping - Nice visualization Sequential Analysis - Wikipedia.
sequential - Exact Sequential Analysis for Poisson and Binomial Data (R package).
confseq - Uniform boundaries, confidence sequences, and always-valid p-values.

Visualizations

Friends don't let friends make certain types of data visualization
Great Overview over Visualizations
1 dataset, 100 visualizations
Dependent Propabilities
Null Hypothesis Significance Testing (NHST) and Sample Size Calculation
estimationstats - Online Tool for visualizing mean differences, effect sizes (Cohen's d) and others.
Sample Size / Duration Calculator
Correlation
Cohen's d
Confidence Interval
Equivalence, non-inferiority and superiority testing
Bayesian two-sample t test
Distribution of p-values when comparing two groups
Understanding the t-distribution and its normal approximation
Statistical Power and Sample Size Calculation Tools

Tidy Tuesday

The Art of Data Visualization with ggplot2, The TidyTuesday Cookbook
Best Practices for Data Visualization
tidytuesday - Weekly challenge for visualization and lots of publicly available datasets for practice.
z3tt/TidyTuesday - Nice charts (R).
nrennie/tidytuesday - Nice charts (R).
poncest/tidytuesday - Nice charts (R).

Talks

Inverse Propensity Weighting
Dealing with Selection Bias By Propensity Based Feature Selection

Texts

Modes, Medians and Means: A Unifying Perspective
Using Norms to Understand Linear Regression
Verifying the Assumptions of Linear Models
Mediation and Moderation Intro
Montgomery et al. - How conditioning on post-treatment variables can ruin your experiment and what to do about it
Lindeløv - Common statistical tests are linear models
Chatruc - The Central Limit Theorem and its misuse
Al-Saleh - Properties of the Standard Deviation that are Rarely Mentioned in Classrooms
Wainer - The Most Dangerous Equation
Gigerenzer - The Bias Bias in Behavioral Economics
Cook - Estimating the chances of something that hasn’t happened yet
Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing, Youtube
How large is that number in the Law of Large Numbers?
The Prosecutor's Fallacy
The Dunning-Kruger Effect is Autocorrelation
Rafi, Greenland - Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise
Carlin et al. - On the uses and abuses of regression models: a call for reform of statistical practice and teaching
Chen, Roth - Logs with zeros? Some problems and solutions
Wigboldus et al. - Encourage Playing with Data and Discourage Questionable Reporting Practices
Simmons et al. - False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Zhang - An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability - Figure 1 shows difference between inferential uncertainty and outcome variability.

Evaluation

Collins et al. - Evaluation of clinical prediction models (part 1): from development to external validation - Twitter

Epidemiology

Lesko et al. - A Framework for Descriptive Epidemiology
R Epidemics Consortium - Large tool suite for working with epidemiological data (R packages). Github
incidence2 - Computation, handling, visualisation and simple modelling of incidence (R package).
EpiEstim - Estimate time varying instantaneous reproduction number R during epidemics (R package) paper.
researchpy - Helpful summary_cont() function for summary statistics (Table 1).
zEpid - Epidemiology analysis package, Tutorial.
tipr - Sensitivity analyses for unmeasured confounders (R package).
quartets - Anscombe’s Quartet, Causal Quartet, Datasaurus Dozen and others (R package).
episensr - Quantitative Bias Analysis for Epidemiologic Data (=simulation of possible effects of different sources of bias) (R package).

Machine Learning Tutorials

Statistical Inference and Regression
Applied Machine Learning in Python
Convolutional Neural Networks for Visual Recognition - Stanford CS class.
Intuition for the Algorithms in Machine Learning - Lecture Series.

Exploration and Cleaning

Checklist.
pyjanitor - Clean messy column names.
skimpy - Create summary statistics of dataframes. Helpful clean_columns() function.
pandera - Data / Schema validation.
dataframely - Data / Schema validation.
pointblank - Data / Schema validation.
great_expectations - Data validation, documentation, and profiling for production pipelines.
impyute - Imputations.
fancyimpute - Matrix completion and imputation algorithms.
imbalanced-learn - Resampling for imbalanced datasets.
tspreprocess - Time series preprocessing: Denoising, Compression, Resampling.
Kaggler - Utility functions (OneHotEncoder(min_obs=100))
skrub - Bridge the gap between tabular data sources and machine-learning models.

Noisy Labels

cleanlab - Machine learning with noisy labels, finding mislabelled data, and uncertainty quantification. Also see awesome list below.
doubtlab - Find bad or noisy labels.

Train / Test Split

iterative-stratification - Stratification of multilabel data.

Feature Engineering

Vincent Warmerdam: Untitled12.ipynb - Using df.pipe()
Vincent Warmerdam: Winning with Simple, even Linear, Models
sklearn - Pipeline, examples.
pdpipe - Pipelines for DataFrames.
scikit-lego - Custom transformers for pipelines.
categorical-encoding - Categorical encoding of variables, vtreat (R package).
patsy - R-like syntax for statistical models.
mlxtend - LDA.
featuretools - Automated feature engineering, example.
tsfresh - Time series feature engineering.
temporian - Time series feature engineering by Google.
pypeln - Concurrent data pipelines.
feature-engine - Encoders, transformers, etc.

Feature Selection

Overview Paper, Talk, Repo
Blog post series - 1, 2, 3, 4
Tutorials - 1, 2
sklearn - Feature selection.
eli5 - Feature selection using permutation importance.
scikit-feature - Feature selection algorithms.
stability-selection - Stability selection.
scikit-rebate - Relief-based feature selection algorithms.
scikit-genetic - Genetic feature selection.
boruta_py - Feature selection, explaination, example.
Boruta-Shap - Boruta feature selection algorithm + shapley values.
linselect - Feature selection package.
mlxtend - Exhaustive feature selection.
BoostARoota - Xgboost feature selection algorithm.
INVASE - Instance-wise Variable Selection using Neural Networks.
SubTab - Subsetting Features of Tabular Data for Self-Supervised Representation Learning, AstraZeneca.
mrmr - Maximum Relevance and Minimum Redundancy Feature Selection, Website.
arfs - All Relevant Feature Selection.
VSURF - Variable Selection Using Random Forests (R package) doc.
FeatureSelectionGA - Feature Selection using Genetic Algorithm.

Subset Selection

apricot - Selecting subsets of data sets to train machine learning models quickly.
ducks - Index data for fast lookup by any combination of fields.

Dimensionality Reduction / Representation Learning

Selection

Check also the Clustering section and self-supervised learning section for ideas!
Review

PCA - link
Autoencoder - link
Isomaps - link
LLE - link
Force-directed graph drawing - link
MDS - link
Diffusion Maps - link
t-SNE - link
NeRV - link, paper
MDR - link
UMAP - link
Random Projection - link
Ivis - link
SimCLR - link
pymde - Minimum-distortion embedding with PyTorch, link

Neural-network based

esvit - Vision Transformers for Representation Learning (Microsoft).
MCML - Semi-supervised dimensionality reduction of Multi-Class, Multi-Label data (sequencing data) paper.

Packages

Dangers of PCA (paper).
Phantom oscillations in PCA.
What to use instead of PCA.
Talk, tsne intro. sklearn.manifold and sklearn.decomposition - PCA, t-SNE, MDS, Isomaps and others.
Additional plots for PCA - Factor Loadings, Cumulative Variance Explained, Correlation Circle Plot, Tweet
sklearn.random_projection - Johnson-Lindenstrauss lemma, Gaussian random projection, Sparse random projection.
sklearn.cross_decomposition - Partial least squares, supervised estimators for dimensionality reduction and regression.
prince - Dimensionality reduction, factor analysis (PCA, MCA, CA, FAMD).
Faster t-SNE implementations: tsne-cuda, MulticoreTSNE, lvdmaaten
umap - Uniform Manifold Approximation and Projection, talk, explorer, explanation, parallel version.
humap - Hierarchical UMAP.
sleepwalk - Explore embeddings, interactive visualization (R package).
somoclu - Self-organizing map.
scikit-tda - Topological Data Analysis, paper, talk, talk, paper.
giotto-tda - Topological Data Analysis.
ivis - Dimensionality reduction using Siamese Networks.
trimap - Dimensionality reduction using triplets.
scanpy - Force-directed graph drawing, Diffusion Maps.
direpack - Projection pursuit, Sufficient dimension reduction, Robust M-estimators.
DBS - DatabionicSwarm (R package).
contrastive - Contrastive PCA.
scPCA - Sparse contrastive PCA (R package).
generalized_contrastive_PCA - Generalized contrastive PCA.
tmap - Visualization library for large, high-dimensional data sets.
lollipop - Linear Optimal Low Rank Projection.
linearsdr - Linear Sufficient Dimension Reduction (R package).
PHATE - Tool for visualizing high dimensional data.
datamapplot - Tool for visualizing high dimensional data.

Visualization

All charts
physt - Better histograms, talk, notebook.
fast-histogram - Fast histograms.
matplotlib_venn - Venn diagrams.
penrose - Venn diagrams.
ridgeplot - Ridge plots.
mosaic plots - Categorical variable visualization, example.
yellowbrick - Visualizations for ML models (similar to scikit-plot).
bokeh - Interactive visualization library, Examples, Examples.
lets-plot - Plotting library.
plotnine - ggplot for Python.
altair - Declarative statistical visualization library.
hvplot - High-level plotting library built on top of holoviews.
dtreeviz - Decision tree visualization and model interpretation.
mpl-scatter-density - Scatter density plots. Alternative to 2d-histograms.
ComplexHeatmap - Complex heatmaps for multidimensional genomic data (R package).
morpheus - Broad Institute tool matrix visualization and analysis software. Source, Tutorial: 1, 2, Code.
jupyter-scatter - Interactive 2D scatter plot widget for Jupyter.
fastplotlib - Fast plotting library using pygfx.
datamapplot - Interactive 2D scatter plot.
SandDance - Interactive visualization tool from Microsoft.

Colors

palettable - Color palettes from colorbrewer2.
colorcet - Collection of perceptually uniform colormaps.
Named Colors Wheel - Color wheel for all named HTML colors.

Dashboards

py-shiny - Shiny for Python, talk.
superset - Dashboarding solution by Apache.
streamlit - Dashboarding solution. Resources, Gallery Components, bokeh-events.
mercury - Convert Python notebook to web app, Example.
dash - Dashboarding solution by plot.ly. Resources.
visdom - Dashboarding library by Facebook.
panel - Dashboarding solution.
altair example - Video.
voila - Turn Jupyter notebooks into standalone web applications.
voila-gridstack - Voila grid layout.

UI

gradio - Create UIs for your machine learning model.

Survey Tools

samplics - Sampling techniques for complex survey designs.

Geographical Tools

folium - Plot geographical maps using the Leaflet.js library, jupyter plugin.
gmaps - Google Maps for Jupyter notebooks.
stadiamaps - Plot geographical maps.
datashader - Draw millions of points on a map.
sklearn - BallTree.
pynndescent - Nearest neighbor descent for approximate nearest neighbors.
geocoder - Geocoding of addresses, IP addresses.
Conversion of different geo formats: talk, repo
geopandas - Tools for geographic data
Low Level Geospatial Tools (GEOS, GDAL/OGR, PROJ.4)
Vector Data (Shapely, Fiona, Pyproj)
Raster Data (Rasterio)
Plotting (Descartes, Catropy)
Predict economic indicators from Open Street Map.
PySal - Python Spatial Analysis Library.
geography - Extract countries, regions and cities from a URL or text.
cartogram - Distorted maps based on population.

Recommender Systems

Examples: 1, 2, 2-ipynb, 3.
surprise - Recommender, talk.
implicit - Fast Collaborative Filtering for Implicit Feedback Datasets.
spotlight - Deep recommender models using PyTorch.
lightfm - Recommendation algorithms for both implicit and explicit feedback.
funk-svd - Fast SVD.

Decision Tree Models

Intro to Decision Trees and Random Forests, Another good visualization, Intro to Gradient Boosting 1, 2, Decision Tree Visualization
lightgbm - Gradient boosting (GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, doc.
xgboost - Gradient boosting (GBDT, GBRT or GBM) library, doc, Methods for CIs: link1, link2.
catboost - Gradient boosting.
h2o - Gradient boosting and general machine learning framework.
pycaret - Wrapper for xgboost, lightgbm, catboost etc.
forestci - Confidence intervals for random forests.
grf - Generalized random forest.
dtreeviz - Decision tree visualization and model interpretation.
Nuance - Decision tree visualization.
rfpimp - Feature Importance for RandomForests using Permuation Importance.
Why the default feature importance for random forests is wrong: link
bartpy - Bayesian Additive Regression Trees.
merf - Mixed Effects Random Forest for Clustering, video
groot - Robust decision trees.
linear-tree - Trees with linear models at the leaves.
supertree - Decision tree visualization.

Natural Language Processing (NLP) / Text Processing

talk-nb, nb2, talk.
Text classification Intro, Preprocessing blog post.
gensim - NLP, doc2vec, word2vec, text processing, topic modelling (LSA, LDA), Example, Coherence Model for evaluation.
Embeddings - GloVe ([1], [2]), StarSpace, wikipedia2vec, visualization.
magnitude - Vector embedding utility package.
pyldavis - Visualization for topic modelling.
spaCy - NLP.
NTLK - NLP, helpful KMeansClusterer with cosine_distance.
pytext - NLP from Facebook.
fastText - Efficient text classification and representation learning.
annoy - Approximate nearest neighbor search.
faiss - Approximate nearest neighbor search.
infomap - Cluster (word-)vectors to find topics.
datasketch - Probabilistic data structures for large data (MinHash, HyperLogLog).
flair - NLP Framework by Zalando.
stanza - NLP Library.
Chatistics - Turn Messenger, Hangouts, WhatsApp and Telegram chat logs into DataFrames.
textdistance - Collection for comparing distances between two or more sequences.

Bio Image Analysis

Lee et al. - A beginner's guide to rigor and reproducibility in fluorescence imaging experiments
Awesome Cytodata

Tutorials

MIT 7.016 Introductory Biology, Fall 2018 - Videos 27, 28, and 29 talk about staining and imaging.
Bio-image Analysis Notebooks - Large collection of image processing workflows, including point-spread-function estimation and deconvolution, 3D cell segmentation, feature extraction using pyclesperanto and others.
python_for_microscopists - Notebooks and associated youtube channel for a variety of image processing tasks.

Datasets

jump-cellpainting - Cellpainting dataset.
MedMNIST - Datasets for 2D and 3D Biomedical Image Classification.
CytoImageNet - Huge diverse dataset like ImageNet but for cell images.
Haghighi - Gene Expression and Morphology Profiles.
broadinstitute/lincs-profiling-complementarity - Cellpainting vs. L1000 assay.

Biostatistics / Robust statistics

MinCovDet - Robust estimator of covariance, RMPV, Paper, App1, App2.
moderated z-score - Weighted average of z-scores based on Spearman correlation.
winsorize - Simple adjustment of outliers.

High-Content Screening Assay Design

Zhang XHD (2008) - Novel analytic criteria and effective plate designs for quality control in genome-wide RNAi screens
Iversen - A Comparison of Assay Performance Measures in Screening Assays, Signal Window, Z′ Factor, and Assay Variability Ratio Z-factor - Measure of statistical effect size.
Z'-factor - Measure of statistical effect size.
CV - Coefficient of variation.
SSMD - Strictly standardized mean difference.
Signal Window - Assay quality measurement.

Microscopy + Assay

BD Spectrum Viewer - Calculate spectral overlap, bleed through for fluorescence microscopy dyes.
SpectraViewer - Visualize the spectral compatibility of fluorophores (PerkinElmer).
Thermofisher Spectrum Viewer - Thermofisher Spectrum Viewer.
Microscopy Resolution Calculator - Calculate resolution of images (Nikon).
PlateEditor - Drug Layout for plates, app, zip, paper.

Image Formats and Converters

OME-Zarr - paper, standard
bioformats2raw - Various formats to zarr.
raw2ometiff - Zarr to tiff.
BatchConvert - Wrapper for bioformats2raw to parallelize conversions with nextflow, video.
REMBI model - Recommended Metadata for Biological Images, BioImage Archive: Study Component Guidance, File List Guide, paper, video, spreadsheet

Matrix Formats

anndata - annotated data matrices in memory and on disk, Docs.
muon - Multimodal omics framework.
mudata - Multimodal Data (.h5mu) implementation.
bdz - Zarr-based format for storing quantitative biological dynamics data.

Image Viewers

napari - Image viewer and image processing tool.
Fiji - General purpose tool. Image viewer and image processing tool.
vizarr - Browser-based image viewer for zarr format.
avivator - Browser-based image viewer for tiff files.
OMERO - Image viewer for high-content screening. IDR uses OMERO. Intro
fiftyone - Viewer and tool for building high-quality datasets and computer vision models.
Image Data Explorer - Microscopy Image Viewer, Shiny App, Video.
ImSwitch - Microscopy Image Viewer, Doc, Video.
pixmi - Web-based image annotation and classification tool, App.
DeepCell Label - Data labeling tool to segment images, Video.
lightly-studio - Image annotation.

Napari Plugins

napari-sam - Segment Anything Plugin.
napari-chatgpt - ChatGPT Plugin.

Image Restoration and Denoising

aydin - Image denoising.
DivNoising - Unsupervised denoising method.
CSBDeep - Content-aware image restoration, Project page.
gibbs-diffusion - Image denoising.

Illumination correction

skimage - Illumination correction (CLAHE).
cidre - Illumination correction method for optical microscopy.
BaSiCPy - Background and Shading Correction of Optical Microscopy Images, BaSiC.

Bleedthrough correction / Spectral Unmixing

PICASSO - Blind unmixing without reference spectra measurement, Paper
cytoflow - Flow cytometry. Includes Bleedthrough correction methods.
Linear unmixing in Fiji for Bleedthrough Correction - Youtube.
Bleedthrough Correction using Lumos and Fiji - Link.
AutoUnmix - Link.

Platforms and Pipelines

CellProfiler, CellProfilerAnalyst - Create image analysis pipelines.
fractal - Framework to process high-content imaging data from UZH, Github.
atomai - Deep and Machine Learning for Microscopy.
py-clesperanto - Tools for 3D microscopy analysis, deskewing and lots of other tutorials, interacts with napari.
qupath - Image analysis.

Microscopy Pipelines

Labsyspharm Stack see below.
BiaPy - Bioimage analysis pipelines, paper.
SCIP - Image processing pipeline on top of Dask.
DeepCell Kiosk - Image analysis platform.
IMCWorkflow - Image analysis pipeline using steinbock, Twitter, Paper, workflow.

Labsyspharm

mcmicro - Multiple-choice microscopy pipeline, Website, Paper.
MCQuant - Quantification of cell features.
cylinter - Quality assurance for microscopy images, Website.
ashlar - Whole-slide microscopy image stitching and registration.
scimap - Spatial Single-Cell Analysis Toolkit.

Cell Segmentation

microscopy-tree - Review of cell segmentation algorithms, Paper.
Review of organoid pipelines - Paper.
BioImage.IO - BioImage Model Zoo.
MEDIAR - Cell segmentation.
cellpose - Cell segmentation. Paper, Dataset.
stardist - Cell segmentation with Star-convex Shapes.
instanseg - Cell segmentation.
UnMicst - Identifying Cells and Segmenting Tissue.
ilastik - Segment, classify, track and count cells. ImageJ Plugin.
nnUnet - 3D biomedical image segmentation.
allencell - Tools for 3D segmentation, classical and deep learning methods.
Cell-ACDC - Python GUI for cell segmentation and tracking.
ZeroCostDL4Mic - Deep-Learning in Microscopy.
DL4MicEverywhere - Bringing the ZeroCostDL4Mic experience using Docker.
EmbedSeg - Embedding-based Instance Segmentation.
segment-anything - Segment Anything (SAM) from Facebook.
micro-sam - Segment Anything for Microscopy.
Segment-Everything-Everywhere-All-At-Once - Segment Everything Everywhere All at Once from Microsoft.
deepcell-tf - Cell segmentation, DeepCell.
labkit - Fiji plugin for image segmentation.
MedImageInsight - Embedding Model for General Domain Medical Imaging.
CHIEF - Clinical Histopathology Imaging Evaluation Foundation Model.

Cell Segmentation Datasets

cellpose - Cell images.
omnipose - Cell images.
LIVECell - Cell images.
Sartorius - Neurons.
EmbedSeg - 2D + 3D images.
connectomics - Annotation of the EPFL Hippocampus dataset.
ZeroCostDL4Mic - Stardist example training and test dataset.

Evaluation

seg-eval - Cell segmentation performance evaluation without Ground Truth labels, Paper.

Feature Engineering Images

Computer vision challenges in drug discovery - Maciej Hermanowicz
CellProfiler - Biological image analysis.
scikit-image - Image processing.
scikit-image regionprops - Regionprops: area, eccentricity, extent.
mahotas - Zernike, Haralick, LBP, and TAS features, example.
pyradiomics - Radiomics features from medical imaging.
pyefd - Elliptical feature descriptor, approximating a contour with a Fourier series.
pyvips - Faster image processing operations.

Domain Adaptation / Batch-Effect Correction

Tran - A benchmark of batch-effect correction methods for single-cell RNA sequencing data, Code.
R Tutorial on correcting batch effects.
harmonypy - Fuzzy k-means and locally linear adjustments.
pyliger - Batch-effect correction, R package.
nimfa - Nonnegative matrix factorization.
scgen - Batch removal. Doc.
CORAL - Correcting for Batch Effects Using Wasserstein Distance, Code, Paper.
adapt - Awesome Domain Adaptation Python Toolbox.
pytorch-adapt - Various neural network models for domain adaptation.

Sequencing

Single cell tutorial.
PyDESeq2 - Analyzing RNA-seq data.
cellxgene - Interactive explorer for single-cell transcriptomics data.
scanpy - Analyze single-cell gene expression data, tutorial.
besca - Beyond single-cell analysis.
janggu - Deep Learning for Genomics.
gdsctools - Drug responses in the context of the Genomics of Drug Sensitivity in Cancer project, ANOVA, IC50, MoBEM, doc.
monkeybread - Analysis of single-cell spatial transcriptomics data.

Drug discovery

TDC - Drug Discovery and Development.
DeepPurpose - Deep Learning Based Molecular Modelling and Prediction Toolkit.

Neural Networks

mit6874 - Computational Systems Biology: Deep Learning in the Life Sciences.
ConvNet Shape Calculator - Calculate output dimensions of Conv2D layer.
Great Gradient Descent Article.
Intro to semi-supervised learning.

Tutorials & Viewer

Google Tuning Playbook - A playbook for systematically maximizing the performance of deep learning models by Google.
fast.ai course - Practical Deep Learning for Coders.
Tensorflow without a PhD - Neural Network course by Google.
Feature Visualization: Blog, PPT
Tensorflow Playground
Visualization of optimization algorithms, Another visualization
cutouts-explorer - Image Viewer.

Image Related

imgaug - More sophisticated image preprocessing.
Augmentor - Image augmentation library.
keras preprocessing - Preprocess images.
albumentations - Wrapper around imgaug and other libraries.
augmix - Image augmentation from Google.
kornia - Image augmentation, feature extraction and loss functions.
augly - Image, audio, text, video augmentation from Facebook.
pyvips - Faster image processing operations.

Lossfunction Related

SegLoss - List of loss functions for medical image segmentation.

Activation Functions

rational_activations - Rational activation functions.

Text Related

ktext - Utilities for pre-processing text for deep learning in Keras.
textgenrnn - Ready-to-use LSTM for text generation.
ctrl - Text generation.

Neural network and deep learning frameworks

OpenMMLab - Framework for segmentation, classification and lots of other computer vision tasks.
caffe - Deep learning framework, pretrained models.
mxnet - Deep learning framework, book.

Libs General

keras - Neural Networks on top of tensorflow, examples.
keras-contrib - Keras community contributions.
keras-tuner - Hyperparameter tuning for Keras.
hyperas - Keras + Hyperopt: Convenient hyperparameter optimization wrapper.
elephas - Distributed Deep learning with Keras & Spark.
tflearn - Neural Networks on top of TensorFlow.
tensorlayer - Neural Networks on top of TensorFlow, tricks.
tensorforce - TensorFlow for applied reinforcement learning.
autokeras - AutoML for deep learning.
PlotNeuralNet - Plot neural networks.
lucid - Neural network interpretability, Activation Maps.
tcav - Interpretability method.
AdaBound - Optimizer that trains as fast as Adam and as good as SGD, alt.
foolbox - Adversarial examples that fool neural networks.
hiddenlayer - Training metrics.
imgclsmob - Pretrained models.
netron - Visualizer for deep learning and machine learning models.
ffcv - Fast dataloader.

Libs PyTorch

Good PyTorch Introduction
skorch - Scikit-learn compatible neural network library that wraps PyTorch, talk, slides.
fastai - Neural Networks in PyTorch.
timm - PyTorch image models.
ignite - Highlevel library for PyTorch.
torchcv - Deep Learning in Computer Vision.
pytorch-optimizer - Collection of optimizers for PyTorch.
pytorch-lightning - Wrapper around PyTorch.
litserve - Serve models.
lightly - MoCo, SimCLR, SimSiam, Barlow Twins, BYOL, NNCLR.
MONAI - Deep learning in healthcare imaging.
kornia - Image transformations, epipolar geometry, depth estimation.
torchinfo - Nice model summary.
lovely-tensors - Inspect tensors, mean, std, inf values.

Distributed Libs

flexflow - Distributed TensorFlow Keras and PyTorch.
horovod - Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

Architecture Visualization

Awesome List.
netron - Viewer for neural networks.
visualkeras - Visualize Keras networks.

Computer Vision General

roboflow - Reusable computer vision tools.

Object detection / Instance Segmentation

Metrics reloaded: Recommendations for image analysis validation - Guide for choosing correct image analysis metrics, Code, Twitter Thread
Good Yolo Explanation
ultralytics - Easily accessible Yolo and SAM models.
yolact - Fully convolutional model for real-time instance segmentation.
EfficientDet Pytorch, EfficientDet Keras - Scalable and Efficient Object Detection.
detectron2 - Object Detection (Mask R-CNN) by Facebook.
simpledet - Object Detection and Instance Recognition.
CenterNet - Object detection.
FCOS - Fully Convolutional One-Stage Object Detection.
norfair - Real-time 2D object tracking.
Detic - Detector with image classes that can use image-level labels (facebookresearch).
EasyCV - Image segmentation, classification, metric-learning, object detection, pose estimation.

Image Classification

nfnets - Neural network.
efficientnet - Neural network.
pycls - PyTorch image classification networks: ResNet, ResNeXt, EfficientNet, and RegNet (by Facebook).

Applications and Snippets

SPADE - Semantic Image Synthesis.
Entity Embeddings of Categorical Variables, code, kaggle
Image Super-Resolution - Super-scaling using a Residual Dense Network.
Cell Segmentation - Talk, Blog Posts: 1, 2
deeplearning-models - Deep learning models.

Variational Autoencoders (VAEs)

Variational Autoencoder Explanation Video
disentanglement_lib - BetaVAE, FactorVAE, BetaTCVAE, DIP-VAE.
ladder-vae-pytorch - Ladder Variational Autoencoders (LVAE).
benchmark_VAE - Unifying Generative Autoencoder implementations.

Generative Adversarial Networks (GANs)

Awesome GAN Applications
The GAN Zoo - List of Generative Adversarial Networks.
CycleGAN and Pix2pix - Various image-to-image tasks.
TensorFlow GAN implementations
PyTorch GAN implementations
PyTorch GAN implementations
StudioGAN - PyTorch GAN implementations.

Transformers

The Annotated Transformer - Intro to transformers.
Transformers from Scratch - Intro.
Neural Networks: Zero to Hero - Video series on building neural networks.
SegFormer - Simple and Efficient Design for Semantic Segmentation with Transformers.
esvit - Efficient self-supervised Vision Transformers.
nystromformer - More efficient transformer because of approximate self-attention.

Deep learning on structured data

Great overview for deep learning for tabular data
TabPFN - Foundation Model for Tabular Data.

Graph-Based Neural Networks

How to do Deep Learning on Graphs with Graph Convolutional Networks
Introduction To Graph Convolutional Networks
An attempt at demystifying graph deep learning
ogb - Open Graph Benchmark, Benchmark datasets.
networkx - Graph library.
cugraph - RAPIDS, Graph library on the GPU.
pytorch-geometric - Various methods for deep learning on graphs.
dgl - Deep Graph Library.
graph_nets - Build graph networks in TensorFlow, by DeepMind.

Model conversion

hummingbird - Compile trained ML models into tensor computations (by Microsoft).

GPU

cuML - RAPIDS, Run traditional tabular ML tasks on GPUs, Intro.
flashlib - A GPU library for classical machine-learning operators.

Regression

Ordinal Regression: paper
Understanding SVM Regression: slides, forum, paper
Generalized Additive Models - Tutorial in R.

pyearth - Multivariate Adaptive Regression Splines (MARS), tutorial.
pygam - Generalized Additive Models (GAMs), Explanation.
GLRM - Generalized Low Rank Models.
tweedie - Specialized distribution for zero inflated targets, Talk.
MAPIE - Estimating prediction intervals.

Classification

Talk, Notebook
Blog post: Probability Scoring
All classification metrics
DESlib - Dynamic classifier and ensemble selection.
human-learn - Create and tune classifier based on your rule set.

Metric Learning

Contrastive Representation Learning

metric-learn - Supervised and weakly-supervised metric learning algorithms.
pytorch-metric-learning - PyTorch metric learning.
deep_metric_learning - Methods for deep metric learning.
ivis - Metric learning using siamese neural networks.
TensorFlow similarity - Metric learning.

Distance Functions

Steck et al. - Is Cosine-Similarity of Embeddings Really About Similarity?
scipy.spatial - All kinds of distance metrics.
vegdist - Distance metrics (R package).
pyemd - Earth Mover's Distance / Wasserstein distance, similarity between histograms. OpenCV implementation, POT implementation
dcor - Distance correlation and related Energy statistics.
GeomLoss - Kernel norms, Hausdorff divergences, Debiased Sinkhorn divergences (=approximation of Wasserstein distance).

Self-supervised Learning

lightly - MoCo, SimCLR, SimSiam, Barlow Twins, BYOL, NNCLR.
vissl - Self-Supervised Learning with PyTorch: RotNet, Jigsaw, NPID, ClusterFit, PIRL, SimCLR, MoCo, DeepCluster, SwAV.

Clustering

Overview of clustering algorithms applied image data (= Deep Clustering).
Clustering with Deep Learning: Taxonomy and New Methods.
Hierarchical Cluster Analysis (R Tutorial) - Dendrogram, Tanglegram
Schubert - Stop using the elbow criterion for k-means and how to choose the number of clusters instead
hdbscan - Clustering algorithm, talk, blog.
pyclustering - All sorts of clustering algorithms.
FCPS - Fundamental Clustering Problems Suite (R package).
GaussianMixture - Generalized k-means clustering using a mixture of Gaussian distributions, video.
nmslib - Similarity search library and toolkit for evaluation of k-NN methods.
merf - Mixed Effects Random Forest for Clustering, video
tree-SNE - Hierarchical clustering algorithm based on t-SNE.
MiniSom - Pure Python implementation of the Self Organizing Maps.
distribution_clustering, paper, related paper, alt.
phenograph - Clustering by community detection.
FastPG - Clustering of single cell data (RNA). Improvement of phenograph, Paper.
HypHC - Hyperbolic Hierarchical Clustering.
BanditPAM - Improved k-Medoids Clustering.
dendextend - Comparing dendrograms (R package).
DeepDPM - Deep Clustering With An Unknown Number of Clusters.
generalized-kmeans-clustering - Generalized k-means clustering.
evoc - Embedding Vector Oriented Clustering.

Clustering Evalutation

• Wagner, Wagner - Comparing Clusterings - An Overview
  • Adjusted Rand Index
  • Normalized Mutual Information
  • Adjusted Mutual Information
  • Fowlkes-Mallows Score
  • Silhouette Coefficient
  • Variation of Information, Julia
  • Pair Confusion Matrix
  • Consensus Score - The similarity of two sets of biclusters.
• Assessing the quality of a clustering (video)
• fpc - Various methods for clustering and cluster validation (R package).
  • Minimum distance between any two clusters
  • Distance between centroids
  • p-separation index: Like minimum distance. Look at the average distance to nearest point in different cluster for p=10% "border" points in any cluster. Measuring density, measuring mountains vs valleys
  • Estimate density by weighted count of close points
• Other measures:
  • Within-cluster average distance
  • Mean of within-cluster average distance over nearest-cluster average distance (silhouette score)
  • Within-cluster similarity measure to normal/uniform
  • Within-cluster (squared) distance to centroid (this is the k-Means loss function)
  • Correlation coefficient between distance we originally had to the distance the are induced by the clustering (Huberts Gamma)
  • Entropy of cluster sizes
  • Average largest within-cluster gap
  • Variation of clusterings on bootstrapped data

Multi-label classification

scikit-multilearn - Multi-label classification, talk.

Critical AI Texts