AR/VR Guide

A guide covering Augmented Reality (AR) & Virtual Reality (VR) including the applications, libraries and tools that will make you a better and more efficient Augmented Reality (AR) & Virtual Reality (VR) development.

Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.

Getting Started with AR/VR

Augmented Reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information.

Virtual Reality (VR) is a simulated experience that can be similar to or completely different from the real world. The applications of virtual reality include entertainment (video games), education (medical or military training) and business (virtual meetings).

Mixed Reality (MR) is the merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real time.

Extended Reality (XR) is a concept referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. Including augmented reality (AR), mixed reality (MR) and virtual reality (VR).

Developer Resources

Back to the Top

AR/VR Training Courses

Back to the Top

AR/VR Tools, Frameworks, and Hardware

Back to the Top

Tools & Frameworks

RealityCapture is a state-of-the-art photogrammetry software solution that creates virtual reality scenes, textured 3D meshes, orthographic projections, geo-referenced maps and much more from images and/or laser scans completely automatically.

Reality Composer is a powerful tool that makes it easy for you to create interactive augmented reality experiences with no prior 3D experience. The Reality Converter quickly converts your existing 3D models to USDZ so it works seamlessly in our tools and on all AR-enabled iPhone and iPad devices.

ARKit is a set set of software development tools to enable developers to build augmented-reality apps for iOS developed by Apple. The latest version ARKit 3.5 takes advantage of the new LiDAR Scanner and depth sensing system on iPad Pro(2020) to support a new generation of AR apps that use Scene Geometry for enhanced scene understanding and object occlusion.

RealityKit is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.

RealityUI is a Swift Package for creating familiar UI Elements and animations in a RealityKit rendered Augmented Reality or Virtual Reality scene.

SceneKit is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.

GPUImage framework is a BSD-licensed iOS library that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies. In comparison to Core Image (part of iOS 5.0), GPUImage allows you to write your own custom filters, supports deployment to iOS 4.0, and has a simpler interface. However, it currently lacks some of the more advanced features of Core Image, such as facial detection.

GPUImage3 is the third generation of the GPUImage framework, an open source project for performing GPU-accelerated image and video processing on Mac and iOS. The original GPUImage framework was written in Objective-C and targeted Mac and iOS, the second iteration rewritten in Swift using OpenGL to target Mac, iOS, and Linux, and now this third generation is redesigned to use Apple's Metal in place of OpenGL.

ARCore is a software development kit developed by Google that allows for augmented reality applications in the real world. These tools include environmental understanding, which allows devices to detect horizontal and vertical surfaces and planes. It also includes motion tracking, which lets phones understand and track their positions relative to the world. Also ARCore’s Light Estimation API lets your digital objects appear realistically as if they’re actually part of the physical world.

Adobe Scantastic is a tool that makes the creation of 3D assets accessible to everyone. It can be used with just a mobile device (combined with Adobe's server-based photogrammetry pipeline), users can easily scan objects in their physical environment and turn them into 3D models which can then be imported into tools like Adobe Dimension and Adobe Aero.

Adobe Aero is a tool that helps you build, view, and share immersive AR experiences. Simply build a scene by bringing in 2D images from Adobe Photoshop and Illustrator, or 3D models from Adobe Dimension, Substance, third-party apps like Cinema 4D, or asset libraries like Adobe Stock and TurboSquid. Aero optimizes a wide array of assets, including OBJ, GLB, and glTF files, for AR, so you can visualize them in real time.

Vuforia is a comprehensive, scalable enterprise AR platform. Our wide-ranging solution suite ensures that we can provide the right AR technology to every customer based on their business needs.

Vuforia Studio is a tool that allows anyone to create beautiful 3D augmented reality (AR) experiences in a matter of minutes with no coding required.

SceneView is a 3D and AR Android Composable and View with Google Filament and ARCore. This is a Sceneform replacement in Kotlin.

Polycam is the world's most popular LiDAR 3D scanning app for iOS(iPhone/iPad), Web, and Android. It let's you scan the world around you with your mobile device, DSLR camera, or drone to get beautiful, accurate 3D models.

3D Scanner App for Mac is a desktop tool for processing photos and videos into 3D models using the power of Photogrammetry; which makes it perfect for 3D Design, CAD, Architecture, Games Assets, AR, VR, XR. It shares USDZ models via iMessage to let friends and family see your models in Augmented Reality.

OpenVX™ is an open-source, royalty-free standard for cross platform acceleration of computer vision applications. OpenVX enables performance and power-optimized computer vision processing, especially important in embedded and real-time use cases such as face, body and gesture tracking, smart video surveillance, advanced driver assistance systems (ADAS), object and scene reconstruction, augmented reality, visual inspection, robotics and more.

OpenXR is an open, royalty-free standard for access to virtual reality and augmented reality platforms and devices.

OpenXR™ SDK is a Software Development Kit that contains OpenXR headers, as well as source code and build scripts for the OpenXR loader.

NVIDIA Flex is a particle-based simulation library designed for real-time applications.

ARToolKit is a fast and modern open source tracking and recognition SDK which enables computers to see and understand more in the environment around them. It's built from the ground up using modern computer vision techniques.

ALVR(Air Light VR) is a Stream VR games from your PC to your headset via Wi-Fi. ALVR uses technologies like Asynchronous Timewarp and Fixed Foveated Rendering for a smoother experience.

ARmedia is a plugin tool that makes it easy to develop and create AR experiences without coding.

Wikitude is a Cross-platform augmented reality SDK to create high-performing object tracking AR experiences using 3D models as an input method (CAD, glTF 2.0 and more) for multiple devices and operating systems.

Kudan AR SDK is a Professional AR SDK to develop compelling applications for mobile devices.

Kundan is a tool that provides proprietary Artificial Perception technologies based on SLAM to enable use cases with significant market potential and impact on our lives such as autonomous driving, robotics, AR/VR and smart cities.

OVR Toolkit is a utility application designed to make viewing the desktop in VR simple and fast, it allows for viewing the desktop within VR, placing desktop windows around the world, mouse input, typing with a virtual keyboard, and quickly switching between windows.

OpenVR is an API and runtime that allows access to VR hardware(Steam Index, HTC Vive, and Oculus Rift) from multiple vendors without requiring that applications have specific knowledge of the hardware they are targeting.

OpenVR Benchmark on Steam is the first benchmark tool for reproducibly testing your real VR performance, rendering inside of your VR headset.

OpenHMD is open source API and drivers that supports a wide range of HMD(head-mounted display) devices such as Oculus Rift, HTC Vive, Sony PSVR, and others.

StereoKit is an easy-to-use open source mixed reality library for building HoloLens and VR applications with C# and OpenXR.

Monado is the first OpenXR™ runtime for GNU/Linux. Monado aims to jump-start development of an open source XR ecosystem and provide the fundamental building blocks for device vendors to target the GNU/Linux platform.

Libsurvive is a set of tools and libraries that enable 6 dof tracking on lighthouse and vive based systems that is completely open source and can run on any device. It currently supports both SteamVR 1.0 and SteamVR 2.0 generation of devices and should support any tracked object commercially available.

Simula is a VR window manager for Linux that runs on top of Godot. It takes less than 1 minute to install. Simula is officially compatible with SteamVR headsets equipped with Linux drivers (e.g. HTC Vive, HTC Vive Pro, & Valve Index). We have also added experimental support to OpenXR headsets that have Monado drivers (e.g. North Star, OSVR HDK, and PSVR). Some people have gotten the Oculus Rift S to run Simula via OpenHMD (see here).

Unreal Engine 5 Development

Back to the Top

Unreal Engine 5 is a game engine developed by Epic Games with the world's most open and advanced real-time 3D creation tool. Continuously evolving to serve not only its original purpose as a state-of-the-art game engine, today it gives creators across industries the freedom and control to deliver cutting-edge content, interactive experiences, and immersive virtual worlds.

Unreal Engine 5 with Big City Sample Project

Unreal Engine Twinmotion. Source: Unreal Engine

Unreal Engine 5 Project browser

Unreal Engine 5 Vehicle Project Demo

Unreal Engine Marketplace

Unreal Engine Tools

Back to the Top

Blueprint Visual Scripting is a complete gameplay scripting system in Unreal Engine based on the concept of using a node-based interface to create gameplay elements from within Unreal Editor. As with many common scripting languages, it is used to define object-oriented (OO) classes or objects in the engine.

Blueprint Visual Scripting UI

Datasmith is a collection of tools and plugins that bring entire pre-constructed scenes and complex assets created in a variety of industry-standard design applications into Unreal Engine.

Lumen is an Unreal Engine 5 feature that uses multiple ray tracing methods to solve Global Illumination and Reflections. Screen traces are done first, followed by a more reliable method. It uses Software Ray Tracing through Signed Distance Fields by default, but can achieve higher quality on supporting video cards when Hardware Ray Tracing is enabled.

Nanite is Unreal Engine 5's new virtualized geometry system which uses a new internal mesh format and rendering technology to render pixel scale detail and high object counts. It intelligently does work on only the detail that can be perceived and no more.

Niagara VFX System is one of two tools you can use to create and adjust visual effects (VFX) inside Unreal Engine 5 (UE5). Before Niagara, the primary way to create and edit visual effects in UE4 was to use Cascade. It has many of the same of particle manipulation methods that Cascade offers, the way you interact and build visual effects with Niagara is vastly different.

Chaos Physics is a Beta feature that is the light-weight physics simulation solution used in Fortnite, and it includes the following major features.

RBAN (Rigid Body Animation Nodes)
Destruction
Cloth
Ragdoll
Vehicles

MetaHuman Creator is a free cloud-based app that empowers anyone to create photorealistic digital humans, complete with hair and clothing, in minutes. MetaHumans come fully rigged and ready to animate in your Unreal Engine projects.

Twinmotion is a real-time 3D immersion software that produces high-quality images, panoramas and standard or 360° VR videos in seconds. Developed for architecture, construction, urban planning and landscaping professionals.

Bridgew by Quixel is a gateway to create a world of 3D content right inside the Unreal Engine. it includes Megascans—the world's largest library of AAA, cinema-quality assets based on real-world scan data.

Unreal Engine Marketplace is Unreal Engine's Store that has a library Of Textures, Models, Animations, Tutorials, and More for creating amazing 3D projects and games.

UnrealBuildTool (UBT) is a tool that manages the process of building UE4 source code across a variety of build configurations.

UnrealHeaderTool (UHT) is a custom parsing and code-generation tool supporting the UObject system.

AutomationTool is a generic system used to automate processes, including testing and building games.

Proxy Geometry Tool is a tool-set that was developed as a way to increase your Unreal Engine 4 (UE4) project's performance while keeping the visual quality of your project uneffected.

Replay System is a tool which can record gameplay for later viewing. This feature is available in all games, from live, multiplayer games played on dedicated servers, to single-player games, and even including Play-In-Editor sessions. At a high level, the Replay system works by using a DemoNetDriver to read data drawn from the built-in replication system, similar to how a NetDriver operates in a live, networked gameplay environment.

Developer Resources

Back to the Top

Note: you will need to create an Epic Games account to download Unreal Engine.

Unity Development

Back to the Top

Unity is a cross-platform game development platform. Unity can be used to build high-quality 3D and 2D games, deploy them across mobile, desktop, VR/AR, consoles or the Web, and connect with loyal and enthusiastic players and customers. Checkout the Unity Manual to get started on your Unity projects.

Unity Terrain project. Source: Unity

Unity Hub Installs

Unity Asset Store

Unity Tools

Back to the Top

Unity Hub is a tool that helps manage multiple installations of the Unity Editor, create new projects, and access your work.

Unity Asset Storeis Unity's Store that has a library Of Textures, Models, Animations, Tutorials, and More for creating amazing 3D projects and games.

Unity Plus is a version of Unity for small businesses and serious hobbyists get more functionality and training resources to power up their projects.

Unity Pro is a version of Unity to create and operate amazing applications and immersive experiences with a complete solution for professionals across industries.

Unity Enterprise is a version of Unity for large teams creating at scale, Unity Enterprise delivers comprehensive technology, resources, and support that fuel innovation, reduce risk, and power your success.

Unity XR is a plug-in framework that enables XR providers to integrate with the Unity engine and make full use of its features.

Unity MARS is a tool that helps you create augmented reality (AR) apps with better workflows and purpose built authoring tools.

Unity Build Server is a tool that offloads your Unity project builds to network hardware, empowering your creative team to iterate more, produce higher quality products, and ship on time.

Unity Teams is a tool that helps you and your team create together, faster. It makes it simple to save, share and sync your Unity projects with anyone.

Unity Automated QA is a tool that shortens the development feedback loop with on-demand tests that anyone on your team can create and run without writing code.

Unity Simulation Pro is a tool that unlocks the true power of scalable simulations through a simulation-optimized version of the Unity runtime.

Unity Forma is a tool that brings the power of real-time 3D to your marketing. It turns 3D product data into interactive 3D configurators, stunning images and more.

Unity Reflect is a tool that helps create an experience immersive and collaborative real-time 3D environments, including in AR and VR, for better decision-making across your entire project lifecycle.

Unity Reflect Develop is a tool that builds your own custom applications to address any problem across the entire building and infrastructure lifecycle for internal or commercial deployment.

Unity ArtEngine is Unity’s material authoring tool. Harnessing the power of assisted creation, the tool helps creators accelerate production pipelines.

Unity Machine Learning Agents is a tol that helps train and embed intelligent agents by leveraging state-of-the-art deep learning technology.

Unity Essential Success is a tool that is your dedicated Support Coordinator and experienced Unity engineers are ready to help.

Bolt Visual scripting is a tool in Unity empowers creators to develop gameplay mechanics or interaction logic with a visual, graph-based system, instead of writing traditional lines of code.

VisualLive is a tool for construction professionals can visualize designs and collaborate in real-time by overlaying large BIM and CAD files onto jobsites using AR.

Professional Services: Optimization Accelerator is atool that boosts your project and your productivity in a four-day sprint with hands-on guidance from a Unity engineer. It will help you analyze performance problems, test solutions, and implement improvements.

Odin Inspector Enterprise is a tool that optimizes your project workflows and save thousands of development hours.

Multiplay is a resilient, multicloud hybrid server hosting and matchmaking platform without having to build and maintain your own infrastructure. Proven at scale in some of the world’s biggest games.

Vivox is a voice and text comms tool that boosts player engagement, retention rates and create immersive multiplayer experiences by leveraging an easy-to-implement, feature-rich voice and text chat service.

Relayᴮᴱᵀᴬ is a tool that provides a great multiplayer gaming experiences with Unity’s Relay service that's now in open beta.

Lobbyᴮᴱᵀᴬ is a tool that connects your players for great multiplayer gaming experiences with Lobby that's now in Open Beta.

Cloud Codeᴮᴱᵀᴬ is a tool that runs your game logic in the cloud as serverless functions and interact with other backend services.

Cloud Saveᴮᴱᵀᴬ is a tool that builds better player experiences by storing game data to the cloud.

Furioos is a tool that streams fully interactive 3D experiences from Unity, other real-time 3D platforms, and Windows applications with peak visual quality in any web browser.

OctaneRender® for Unity is a tool that lets you harness physically accurate rendering directly in Unity and compose gorgeous, cinematic-quality scenes with OctaneRender materials and lighting.

Pacelab WEAVR is a tool that provides a complete XR platform that enables any company, in any industry, to develop and manage enterprise-scale immersive training.

Interact is a tool that creates advanced real-time, human-centric simulations on any VR configuration for training, visualization, and safety.

Prespective is a tool that helps you design and simulate your machine, factory, or system in Unity real-time 3D. Create common ground among stakeholders and speed up decision processes with improved visualization tools.

Pixyz is a tool that lets you can quickly import, prep, and optimize your large CAD, mesh and point cloud models for real-time visualizations in Unity.

Backtrace is a game crash management platform that delivers uninterrupted game experiences by automating your response to errors.

SpeedTree® is a group of vegetation programming and modeling software products developed and sold by Interactive Data Visualization, Inc. that generates virtual foliage for animations, architecture and in real time for video games and demanding real time simulations. Unity Technologies acquired SpeedTree in July 2021.

SyncSketch is a collaboration software for visual artists that allows a team to view, review, and provide feedback on each other’s work. It includes support for both 2D and 3D assets as well as video. As of December 2021 SyncSketch has been acquired by Unity Technologies.

Unity Developer Resources

Back to the Top

Unity Knowledge Base is a library of articles that helps you troubleshoot common problems, whether it’s creating an account, importing assets, or baking a scene.
Unity Forum
Unity Issue Tracker
Unity Certifications
Learn game development with Unity
Autodesk for Games
Getting Started with DirectX 12 Ultimate
Getting Started with Vulkan
Getting Started with Apple Metal

AMD FidelityFX Super Resolution (FSR)

Back to the Top

AMD FidelityFX Super Resolution (FSR) is an open source, high-quality solution for producing high resolution frames from lower resolution inputs. FSR enables “practical performance” for costly render operations, such as hardware ray tracing for the AMD RDNA™ and AMD RDNA™ 2 architectures.
AMD FidelityFX Super Resolution (FSR) 2.0 is an open source, high-quality solution for producing high resolution frames from lower resolution inputs. It uses temporal data and optimized anti-aliasing to boost framerates in supported games while delivering similar or better image quality than native resolution.
AMD Open Source Driver for Vulkan® is an open-source Vulkan driver for Radeon™ graphics adapters on Linux®. It is built on top of AMD's Platform Abstraction Library (PAL), a shared component that is designed to encapsulate certain hardware and OS-specific programming details for many of AMD's 3D and compute drivers.
Vulkan® Memory Allocator (VMA) is a library provides a simple and easy to integrate API to help you allocate memory for Vulkan® buffer and image storage.
Radeon™ Raytracing Analyzer (RRA) is a tool that investigates the performance of your raytracing applications and highlight potential bottlenecks.
Radeon™ GPU Profiler is a low-level optimization tool that provides detailed information on Radeon™ GPUs.
Radeon™ GPU Analyzer is an offline compiler and performance analysis tool for DirectX®, Vulkan®, SPIR-V™, OpenGL®, and OpenCL™. It can be used together with RGP, RMV, and RDP.
Radeon™ Developer Panel (RDP) is an essential part of the Radeon™ Developer Tool Suite. It provides the communication channel that delivers requests to, and receives data from, the AMD Radeon™ driver.

NVIDIA DLSS

Back to the Top

NVIDIA DLSS (Deep Learning Super Sampling) is a temporal image upscaling AI rendering technology that increases graphics performance using dedicated Tensor Core AI processors on GeForce RTX™ GPUs. DLSS uses the power of a deep learning neural network to boost frame rates and generate beautiful, sharp images for your games.
NVIDIA Image Scaling (NIS) is an open-source spatial upscaler and sharpening algorithm that works cross-platform on all GPUs. It's similar to AMD's FidelityFX Super Resolution (FSR) solution.

AMID EVIL NVIDIA DLSS demo. Credit: NVIDIA and AMID EVIL

Microsoft HoloLens Headset

Back to the Top

Microsoft HoloLens Headset is a pair of mixed reality smartglasses developed and manufactured by Microsoft. It's also the first fully self-contained holographic computer to run Windows 10.

Microsoft HoloLens Headset. Source: Microsoft

PlayStation® VR Headset

Back to the Top

PlayStation® VR Headset is a virtual reality headset developed by Sony Interactive Entertainment. It gives the user an amazing gaming experience with stunning High Dynamic Range visuals and up to 120FPS, as two 1920 x 1080 OLED displays deliver a vast 100º field of view. PSVR is fully functional with the PlayStation 4™ and is forward compatible with the PlayStation 5™ home video game console.

PlayStation VR Headset. Source: PlayStation

PlayStation® VR 2 Headset

Back to the Top

PlayStation®VR 2 is the next generation of virtual reality gaming by PlayStation. It gives the user an amazing gaming experience with new stunning 4K High Dynamic Range visuals and up to 120fps, as two 2000 x 2040 OLED displays deliver over four times the resolution generated by the original PlayStation®VR headset. A vast 110º field of view and a light, well-balanced Fresnel lens. The lens adjustment dial allows you to customize spacing between lenses to match your eye position for an optimal onscreen image. Along with ground-breaking new controllers and genre-defining games, all made possible by the power of PS5™.

PlayStation VR 2 Headset. Source: PlayStation

Oculus Quest Headsets

Back to the Top

Oculus Developer Tools

Getting Started with Oculus

Oculus Developer Hub (ODH) for Windows is a desktop companion tool that streamlines app development for the Oculus Quest.

Oculus Developer Hub (ODH)

Oculus Developer Hub (ODH) for macOS is a desktop companion tool that streamlines app development for the Oculus Quest.

Oculus Integration SDK is a toolkit that provides tools for Unity packages OVRPlugin, Interaction SDK (Experimental), Platform SDK, Voice SDK, Audio SDK, and Lipsync SDK.

OVR Metrics Tool is a tool that provides performance metrics for Quest and Quest 2, including frame rate, heat, GPU & CPU throttling values, tears and stale frames per second.

GPU Systrace is a utility that contains required files to modify the Android SDK tool Systrace in order to provide low-level GPU pipeline data for apps running on Quest and Quest 2.

Meta Presence Platform is a suite of tools that let's you build the future of spatial computing with state-of-the-art AI, machine perception, graphics, and audio technologies, so you can invent experiences that blend the physical and virtual worlds.

Arkio is a collaborative design tool that enables you to sketch buildings, interiors or urban plans with others using VR, desktops, tablets and phones.

Gravity Sketch is a 3D design and collaboration tool that enables you to create vehicles, sneakers, furniture, characters, art, and more. Express your ideas with 3D strokes and geometry; solve complex design challenges working directly in 3D at any scale, all with a fun and intuitive toolset that makes working in 3D easy.

Immersed is a free app for multiple virtual screens in VR for Mac/PC/Linux (with no extra hardware) in stunning virtual worlds. It's great for solo work with multiple screens in a virtual coffee shop, or for collaborating with your team around even more screens and whiteboards.

Meta Quest 3

Back to the Top

Checkout the Meta Quest 3 (Available October 10th)

Meta Quest 3 Mixed Reality headset

Hardware Specs:

CPU: Snapdragon XR2 "Gen 2" chipset (double the GPU processing power of the Meta Quest 2).
RAM: 8GB.
Display Resolution: 2064x2208 pixels per eye nearly 30% better than the Meta Quest 2.
Field of View (FOV): 110 degrees horizontal and 96 degrees vertical.
Refresh Rate: 90Hz, 120Hz (experimental).
Wi-Fi: Wi-Fi 6E enabled (region permitting).

Price:

$499 for a 128GB model.
$649 for a 512GB model.

Battery Life:

Up to 2.2 hours of usage on average.
Gaming: 2.4 hours of usage on average.
Social: 2.2 hours of usage on average.
Productivity: 1.5 hours of usage on average.
Media: 2.9 hours of usage on average.
Charging Time: Approximately 2.3 hours with the included 18W power adapter (headset only).

Privacy Options

External LED: Signal notifies bystanders that the external cameras are in use.
Data Protection: Advanced encryption plus customized settings include data sharing opt-in/opt-out, electronic mic/camera disconnect and password/PIN protection to let multiple users explore safely.

Quest 2

Back to the Top

Oculus Quest 2 is a virtual reality headset developed by Facebook AKA Meta Technologies, LLC. The Quest 2 provides multiplayer games to unique social experiences to joining up with friends at a live show or instructor-led workout. With High Dynamic Range visuals and up to 120fps, as two 1832x1920 per-eye and vast 97° field of view horizontal. Oculus Quest 2 can be connected to a PC via Oculus Link, which can be performed using the official Link cable or a third party USB-C cable. This allows the headset to be used to run content from SteamVR and the desktop Oculus store. Note that a high-bandwidth USB specification (USB 3.0 at a minimum) is required to achieve a high-quality and smooth picture.

Oculus Quest 2 Headset. Source: Oculus

Quest Pro

Back to the Top

Meta Quest Pro($1500 USD, out October 25) is the first-ever device powered by the new Qualcomm Snapdragon XR2+ platform, which is optimized for VR to run at 50% more power than Meta Quest 2 with better thermal dissipation. Each Meta Quest Pro comes with the following specs:

Snapdragon XR2+ processor
12GB of memory(RAM)
Up to 256GB of storage.
90 Hz refresh rate.
50% more performance than Quest 2.
1.3x larger color gamut than Quest 2.
1800 by 1920 pixels per display, which is 37% more pixels per inch and 10% greater pixels per degree than Meta Quest 2.
10 high-res sensors (five inside the headset and five outside).
New sensor architecture with high-res outward-facing cameras that capture 4x the pixels as Meta Quest 2 cameras.
Charging dock and 45W USB-C power adapter.

Oculus Quest Pro Headset. Credit: Meta

Quest Pro Accessories:

Meta Quest Pro Compact Charging Dock ($79.99 USD, out October 25)
Meta Quest Pro VR Earphones ($49.99 USD, out October 25)
Meta Quest Pro Carry Case from Incase ($119.95 USD, out October 25)
Meta Quest Pro Full Light Blocker ($49.99 USD, out November 22)

Oculus Quest Pro Headset & Accessories. Credit: Meta

Oculus Quest Pro Controllers. Credit: Meta

Apple Vision Pro Headset

Back to the Top

Vision Pro Headset

Vision Pro Headset with battery pack

R1 & M2 chips in the Vision Pro Headset

VisionOS

Patents:

Apple Vision Pro Headset Specs:

Powered by an M2 chip and R1 chip(dedicated toward real-time sensor processing for the 12 cameras and 5 sensors on the Vision Pro headset).
High-resolution 4K micro OLED displays.
Display refresh rate of 90Hz and supports a special 96Hz mode for 24fps video.
Display Pixel Size will be between 3000 ppi-4000 ppi.
Display offers a brightness of 5,000 nits.
The color gamut is quoted as DCI > 97% DCI.
96W USB-C power adapter.
A USB-C port for data transfer.
A proprietary magnetic port(MagSafe) to attach the battery pack.
Expected battery life of about 2 hours per pack.
12 optical cameras and 5 sensors for tracking movements, mapping the environment, and projecting visual experiences.
WiFi 6E, which adds 6GHz spectrum to the 2.4GHz & 5GHz bands for increased bandwidth and less device interference.

OLEDoS (OLED on Silicon) is a display panel that typically has a diagonal length of less than 1 inch and meets the 3000 ppi-4000 ppi resolution criteria of AR/VR device displays. Existing OLED displays use Low-Temperature-Poly-Silicon (LTPS) or Oxide TFT based on glass substrates. OLEDoS uses silicon-wafer-based CMOS substrates. Using silicon substrates, ultra-fine circuit structures typically used in semiconductor processes can be reproduced, which in turn lead to the creation of ultra-high-resolution OLEDs when organic matter is deposited on them.

LG OLEDoS Display Specs:

High-resolution 4K micro OLED displays.
Display Pixel Size will be between 3000 ppi-4000 ppi.
Display offers a brightness of 5,000 nits.
The color gamut is quoted as DCI > 97% DCI.

OLEDoS: Solution for ultra-high resolution

Credit: LG

OLEDoS is also called Micro OLED in the market and features high efficiency, high luminance, infinite contrast, fast response and long LED life compared to OLED. The size is smaller than 1 inch, the user does not see the panel directly but sees the enlarged image through the optical lens. When used on AR/VR equipment, it shows high resolution in a small, lightweight wearable device.

Specs for OLED and OLEDoS

Credit: Sony

VisionOS

Back to the Top

VisionOS is Apple's iOS-like operating system for their upcoming AR/VR (Augmented Reality/Virtual Reality) headset powered by Apple's Silicon M2 and R1 chips. It will have an App Store with apps that include mixed-reality versions of their core Apple apps like Messages, FaceTime, Maps, and AR/VR games from Apple Arcade. Along with features such as Memojis and SharePlay that could be central to the user experience.

VisionOS. Image Credit: Apple

visionOS Resources

Apple AR Toolkits and Frameworks

Back to the Top

Reality Composer Pro is a tool that let's you discover how to easily compose, edit, and preview 3D content with Reality Composer Pro. Follow along as we explore this developer tool by setting up a new project, composing scenes, adding particle emitters and audio, and even previewing content on device.
Reality Composer is a powerful tool that makes it easy for you to create interactive augmented reality experiences with no prior 3D experience. The Reality Converter quickly converts your existing 3D models to USDZ so it works seamlessly in our tools and on all AR-enabled iPhone and iPad devices.
Apple Vision is a framework that performs face and face landmark detection, text detection, barcode recognition, image registration, and general feature tracking. Vision also allows the use of custom Core ML models for tasks like classification or object detection.
USDZ is a file format that contains a 3D scene or object saved in the USDZ Universal format, which is developed by Apple and Pixar Animation Studios. It is an uncompressed and unencrypted .ZIP archive that stores a Universal Scene Description (.USD, USDA, or USDC) file, which includes 3D geometry and shading data. USDZ files may also contain .PNG and .JPEG image textures and .M4A, .MP3, or .WAV audio files utilized in the 3D object or scene.
ARKit is a set set of software development tools to enable developers to build augmented-reality apps for iOS developed by Apple. The latest version ARKit 3.5 takes advantage of the new LiDAR Scanner and depth sensing system on iPad Pro(2020) to support a new generation of AR apps that use Scene Geometry for enhanced scene understanding and object occlusion.
RealityKit is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.
RealityUI is a Swift Package for creating familiar UI Elements and animations in a RealityKit rendered Augmented Reality or Virtual Reality scene.
SceneKit is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.
UIKit is a framework provides the required infrastructure for your iOS or tvOS apps. It provides the window and view architecture for implementing your interface, the event handling infrastructure for delivering Multi-Touch and other types of input to your app, and the main run loop needed to manage interactions among the user, the system, and your app.
SwiftUI is a user interface toolkit that provides views, controls, and layout structures for declaring your app's user interface. The SwiftUI framework provides event handlers for delivering taps, gestures, and other types of input to your application.
MetalFX is a new API that provides platform optimized graphics effects for Metal applications. With MetalFX Upscaling, your application can now render frames at a lower resolution, reducing rendering time, without compromising rendering quality. We'll also show you how and when to use its two effects: spatial upscaling, which delivers substantial performance gains, and temporal AA and upscaling, which delivers the highest quality rendering.

MetalFX Rendering. Image Credit: Apple

Apple Core Animation Framework is a graphics rendering and animation infrastructure that provides high frame rates and smooth animations without burdening the CPU and slowing down your app.
Apple Core Graphics Framework is a framework based on the Quartz advanced drawing engine. It provides low-level, lightweight 2D rendering with unmatched output fidelity.
GPUImage3 is the third generation of the GPUImage framework, an open source project for performing GPU-accelerated image and video processing on Mac and iOS. The original GPUImage framework was written in Objective-C and targeted Mac and iOS, the second iteration rewritten in Swift using OpenGL to target Mac, iOS, and Linux, and now this third generation is redesigned to use Apple's Metal in place of OpenGL.
Unity AR Foundation is a framework purpose-built for augmented reality development that allows you to build rich experiences once, then deploy across multiple mobile and wearable AR devices.
Universal Render Pipeline (URP) is a prebuilt Scriptable Render Pipeline, made by Unity. It provides artist-friendly workflows that let you quickly and easily create optimized graphics across a range of platforms, from mobile to high-end consoles and PCs.

HTC Vive Headset

Back to the Top

HTC Vive Headset is a virtual reality headset developed by HTC.

SteamVR

Back to the Top

SteamVR is the ultimate tool for experiencing VR content on the hardware of your choice. SteamVR supports the Valve Index, HTC Vive, Oculus Rift, Windows Mixed Reality headsets, and others.

SteamVR Home

Valve Index VR Headset. Source: Steam

Nreal Air AR Glasses

Back to the Top

Nreal Air AR Glasses & Nreal Adapter is a pair AR Glasses that you Watch, Stream, and Game on PC, Android, iOS, Nintendo Switch, Steam Deck, Playstation 4|5, Xbox Series X|S, and Cloud Gaming.

Samsung Gear VR Headset

Samsung Gear VR Headset is a virtual reality headset developed by Samsung along with collaboration from Oculus VR.

Perception Neuron Motion Capture Suit

Back to the Top

Perception Neuron Motion Capture Suit is a series of industry-leading motion capture solutions designed for every type of creator, from the amateur 3D animator to the VFX pro.

Vulkan Development

Back to the Top

Vulkan Learning Resources

Vulkan® is a modern cross-platform graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in a wide variety of devices from PCs and consoles to mobile phones and embedded platforms. Vulkan is currently in development by the Khronos consortium.

Khronos Group GitHub

Vulkan Documentation

HLSL to SPIR-V Feature Mapping Manual

Vulkan GLSL Ray Tracing Emulator Tutorial

Getting Started with Vulkan

Vulkan Samples

Khronos Community Forums

Vulkan Tools, Libraries, and Frameworks

Vulkan SDK is a set of tools that enables Vulkan developers to develop Vulkan applications.

SPIR-V is a set of tools that enables high-level language front-ends to emit programs in a standardized intermediate form to be ingested by Vulkan, OpenGL or OpenCL drivers. It eliminates the need for high-level language front-end compilers in device drivers, significantly reducing driver complexity, enables a broad range of language and framework front-ends to run on diverse hardware architectures and encourages a vibrant ecosystem of open source analysis, porting, debug and optimization tools.

SPIRV-Reflect is a lightweight library that provides a C/C++ reflection API for SPIR-V shader bytecode in Vulkan applications.

Vulkan® Tools is a project that provides Khronos official Vulkan Tools and Utilities for Windows, Linux, Android, and macOS.

Vulkan-Hpp is a API that provides a header only C++ bindings for the Vulkan C API to improve the developers Vulkan experience without introducing CPU runtime cost. It adds features like type safety for enums and bitfields, STL container support, exceptions and simple enumerations.

Vulkan® Memory Allocator (VMA) is a library that provides a simple and easy to integrate API to help you allocate memory for Vulkan® buffer and image storage.

AMD Open Source Driver for Vulkan® is an open-source Vulkan driver for AMD Radeon™ graphics adapters on Linux®.

NVIDIA® Nsight™ Visual Studio Edition is an application development environment for heterogeneous platforms which brings GPU computing into Microsoft Visual Studio. NVIDIA Nsight™ VSE allows you to build and debug integrated GPU kernels and native CPU code as well as inspect the state of the GPU and memory.

Radeon™ GPU Profiler is a performance tool that can be used by developers to optimize DirectX®12, Vulkan® and OpenCL™ applications for AMD RDNA™ and GCN hardware.

Radeon™ GPU Analyzer is a compiler and code analysis tool for Vulkan®, DirectX®, OpenGL® and OpenCL™.

Radeon™ Memory Visualizer (RMV) is a tool provided by AMD for use by game engine developers. It allows engineers to examine, diagnose, and understand the GPU memory management within their projects.

DXVK is a Vulkan-based translation layer for Direct3D 9/10/11 which allows running 3D applications on Linux using Wine.

MoltenVK is an implementation of Vulkan running on iOS and macOS using Apple's Metal graphics framework.

RenderDoc is a stand-alone graphics debugger that allows quick and easy single-frame capture and detailed introspection of any application using Vulkan, D3D11, OpenGL & OpenGL ES or D3D12 across Windows, Linux, Android, Stadia, or Nintendo Switch™.

PerfDoc is a cross-platform Vulkan layer which checks Vulkan applications for best practices on Arm Mali devices.

GLFW is an Open Source, multi-platform library for OpenGL, OpenGL ES and Vulkan application development. It provides a simple, platform-independent API for creating windows, contexts and surfaces, reading input, handling events, etc. GLFW natively supports Windows, macOS and Linux and other Unix-like systems. On Linux both X11 and Wayland are supported.

VulkanSharp is a project provides a .NET binding for the Vulkan API.

Vortice.Vulkan is a .NET Standard 2.0 and .NET5 low-level bindings for Vulkan API.

VKD3D-Proton is a fork of VKD3D, which aims to implement the full Direct3D 12 API on top of Vulkan.

ImGui is a bloat-free graphical user interface library for C++. It outputs optimized vertex buffers that you can render anytime in your 3D-pipeline enabled application. It is fast, portable, renderer agnostic and self-contained (no external dependencies).

Ash is a very lightweight wrapper around Vulkan.

gfx-rs is a low-level, cross-platform graphics and compute abstraction library in Rust.

Vulkan.jl is a lightweight wrapper around the Vulkan graphics and compute library. It exposes abstractions over the underlying C interface, primarily geared towards developers looking for a more natural way to work with Vulkan with minimal overhead.

Metal Development

Back to the Top

Metal Learning Resources

Metal is a low-level API that provides a platform-optimized, low-overhead API for developing the latest 3D pro applications and amazing games using a rich shading language with tighter integration between graphics and compute programs. To help you do more while managing ever more complex shader code, Metal adds an unparalleled suite of advanced GPU debugging tools to help you realize the full potential of your graphics code.

Apple Developer Documentation

MetalKit

Metal Shading Language Specification

Using Metal Feature Set Tables

Metal Performance Shaders

Optimizing Performance with the GPU Counters Instrument

Enabling Frame Capture

Reducing the Memory Footprint of Metal Apps

Metal Developer Tools for Windows

Metal Sample code

Metal plugin for TensorFlow

Metal Developer discussions

Metal Tools, Libraries, and Frameworks

Apple Foundation Framework is a framework provides a base layer of functionality for apps and frameworks, including data storage and persistence, text processing, date and time calculations, sorting and filtering, and networking. The classes, protocols, and data types defined by Foundation are used throughout the macOS, iOS, watchOS, and tvOS SDKs.

Apple Core Animation Framework is a graphics rendering and animation infrastructure that provides high frame rates and smooth animations without burdening the CPU and slowing down your app.

Apple Core Graphics Frameworkis a framework based on the Quartz advanced drawing engine. It provides low-level, lightweight 2D rendering with unmatched output fidelity.

Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The macOS operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.

Xcode includes everything developers need to create great applications for Mac, iPhone, iPad, Apple TV, and Apple Watch. Xcode provides developers a unified workflow for user interface design, coding, testing, and debugging. Xcode 12 is built as an Universal app that runs 100% natively on Intel-based CPUs and Apple Silicon. It includes a unified macOS SDK that features all the frameworks, compilers, debuggers, and other tools you need to build apps that run natively on Apple Silicon and the Intel x86_64 CPU.

SwiftUI is a user interface toolkit that provides views, controls, and layout structures for declaring your app's user interface. The SwiftUI framework provides event handlers for delivering taps, gestures, and other types of input to your application.

UIKit is a framework provides the required infrastructure for your iOS or tvOS apps. It provides the window and view architecture for implementing your interface, the event handling infrastructure for delivering Multi-Touch and other types of input to your app, and the main run loop needed to manage interactions among the user, the system, and your app.

AppKit is a graphical user interface toolkit that contains all the objects you need to implement the user interface for a macOS app such as windows, panels, buttons, menus, scrollers, and text fields, and it handles all the details for you as it efficiently draws on the screen, communicates with hardware devices and screen buffers, clears areas of the screen before drawing, and clips views.

RealityKit is a framework to implement high-performance 3D simulation and rendering with information provided by the ARKit framework to seamlessly integrate virtual objects into the real world.

SceneKit is a high-level 3D graphics framework that helps you create 3D animated scenes and effects in your iOS apps.

Instruments is a powerful and flexible performance-analysis and testing tool that’s part of the Xcode tool set. It’s designed to help you profile your iOS, watchOS, tvOS, and macOS apps, processes, and devices in order to better understand and optimize their behavior and performance.

Cocoapods is a dependency manager for Swift and Objective-C used in Xcode projects by specifying the dependencies for your project in a simple text file. CocoaPods then recursively resolves dependencies between libraries, fetches source code for all dependencies, and creates and maintains an Xcode workspace to build your project.

AppCode is constantly monitoring the quality of your code. It warns you of errors and smells and suggests quick-fixes to resolve them automatically. AppCode provides lots of code inspections for Objective-C, Swift, C/C++, and a number of code inspections for other supported languages.

MoltenVK is an implementation of Vulkan running on iOS and macOS using Apple's Metal graphics framework.

DirectX Development

Back to the Top

DirectX Learning Resources

Microsoft DirectX® is a low-level API that handles tasks related to multimedia for game programming and video on Microsoft platforms(Windows & Xbox).

Getting Started with DirectX 12 Ultimate

Getting Started with the DirectX 12 Agility SDK

DirectX 12 and Graphics Education | YouTube

DirectX— Feature Level 12_2

DirectX 12 Technology | NVIDIA

AMD DirectX® 12 (DX12) Technology | AMD

Top Microsoft DirectX Courses Online | Udemy

DirectX - Learn Microsoft DirectX from Scratch Course | Udemy

DirectX 11 Programming Course | Udemy

DirectX Tools, Libraries, and Frameworks

Visual Studio is an integrated development environment (IDE) from Microsoft; which is a feature-rich application that can be used for many aspects of software development. Visual Studio makes it easy to edit, debug, build, and publish your app. By using Microsoft software development platforms such as Windows API, Windows Forms, Windows Presentation Foundation, and Windows Store.

Visual Studio Code is a code editor redefined and optimized for building and debugging modern web and cloud applications.

DirectX-Graphics-Samples is a project that contains the DirectX 12 Graphics samples that demonstrate how to build graphics intensive applications for Windows 10.

PIX on Windows is a performance tuning and debugging tool for DirectX 12 games on Windows.

DirectStorage API is an API in the DirectX family originally designed for the Velocity Architecture to Windows. The DirectX API is architected in a way that takes all this into account and maximizes performance throughout the entire pipeline from NVMe drive all the way to the GPU. It does this in several ways: by reducing per-request NVMe overhead, enabling batched many-at-a-time parallel IO requests which can be efficiently fed to the GPU, and giving games finer grain control over when they get notified of IO request completion instead of having to react to every tiny IO completion. The DirectStorage API will be available on Windows 11 PCs with NVMe SSDs, but will also be support in Windows 10 version 1909 and newer.

NVRHI (NVIDIA Rendering Hardware Interface) is a library that implements a common abstraction layer over multiple graphics APIs (GAPIs): Direct3D 11, Direct3D 12, and Vulkan 1.2. It works on Windows (x64 only) and Linux (x64 and ARM64).

RTXMU - RTX Memory Utility SDK is an SDK tool that batchs up all of the acceleration structure build inputs and pass them to RTXMU which in turn will perform all the suballocation memory requests and build details including compaction. Then post build info is abstracted away by the SDK in order to do compaction under the hood. RTXMU returns acceleration structure handle ids that are used to reference the underlying memory buffers. These handle ids are passed into RTXMU to create compaction copy workloads, deallocate unused build resources or remove all memory associated with an acceleration structure.

Radeon™ GPU Profiler is a performance tool that can be used by developers to optimize DirectX®12, Vulkan® and OpenCL™ applications for AMD RDNA™ and GCN hardware.

Radeon™ GPU Analyzer is a compiler and code analysis tool for Vulkan®, DirectX®, OpenGL® and OpenCL™.

FNA is an XNA4 reimplementation that focuses solely on developing a fully accurate XNA4 runtime for the desktop.

FAudio is an XAudio reimplementation that focuses solely on developing fully accurate DirectX Audio runtime libraries for the FNA project, including XAudio2, X3DAudio, XAPO, and XACT3.

Simple DirectMedia Layer is a cross-platform development library designed to provide low level access to audio, keyboard, mouse, joystick, and graphics hardware via OpenGL and Direct3D. It is used by video playback software, emulators, and popular games including Valve's award winning catalog.

DXVK is a Vulkan-based translation layer for Direct3D 9/10/11 which allows running 3D applications on Linux using Wine.

VKD3D-Proton is a fork of VKD3D, which aims to implement the full Direct3D 12 API on top of Vulkan.

Computer Vision Development

Back to the Top

Computer Vision Learning Resources

Computer Vision is a field of Artificial Intelligence (AI) that focuses on enabling computers to identify and understand objects and people in images and videos.

OpenCV Courses

Exploring Computer Vision in Microsoft Azure

Top Computer Vision Courses Online | Coursera

Top Computer Vision Courses Online | Udemy

Learn Computer Vision with Online Courses and Lessons | edX

Computer Vision and Image Processing Fundamentals | edX

Introduction to Computer Vision Courses | Udacity

Computer Vision Nanodegree program | Udacity

Machine Vision Course |MIT Open Courseware

Computer Vision Training Courses | NobleProg

Visual Computing Graduate Program | Stanford Online

Computer Vision Tools, Libraries, and Frameworks

OpenCV is a highly optimized library with focus on real-time computer vision applications. The C++, Python, and Java interfaces support Linux, MacOS, Windows, iOS, and Android.

Microsoft Cognitive Toolkit (CNTK) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

Scikit-Learn is a Python module for machine learning built on top of SciPy, NumPy, and matplotlib, making it easier to apply robust and simple implementations of many popular machine learning algorithms.

NVIDIA cuDNN is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN accelerates widely used deep learning frameworks, including Caffe2, Chainer, Keras, MATLAB, MxNet, PyTorch, and TensorFlow.

Automated Driving Toolbox™ is a MATLAB tool that provides algorithms and tools for designing, simulating, and testing ADAS and autonomous driving systems. You can design and test vision and lidar perception systems, as well as sensor fusion, path planning, and vehicle controllers. Visualization tools include a bird’s-eye-view plot and scope for sensor coverage, detections and tracks, and displays for video, lidar, and maps. The toolbox lets you import and work with HERE HD Live Map data and OpenDRIVE® road networks. It also provides reference application examples for common ADAS and automated driving features, including FCW, AEB, ACC, LKA, and parking valet. The toolbox supports C/C++ code generation for rapid prototyping and HIL testing, with support for sensor fusion, tracking, path planning, and vehicle controller algorithms.

LRSLibrary is a Low-Rank and Sparse Tools for Background Modeling and Subtraction in Videos. The library was designed for moving object detection in videos, but it can be also used for other computer vision and machine learning problems.

Image Processing Toolbox™ is a tool that provides a comprehensive set of reference-standard algorithms and workflow apps for image processing, analysis, visualization, and algorithm development. You can perform image segmentation, image enhancement, noise reduction, geometric transformations, image registration, and 3D image processing.

Computer Vision Toolbox™ is a tool that provides algorithms, functions, and apps for designing and testing computer vision, 3D vision, and video processing systems. You can perform object detection and tracking, as well as feature detection, extraction, and matching. You can automate calibration workflows for single, stereo, and fisheye cameras. For 3D vision, the toolbox supports visual and point cloud SLAM, stereo vision, structure from motion, and point cloud processing.

Statistics and Machine Learning Toolbox™ is a tool that provides functions and apps to describe, analyze, and model data. You can use descriptive statistics, visualizations, and clustering for exploratory data analysis; fit probability distributions to data; generate random numbers for Monte Carlo simulations, and perform hypothesis tests. Regression and classification algorithms let you draw inferences from data and build predictive models either interactively, using the Classification and Regression Learner apps, or programmatically, using AutoML.

Lidar Toolbox™ is a tool that provides algorithms, functions, and apps for designing, analyzing, and testing lidar processing systems. You can perform object detection and tracking, semantic segmentation, shape fitting, lidar registration, and obstacle detection. Lidar Toolbox supports lidar-camera cross calibration for workflows that combine computer vision and lidar processing.

Mapping Toolbox™ is a tool that provides algorithms and functions for transforming geographic data and creating map displays. You can visualize your data in a geographic context, build map displays from more than 60 map projections, and transform data from a variety of sources into a consistent geographic coordinate system.

UAV Toolbox is an application that provides tools and reference applications for designing, simulating, testing, and deploying unmanned aerial vehicle (UAV) and drone applications. You can design autonomous flight algorithms, UAV missions, and flight controllers. The Flight Log Analyzer app lets you interactively analyze 3D flight paths, telemetry information, and sensor readings from common flight log formats.

Parallel Computing Toolbox™ is a tool that lets you solve computationally and data-intensive problems using multicore processors, GPUs, and computer clusters. High-level constructs such as parallel for-loops, special array types, and parallelized numerical algorithms enable you to parallelize MATLAB® applications without CUDA or MPI programming. The toolbox lets you use parallel-enabled functions in MATLAB and other toolboxes. You can use the toolbox with Simulink® to run multiple simulations of a model in parallel. Programs and models can run in both interactive and batch modes.

Partial Differential Equation Toolbox™ is a tool that provides functions for solving structural mechanics, heat transfer, and general partial differential equations (PDEs) using finite element analysis.

ROS Toolbox is a tool that provides an interface connecting MATLAB® and Simulink® with the Robot Operating System (ROS and ROS 2), enabling you to create a network of ROS nodes. The toolbox includes MATLAB functions and Simulink blocks to import, analyze, and play back ROS data recorded in rosbag files. You can also connect to a live ROS network to access ROS messages.

Robotics Toolbox™ provides a toolbox that brings robotics specific functionality(designing, simulating, and testing manipulators, mobile robots, and humanoid robots) to MATLAB, exploiting the native capabilities of MATLAB (linear algebra, portability, graphics). The toolbox also supports mobile robots with functions for robot motion models (bicycle), path planning algorithms (bug, distance transform, D*, PRM), kinodynamic planning (lattice, RRT), localization (EKF, particle filter), map building (EKF) and simultaneous localization and mapping (EKF), and a Simulink model a of non-holonomic vehicle. The Toolbox also including a detailed Simulink model for a quadrotor flying robot.

Deep Learning Toolbox™ is a tool that provides a framework for designing and implementing deep neural networks with algorithms, pretrained models, and apps. You can use convolutional neural networks (ConvNets, CNNs) and long short-term memory (LSTM) networks to perform classification and regression on image, time-series, and text data. You can build network architectures such as generative adversarial networks (GANs) and Siamese networks using automatic differentiation, custom training loops, and shared weights. With the Deep Network Designer app, you can design, analyze, and train networks graphically. It can exchange models with TensorFlow™ and PyTorch through the ONNX format and import models from TensorFlow-Keras and Caffe. The toolbox supports transfer learning with DarkNet-53, ResNet-50, NASNet, SqueezeNet and many other pretrained models.

Reinforcement Learning Toolbox™ is a tool that provides an app, functions, and a Simulink® block for training policies using reinforcement learning algorithms, including DQN, PPO, SAC, and DDPG. You can use these policies to implement controllers and decision-making algorithms for complex applications such as resource allocation, robotics, and autonomous systems.

Deep Learning HDL Toolbox™ is a tool that provides functions and tools to prototype and implement deep learning networks on FPGAs and SoCs. It provides pre-built bitstreams for running a variety of deep learning networks on supported Xilinx® and Intel® FPGA and SoC devices. Profiling and estimation tools let you customize a deep learning network by exploring design, performance, and resource utilization tradeoffs.

Model Predictive Control Toolbox™ is a tool that provides functions, an app, and Simulink® blocks for designing and simulating controllers using linear and nonlinear model predictive control (MPC). The toolbox lets you specify plant and disturbance models, horizons, constraints, and weights. By running closed-loop simulations, you can evaluate controller performance.

Vision HDL Toolbox™ is a tool that provides pixel-streaming algorithms for the design and implementation of vision systems on FPGAs and ASICs. It provides a design framework that supports a diverse set of interface types, frame sizes, and frame rates. The image processing, video, and computer vision algorithms in the toolbox use an architecture appropriate for HDL implementations.

Microsoft AirSim is a simulator for drones, cars and more, built on Unreal Engine (with an experimental Unity release). AirSim is open-source, cross platform, and supports software-in-the-loop simulation with popular flight controllers such as PX4 & ArduPilot and hardware-in-loop with PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. AirSim is being developed as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles.

Photogrammetry Development

Back to the Top

Photogrammetry Learning Resources

Photogrammetry is the art and science of extracting 3D information from photographs. The process involves taking overlapping photographs of an object, structure, or space, and converting them into 2D or 3D digital models. Photogrammetry is often used by surveyors, architects, engineers, and contractors to create topographic maps, meshes, point clouds, or drawings based on the real-world.

Aerial photogrammetry is process of utilizing aircrafts to produce aerial photography that can be turned into a 3D model or mapped digitally. Now, it is possible to do the same work with a drone.

Terrestrial(Close-range) photogrammetry is when images are captured using a handheld camera or with a camera mounted to a tripod. The output of this method is not to create topographic maps, but rather to make 3D models of a smaller object.

Top Photogrammetry Courses Online | Udemy

Photogrammetry With Drones: In Mapping Technology | Udemy

Introduction to Photogrammetry Course | Coursera

Photogrammetry Online Classes and Training | Linkedin Learning

Pix4D training and certification for mapping professionals

Drone mapping and photogrammetry workshops with Pix4D

Digital Photogrammetric Systems Course | Purdue Online Learning

Photogrammetry Training | Deep3D Photogrammetry

ASPRS Certification Program

Photogrammetry Tools, Libraries, and Frameworks

Autodesk® ReCap™ is a software tool that converts reality captured from laser scans or photos into a 3D model or 2D drawing that's ready to be used in your design built for UAV and drone processes.

Autodesk® ReCap™ Photo is a cloud-connected solution tailored for drone/UAV photo capturing workflows. Using ReCap Photo, you can create textured meshes, point clouds with geolocation, and high-resolution orthographic views with elevation maps.

Pix4D is a unique suite of photogrammetry software for drone mapping. Capture images with our app, process on desktop or cloud and create maps and 3D models.

PIX4Dmapper is the leading photogrammetry software for professional drone mapping.

Agisoft Metashape is a stand-alone software product that performs photogrammetric processing of digital images and generates 3D spatial data to be used in GIS applications, cultural heritage documentation, and visual effects production as well as for indirect measurements of objects of various scales.

MicroStation is a CAD software platform for 2D and 3D dimensional design and drafting, developed and sold by Bentley Systems. It generates 2D/3D vector graphics objects and elements and includes building information modeling (BIM) features.

Leica Photogrammetry Suite (LPS) is a powerful photogrammetry system that delivers full analytical triangulation, the generation of digital terrain models, orthophoto production, mosaicking, and 3D feature extraction in a user-friendly environment that guarantees results even for photogrammetry novices.

PhotoCatch is the first app for Apple's Object Capture API, enabling anyone to create stunning 3D models in minutes, not days, with no code or 3D experience required.

PhotoCatch Desktop is a Professional photogrammetry workstation for exporting beautiful 3D assets ready for Augmented Reality and Visual Effects. It provides performance optimizations for M1 Max and M1 Ultra on MacBook Pro and Mac Studio, so you can process up to 4 3D models at the same time.

PhotoCatch Cloud is a professional service that brings desktop class photogrammetry and 3D editing tools to mobile devices, so you can capture, edit, and share 3D content on-site. It Syncs photo capture with your equipment and use Depth Capture so your 3D models stay the same size as the real world objects.

Terramodel is a powerful software package for the surveyor, civil engineer or contractor who requires a CAD and design package with integrated support for raw survey data.

MicMac is a free and open-source photogrammetry software tools for 3D reconstruction.

[3DF Zephyr] (https://www.3dflow.net/3df-zephyr-photogrammetry-software/) is a photogrammetry software solution by 3Dflow. It allows you automatically reconstruct 3D models from photos and deal with any 3D reconstruction and scanning challenge. No matter what camera sensor, drone or laser scanner device you are going to use.

COLMAP is a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline with a graphical and command-line interface. It offers a wide range of features for reconstruction of ordered and unordered image collections.

Multi-View Environment (MVE) is an effort to ease the work with multi-view datasets and to support the development of algorithms based on multiple views. It features Structure from Motion, Multi-View Stereo and Surface Reconstruction. MVE is developed at the TU Darmstadt.

AliceVision is a Photogrammetric Computer Vision Framework which provides 3D Reconstruction and Camera Tracking algorithms. AliceVision comes up with strong software basis and state-of-the-art computer vision algorithms that can be tested, analyzed and reused.

Meshroom is a free, open-source 3D Reconstruction Software based on the AliceVision framework.

PhotoModeler is a software extracts Measurements and Models from photographs taken with an ordinary camera. A cost-effective way for accurate 2D or 3D measurement, photo-digitizing, surveying, 3D scanning, and reality capture.

ODM is an open source command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images.

WebODM is a user-friendly, commercial grade software for drone image processing. Generate georeferenced maps, point clouds, elevation models and textured 3D models from aerial images. It supports multiple engines for processing, currently ODM and MicMac.

NodeODM is a standard API specification for processing aerial images with engines such as ODM. The API is used by clients such as WebODM, CloudODM and PyODM.

[ClusterODM]https://www.opendronemap.org/clusterodm/) is a reverse proxy, load balancer and task tracker with optional cloud autoscaling capabilities for NodeODM API compatible nodes. In a nutshell, it's a program to link together multiple NodeODM API compatible nodes under a single network address.

FIELDimageR is an R package to analyze orthomosaic images from agricultural field trials.

Regard3D is a free and open source structure-from-motion program. It converts photos of an object, taken from different angles, into a 3D model of this object.

AR-VR-Guide

AR/VR Guide

A guide covering Augmented Reality (AR) & Virtual Reality (VR) including the applications, libraries and tools that will make you a better and more efficient Augmented Reality (AR) & Virtual Reality (VR) development.

Table of Contents

Getting Started with AR/VR

Developer Resources

AR/VR Training Courses

AR/VR Tools, Frameworks, and Hardware

Tools & Frameworks

Unreal Engine 5 Development

Unreal Engine Tools

Developer Resources

Unity Development

Unity Tools

Unity Developer Resources

AMD FidelityFX Super Resolution (FSR)

NVIDIA DLSS

Microsoft HoloLens Headset

PlayStation® VR Headset

PlayStation® VR 2 Headset

Oculus Quest Headsets

Oculus Developer Tools

Meta Quest 3

Quest 2

Quest Pro

Quest Pro Accessories:

Apple Vision Pro Headset

VisionOS

Apple AR Toolkits and Frameworks

HTC Vive Headset

SteamVR

Nreal Air AR Glasses

Samsung Gear VR Headset

Perception Neuron Motion Capture Suit

Vulkan Development

Vulkan Learning Resources

Vulkan Tools, Libraries, and Frameworks

Metal Development

Metal Learning Resources

Metal Tools, Libraries, and Frameworks

DirectX Development

DirectX Learning Resources

DirectX Tools, Libraries, and Frameworks

Computer Vision Development

Computer Vision Learning Resources

Computer Vision Tools, Libraries, and Frameworks

Photogrammetry Development

Photogrammetry Learning Resources

Photogrammetry Tools, Libraries, and Frameworks

Autodesk Development

ARVoxelKit

Reason3D-PyTorch

Paper-Piano

erizo

search