Vision Framework API Reference

Comprehensive reference for Vision framework computer vision: subject segmentation, hand/body pose detection, face analysis, OCR, barcode scanning, document scanning, and sensitive content analysis.

Overview

Vision provides computer vision algorithms for still images and video. This reference covers all people-focused APIs with complete code examples.

Core workflow:

1. Create request (e.g., VNDetectHumanHandPoseRequest())
2. Create handler with image (VNImageRequestHandler(cgImage: image))
3. Perform request (try handler.perform([request]))
4. Access observations from request.results

API Categories

Subject Segmentation

• VNGenerateForegroundInstanceMaskRequest (iOS 17+)

  • Class-agnostic subject detection
  • Returns instance masks for individual objects
  • Use with CoreImage for HDR compositing

• VisionKit APIs (iOS 16+)

  • ImageAnalysisInteraction / ImageAnalysisOverlayView
  • System UI for subject lifting
  • Programmatic access via ImageAnalyzer

Person Segmentation

• VNGeneratePersonInstanceMaskRequest (iOS 17+)

  • Separate masks for up to 4 people
  • Per-person effects and counting

• VNGeneratePersonSegmentationRequest (iOS 15+)

  • Single mask for all people
  • Background removal

Hand Pose Detection

• VNDetectHumanHandPoseRequest (iOS 14+)
  • 21 hand landmarks per hand
  • Gesture recognition support
  • Configurable maximumHandCount

Body Pose Detection

• VNDetectHumanBodyPoseRequest (iOS 14+)

  • 18 body landmarks (2D)
  • Normalized coordinates

• VNDetectHumanBodyPose3DRequest (iOS 17+)

  • 17 body joints (3D)
  • Real-world coordinates in meters
  • Supports depth data input

Iterative Segmentation (Tap-to-Segment)

• GenerateIterativeSegmentationRequest (iOS 27+, not watchOS)
  • Segment any object the user selects — by tap, box, or scribble/lasso
  • Refine the mask with included/excluded points
  • Requires a one-time model download (downloadAssets / assetStatus)

Visual Intelligence

• Visual Intelligence system integration (iOS 26+; iPadOS and macOS from the 27 releases)
  • System-level visual analysis integration
  • System store integrations: events, contacts, and medical-device readings captured by Visual Intelligence reach your app via EventKit, Contacts, and HealthKit

Vision on watchOS

• Modern Swift Vision API on watchOS (watchOS 27+)
  • Subset: faces, classification, segmentation, saliency, barcodes, feature prints, tracking
  • No text recognition or pose requests on the watch

Vision Tools for Foundation Models

• BarcodeReaderTool / OCRTool (from the 27 releases; neither on tvOS, OCRTool also not on watchOS)
  • Ready-made tools the on-device LLM can call on attached images

Face Detection

• VNDetectFaceRectanglesRequest (iOS 11+)

  • Face bounding boxes only

• VNDetectFaceLandmarksRequest (iOS 11+)

  • Detailed facial landmarks
  • Pupil locations (Revision 3+)

Sensitive Content Analysis

• SCSensitivityAnalyzer (iOS 17+, macOS 14+, visionOS 2+; not watchOS/tvOS)
  • Detects nudity, gore, and violence in images and video
  • Gated by the user's Sensitive Content Warning setting — check analysisPolicy != .disabled before analyzing; .disabled means the feature is off, not an error
  • iOS 27: detectedTypes (Set<SCSensitivityAnalysis.ContentType>) adds per-category results (.sexuallyExplicit, .goreOrViolence); prior to iOS 27 only the boolean isSensitive is available
  • The verdict must stay on-device — the Apple Developer Program License Agreement prohibits transmitting whether content was flagged off the device (no analytics, server-side moderation queue, or synced cache); reporting has to be a separate explicit user action
  • No public deep-link to the Sensitive Content Warning toggle — openSettingsURLString opens your app's own settings page, so guide users to the setting with on-screen instructions
  • Testing: install Apple's configuration profile (it flags a harmless test image) via Settings > General > VPN & Device Management — downloading the profile alone does nothing

Code Examples

Subject Segmentation

let request = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

guard let observation = request.results?.first as? VNInstanceMaskObservation else {
    return
}

// Get soft segmentation mask
let mask = try observation.createScaledMask(
    for: observation.allInstances,
    croppedToInstancesContent: false
)

Hand Pose Detection

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 2

try handler.perform([request])

for observation in request.results as? [VNHumanHandPoseObservation] ?? [] {
    let thumbTip = try observation.recognizedPoint(.thumbTip)
    let indexTip = try observation.recognizedPoint(.indexTip)

    // Check confidence
    guard thumbTip.confidence > 0.5, indexTip.confidence > 0.5 else {
        continue
    }

    // Calculate distance for pinch gesture
    let distance = hypot(
        thumbTip.location.x - indexTip.location.x,
        thumbTip.location.y - indexTip.location.y
    )
}

CoreImage Compositing

// Preserve HDR with CoreImage
let filter = CIFilter(name: "CIBlendWithMask")!
filter.setValue(sourceImage, forKey: kCIInputImageKey)
filter.setValue(CIImage(cvPixelBuffer: mask), forKey: kCIInputMaskImageKey)
filter.setValue(newBackground, forKey: kCIInputBackgroundImageKey)

let output = filter.outputImage

Quick Reference Tables

Subject Segmentation APIs

API	Platform	Purpose
VNGenerateForegroundInstanceMaskRequest	iOS 17+	Class-agnostic subject instances
VNGeneratePersonInstanceMaskRequest	iOS 17+	Up to 4 people separately
VNGeneratePersonSegmentationRequest	iOS 15+	All people (single mask)
ImageAnalysisInteraction (VisionKit)	iOS 16+	UI for subject lifting
GenerateIterativeSegmentationRequest	iOS 27+	Tap/box/scribble-seeded segmentation

Pose Detection APIs

API	Platform	Landmarks	Coordinates
VNDetectHumanHandPoseRequest	iOS 14+	21 per hand	2D normalized
VNDetectHumanBodyPoseRequest	iOS 14+	18 body joints	2D normalized
VNDetectHumanBodyPose3DRequest	iOS 17+	17 body joints	3D meters

Observation Types

Observation	Returned By
VNInstanceMaskObservation	Foreground/person instance masks
VNPixelBufferObservation	Person segmentation (single mask)
VNHumanHandPoseObservation	Hand pose
VNHumanBodyPoseObservation	Body pose (2D)
VNHumanBodyPose3DObservation	Body pose (3D)
VNFaceObservation	Face detection/landmarks
SCSensitivityAnalysis.detectedTypes	Sensitive content analysis (iOS 27+)

Coordinate System

Vision uses lower-left origin with normalized coordinates (0.0-1.0).

Converting to UIKit (top-left origin):

let visionPoint = recognizedPoint.location

let uiPoint = CGPoint(
    x: visionPoint.x * imageWidth,
    y: (1 - visionPoint.y) * imageHeight  // Flip Y axis
)

Performance Considerations

• Always use background queue – Vision is resource intensive
• Set maximumHandCount appropriately – Pose computed for all detected hands ≤ max
• Check confidence scores – Low confidence landmarks are unreliable
• Downscale images – 1280x720 sufficient for most real-time use cases

Resources

• Vision Framework (Main Skill) – Decision trees and patterns
• Vision Diagnostics – Troubleshooting guide

WWDC Sessions

• WWDC 2026 - Session 237: What's new in image understanding
• WWDC 2026 - Session 297: Best practices for integrating visual intelligence in your app
• WWDC 2023 - Session 10176: Lift subjects from images in your app
• WWDC 2023 - Session 111241: 3D body pose and person segmentation
• WWDC 2020 - Session 10653: Detect Body and Hand Pose with Vision

Apple Documentation

• Vision Framework
• VisionKit Framework