算法艺术创作
algorithmic-art
通过代码和可调参数创作生成艺术与交互式视觉效果。
mobile-ar-vr【设计与创意】Use this skill when the user says 'AR', 'VR', 'augmented reality', 'virtual reality', 'ARKit', 'ARCore', 'Unity AR', '3D rendering', 'SceneView', 'AR scene', 'AR interaction', 'AR performance'. This skill enforces: platform-specific AR configuration (ARKit vs ARCore), scene setup with anchor management, optimal 3D model handling with LODs and compres…
独立 GitHub 快照 · 自动分类 · 以原作者文档为准
Design and implement AR/VR experiences for mobile platforms with platform-aware setup, optimized 3D content, intuitive interaction patterns, and performance within device constraints.
"AR", "VR", "augmented reality", "virtual reality", "ARKit", "ARCore", "Unity AR", "3D rendering", "SceneView", "AR scene", "AR interaction", "AR performance", "AR placement", "AR anchor", "AR lighting", "AR tracking", "AR model", "AR filter", "AR face tracking", "AR image tracking", "AR plane detection".
AR/VR integration plan with platform selection, scene architecture, anchor management strategy, 3D model pipeline, interaction model, and performance budget.
No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output.
300 lines
Is the app cross-platform (iOS + Android)?
├── Yes → Use ARFoundation (Unity) or SceneView (React Native / Flutter)
│ ├── Unity project? → ARFoundation with ARCore/ARKit XR Plugins
│ └── React Native / Flutter? → SceneView / ar_flutter_plugin
├── iOS only → ARKit (native Swift) — best iOS experience
└── Android only → ARCore (native Kotlin/Java)
Does the app need LiDAR features?
├── Yes → ARKit (LiDAR on iOS), ARCore Depth API (limited Android)
└── No → Any framework works
What type of AR tracking is needed?
├── World tracking (object placement on surfaces)
│ ├── Indoor → ARWorldTrackingConfiguration + plane detection
│ └── Outdoor → GPS + ARKit GeoTracking / ARCore + VPS
├── Image tracking (detect 2D images)
│ └── ARImageTrackingConfiguration / Augmented Images
├── Face tracking (AR filters, masks)
│ └── ARFaceTrackingConfiguration (iOS TrueDepth) / ARCore Augmented Faces
├── Body tracking (motion capture)
│ └── ARBodyTrackingConfiguration (iOS A12+) / ARCore 3D Body
└── Object scanning (3D object recognition)
└── ARObjectScanningConfiguration / ARCore Cloud Anchors
Source format?
├── Animated → glTF 2.0 (universal) or USDZ (iOS native with animation)
│ GLTF preferred — broader tool support, Draco compression
├── Static → glTF or compressed OBJ (Draco compressed for streaming)
└── CAD/BIM data → USDZ (Pixar USD ecosystem, retains metadata)
Budle target size?
├── <10MB total → glTF with texture atlases, no LOD needed
├── 10-50MB → LOD generation (3 levels), Draco compression
└── >50MB → Streaming (download on demand), not bundled
ARKit (iOS 11+): A12 Bionic or later for LiDAR, ARWorldTrackingConfiguration for 6DOF, ARImageTrackingConfiguration for image targets, ARFaceTrackingConfiguration for face AR. ARCore (Android 7+): Google Play Services for AR, supported devices list at developers.google.com/ar/devices, Depth API for occlusion, Augmented Images for image tracking. Cross-platform: use ARFoundation (Unity) or SceneView (React Native / Flutter).
| Capability | ARKit | ARCore |
|---|---|---|
| Plane detection | Horizontal + vertical + LiDAR mesh | Horizontal + vertical |
| Image tracking | Concurrent, up to 100 images | Up to 20 images |
| Face tracking | TrueDepth camera (iPhone X+) | Front camera, less precise |
| World mapping | Persistent via ARWorldMap | Cloud Anchors |
| LiDAR support | iPad Pro 2020+, iPhone 12 Pro+ | Select Android devices |
Configure ARSession. For world tracking: set ARWorldTrackingConfiguration (iOS) or Config with PlaneFindingMode (Android). Enable auto-focus, ambient light estimation, and environment texturing. Handle session interruptions.
ARSession
├── Configuration (plane detection, light estimation, environment texturing, frame semantics)
├── Anchors (Plane anchors, Image anchors, Object anchors)
└── State management (Running -> Paused -> Running, Limited tracking -> Recovery prompt)
Preferred formats: USDZ (iOS native, animation + materials), glTF 2.0 (cross-platform, Draco compression). Model pipeline: source -> optimize (decimate to target poly count) -> compress (Draco for glTF) -> LOD generation (3 levels at 100%, 60%, 30% detail) -> bundle. Texture atlas: combine textures, max 1024x1024 for mobile, ASTC (iOS) or ETC2 (Android).
Placement: hit-test against detected planes (raycast), show ghost preview, confirm placement with haptic + animation. Manipulation: one-finger rotate, two-finger scale, long-press drag. Selection: tap to select (visual highlight + bounding box), double-tap for context menu. Feedback: light impact for selection, medium for placement, ripple animation, glow effect.
GPU instancing for repeated objects. Occlusion culling (LiDAR depth on iOS, Depth API on Android). Limit tracked images to 10 max. Batch ARAnchor operations. LOD switching based on camera distance. Profile with Xcode SceneKit debugger or Android GPU Inspector.
VR requires dedicated headset (Meta Quest, Apple Vision Pro). Mobile VR is primarily ARKit or Unity VR build. For Vision Pro: RealityKit, SwiftUI with volumetric windows, immersive spaces. AR: 60fps. VR: 72fps minimum (90fps preferred) to prevent motion sickness.
AR sessions are interrupted by: incoming call, app backgrounding, tracking loss (poor lighting, featureless surfaces). Implement session interruption handlers: (a) On interruption: pause AR, save state/snapshot. (b) On resume: check tracking state, reload anchors if needed. (c) If tracking state is limited: check tracking reason (excessive motion, insufficient features, initializing), inform user with action prompt (move device to well-lit area with texture). (d) After prolonged interruption: reset session configuration and re-establish anchors.
ARKit: environment texturing (automatic or manual). Automatic generates environment probes from camera feed for realistic reflections. Manual: provide pre-baked environment maps for consistent lighting. ARCore: Environmental HDR mode with LightEstimationMode. Both: ambient light estimation (intensity + color temperature) for consistent virtual object rendering. Enable shadow casting from virtual objects onto real surfaces (ARKit directional shadows, ARCore Depth Occlusion).
Running AR features on unsupported devices crashes the app. Always check ARConfiguration.isSupported (iOS) or ArCoreApk.checkAvailability (Android) before activating AR.
Setting a fixed tracking configuration without handling runtime failures. Always monitor session state changes and provide user recovery prompts for limited tracking.
Shipping raw OBJ or FBX files in production bundles. These are 5-10x larger than compressed glTF or USDZ, causing long load times and memory pressure.
Adding ARAnchors in the update loop creates anchor overload, degrading tracking quality. Batch anchor operations and reuse existing anchors when possible.
Rendering full-detail models at every distance wastes GPU resources. Three LOD levels reduce draw calls by up to 60%.
AR sessions consume significant battery power (camera + sensors + rendering). Monitor battery level and reduce rendering quality when battery is low.
AR performance on simulators/emulators bears no relation to real-device performance. Test on real devices with varied lighting, surfaces, and motion conditions.
For shared AR experiences where multiple users see the same virtual content at the same location: iOS ARKit supports ARWorldMap for saving and sharing AR scene state. Host device exports the world map (ARSession.createWorldMap) and sends it to other devices via network (WebSocket, custom signaling server). Receiving devices import the world map (ARSession.run(with: worldMapConfiguration)) to align their AR coordinate system. Android ARCore provides Cloud Anchons via CloudAnchorManager. Host resolves anchors to the cloud (ArCoreResolveAndAttachAnchor), share the cloud anchor ID, guests resolve. For cross-platform: use ARCore Cloud Anchors from both platforms via the ARCore SDK. Multi-user features require: (1) real-time anchor state sync via network, (2) conflict resolution for simultaneous moves, (3) late-joining support (new users see existing content), (4) persistent content across sessions. Network sync should use WebRTC or a lightweight WebSocket protocol (not polling). Sync frequency: position updates at 20Hz, anchor events on change.
For AR experiences that need dynamic content without pre-bundled 3D assets, use procedural generation. Generate geometry at runtime using SCNGeometry (iOS SceneKit) or Mesh API (Unity/ARFoundation). Patterns: (a) terrain mesh from depth data — sample LiDAR/Depth API grid, generate vertex buffer, apply surface shader, (b) parametric objects — cylinders, spheres, extrusions with runtime parameter control, (c) text extrusion — convert string to 3D geometry using platform text-to-mesh APIs, (d) particle systems for ambient effects (sparkles, floating particles, smoke). Procedural content reduces bundle size significantly (no 3D assets to ship) at the cost of runtime compute. Use compute shaders (iOS Metal, Android Vulkan) for vertex-heavy generation to avoid GPU stalls.
Rendering complexity for AR scene?
├── Simple (static models, flat lighting) → SceneKit (iOS) / Sceneform (Android)
│ Built-in rendering, PBR materials, shadows — good for most product placement
├── Medium (animated models, environment lighting) → RealityKit (iOS) / ARCore + Filament (Android)
│ RealityKit: entity-component, animations, physics, environment probes
│ Filament: PBR renderer from Google, used by ARCore internally
├── Complex (custom shaders, post-processing, 50k+ polys)
│ → Metal (iOS) / Vulkan (Android) — full GPU control
│ Significant engineering cost, only for specialized rendering apps
└── Cross-platform 3D → Unity ARFoundation with Universal Render Pipeline
Write once, render on both platforms with URP feature sets
| Technique | Quality | Performance | Use Case |
|---|---|---|---|
| Ambient light estimation | Low | Free | Quick placeholder, matches scene brightness |
| Directional light from environment | Medium | Cheap | Product placement, single shadow cast |
| Environment texturing (automatic) | High | Moderate | Realistic reflections on glossy objects |
| Image-based lighting (IBL) | High | Moderate | Pre-baked HDR environment maps |
| Real-time shadow casting | High | Expensive | Ground contact shadows for virtual objects |
| LiDAR mesh with occlusion | Best | Heavy | Physical interaction, object occlusion behind real surfaces |
AR interaction type?
├── Tap to place → Hit-test against detected planes
│ Show ghost preview (transparent model at hit location)
│ Confirm on tap with haptic feedback and scale animation
├── Drag to move → Continuous hit-test while dragging
│ Constrain to detected plane surface (Y-axis locked)
│ Show position indicator, snap to grid if needed
├── Pinch to scale → Two-finger gesture with uniform or axis-locked scaling
│ Clamp scale: min 0.1x, max 10x — prevent impossibly small/large
├── Rotate → One or two-finger rotation gesture
│ Snap rotation by 45° increments for structured placement
├── Long-press context → Show action menu: delete, info, share
│ Use radial menu or bottom sheet for consistent UX
└── Gaze-based (hands-free AR glasses) → Eye tracking + dwell time
Look at object for 1.5s to select, blink to confirm
Track AR session quality metrics to diagnose user experience issues: (1) tracking state duration — time spent in limited/normal tracking, (2) average light estimation values — too dark reduces tracking quality, (3) anchor count over time — anchor leaks degrade performance, (4) session interruption frequency — correlates with app backgrounding and poor conditions, (5) average frame rate — drop below 30fps causes discomfort, (6) depth data availability — LiDAR vs. no LiDAR session path. Log these as analytics events prefixed with ar_. Monitor dashboards for: sessions with >50% time in limited tracking, anchor count >100 sustained, fps consistently below 30. Alerts trigger when any metric exceeds 2 standard deviations from baseline.
| Failure | Symptom | Mitigation |
|---|---|---|
| Device not supported | Crash/can't start | Check ARConfiguration.isSupported pre-launch |
| Tracking lost | Content floats | Visual indicator + recovery prompt |
| Camera permission denied | Black view | Graceful message, redirect to settings |
| Low light | Poor tracking | Torch prompt, adjust rendering |
| Memory pressure | App terminated | LOD streaming, texture budget <200MB |
| Battery drain | Phone hot | Reduce render quality when battery <20% |
| Condition | iOS Checklist | Android Checklist |
|---|---|---|
| Bright sunlight | Tracking stable, shadows | Same + auto-exposure |
| Dim interior | Low light prompt shows | Same |
| Featureless surface (white wall) | Tracking limited recovery | Same |
| Moving object detection | Works within 2s | Same |
| Multiple planes | 5+ concurrent | Same |
| App background + resume | Sessions resumes | Same |
| LiDAR / no LiDAR | Both test paths | Depth API availability |
ARConfiguration.isSupported / ArCoreApk.checkAvailabilityModel detail vs frame rate:
├── fps stable at 60, model count <10 → Add detail (4K textures, more polys)
├── fps 30-45, model count 10-50 → Enable LOD, GPU instancing, texture compression
├── fps <30, any count → Reduce render scale, disable shadows, limit draw distance
└── fps variable with battery → Implement battery-aware quality scaling
Battery <20%: reduce render scale to 0.7, disable HDR, lower draw distance
import ARKit
class ARViewController: UIViewController, ARSCNViewDelegate {
@IBOutlet var sceneView: ARSCNView!
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
guard ARWorldTrackingConfiguration.isSupported else {
showUnsupportedAlert(); return
}
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal, .vertical]
config.environmentTexturing = .automatic
config.frameSemantics = [.personSegmentationWithDepth]
config.isAutoFocusEnabled = true
sceneView.session.run(config, options: [.removeExistingAnchors, .resetTracking])
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
sceneView.session.pause()
}
func session(_ session: ARSession, didFailWithError error: Error) {
// App should present recovery UI, not just log
showRecoveryPrompt(message: error.localizedDescription)
}
func sessionWasInterrupted(_ session: ARSession) {
// Pause rendering, show overlay
}
func sessionInterruptionEnded(_ session: ARSession) {
// Reset tracking and reload anchors
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal, .vertical]
session.run(config, options: [.resetTracking, .removeExistingAnchors])
}
}
class ArActivity : AppCompatActivity() {
private lateinit var arFragment: ArFragment
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_ar)
arFragment = supportFragmentManager.findFragmentById(R.id.ar_fragment) as ArFragment
// Check ARCore availability
when (ArCoreApk.getInstance().checkAvailability(this)) {
ArCoreApk.Availability.UNKNOWN_CHECKING -> { /*wait*/ }
ArCoreApk.Availability.UNSUPPORTED_DEVICE_NOT_CAPABLE -> { showUnsupported() }
ArCoreApk.Availability.SUPPORTED_APK_TOO_OLD -> { /*update*/ }
ArCoreApk.Availability.SUPPORTED_INSTALLED -> { /*ready*/ }
}
val session = arFragment.arSceneView.session
val config = Config(session)
config.depthMode = Config.DepthMode.AUTOMATIC
config.lightEstimationMode = Config.LightEstimationMode.ENVIRONMENTAL_HDR
session.configure(config)
arFragment.setOnTapArPlaneListener { hitResult, _, _ ->
// Place anchor
val anchor = hitResult.createAnchor()
placeModel(anchor)
}
}
}
// Three.js / React Three Fiber in mobile AR
import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader';
import { DRACOLoader } from 'three/examples/jsm/loaders/DRACOLoader';
export async function loadARModel(url: string) {
const loader = new GLTFLoader();
const dracoLoader = new DRACOLoader();
dracoLoader.setDecoderPath('/draco-gltf/');
loader.setDRACOLoader(dracoLoader);
const gltf = await loader.loadAsync(url);
const model = gltf.scene;
// Apply LOD
model.traverse(child => {
if (child.isMesh) {
child.frustumCulled = true;
// Reduce shadow map if on mobile
child.receiveShadow = true;
child.castShadow = true;
}
});
return model;
}
ARWorldMap (iOS) or Cloud Anchors (Android) for session restoration.NSPhotoLibraryUsageDescription: AR apps that save photos/videos of AR scenes need photo library permission. Missing description = App Store rejection.App ready for AR release?
├── [ ] Device capability check at first launch (graceful fallback if unsupported)
├── [ ] Camera permission requested in context with rationale
├── [ ] All 3D models compressed (Draco/USDZ) and LOD levels generated
├── [ ] Session interruption handlers save/restore AR state
├── [ ] Performance profiled: 60fps sustained, <500MB total memory
├── [ ] Tested on: bright sunlight, dim interior, featureless surfaces
├── [ ] Tested on: LiDAR devices and non-LiDAR devices (both paths)
├── [ ] Anchor count never exceeds 50 per session (remove unused)
├── [ ] Battery-aware rendering: reduce quality at <20% battery
├── [ ] Analytics: AR session tracking events implemented
├── [ ] Haptic + visual feedback on every user action
├── [ ] Low-light detection triggers torch suggestion
├── [ ] Accessibility: voice-guided or manual placement fallback
├── [ ] App Store info.plist: ARKit usage description, camera usage
├── [ ] CI/CD: model size check, performance trace capture, real device test
mobile/universal/testing for AR testing strategy (real device, varied lighting, occlusion scenarios) mobile/universal/performance for profiling AR-specific performance metrics
` interface EventObserver { onEvent(event: T): Promise; }
class EventBus { private observers: Set<EventObserver> = new Set(); subscribe(observer: EventObserver): void { this.observers.add(observer); } unsubscribe(observer: EventObserver): void { this.observers.delete(observer); } async emit(event: T): Promise { const results = Array.from(this.observers).map(o => o.onEvent(event)); await Promise.allSettled(results); } } `
config: defaults: timeout: 30s retryCount: 3 overrides: production: timeout: 60s retryCount: 5 development: timeout: 300s retryCount: 1
| Metric | Threshold | Severity | Action |
|---|---|---|---|
| Error rate | > 1% over 5min | Critical | Page on-call |
| p99 latency | > 2s over 5min | Warning | Investigate |
| Throughput drop | > 50% over 1min | Critical | Check upstream |
| Queue depth | > 1000 over 1min | Warning | Scale consumers |
| Disk usage | > 85% | Warning | Clean or expand |
| Memory usage | > 90% heap | Critical | Restart or scale |
| Anti-Pattern | Symptom | Root Cause | Solution |
|---|---|---|---|
| Premature optimization | Complex code for no measured benefit | Guessing instead of profiling | Measure first, optimize based on data |
| Copy-paste reuse | Duplicate code across codebase | Lack of abstraction | Extract shared logic into libraries |
| Gold-plating | Features with no current requirement | Over-engineering | YAGNI — build what's needed now |
| Magical thinking | Assumptions without validation | Skipping error handling | Handle all failure modes explicitly |
Cache hierarchy: L1 (in-memory local) → L2 (distributed Redis/Memcached) → L3 (CDN/Edge). Cache invalidation: TTL-based (simple, stale), event-based (complex, fresh), write-through (consistent, higher write latency), write-behind (fast writes, eventual consistency).