iOS ARKit Face Tracking Vertices   18 comments

I’m currently working on a project that involves face tracking, and as a first prototype am using the built-in features in the ARKit library, Apple’s augmented reality API for iOS.

Using a device with a front-facing TrueDepth camera, an augmented reality session can be started to return quite a bit of information about faces it is tracking in an ARFaceAnchor object. One of these details is the face geometry itself, contained in the aptly named ARFaceGeometry object. For those who care about face shape landmarks, the 1220 individual vertices (points) can provide a wealth of information. However, there is little to no published information on which vertex corresponds to what point on the face.

While keeping in mind that this means these vertices could easily change in future versions of ARKit, I’ve taken the liberty of labeling points in case it is helpful for a future developer.

Update: A few people have asked how I generated this map. Details below.

Update: Since posting these pictures, a few people have asked how I generated this map and/or to share some code. I recommend starting with Yono Mittlefehldt’s AR Face Tracking Tutorial. That page goes through the basics of setting up your ARSession, ARFaceTrackingConfiguration, etc etc. It even goes so far as to attach SCNNode and child nodes that move with the face.

This is where Mittlefehldt stops short: he gives a few “magic numbers” corresponding to key points (eyes, nose, mouth, and hat) to attach emojis. To generate the maps above, I took this one short step further and generated a child node that puts the number itself on each node. If you’re reading through the tutorial, the difference is to add the following to your extension ViewController: ARSCNViewDelegate extension:

func renderer(_ renderer: SCNSceneRenderer,
              nodeFor anchor: ARAnchor) -> SCNNode? {
    guard let device = cameraView.device else {
        return nil
    }

    let faceGeometry = ARSCNFaceGeometry(device: device)
    let node = SCNNode(geometry: faceGeometry)
        
    for x in [1076, 1070, 1163, 1168, 1094, 358, 1108, 1102, 20, 661, 888, 822, 1047, 462, 376, 39] {
        let text = SCNText(string: "\(x)", extrusionDepth: 1)
        let txtnode = SCNNode(geometry: text)
        txtnode.scale = SCNVector3(x: 0.0002, y: 0.0002, z: 0.0002)
        txtnode.name = "\(x)"
        node.addChildNode(txtnode)
        txtnode.geometry?.firstMaterial?.fillMode = .fill
    }
                
    node.geometry?.firstMaterial?.fillMode = .lines

    return node
}

func renderer(_ renderer: SCNSceneRenderer,
              didUpdate node: SCNNode,
              for anchor: ARAnchor) {
   
    guard let faceAnchor = anchor as? ARFaceAnchor,
          let faceGeometry = node.geometry as? ARSCNFaceGeometry
        else {
            return
        }
    
    for x in 0..<1220 {
        let child = node.childNode(withName: "\(x)", recursively: false)
        child?.position = SCNVector3(faceAnchor.geometry.vertices[x])
    }
    
    faceGeometry.update(from: faceAnchor.geometry)
}

You can replace the list of vertex numbers with anything you’d like. The images above include all vertices, followed by every 4th vertex to clear out particularly “busy” areas.

Posted 18 Dec 2019 by John McManigle in Technical

18 responses to iOS ARKit Face Tracking Vertices

Subscribe to comments with RSS.

  1. hi could you explain the code to find out each vertex point?

  2. Hi, did you follow the emoji tutorial throughly? I tried to implement you suggestion code. I only got the mesh but with no vertiex shown.

    • John McManigle

      Hi Andrew,

      Wow; I am so sorry! When I updated this post to include the code to add vertices to the ARSCNView, I neglected to add the code that updates their position. That is now included. So sorry to make you struggle and wait for this!

      John

  3. thanks man , u help me alot ❤️

  4. This is a very interesting, thanks for sharing. Is it possible to get the distance between two vertices, say you wanted to get the distance of the path from the left bridge of the nose to the right?

  5. This is fantastic example. Thanks so much! Can we approximately or accurately measure the distance between left ear and right ear SCNNodes and vertical distance of the face of a user using this mesh?

  6. This is a great example but I do have a question. Is it possible to display only the mesh without the face? If yes, I would be grateful if you could help. Thank you.

  7. hey, do you have the wireframe with the vertexes. i need to know exact position of those 1220 vertex.

  8. This helps a lot. Amazing example.

  9. How can I calculate the “Bitragion frontal arc” using this nord?

  10. is it possible to track and anchor full side face profile? from the side?

    • John McManigle

      In my brief testing, the face model does not acquire from the side. That said, my testing was long ago, and Apple’s approach may have changed since. (And of course, you could train a different machine learning model to do something similar with face profiles. This is just the one Apple has bundled in iOS.)

  11. Hi John,
    Thanks for sharing these amazing examples. I have two questions.
    Do you know the reason why point 0 is determined there?
    Do teeth or gums have anything to do with determining this point?
    Thank you,
    Okan AYDIN

Leave a Reply

Your email address will not be published. Required fields are marked *