At our recent KEEN National Conference, I often got asked as I wore the new Apple Vision Pro, what are you looking at or what are you doing? We take for granted at a conference or meeting, people are looking at their smartphones or laptops, while also participating in the conference. Sometimes that’s what I was doing. When I spoke to a room full of Engineering Dean’s, I was using the Apple Vision Pro (AVP) to also look at my notes so I didn’t miss anything I wanted to say. Before I talk more about how it might transform engineering education, I’ll let you know what it was like using it. I wasn’t just using it to multitask. I wanted to ask the question: Can the Apple Vision Pro make me more productive than a desktop or laptop? To me the AVP obviously is a superior media consumption device because you can use it for augmented reality or virtual reality. But I wanted to see if it can be a superior “producer” device. Can I generate content and get work done more easily and faster? Let me share with you what it was like to wear it, and some surprises and delights I experienced.
Wearing the Apple Vision Pro on the Plane
I have since taken a few trips on a plane with the AVP. One problem I had was putting it in Travel Mode on the plane when it was dark. The AVP has all kinds of sensors to track your eye, limb, and body motion but normally it assumes you are stationary. When you are on a plane, it has to know that you are moving in order to accurately track you. The plane had taken off and once we reached cruising altitude, I took it out of my bag to use it. But I had to enter my password first. Since I wasn’t in Travel Mode, my eyes had a hard time looking at the numbers accurately. I got a pop up message saying that I needed to put it in tracking mode, but no matter how hard I tried to look at the “button” to put it in tracking mode, I couldn’t do it. Not sure if it was because the plane was dark but I tried to no avail. I also couldn’t accurately put in the passcode so it said try again in a minute. I propose a simple solution: have a button sequence that I can press manually to put me in Travel Mode. The next time I traveled , I set the Travel Mode while in the airport lobby but left the battery connected. Not the best way to do things , but I didn’t want to be on the plane again and not be able to use my AVP.
Using the AVP on a plane is delightful. I was able to connect to American Airlines and Delta‘s Wi-Fi easily and watch movies and TV or listen to music. I enjoy how you can turn on immersive environment so that you are surrounded by a starry sky or mountains. At the same time, as the flight attendant approached me, I could see them and request soda and chips. On one trip I forgot to bring my AirPods, but the sound from the AVP itself was good enough for me to hear. In a loud environment, the AirPods work much better.
Walking Around with AVP
Not advisable. In fact, there should probably be warning mode so that Apple is not liable for any accidents occurred. With that said, I couldn’t help but want to try to walk around with it. I quickly found that it’s not built for that. For example, I wanted to FaceTime my wife while I was walking in the airport. However, the FaceTime screen stood still and then went behind me as I traveled forward. I could hear her voice, but I couldn’t see her. I would like to be able to see her on the FaceTime as I walked much like I do when I’m with her walking in the airport. Currently, this is not possible.
On the other hand I was very surprised how I could actually walk with the AVP on. I use the custom optical inserts that match my prescriptions for my glasses. The cameras on the AVP allow me to navigate successfully in the airport and on the streets . I noticed that they even work well on a sunny day. Again, I definitely would strongly recommend not using them while driving.
Can I Do Real Work on It?
Yes. But make sure you learn how to turn on Voice Control in the Accessibility menu. You can put it in Command Mode or Dictation Mode (there is also a Spelling Mode but I haven’t got that to work yet for some reason). If you have ever seen the original Star Trek, you remember Captain Kirk could say, “Computer, tell me where the Klingons are” or something like that. You can do this on your Mac as well, but with the AVP, I can tell it to open or close different apps. Move the cursor around or select text with my voice. The better I get with the canned commands or customizing my own, this will make things much more productive as I manage the many screens that I can have open. Another way to visualize how it works is to think of the movie Minority Report where Tom Cruise would use his fingers to swipe different virtual compute screens from the air. Using the AVP is a lot like using that holographic computing device combined with voice commands. But you can use it discreetly by just controlling the cursor with your eyes and tapping your fingers.
If needed, you can use a Bluetooth keyboard like the Apple Magic Keyboard to type. I find this helpful as I’m learning how to use voice dictation and voice commands better. I was surprised to see how the AVP detects that you are using a Apple Magic Keyboard and then puts the text preview above the keys on the keyboard, as though it’s magically attached. Very cool how the AVP tracks and knows the locations of your body, equipment, windows, and other objects.
Your Eyes and Hands as a 3D Mouse
What you don’t initially realize is that the AVP’s phenomenal eye and hand tracking allows you to have a “mouse”, or point and click device, that you allows you to move your windows and objects in a 3D space. With a regular mouse you can move up and down and side to side. With its 3D “mouse”, you can move up, down, side to side, and back and forth. The eye tracking is very accurate. I can look at a precise place that I want to move the cursor and it will move it. I can “pinch” the cursor and move it around a text page very accurately. I did have trouble selecting a phrase of text without getting the last blank character. I ended up using the Command Mode “select phrase” command to accurately select a phrase of text. The cool thing is that the tracking also tracks your lips so that when I’m in a Zoom, FaceTime, or Google Meet, people can see my avatar, or persona, moving its lips in sync with mine. Since I’m not using a laptop with a camera to show me live, Apple has come up with a cool, somewhat creepy, way to make an avatar that tracks your smile, lips, eyes, and head movements so it looks like you. Instead of it being a DeepFake, it’s a RealFake (coined here first :-)) .
Dealing with Motion Sickness
I admit, as a youngster, I would get motion sick riding a bus. I used to get motion sick riding a plane and would take Dramamine. But I got used to it. After my initial demo of the AVP at the Apple Store by a phenomenal worker named Jalen, I noticed that my stomach was a little queezy after I took the AVP off. So I started using it 10 or 20 minutes at a time and started to get used to it. I now can go a “long” time using it. It’s actually more fun to use than a MacBook. I still think it’s advisable to take breaks and rest your eyes.
Transforming Engineering Education
I could write a series of blogs why I think the AVP can and possibly will transform engineering education. The main reason I believe it can is because it allows the student to experience augmented or virtual reality in such a realistic why, it will make learning experiences more engaging, realistic, exciting, and cheaper. With the new iPhone 15 and the AVP, you can record 3D videos that allow you to be immersed in the experience. I imagine that you could train a surgeon to conduct surgeons without having real patients. We could take our construction engineers to different environments and see how to construct buildings in Antarctica without having to travel there. The warning is that like any technology, people can use things for bad things too. For example, I wouldn’t want to see a video game where the player is stabbed realistically. It could potentially cause a heart attack. I hope that Apple and others make sure that all games and apps come with sufficient warnings and protections.
Opportunity, Equity and Access
If you are an engineering educator, I highly recommend it. I foresee new engineering buildings having labs that are equipped with tons of these. Same with museums. Like the iPhone in its inception, we will need the developers, content creators, and individual users to create the apps and content that will make it useful for education. I hope that public libraries will soon carry them so that they are accessible to anyone in the public no matter their socio-economic status. I was the only one I saw using one publicly in the airport or at a conference. If the AVP takes off, in the next year or two I’ll see hundreds of them. By the way, this blog post was created completely by a human using the Apple Vision Pro.
© 2024 Andrew B. Williams
About the Author: Andrew B. Williams is Dean of Engineering and Louis S. LeTellier Chair for The Citadel School of Engineering. He was recently named on of Business Insider’s Cloudverse 100 and humbly holds the designation of AWS Education Champion. He sits on the AWS Machine Learning Advisory Board and is a certified AWS Cloud Practitioner. He is proud to have recently received a Generative AI for Large Language Models certification from DeepLearning.AI and AWS. Andrew has also held positions at Spelman College, University of Kansas, University of Iowa, Marquette University, Apple, GE, and Allied Signal Aerospace Company. He is author of the book, Out of the Box: Building Robots, Transforming Lives.
In Search of Innovation 