Tesla AI Day 2022 全程回顾

14:33All right, welcome everybody.

14:36Let's give everyone a moment to get back in the audience.

14:41And welcome to Tesla, AID 2022.

14:57We've got some really exciting things to show you.

15:01I think you'll be pretty impressed.

15:03I do want to set some expectations with respect

15:06to our Optimus robot.

15:11As you know last year,

15:12it was just a person in a robot suit.

15:16But we've come a long way and it's I think we're you know compared to that it's gonna be very impressive

15:23and

15:26We're gonna talk about

15:28the advancements in AI for full self-driving as well as how they apply to

15:33More generally to real-world AI problems like a humanoid robot and even going beyond that

15:39I think there's some potential that or we're doing here at Tesla could

15:43make a meaningful contribution to AGI.

15:47And I think actually Tesla's a good

15:51entity to do it from a governance standpoint because

15:55we're a publicly traded company, we have one class of stock,

15:59and that means the public controls Tesla, and I think that's

16:03actually a good thing. So if I go crazy, you can fire me.

16:07This is important. Maybe I'm not crazy.

16:11So, yeah, so we're going to talk a lot about our progress in AI, an autopilot, as well as

16:20a progress with Dojo.

16:23And then we're going to bring the team out and do a long Q&A.

16:27So you can ask tough questions, whatever you'd like, existential questions, technical questions,

16:34but we want to have as much time for Q&A as possible.

16:39So let's see with that, because what,

16:43same thing?

16:44Hey guys, I'm Milana, working on a pilot and it is not bad.

16:48And I'm Lizzie, mechanical engineer on the project as well.

16:53Okay. So should we bring out the bot?

16:57Before we do that, we have one little bonus tip for the day.

17:02This is actually the first time we try this robot without any backup support,

17:07We're at Crane's mechanical mechanisms, no cables, nothing.

17:11Yeah, I want to do it with you guys tonight.

17:13That's the first time.

17:14You want to see?

17:15You ready?

17:16Let's go!

17:17The thing about Gossip Moves too.

18:12So this is essentially the same food self-driving home shooter

18:27that runs in your Tesla cars by the way.

18:30This is literally the first time the robot has operated

18:38without a Tesla response stage tonight.

19:12So the robot can actually do a lot more than we just showed you.

19:17We just didn't want it to fall on its face.

19:20So we'll show you some videos now of the robot doing a bunch of other things.

19:27Yeah, which are less risky.

19:29Yeah.

19:30We should close the screen, guys.

19:33Yeah.

19:34Yeah, we wanted to show it a little bit more what we've done over the past few months with

19:44the bot and just walking around and dancing on stage.

19:49Just humble beginnings, but you can see the autopilot

19:53neural networks running as it's just retrained

19:55for the bot directly on that new platform.

19:59That's why watering can.

20:00Yeah, when you see a rendered view, that's the robot.

20:03What's the world the robot sees?

20:06So it's very clearly identifying objects.

20:09Like this is the object they should pick up,

20:11picking it up.

20:12Yeah.

20:18We use the same process as we did for Autopiedout to connect data and train on networks that

20:23we didn't deploy on the robot.

20:25That's an example that illustrates the upper body a little bit more.

20:32Something that we'll try to nail down in a few months, over the next few months I would

20:35say to perfection.

20:38This is really an actual station in the Fremont factory as well that it's working at.

20:43Yep.

20:44So.

20:54And that's not the only thing we have to show today, right?

20:56Yeah, absolutely.

20:57So, what you saw was what we call Bumble-C,

21:02that's our sort of rough development robot,

21:06using semi off the shelf actuators.

21:10But we actually have gone a step further than that already.

21:13The team's done an incredible job.

21:15And we actually have an optimist bot

21:17with fully Tesla designed and built actuators,

21:22battery pack, control system, everything.

21:25It wasn't quite ready to walk, but I think it will walk in a few weeks.

21:31But we wanted to show you the robot, something that's actually fairly close to what we'll

21:36go into production and show you all the things it can do.

21:41So let's bring it up.

21:45Do it.

22:33So here you're seeing optimists with the degrees of freedom that we expect to have in

22:42optimist production unit one, which is the ability to move all the fingers independently, move

22:47the thumb have two degrees of freedom, so it has opposable thumbs and both left and right

22:54hand.

22:55So it's able to operate tools and do useful things.

22:58Our goal is to make a useful humanoid robot as quickly as possible.

23:04And we've also designed it using the same discipline that we use in designing the car,

23:10which is to say to design a form manufacturing,

23:14such that it's possible to make the robot

23:16at high volume at low cost with high reliability.

23:21So that's incredibly important.

23:23I mean, you've all seen very impressive humanoid robot

23:26demonstrations, and that's great.

23:29But what are they missing?

23:31They're missing a brain.

23:33They don't have the intelligence to navigate the world

23:36by themselves.

23:37And they're also very expensive and made in low volume.

23:42Whereas this is, the optimist is designed to be

23:45an extremely capable robot, but made in very high volume,

23:49probably ultimately millions of units.

23:52And it is expected to cost much less than a car.

23:56I'm just bringing you back to the right here.

23:59I would say probably less than $20,000,

24:01would be my guess.

24:09The potential for optimists is, I think, appreciated

24:13by very few people.

24:15Hey.

24:19As usual, Tesla demos are coming in hot.

24:21So.

24:22That's good.

24:24That's good.

24:26Yeah.

24:28The team has put in an incredible amount of work

24:33working seven days a week, running the 3 AM oil

24:38to get to the demonstration today.

24:41Super proud of what they've done.

24:42I've really done a great job.

24:43I'd just like to give a hand to the whole optimist team.

24:57So, you know, there's still a lot of work to be done

25:00to refine optimists and improve it.

25:04Obviously, this is just optimist version one.

25:07And that's really why we're holding this event,

25:10which is to convince some of the most talented people

25:13in the world, like you guys, to join Tesla

25:18and help make it a reality.

25:20and bring it to fruition at scale,

25:23such that it can help millions of people.

25:27And the potential, like I said, is really

25:31bubbles the mind because, you have to say,

25:33what is an economy?

25:35An economy is sort of productive entities

25:39times their productivity,

25:42capita times productivity per capita.

25:45But the point of which there is not a limitation

25:47on capital, it's not clear what an economy even means at that point. An economy becomes

25:53quasi-infinite. So what, you know, take into fruition in the publicly benign scenario,

26:06this means a future of abundance, a future where there is no poverty, where people, you

26:15You can have whatever you want in terms of products and services.

26:22It really is a fundamental transformation of civilization as we know it.

26:28Obviously, we want to make sure that transformation is a positive one and safe.

26:37But that's also why I think Tesla as an entity doing this being a single class of stock publicly

26:43traded owned by the public is very important and should not be overlooked. I think this is

26:51essential because then if the public doesn't like what Tesla is doing, the public can buy

26:56shares in Tesla and vote differently. This is a big deal. Like it's very important that

27:04I can't just do what I want. Sometimes people think that it's not true. So it's very important

27:15that the corporate entity that makes this happen

27:19is something that the public can properly influence.

27:25So I think the Tesla structure is ideal for that.

27:32And like I said, self-driving cars

27:35will certainly have a tremendous impact on the world.

27:40I think they will improve the productivity

27:43transport by at least half order of magnitude, perhaps an order of magnitude, perhaps more.

27:53Optimists, I think, has maybe a two order of magnitude, potential improvement in economic

28:03output.

28:06It's not clear what the limit actually even is.

28:11So, but we need to do this in the right way.

28:15We need to do it carefully and safely

28:18and ensure that the outcome is one that is beneficial

28:21to civilization and one that humanity wants.

28:27This is extremely important, obviously.

28:34And I hope you will consider joining Tesla

28:39to achieve those goals.

28:43It tells us we really care about doing the right thing here or aspired to the right thing

28:48and really not pay the road to hell with good intentions.

28:53I think the road to hell is mostly paid with bad intentions but every now and again there's

28:56a good intention in there.

28:58So we want to do the right thing.

29:00So consider joining us and helping make it happen.

29:05With that, we want to the next phase.

29:07All right.

29:08Thank you, Elon.

29:16All right.

29:17A couple of robots today.

29:18Let's do a quick timeline recap.

29:20So last year we unveiled the Tesla Bot concept,

29:23but a concept doesn't get us very far.

29:25We knew we needed a real development

29:27and integration platform to get real life learnings

29:29as quickly as possible.

29:31So that robot that came out and did the little routine

29:33for you guys, we had that within six months.

29:36Built working on software integration,

29:39hardware upgrades over the months since then.

29:42But in parallel, we've also been designing

29:44the next generation, this one over here.

29:47So this guy is rooted in the the foundation of sort of the vehicle design process, you know

29:52We're leveraging all of those learnings that we already have

29:56Obviously, there's a lot that's changed since last year, but there's a few things that are still the same you'll notice

30:01We still have this really detailed focus on the true human form. We think that matters for a few reasons

30:07But it's fun. We spend a lot of time thinking about how amazing the human body is

30:11We have this incredible range of motion typically really amazing strength

30:17A fun exercise is if you put your fingertip on the chair in front of you, you'll notice

30:21that there's a huge range of motion that you have in your shoulder and your elbow, for

30:26example.

30:27Without moving your fingertip, you can move those joints all over the place.

30:31But the robot, you know, its main function is to do real useful work, and it maybe doesn't

30:36necessarily need all of those degrees of freedom right away.

30:40So we have stripped it down to a minimum sort of 28 fundamental degrees of freedom, and then

30:44And then, of course, our hands in addition to that.

30:47Humans are also pretty efficient at some things,

30:50and not so efficient in other times.

30:52So for example, we can eat a small amount of food

30:54to sustain ourselves for several hours.

30:56That's great.

30:58But when we're just kind of sitting around, no offense,

31:00but we're kind of inefficient.

31:02We're just sort of burning energy.

31:04So on the robot platform, what we're going to do

31:06is we're going to minimize that idle power consumption,

31:09drop it as low as possible.

31:11And that way, we can just flip a switch,

31:12and immediately the robot turns into something

31:15that does useful work.

31:18So let's talk about this latest generation

31:19in some detail, shall we?

31:21So on the screen here, you'll see in orange

31:23our actuators, which we'll get to in a little bit,

31:26and in blue our electrical system.

31:29So now that we have our sort of human-based research,

31:33and we have our first development platform,

31:35we have both research and execution to draw from

31:37for this design.

31:39Again, we're using that vehicle design foundation,

31:41So we're taking it from concept through design and analysis,

31:45and then build and validation.

31:48Along the way, we're going to optimize

31:50for things like cost and efficiency,

31:52because those are critical metrics

31:53to take this product to scale eventually.

31:56How are we going to do that?

31:58Well, we're going to reduce our part count

32:00and our power consumption of every element possible.

32:03We're going to do things like reduce the sensing

32:05and the wiring at our extremities.

32:07You can imagine a lot of mass in your hands and feet

32:09is going to be quite difficult and power

32:11consumptive to move around.

32:14And we're going to centralize both our power distribution

32:16and our compute to the physical center of the platform.

32:21So in the middle of our torso, actually it is the torso.

32:24We have our battery pack.

32:25This is sized at 2.3 kilowatt hours,

32:28which is perfect for about a full day's worth of work.

32:31What's really unique about this battery pack

32:33is it has all of the battery electronics integrated

32:36into a single PCB within the pack.

32:39So that means everything from sensing to fusing,

32:42charge management, and power distribution

32:45is all in one place.

32:48We're also leveraging both our vehicle products

32:52and our energy products to roll all of those key features

32:56into this battery.

32:57So that's streamlined manufacturing,

32:59really efficient and simple cooling methods,

33:02battery management, and also safety.

33:05And of course, we can leverage Tesla's existing

33:07infrastructure and supply chain to make it.

33:11So going on to sort of our brain, it's not in the head,

33:14but it's pretty close.

33:16Also in our torso, we have our central computer.

33:19So as you know, Tesla already ships full self-driving

33:22computers in every vehicle we produce.

33:25We want to leverage both the autopilot hardware and the

33:28software for the humanoid platform, but because it's

33:31different in requirements and in form factor, we're going

33:34to change a few things first.

33:36So we still are going to do everything that a human

33:39brain does, processing vision data, making split-sescend decisions based on multiple sensory

33:45inputs and also communications.

33:48So to support communications, it's equipped with wireless connectivity as well as audio

33:52support.

33:53And then it also has hardware level security features which are important to protect both

33:57the robot and the people around the robot.

34:02So now that we have our sort of core, we're going to need some limbs on the sky.

34:07And we'd love to show you a little bit about our actuators and our fully functional hands

34:11as well.

34:12But before we do that, I'd like to introduce Malcolm, who's going to speak a little bit about

34:16our structural foundation for the robot.

34:23Thank you, Rudy.

34:25Tesla, up to capabilities, finalised highly complex systems.

34:33It doesn't get much more complex than a crash.

34:36You can see here a simulated crash in bottle three, superimposed on top of the actual physical

34:40crash.

34:41It's actually incredible how accurate it is.

34:45give you an idea of the complexity of this model, it includes every not-bought-in-washer,

34:50every spot world, and it has 35 million degrees of freedom.

34:53Quite amazing.

34:54And it's true to say that if we didn't have models like this, we wouldn't be able to

34:58make the safest cars in the world.

35:01So can we utilise our capabilities and methods from the automotive side to influence a robot?

35:09Well, we can make a model, and since we had crashed software, we used the same software

35:14here, we can make it fall down.

35:16The purpose of this is to make sure that if it falls down, ideally it doesn't, but it's

35:21superficial damage.

35:23We don't want it to, for example, break its gearbox and its arms.

35:26That's equivalent of a dislocated shoulder of a robot, difficult and expensive to fix.

35:30So we wanted to dust itself off, get on with the job that's been given.

35:35We've also taken the same model and we can drive the actuators using the input from a

35:43previously solved model, bringing it to life.

35:47So this is producing the motions for the tasks we want the robot to do.

35:51These tasks are picking up boxes, turning, squatting, and walking upstairs.

35:55Whatever the set of tasks are, we can play for the model.

35:59This is showing just simple walking.

36:00We can create the stresses in all the components that helps us to optimize the components.

36:08These are not dancing robots.

36:10These are actually the modal behavior, the first five modes of the robot.

36:14And typically, when people make robots, they make sure the first mode is up around

36:19top single figures up towards 10 hertz.

36:22The reason to do this is to make the controls of walking easier.

36:25It's very difficult to walk if you can't guarantee where you're foots wobbling around.

36:30That's okay if you make one robot. We want to make thousands, maybe millions.

36:34We haven't got the luxury of making from carbon fiber to titanium.

36:37We want to make them unplastic. Things are not quite as stiff.

36:41So we can't have these high targets. I'll call them dumb targets.

36:46We've got to make them work at lower targets.

36:49So is that good to work?

36:50Well, if you think about it, sorry about this,

36:53but we're just bags of soggy, jelly and bone-throwning.

36:57We're not high-frequency.

36:58If I stun on my leg, I don't vibrate at 10 hertz.

37:03We people operate at a lot of frequency,

37:05so we know the robot actually can.

37:06It just makes controls harder.

37:08So we take the information from this,

37:10the modal data and the stiffness,

37:12feed the into the control system,

37:14That allows it to walk.

37:18Just changing the tax likely, looking at the knee.

37:21We can take some inspiration from biology.

37:24And we can look to see what the mechanical advantage of the knee

37:26is.

37:27It turns out it actually represents quite

37:29similar to four-bar link.

37:31And that's quite non-linear.

37:33That's not surprising really, because if you think when

37:35you bend your leg down, the torque on your knee is much more

37:38when it's bent, and it is when it's straight.

37:41So you'd expect a non-linear function.

37:43And in fact, the biology is non-linear.

37:46This matches it quite accurately.

37:50So that's a representation.

37:51The forebarling is obviously not physically forebarling,

37:53so that's said the characteristic is similar.

37:55But me bending down, that's not very scientific.

37:58Let's be a bit more scientific.

38:00We've played all the tasks through this graph,

38:04and this is showing picketings up, walking, squatting,

38:07the tasks I said we did on the stress.

38:09And that's the talk, seen at the knee,

38:14against the knee bend on the horizontal axis.

38:16This is showing the requirement for the knee to do all these tasks.

38:20And then put a curve through it, surfing over the top of the piece.

38:23And that's saying this is what's required to make the robot do these tasks.

38:32So if we look at the four-bar link, that's actually the green curve.

38:36And it's saying that the non-linearity of the four-bar link

38:38is actually linearized the characteristic of the force.

38:42What that really says is that's lower the force.

38:44That's what makes the actuator have the lowest possible force, which

38:47is the most efficient.

38:48We want to burn energy up slowly.

38:50What's the blue curve?

38:51The blue curve is actually if we didn't have a four bar link,

38:54we just had an arm sticking out of my leg here

38:57with an actuator on it, a simple two bar link.

39:00That's the best we could do with a simple two bar link.

39:03And it shows that that would create a much more

39:05force in the actuator, which would not be efficient.

39:10So what's that look like in practice?

39:12Well, as you'll see, it's very tightly packaged in the knee.

39:16You'll see it go transparent on the second.

39:18You'll see the four bar link there.

39:19It's operating on the actuator.

39:21This is determined the force and the displacements on the actuator.

39:26I now pass you over to Konstantinos to tell you a lot more detail about how these actuators

39:29have made and designed optimised.

39:32Thank you.

39:34Thank you, Michael.

39:41So I would like to talk to you about the design process and the actuator portfolio in our robot.

39:51So there are many similarities between a car and the robot when it comes to powertrain design.

39:55The most important thing that matters here is energy,

39:59mass, and cost.

40:01We are carrying over most of our designing experience

40:04from the car to the robot.

40:08So in the particular case, you see a car with two drive units.

40:13And the drive units are used in order

40:16to accelerate the car 0 to 60 miles per hour time

40:19or drive a city drive side.

40:22while the robot that has 28 actuators,

40:28it's not obvious what are the task at actuator level.

40:32So we have tasks that are higher level

40:34like walking or climbing stairs

40:37or carrying a heavy object

40:39which need to be translated into joint specs.

40:44Therefore we use our model that generates

40:48the torque speed trajectories for our joints

40:52which subsequently is going to be fed in our optimization model

40:56to run through the optimization process.

41:01This is one of the scenarios that the robot is capable of doing,

41:05which is turning and walking.

41:08So when we have this torque speed trajectory,

41:10we lay it over in a efficiency map of an actuator.

41:14And we are able, along the trajectory,

41:17to generate the power consumption and the energy,

41:22cumulative energy for the task versus time.

41:26So this allows us to define the system cost for the particular

41:29actuator and put a simple point into the cloud.

41:33Then we do this for hundreds of thousands of actuators by

41:36solving in our cluster.

41:38And the red line denotes the poreto front, which is the

41:41preferred area where we will look for optimal.

41:44So the X denotes the preferred actuator design we have picked

41:49for this particular joint.

41:50So now we need to do this for every joint.

41:53We have 28 joints to optimize.

41:55And we parse our cloud again for every joint spec.

42:01And the red axis this time denotes the bespoke actuator designs

42:06for every joint.

42:07The problem here is that we have too many unique actuator

42:11designs.

42:12And even if we take advantage of the symmetry,

42:14still there are too many.

42:16In order to make something much manufacturable,

42:19We need to be able to reduce the amount of unique actuator

42:22designs.

42:23Therefore, we run something called commonality study,

42:26which we parse our cloud again, looking this time

42:30for actuators that simultaneously meet the joint

42:33performance requirements for more than one

42:35joint at the same time.

42:37So the resulting portfolio is six actuators,

42:39and they show in a color map at the middle figure.

42:44And the actuators can be also viewed in this slide.

42:48We have three rotary and three linear actuators,

42:52all of which have a great output force or torque per mass.

42:59The rotary actuator in particular

43:01has a mechanical clots integrated on the high speed side,

43:05angular contact ball bearing, and on the high speed side,

43:08and on the low speed side, a cross roller bearing,

43:12and the gear train is a strain wave gear.

43:17there are three integrated sensors here

43:20and the bespoke permanent magnet machine.

43:25The linear actuator, I'm sorry,

43:33the linear actuator has planetary rollers

43:36and an inverted planetary screw as a gear train

43:40which allows efficiency and compaction and durability.

43:44So in order to demonstrate the force capability

43:47of our linear actuators, we have set up an experiment

43:52in order to test it under its limits.

43:58And I will let you enjoy the video.

44:07So our actuator is able to lift a half-ton, nine-foot concert

44:22grand piano.

44:26And this is a requirement.

44:32It's not something nice to have, because our muscles

44:36can do the same when they are directly driven.

44:39When they are directly driven, our quadriceps muscles

44:42can do the same thing.

44:43It's just that the knee is an up-gearing,

44:46linked system that converts the force

44:49into velocity at the end effect of our heels

44:52for purposes of giving to the human body agility.

44:56So this is one of the main things

44:58that are amazing about the human body.

45:00And I'm concluding my part at this point,

45:02and I would like to welcome my colleague Mike,

45:04who's going to talk to you about hand design.

45:07Thank you very much.

45:10Thanks, Tazitinas.

45:13So we just saw how powerful a human

45:15and a humanoid actuator can be.

45:18However, humans are also incredibly dexterous.

45:23The human hand has the ability to move at 300 degrees per second.

45:27There's tens of thousands of tactile sensors.

45:31It has the ability to grasp and manipulate almost every object

45:34in our daily lives.

45:36For our robotic hand design, we were inspired by biology.

45:40We have five fingers and opposable thumb.

45:43Our fingers are driven by metallic tendons

45:45that are both flexible and strong.

45:48We have the ability to complete wide-adventure power

45:50grasps while also being optimized for precision

45:53gripping of small, thin, and delicate objects.

45:57So why a human-like robotic hand?

46:00Well, the main reason that our factories and the world

46:02around us is designed to be ergonomic.

46:05So what that means is that it ensures

46:07that objects in our factory are graspable,

46:09but it also ensures that new objects that we may have never

46:12seen before can be grasped by the human hand.

46:15and by our robotic hand as well.

46:18The converse there is pretty interesting

46:19because it's saying that these objects are designed

46:21to our hand instead of having to make changes to our hand

46:24to accompany a new object.

46:27Some basic stats about our hand is that it has six actuators

46:29and 11 degrees of freedom.

46:31It has an in hand controller which drives the fingers

46:35and receives sensor feedback.

46:37Sensor feedback is really important to learn a little bit

46:39more about the objects that we're grasping

46:41and also for proprioception.

46:43And that's the ability for us to recognize

46:45where our hand is in space.

46:48One of the important aspects of our hand

46:49is that it's adaptive.

46:51This adaptability is involved essentially

46:53as complex mechanisms that allow the hand

46:55to adapt the objects that's being grasped.

46:58Another important part is that we have a non-back driveable

47:00finger drive.

47:01This clutching mechanism allows us to hold and transport

47:04objects without having to turn on the hand motors.

47:08You just heard how we went about designing the Tesla Bot

47:11hardware.

47:12Now we'll hand it off to Milan and our autonomy team

47:14to bring this robot to life.

47:18Thanks, Mikey.

47:24All right, so all those cool things

47:27we've shown earlier in the video

47:29were possible just in a matter of a few months,

47:32thanks to the amazing work that we've done

47:34on autopilot over the past few years.

47:36Most of those components bought it quite easily

47:39over to the bots environment.

47:40If you think about it, we're just moving

47:42from a robot on wheels to a robot on legs.

47:45So some of the components are pretty similar,

47:47and some of them require more heavy lifting.

47:50So for example, our computer vision neural networks were pointed directly from autopilot

47:57to the bots situation.

47:59It's exactly the same occupancy network that we'll talk into a little bit more details

48:03later with the autopilot team that is now running on the bot here in this video.

48:08The only thing that changed really is the training data that we had to recollect.

48:14We also trying to find ways to improve those occupancy networks using work made on your

48:19radiance fields to get really great volumetric rendering of the bots environments.

48:25For example here, some machinery that the bot might have to interact with.

48:32Another interesting problem to think about is, in indoor environments mostly with that

48:37sense of GPS signal, how do you get the bot to navigate to its destination?

48:42Say for instance, to find its nearest charging station.

48:45So we've been training more neural networks to identify how frequency features, key points

48:51within the bots camera streams and track them across frames over time as the bot navigates

48:57with its environment.

48:59And we're using those points to get a better estimate of the bots pose and trajectory within

49:04its environment as it's walking.

49:09We also did quite some work on the simulation side and this is literally the autopilot simulator

49:14to which we've integrated the robot's locomotion code.

49:18And this is a video of the motion control code running in the opilot simulator showing

49:23the evolution of the robot's work over time.

49:27And so as you can see, we started quite slowly in April

49:29and started accelerating as we unlock more joints

49:31and deeper, more advanced techniques

49:33like arms balancing over the past few months.

49:37And so local motion is specifically one component

49:39that's very different as we're moving from the car

49:42to the bots environment.

49:43And so I think it warrants a little bit more depth,

49:45and I'd like my colleagues to start talking about this now.

49:57Thank you, Milan.

49:58Hi, everyone.

49:59I'm Felix.

50:00I'm a robotics engineer on the project.

50:01and I'm gonna talk about walking.

50:04Walking seems easy, right?

50:06People do it every day.

50:08You don't even have to think about it.

50:10But there are some aspects of walking

50:12which are challenging from engineering perspective.

50:15For example, physical self-awareness,

50:19that means having a good representation of yourself.

50:22What is the length of your limbs?

50:23What is the mass of your limbs?

50:25What is the size of your feet?

50:26All that matters.

50:28Also, having an energy-efficient gate,

50:31You can imagine there's different styles of walking

50:33and all of them are equally efficient.

50:37Most important, keep balanced, don't fall.

50:41And of course also coordinate the motion

50:43of all of your limbs together.

50:45So now humans do all of this naturally,

50:48but as engineers or roboticists,

50:50we have to think about these problems.

50:52And the following gonna show you how we address them

50:54in our locomotion planning and control stack.

50:57So we start with locomotion planning

51:00and our representation of the bot

51:01that means a model of the robots' kinematics,

51:04dynamics, and the contact properties.

51:07And using that model and the desired path for the bots,

51:11our locomotion planner generates reference

51:13trajectories for the entire system.

51:16This means feasible trajectories

51:18with respect to the assumptions of our model.

51:22The planner currently works in three stages.

51:24It starts planning footsteps

51:26and ends with the entire motion for the system.

51:29and let's dive a little deeper in how this works.

51:32So in this video, we see footsteps being planned

51:35over a planning horizon following the desired path.

51:39And we start from this and add them for trajectories

51:43that connect these footsteps using toe-off and heel strike

51:46just as humans do.

51:49And this gives us the largest stride and less knee bend

51:52for high efficiency of the system.

51:55The last stage is then finding a sense of mass trajectory,

51:59which gives us a dynamically feasible motion

52:01of the entire system to keep balance.

52:04As we all know, plans are good,

52:06but we also have to realize them in reality.

52:09Let's say how, see how we can do this.

52:20Thank you Felix.

52:21Hello everyone, my name is Anand.

52:23And I'm gonna talk to you about controls.

52:26So let's take the motion plan that Felix just talked about

52:30and put it in the real world on a real robot.

52:33Let's see what happens.

52:37It takes a couple steps and falls down.

52:41Well, that's a little disappointing.

52:43But we are missing a few key pieces here,

52:45which will make it walk.

52:48Now, as Felix mentioned, the motion planner

52:51is using an idealized version of itself

52:54and a version of reality around it.

52:57This is not exactly correct.

52:59It also expresses its intention through trajectories

53:04and wrenches, wrenches of forces and torques,

53:07that it wants to exert on the world to local mode.

53:12Reality is way more complex than any similar model.

53:16Also, the robot is not simplified.

53:18It's got vibrations and modes, compliance, sensor noise,

53:23and on and on and on.

53:26So what does that do to the real world when

53:28you put the bot in the real world?

53:30Well, the unexpected forces cause un-model dynamics,

53:34which essentially the planet doesn't know about,

53:36and that causes destabilization,

53:38especially for a system that is dynamically stable,

53:42like biped locomotion.

53:44So what can we do about it?

53:46Well, we measure reality.

53:48We use sensors and our understanding of the world

53:51to do state estimation.

53:53And here you can see the attitude and pelvis pose,

53:56which is essentially the vestibular system in a human,

54:00along with the center of mass trajectory being tracked.

54:03when the robot's walking in the office environment.

54:07Now we have all the pieces we need in order to close the loop.

54:11So we use our better bot model.

54:14We use the understanding of reality

54:16that we've gained through state estimation.

54:18And we compare what we want versus what

54:21we expect the reality is doing to us

54:24in order to add corrections to the behavior of the robot.

54:30Here, the robot certainly doesn't appreciate being poked,

54:34but it has an admirable job of staying upright.

54:38The final point here is a robot that walks is not enough.

54:43We need it to use its hands and arms to be useful.

54:48Let's talk about manipulation.

55:00Hi, everyone.

55:01My name is Eric, robotics engineer on Tesla Bot.

55:04And I want to talk about how we've made the robot manipulate

55:08things in the real world.

55:10We wanted to manipulate objects while looking as natural as possible and also get there quickly.

55:17So what we've done is we've broken this process down into two steps.

55:20First, is generating a library of natural motion references or we could call them demonstrations

55:26and then we've adapted these motion references online to the current real world situation.

55:33So let's say we have a human demonstration of picking up an object.

55:36We can get a motion capture of that demonstration, which is visualized right here as a bunch of

55:42keyframes representing the location of the hands, the elbows, the torso.

55:46We can map that to the robot using inverse kinematics.

55:49And if we collect a lot of these, now we have a library that we can work with.

55:55But a single demonstration is not generalizable to the variation in the real world.

56:01For instance, this would only work for a box in a very particular location.

56:05So what we've also done is run these reference trajectories

56:09through a trajectory optimization program,

56:12which solves for where the hand should be,

56:15how the robot should balance when it needs to adapt the motion

56:20to the real world.

56:21So for instance, if the box is in this location,

56:25then our optimizer will create this trajectory instead.

56:31Next, Mulan's going to talk about what's next

56:34for the optimist, Tesla and Ly.

56:36Thanks, Eric.

56:45Right, so hopefully by now you guys got a good idea of what

56:47we've been up to over the past few months.

56:50We started doing something that's usable,

56:52but it's far from being useful.

56:54They're still a long and exciting road ahead of us.

56:57I think the first thing within the next few weeks

56:59is to get up to us at least apart with BumbleC, the other

57:03bug prototype you saw earlier, and probably beyond.

57:07We are also going to start focusing on the real use case

57:10at one of our factories, and really going to try to nail this down.

57:15And I run out all the elements needed to deploy this product

57:19in the real world.

57:20I was mentioning earlier, indoor navigation, graceful

57:24for management, or even servicing.

57:27All components needed to scale this product up.

57:31But I don't know about you, but after seeing what we've shown

57:34tonight, I'm pretty sure we can get this down

57:36within an extra month or years.

57:38and make this project a real-life and change the entire economy.

57:42So I would like to thank the entire optimist team

57:46for their whole work over the past few months.

57:47I think it's pretty amazing.

57:48All of this was done in barely six or eight months.

57:51Thank you very much.

58:07Hey, everyone.

58:10Hi, I'm Ashok.

58:11I lead the autopilot team alongside Milan.

58:15God, it's coming so hard to top that optimist section.

58:19He'll try nonetheless.

58:21Anyway, every Tesla that has been built over the last several years, we think as the hardware

58:28to make the car drive itself.

58:31We have been working on the software to add higher and higher levels of autonomy.

58:37This time around last year, we had roughly 2000 cars driving our FSD beta software.

58:43Since then, we have significantly improved the software's robustness and capability that

58:48We have now shipped it to 160,000 customers as of today.

59:00This has not come for free.

59:01It came from the sweat and blood of the engineering team

59:04over the last one year.

59:06For example, between 75,000 neural network models

59:10just last one year.

59:11That's roughly a model every eight minutes coming out

59:15of the team.

59:16And then we evaluate them on our large clusters.

59:19And then we shipped 281 of those models that actually

59:22improve the performance of the car.

59:24And this pace of innovation is happening throughout the stack.

59:28The planning software, the infrastructure, the tools,

59:31even hiring, everything is progressing to the next level.

59:37The FSD beta software is quite capable of driving the car.

59:42You should be able to navigate from parking lot to parking lot,

59:44handling CVC driving, stopping for traffic lights and stop signs,

59:49negotiating with objects at intersections, making turns, and so on.

59:56All of this comes from the camera streams

59:59that go through neural networks that run on the car itself.

1:00:02It's not coming back to the server or anything.

1:00:03It runs on the car and produces all the outputs

1:00:06to form the world model or on the car.

1:00:08And the planning software drives the car based on that.

1:00:13Today, we'll go into a lot of the components

1:00:15that make up the system.

1:00:17The occupancy network acts as the base geometry layer

1:00:21of the system.

1:00:22This is a multi-camera video neural network

1:00:27that from the images predicts the full physical occupancy

1:00:31of the world around the robot.

1:00:34So anything that's physically present, trees, walls, buildings,

1:00:37cars, walls, what have you, it predicts,

1:00:39if it's physically present, it predicts them,

1:00:41along with their future motion.

1:00:46On top of this base level of geometry,

1:00:49we have more semantic layers.

1:00:51In order to navigate the roadways,

1:00:52we need the lanes, of course.

1:00:55But then the roadways have lots of different lanes,

1:00:57and they connect in all kinds of ways.

1:00:59So it's actually a really difficult problem

1:01:01for typical computer vision techniques

1:01:03to predict the set of lens and the connectivity.

1:01:06So we reached all the way into language technologies

1:01:09and then pull the state of the art from other domains

1:01:11and not just computer vision to make this task possible.

1:01:16For vehicles, we need their full kinematic state

1:01:18to control for them.

1:01:21All of this directly comes from neural networks.

1:01:24Video streams, raw video streams,

1:01:26come into the networks,

1:01:28goes through a lot of processing, and then outputs

1:01:30the full kinematic state, the positions, velocities,

1:01:33acceleration, jerk, all of that.

1:01:35Directly comes out on networks with minimal post-processing.

1:01:38That's really fascinating to me because how is this even

1:01:41possible?

1:01:41What world do we live in that this magic is possible,

1:01:44that these networks predicts fourth derivatives of these

1:01:47positions, and people thought we couldn't even detect these

1:01:50objects.

1:01:53My opinion is that it did not come for free.

1:01:55It required tons of data.

1:01:57So we had to bit sophisticated auto labeling systems that

1:02:01churn through raw sensor data, run a ton of off-line compute

1:02:05on the servers.

1:02:05It can take a few hours, run expensive neural networks,

1:02:09distill the information into labels that

1:02:11train our in-car neural networks.

1:02:15On top of this, we also use our simulation system

1:02:18to synthetically create images.

1:02:20And since it's a simulation, we trivially

1:02:22have all the labels.

1:02:26All of this goes through a well-oiled data engine pipeline,

1:02:29where we first train a baseline model with some data,

1:02:33ship it to the car, see what the failures are.

1:02:36And once we know the failures, we

1:02:38mind the fleet for the KXS where it fails,

1:02:41provide the correct labels, and add the data

1:02:43to the training set.

1:02:45This process systematically fixes the issues.

1:02:48And we do this for every task that runs in the car.

1:02:52Yeah, and to train these new, massive neural networks,

1:02:55This year we expanded our training infrastructure by roughly 40 to 50 percent, so that sits

1:03:00us at about 14,000 GPUs today across multiple training clusters in the United States.

1:03:07We also worked on our AI compiler, which now supports new operations needed by those neural

1:03:12networks, and mapped them to the best of our underlying hardware resources.

1:03:17And our inference engine today is capable of distributing the execution of a single neural

1:03:22network across two independent system on chips.

1:03:26Essentially, two independent computers

1:03:28interconnected within the same full self-driving computer.

1:03:32And to make this possible, we had to keep a tight control on

1:03:35the end to end latency of this new system.

1:03:37So we deployed more advanced scheduling code across the

1:03:39full FOSI platform.

1:03:43All of these neural networks running in the car, together

1:03:46produce the vector space, which is again the model of the

1:03:48world around the robot or the car.

1:03:50Then the planning system operates on top of this, coming up

1:03:53with trajectories that avoid collisions or smooth,

1:03:56make progress towards the destination,

1:03:58using a combination of model-based optimization,

1:04:00plus neural network that helps optimize it to be really fast.

1:04:07Today, we are really excited to present progress

1:04:09on all of these areas.

1:04:11We have the engineering leads standing by to come in

1:04:13and explain these various blocks,

1:04:14and these power not just the car,

1:04:17but the same components also around the Optimus robot

1:04:19that Milan showed earlier.

1:04:22With that, I welcome Puddle to start talking

1:04:24the planning section. Hi, I'm Pearl Jain. Let's use this intersection scenario to dive straight into

1:04:39how we do the planning and decision making in autopilot. So we are approaching this intersection

1:04:44from a side street and we have to yield to all the crossing vehicles. Right with us, we are

1:04:50about to enter the intersection. The pedestrian on the other side of the intersection decides to

1:04:55cross the road without a crosswalk. Now we need to yield to this pedestrian, yield to the vehicles

1:05:01from the right and also understand the relation between the pedestrian and the vehicle on the

1:05:06other side of the intersection.

1:05:08So a lot of these intra-object dependencies that we need to resolve in a quick glance.

1:05:15And humans are really good at this.

1:05:17We look at a scene, understand all the possible interactions, evaluate the most promising

1:05:22ones and generally end up choosing a reasonable one.

1:05:25So let's look at a few of these interactions that autopilot system evaluated.

1:05:30We could have gone in front of this pedestrian with a very aggressive launch and lateral

1:05:36profile.

1:05:37Now obviously we are being a jerk to the pedestrian and we would spook the pedestrian and his

1:05:40cute foot.

1:05:42We could have moved forward slowly, shot for a gap between the pedestrian and the vehicle

1:05:47from the right.

1:05:48Again, we are being a jerk to the vehicle coming from the right, but you should not

1:05:52outright reject this interaction in case this is only safe interaction available.

1:05:57Lastly, the interaction we ended up choosing.

1:06:02Stay slow initially, find the reasonable gap, and then finish the maneuver after all the

1:06:06agents pass.

1:06:09Now, evaluation of all of these interactions is not trivial, especially when you care about

1:06:14modeling the higher-order derivatives for other agents.

1:06:18For example, what is the longitudinal jerk required by the vehicle coming from the ride

1:06:22when you assert in front of it?

1:06:25relying purely on collision checks with marginal predictions

1:06:28will only get you so far because you

1:06:30will miss out on a lot of valid interactions.

1:06:33This basically boils down to solving the multi-agent joint

1:06:36trajectory planning problem over the trajectories of ego

1:06:39and all the other agents.

1:06:42Now, how much ever you optimize, there's

1:06:44going to be a limit to how fast you can run this optimization

1:06:46problem.

1:06:47It will be close to order of 10 milliseconds,

1:06:49even after a lot of incremental approximation.

1:06:54Now, for a typical crowded, un-predictable lift,

1:06:57say you have more than 20 objects, each object having multiple different future modes, the

1:07:02number of relevant interaction combinations will blow up. The planner needs to make a

1:07:09decision every 50 milliseconds. So how do we solve this in real time? We rely on a framework

1:07:16what we call as interaction search, which is basically a paralyzed research over a bunch

1:07:20of manoeuvre trajectories. The state space here corresponds to the kinematic state of

1:07:26ego, the kinematic state of other agents, the nominal future multi-modal predictions, and

1:07:32all the static entities in the scene. The action space is where things get interesting.

1:07:40We use a set of maneuver trajectory candidates to branch over a bunch of interaction decisions

1:07:46and also incremental goals for a longer horizon maneuver. Let's walk through this research

1:07:51very quickly to get a sense of how it works. We start with a set of vision measurements,

1:07:57namely lanes, occupancy, moving objects.

1:08:00These get represented as sparse abstractions,

1:08:03as well as latent features.

1:08:05We use this to create a set of goal candidates.

1:08:08Lanes, again, from the Lanes network,

1:08:10are unstructured regions which correspond

1:08:12to a probability mask derived from human demonstrations.

1:08:17Once we have a bunch of these goal candidates,

1:08:19we create three trajectories using a combination

1:08:21of classical optimization approaches,

1:08:23as well as our network planner, again,

1:08:25trained on data from the customer fleet.

1:08:28Now, once we get a bunch of these three trajectories,

1:08:31we use them to start branching on the interactions.

1:08:35We find the most critical interaction.

1:08:37In our case, this would be the interaction

1:08:39with respect to the pedestrian, whether we

1:08:41are stored in front of it or yield to it.

1:08:43Obviously, the option on the left is a high penalty option.

1:08:47It likely won't get prioritized.

1:08:49So we branch further onto the option on the right.

1:08:51And that's where we bring in more and more complex interactions.

1:08:54building this optimization problem incrementally

1:08:57with more and more constraints.

1:08:59And the research keeps flowing,

1:09:01branching on more interactions,

1:09:02branching on more goals.

1:09:04Now, a lot of pricks here lie in evaluation

1:09:07of each of this node of the research.

1:09:11Inside each node,

1:09:12initially we started with creating trajectories

1:09:14using classical optimization approaches,

1:09:17where the constraints like I described

1:09:18would be added incrementally.

1:09:21And this would take close to one to five milliseconds

1:09:23per action.

1:09:24Now, even though this is a fairly good number, when you want to evaluate more than 100%

1:09:29fractions, this does not scale.

1:09:31So, we ended up building lightweight, queryable networks that we can run in the loop of the

1:09:37planner.

1:09:38These networks are trained on human demonstrations from the fleet, as well as offline solvers

1:09:43with relaxed time limits.

1:09:46With this, we were able to bring the runtime down to close to 100 microseconds per action.

1:09:58Now doing this alone is not enough

1:10:01because you still have this massive research

1:10:02that you need to go through

1:10:04and you need to efficiently prune the search space.

1:10:07So you need to do a discerning on each of these trajectories.

1:10:11Few of these are fairly standard.

1:10:12You do a bunch of collision checks,

1:10:14you do a bunch of comfort analysis,

1:10:16what is the jerk and acts are required

1:10:17for a given maneuver.

1:10:19The customer fleet data plays an important role

1:10:21here again.

1:10:24We run two sets of again lightweight queryable networks,

1:10:27both really augmenting each other.

1:10:29One of them trained from interventions

1:10:31from the FSD beta fleet, which gives us

1:10:33score on how likely is a given maneuver

1:10:35to result in interventions over the next few seconds.

1:10:38And second, which is purely on human demonstrations,

1:10:41human driven data, giving a score on how close

1:10:44is your given selected action to a human driven trajectory.

1:10:48The scoring helps us tune the search space,

1:10:51keep branching further on the interactions,

1:10:53and focus the compute on the most promising outcomes.

1:10:59The cool part about this architecture

1:11:01is that it allows us to create a cool blend between data

1:11:05driven approaches where you don't have to rely on a lot

1:11:07of hand engineered costs, but also grounded in reality

1:11:10with physics based checks.

1:11:14Now, a lot of what I described was

1:11:15with respect to the agents we could observe in the scene.

1:11:19But the same framework extends to objects behind occlusions.

1:11:24We use the video feed from eight cameras

1:11:27to generate the 3D occupancy of the world.

1:11:30The blue mask here corresponds to the visibility region,

1:11:33we call it.

1:11:35It basically gets blocked at the first occlusion

1:11:37you see in the scene.

1:11:39We consume this visibility mask to generate what we call

1:11:41as ghost objects, which you can see on the top left.

1:11:45Now, if you model the spawn regions and the state

1:11:47transitions of this ghost objects correctly,

1:11:51if you tune your control response as a function

1:11:54of the existence likelihood, you can extract some really nice

1:11:57human-like behaviors.

1:11:59Now I'll pass it on to Phil to describe more on how we generate this occupancy networks.

1:12:04Thank you.

1:12:11Hey guys, my name is Phil.

1:12:12I will share the details of the occupancy network we built over the past year.

1:12:18This network is our solution to model the physical work in 3D around our cars.

1:12:23And it is currently not shown in our customer facing visualization.

1:12:27What you will see here is the road network output from our internal dev tool.

1:12:35The acoustic network takes video streams of all our 8 cameras as input, produces a single

1:12:41unified volumetric occupancy in battery space directly.

1:12:46For every 3D location around our car, it predicts the probability of that location being occupied

1:12:52a lot.

1:12:53Since it has video contacts, it is capable of predicting obstacles that are included

1:12:59instantaneously.

1:13:02For each location, it also produces a stat of semantics,

1:13:06such as a curve, car, pedestrian, and road debris,

1:13:10as color coded here.

1:13:16Occupancy flow is also predicted for motion.

1:13:19Since the model is a generalized network,

1:13:21it does not tell static and dynamic objects explicitly.

1:13:26It is able to produce and model the random motion,

1:13:30such as a swerving trainer here.

1:13:34This network is currently running all Tesla's

1:13:36with FSD computers.

1:13:38And it is incredibly efficient, runs about every 10 milliseconds

1:13:42with our neural accelerator.

1:13:45So how does this work?

1:13:46Let's take a look at architecture.

1:13:49First, we rectify each camera images

1:13:52with the camera calibration.

1:13:54And the images we're showing here,

1:13:55what give you the network is actually not

1:13:57a typical AB RGB image.

1:13:59As you can see from the first image on top,

1:14:02we're giving the 12-speed raw photo count

1:14:05image to the network. Since it has four bits more information, it has 16 times better dynamic

1:14:11range as well as reduced latency since we don't have the wrong ISP in the loop anymore.

1:14:18We use a set of regulates and biofpends as a backbone to extract image space features.

1:14:25Next we construct a set of 3D position query along with image space features as keys and

1:14:31values fit into an attention module. The output of the attention module is high-dimensional

1:14:37spatial features. These spatial features are online, temporarily, using vehicle odometry

1:14:46to derive motion.

1:14:48Next, these spatial temporal features go through a set of deconlution to produce the

1:14:54final occupancy and occupancy flow output. They're formed as fixed-sized box of gray,

1:15:00which might not be precise enough for planning on control.

1:15:04In order to get a higher resolution,

1:15:06we also produce per-voxel feature maps,

1:15:08which will fade into MLP with 3D spatial point queries

1:15:13to get position and semantics at any arbitrary location.

1:15:19After knowing the model better,

1:15:21let's take a look at another example.

1:15:23Here we have an articulated bus parked on the right side row,

1:15:26highlighted as an L-shaped box here.

1:15:29As we approach, the bus starts to move.

1:15:31The front of the cart turns blue first,

1:15:35indicating the model predicts.

1:15:37The front of the bus has a long zero occupancy flow.

1:15:42And as the bus keeps moving, the entire bus turns blue.

1:15:46And you can also see that the network predicts

1:15:49the precise curvature of the bus.

1:15:52Well, this is a very complicated problem

1:15:55for traditional object detection network.

1:15:57As you have to see whether I'm going to use one cuboid or perhaps a cube to fit the curvature.

1:16:03But for our occupancy network, since all we care about is the occupancy in the visible

1:16:07space and we will be able to model the curvature precisely.

1:16:13Besides the voxel gray, the occupancy network also produces a dry-boat surface.

1:16:18The dry-boat surface has both 3D geometry and semantics.

1:16:21They are very useful for control, especially on hilly and curvy roads.

1:16:28The surface and the voxel gray are not predicted independently.

1:16:31Instead, the voxel gray actually aligns with the surface implicitly.

1:16:38Here we are at a here quest where you can see the 3D geometry of the surface being predicted

1:16:45nicely.

1:16:46Planner can use this information to decide perhaps when you slow down wall for the here

1:16:51And as you can also see, the box of gray

1:16:54aligns with the surface consistently.

1:16:58Besides the box of and the surface,

1:17:01we're also very excited about the recent breakthrough

1:17:03in neural radians field or LERF.

1:17:07We're looking into both incorporated

1:17:09some of the LISD LERF features into occupancy network training

1:17:13as well as using our network output

1:17:15as an input state for LERF.

1:17:19As a matter of fact, Ashok is very excited about this.

1:17:21This has been his personal weekend project for a while.

1:17:29These nerves, because I think academia

1:17:32is building a lot of these foundation models for language

1:17:35using tons of large data sets for language.

1:17:38But I think for vision, nerves are

1:17:40going to provide the foundation models for computer vision,

1:17:43because they are grounded in geometry.

1:17:46And geometry gives us a nice way to supervise

1:17:48the networks and free sets of the requirement

1:17:50to define an ontology.

1:17:52And the supervision is essentially

1:17:53because you just have to differentiably render these images.

1:17:57So I think in the future, this occupancy network idea

1:18:00where images come in and then the network

1:18:03produces a consistent volume representation

1:18:06of the scene that can then be differentiably rendered

1:18:09into any image that was observed.

1:18:12I personally think is a future of computer vision.

1:18:15And we do some initial work on it right now.

1:18:18But I think in the future, both the Tesla and in the academia,

1:18:22we will see that this combination of one-shot prediction

1:18:26of volumetric occupancy will be the future.

1:18:29That's my personal bet.

1:18:32Thanks, Ashok.

1:18:34So here's an example early result

1:18:36of a 3D reconstruction from our fleet data.

1:18:39Instead of focusing on getting perfect RGB reprojection

1:18:43in image space, our primary goal here

1:18:45is to accurately represent the world

1:18:47in a 3D space for driving.

1:18:49And we want to do this for all our fleet data

1:18:51over the world in all weather and lighting conditions.

1:18:55And obviously, this is a very challenging problem,

1:18:57and we're looking for you guys to help.

1:19:00Finally, the occupancy network is trained

1:19:02with large, ultra-labeled data set

1:19:04without any human in the loop.

1:19:07And with that, I'll pass to PIM to talk about

1:19:09what it takes to train this network.

1:19:12Thanks, Phil.

1:19:18All right, hey, everyone.

1:19:21Let's talk about some training infrastructure.

1:19:24So we've seen a couple of videos,

1:19:25No, four or five, I think, and care more,

1:19:29and worry more about a lot more clips on that.

1:19:32So we've been looking at the occupancy networks

1:19:35just from Phil.

1:19:36Just Phil's videos, it takes 1.4 billion frames

1:19:41to train that network, what you just saw.

1:19:43And if you have 100,000 GPU, it would take one hour,

1:19:47but if you have one GPU, it would take 100,000 hours.

1:19:52So that is not a humane time period

1:19:54you can wait for your training job to run, right?

1:19:56We want to ship faster than that.

1:19:58So that means you're going to need to go parallel.

1:20:01So you need a more compute for that.

1:20:03That means you're going to need a supercomputer.

1:20:06So this is why we've built in-house three supercomputers

1:20:09comprising of 14,000 GPUs, where we use 10,000 GPUs for

1:20:13training and run 4,000 GPUs for all labeling.

1:20:18All these videos are stored in 30 petabytes of a distributed

1:20:22managed video cache.

1:20:25You shouldn't think of our data sets as fixed, let's say,

1:20:28as you think of your image net or something,

1:20:30you know, with like a million frames.

1:20:31You should think of it as a very fluid thing.

1:20:34So we've got half a million of these videos flowing in

1:20:37and out of these clusters every single day.

1:20:42And we track 400,000 of these kind of Python video

1:20:46instantiations every second.

1:20:49So that's a lot of calls.

1:20:51We are going to need to capture that in order to govern the retention policies of this distributed

1:20:56video cache.

1:20:57So underlying all of this is a huge amount of info, all of which we build and manage in-house.

1:21:04So you cannot just buy, you know, 14,000 GPUs and then 30 petabytes of flash NVMe and just

1:21:10put it together and let's go train.

1:21:13It actually takes a lot of work and I'm going to go into a little bit of that.

1:21:17What you actually typically want to do is you want to take your accelerator so that

1:21:21it could be the GPU or Dojo, which we'll talk about later.

1:21:26Because that's the most expensive component, that's where you want to put your bottleneck.

1:21:31And so that means that every single part of your system is going to need to outperform

1:21:36this accelerator.

1:21:37And so that is really complicated.

1:21:39That means that your storage is going to need to have the size and the bandwidth to deliver

1:21:44all the data down into the nodes.

1:21:46These nodes need to have the right amount of CPU and memory capabilities to feed into

1:21:51your machine learning framework.

1:21:53This machine learning framework then needs to hand it off to your GPU and then you can

1:21:56start training.

1:21:57But then you need to do so across hundreds or thousands of GPU in a reliable way, in lock

1:22:04step and in a way that's also fast.

1:22:06So you're also going to need an interconnect.

1:22:09Extremely complicated.

1:22:10We'll talk more about dojo in a second.

1:22:13So first, I want to take you through some optimizations

1:22:16that we've done on our cluster.

1:22:19So we're getting in a lot of videos.

1:22:21And video is very much unlike, let's say,

1:22:23training on images or text, which I think

1:22:26is very well established.

1:22:27Video is quite literally a dimension more complicated.

1:22:32And so that's why we needed to go end to end from the storage

1:22:35layer down to de-accelerate and optimize

1:22:37every single piece of that.

1:22:39Because we train on the photon count

1:22:41videos that come directly from our fleet.

1:22:44We train on those directly.

1:22:45We do not post process those at all.

1:22:48The way it's just done is we seek exactly to the frames we

1:22:52select for our batch.

1:22:53We load those in, including the frames that they depend on.

1:22:56So these are your i-frames or your key frames.

1:22:58We package those up, move them into shared memory, move them

1:23:01into a double bar from the GPU, and then use a hardware decoder

1:23:05that's only accelerated to actually decode the video.

1:23:09So we do that on the GPU natively.

1:23:11And this is all in a very nice PyTorch extension.

1:23:15Doing so unlocks more than 30% training speed increase

1:23:18for the occupancy networks and freed up basically

1:23:21the whole CPU to do any other thing.

1:23:26You cannot just do training with just videos.

1:23:28Of course, you need some kind of a ground truth.

1:23:31And that is actually an interesting problem as well.

1:23:34The objective for storing a ground truth

1:23:36is that you want to make sure you get to your ground truth

1:23:39that you need in the minimal amount of file system operations

1:23:43and load in the minimal size of what you need

1:23:45in order to optimize for aggregate cross cluster throughput.

1:23:49Because you should see a compute cluster

1:23:51as one big device which has internally fixed

1:23:54constraints and thresholds.

1:23:56So for this, we rolled out a format that is native to us

1:24:01that's called small.

1:24:02We use this for our ground truth, our feature cache,

1:24:04and any inference outputs.

1:24:06So a lot of tensors that are in there.

1:24:08And so just the cartoon here, let's say, these are your table

1:24:12that you want to store, then that's how that would look out

1:24:14if you're all down on disk.

1:24:16So what you do is you take anything you'd want to index on.

1:24:18So for example, video timestamps, you

1:24:21put those all in the header so that in your initial header,

1:24:23you know exactly where to go on disk.

1:24:26Then if you have any tensors, you're

1:24:28going to try to transpose the dimensions to put a different

1:24:31dimension last as the contiguous dimension,

1:24:34and then also try different types of compression.

1:24:37Then you check out which one was most optimal,

1:24:39and then store that one.

1:24:41This is actually a huge step if you do feature caching.

1:24:44Unintelligible output from the machine learning network

1:24:47rotated around the dimensions a little bit,

1:24:49you can get up to 20% increase in efficiency of storage.

1:24:53Then when you store that, we also ordered columns by size

1:24:58so that all your small columns and small values

1:25:01are together so that when you seek for a single value,

1:25:04you're likely to overlap with the read on more values,

1:25:06which we'll use later, so that you don't need to do another file system operation.

1:25:12So I could go on and on.

1:25:13I just went on, touched on two projects that we have internally.

1:25:17But this is actually part of a huge continuous effort to optimize the compute that we have

1:25:22in house.

1:25:24So accumulating and aggregating through all these optimizations, we now train our occupancy

1:25:28networks twice as fast just because it's twice as efficient.

1:25:32And now if we add in a bunch more compute and go parallel, we can now train this in hours

1:25:37instead of days.

1:25:39And with that, I'd like to hand it off to the biggest user of compute, John.

1:25:42Hi, everybody.

1:25:51My name is John Emmons.

1:25:52I lead the Autopilot Vision Team.

1:25:54I'm going to cover two topics with you today.

1:25:57The first is how we predict lanes.

1:25:58And the second is how we predict the future behavior of other agents on the road.

1:26:04In the early days of Autopilot, we modeled the lane detection problem as an image space

1:26:08instant segmentation task.

1:26:11Our network was super simple, though.

1:26:13In fact, it was only capable of printing lanes

1:26:15from a few different kinds of geometries.

1:26:18Specifically, it would segment the eagle lane,

1:26:21it could segment adjacent lanes, and then it

1:26:23had some special casing for forks and merges.

1:26:26This simplistic modeling of the problem

1:26:27worked for highly structured roads like highways.

1:26:31But today, we're trying to build a system that's

1:26:33capable of much more complex maneuvers.

1:26:35Specifically, we want to make left and right turns

1:26:37intersections where the road topology can be quite a bit more complex and diverse.

1:26:41When we try to apply the simplistic modeling of the problem here, it just totally breaks

1:26:45down.

1:26:47Taking a step back for a moment, what we're trying to do here is to predict the sparse

1:26:51set of lane instances in their connectivity.

1:26:54And what we want to do is to have a neural network that basically predicts this graph,

1:26:58where the nodes are the lane segments and the edges encode the connectivity between these

1:27:02lanes.

1:27:05So what we have is our lane detection neural network.

1:27:08made up of three components. In the first component, we have a set of convolutional layers, attention

1:27:14layers, and other neural network layers that encode the video streams from our eight cameras

1:27:18on the vehicle and produce a rich visual representation. We then enhance this visual representation with

1:27:25a coarse road level mapped data, which we encode with a set of additional neural network

1:27:32layers that we call the Lane Guidance Module. This map is not an HD map, but it provides

1:27:37lot of useful hints about the topology of lanes inside of intersections, the lane counts on

1:27:41various roads, and a set of other attributes that help us. The first two components here

1:27:48produce a dense tensor that sort of encodes the world. But what we really want to do is

1:27:52to convert this dense tensor into a spark set of lanes in their con activities. We approach

1:27:58this problem like an image captioning task where the input is this dense tensor and the

1:28:03output text is predicted into a special language that we developed at Tesla for encoding lanes

1:28:07in their con activities.

1:28:09In this language of lanes, the words and tokens

1:28:12are the lane positions in 3D space,

1:28:14in the ordering of the tokens,

1:28:16and predicted modifiers in the tokens,

1:28:17encode the connective relationships between these lanes.

1:28:21By modeling the task as a language problem,

1:28:23we can capitalize on recent autoregressive architectures

1:28:26and techniques from the language community

1:28:28for handling the multiplicity of the problem.

1:28:30We're not just solving the computer vision problem

1:28:32at Autopilot, we're also applying the state of the art

1:28:34and language modeling and machine learning more generally.

1:28:38I'm now going to dive into a little bit more detail

1:28:40this language component.

1:28:43What I have depicted on the screen here

1:28:44is a satellite image which sort of represents

1:28:47the local area around the vehicle.

1:28:49The set of nose images is what we refer to as the lane graph,

1:28:52and it's ultimately what we want to come out of this neural network.

1:28:56We start with a blank slate.

1:28:58We're going to want to make our first prediction

1:29:00here at the screen dot.

1:29:03The screen dot's position is encoded as an index

1:29:05into a course grid which discretizes the 3D world.

1:29:09Now we don't predict this index directly,

1:29:10because it would be too computationally expensive to do so.

1:29:13There's just too many grid points,

1:29:15and predicting a categorical distribution over this

1:29:17as both implications at training time and test time.

1:29:20So instead what we do is we discretize the world

1:29:22coarsely first.

1:29:23We predict the heat map over the possible locations,

1:29:26and then we latch in the most probable location.

1:29:29Condition on this, we then refine the prediction

1:29:31and get the precise point.

1:29:35Now we know where the position of this token is, but we don't know its type.

1:29:38In this case, though, it's the beginning of a new lane.

1:29:41So we pricked it as a start token.

1:29:44And because it's a start token, there's no additional attributes in our language.

1:29:49We then take the predictions from this first forward pass, and we encode them using a learned

1:29:52conditional embedding, which produces a set of tensors that we combine together, which is

1:29:58actually the first word in our language of lanes.

1:30:01We add this to the first position in our sentence here.

1:30:04continue this process by printing the next lane point in a similar fashion. Now this lane

1:30:10point is not the beginning of a new lane. It's actually a continuation of the previous lane.

1:30:15So it's a continuation token type. Now it's not enough just to know that this lane is connected

1:30:21to the previously predicted lane. We want to encode its precise geometry, which we do by

1:30:25regressing a set of flying coefficients. We then take this lane, we encode it again, and

1:30:32and add it as the next word in the sentence.

1:30:35We continue predicting these continuation lanes

1:30:37until we get to the end of the prediction grid.

1:30:40We then move on to a different lane segment.

1:30:42So you can see that cyan dot there.

1:30:44Now, it's not topologically connected to that pink point.

1:30:47It's actually forking off of that green point there.

1:30:52So it's got a fork type.

1:30:54And fork tokens actually point back

1:30:57to previous tokens from which the fork originates.

1:31:01So you can see here the fourth point predictor is actually

1:31:03the index 0.

1:31:04So it's actually referencing the back of tokens that is

1:31:05already created, like you would in language.

1:31:10We continue this process over and over again.

1:31:11And so we've enumerated all of the tokens in the link graph.

1:31:15And then the network predicts the end of center token.

1:31:19Yeah, I just wanted to note that the reason we do this is

1:31:22not just because we want to build something complicated.

1:31:24It almost feels like a Turing complete machine here

1:31:26with neural networks though.

1:31:27Is that we tried simple approaches, for example,

1:31:30trying to just segment the lanes along the road

1:31:33or something like that.

1:31:34But then the problem is when this uncertainty, say you cannot

1:31:37see the road clearly, and there could be two lanes or three

1:31:39lanes, and you can't tell a simple segmentation-based

1:31:42approach would just draw both of them.

1:31:44It's kind of a 2.5 lane situation.

1:31:47And the post-processing algorithm would

1:31:48hilariously fail when the predictions are such.

1:31:51Yeah, the problems don't end there.

1:31:53I mean, you need to predict these connective lanes

1:31:56inside of intersections, which is just not possible

1:31:58with the approach that Ashok's mentioning, which is why

1:32:00we had to upgrade to this sort of thing.

1:32:01Like overlaps like this, segmentation would just go haywire.

1:32:04But even if you try very hard to put them on separate layers,

1:32:07it's just a really hard problem.

1:32:09But language just offers a really nice framework

1:32:11for getting a sample from a posterior, as opposed

1:32:15to trying to do all of this in post processing.

1:32:19But this doesn't actually stop for just autopilot.

1:32:21This can be used for optimists.

1:32:23Yeah, I guess they wouldn't be called lanes.

1:32:25But you could imagine, sort of in this stage here,

1:32:28that you might have paths that encode the possible places

1:32:32that people could walk.

1:32:34Yeah, basically, if you're in a factory or in a home setting,

1:32:38you can just ask the robot, OK, let's me.

1:32:40Please shout to the kitchen or please

1:32:42shout to some location in the factory.

1:32:44And then we put a set of pathways that

1:32:46would go through the aisles, take the robot, and say, OK,

1:32:48this is how you get to the kitchen.

1:32:50It just really gives us a nice framework

1:32:52to model these different paths that

1:32:53simplify the navigation problem for the downstream planner.

1:33:00All right, so ultimately what we get from this lane detection

1:33:03network is a set of lanes in their con activities, which

1:33:05comes directly from the network.

1:33:07There's no additional step here for specifying these dense

1:33:10predictions into sparse ones.

1:33:13This is just a direct unfiltered output of the network.

1:33:18So I talked a little bit about lanes.

1:33:20I'm going to briefly touch on how we model and predict

1:33:22the future paths and other semantics on objects.

1:33:26So I'm just going to go really quickly

1:33:27through two examples.

1:33:29The video on the right here, we've

1:33:30got a car that's actually running a red light and turning

1:33:32in front of us.

1:33:34What we do to handle situations like this

1:33:36is we predict a set of short time horizon future trajectories

1:33:39on all objects.

1:33:40We can use these to anticipate the danger situation here

1:33:44and apply whatever breaking and steering actions

1:33:46required to avoid a collision.

1:33:48In the video on the right, there's two vehicles

1:33:50in front of us.

1:33:51The one on the left lane is parked.

1:33:54Apparently it's being unloaded, unloaded.

1:33:55I don't know why the driver decided to park there.

1:33:57But the important thing is that our neural network predicted

1:33:59that it was stopped, which is the red color there.

1:34:03The vehicle and the other lane, as you notice,

1:34:04also is stationary.

1:34:06But that one's obviously just waiting

1:34:07for that red light to turn green.

1:34:08So even though both objects are stationary

1:34:10and have zero velocity, it's the semantics

1:34:12that is really important here,

1:34:14so that we don't get stuck behind that awkwardly parked car.

1:34:19Predicting all of these agent attributes

1:34:21presents some practical problems

1:34:22when trying to build a real-time system.

1:34:24We need to maximize the frame rate

1:34:25of our object section stack,

1:34:27so that Autopilot can quickly react

1:34:28to the changing environment.

1:34:30Every millisecond really matters here.

1:34:33To minimize the inference latency,

1:34:34our neural network is split into two phases.

1:34:37In the first phase, we identified the locations

1:34:39in 3D space where agents exist.

1:34:42In the second stage, we then pull out tensors

1:34:44at those 3D locations, append it with additional data

1:34:47that's on the vehicle, and then we do the rest of the processing.

1:34:52This classification step allows the neural number

1:34:54to focus compute on the areas that matter most,

1:34:57which gives us superior performance

1:34:58for a fraction of the latency cost.

1:35:01So putting it all together, the Autopilot Vision Stack

1:35:04predicts more than just the geometry and kinematics

1:35:06of the world.

1:35:06It also predicts a rich set of semantics,

1:35:09which enables safe and human-like driving.

1:35:11I'm now going to hand things off to Shree, who will tell us

1:35:13how we run all these cool neural networks on our FSD computer.

1:35:15Thank you.

1:35:24Hi, everyone.

1:35:25I'm Shree.

1:35:26Today I'm going to give glimpse of what

1:35:28it takes to run these FSD networks in the car,

1:35:31and how do we optimize for the inference latency.

1:35:34Today I am going to focus just on FSC Lane's network that John just talked about.

1:35:41So when we started this track, we wanted to know if we can run this FSC Lane's network

1:35:46natively on the trip engine, which is our enough neural network accelerator that we built in

1:35:52the FSC computer.

1:35:53When we built this hardware, we kept it simple and we made sure it can do one thing ridiculously

1:35:59fast, dense dot products.

1:36:02But this architecture is auto regressive and iterative where it crunches through multiple

1:36:08attention blocks in the inner loop producing sparse points directly at every step.

1:36:14So, the challenge here was how can we do this sparse point prediction and sparse computation

1:36:19on a dense dot product engine.

1:36:21Let us see how we did this on the trip.

1:36:25So, the network predicts the heat map of most probable spatial locations of the point.

1:36:32Now we do a argmax and a one-hot operation which gives the one-hot encoding of the index

1:36:39of the spatial location.

1:36:41Now we need to select the embedding associated with this index from an embedding table that

1:36:47is learned during training.

1:36:48To do this on trip we actually built a lookup table in S-Tram and we engineered the dimensions

1:36:56of this embedding such that we could achieve all of this thing with just matrix multiplication.

1:37:03Not just that, we also wanted to store this embedding into a token cache so that we do

1:37:08not recompute this for every iteration rather reuse it for future point prediction. Again,

1:37:13we put some tricks here where we did all these operations just on the dot product engine. It

1:37:19is actually cool that our team found creative ways to map all these operations on the trip

1:37:24engine in ways that were not even imagined when this hardware was designed.

1:37:31But that is not the only thing we had to do to make this work.

1:37:34We actually implemented a whole lot of operations and features to make this model compilable

1:37:41to improve the intate accuracy as well as to optimize performance.

1:37:45All of these things helped us run this 75 million parameter model just under 10 millisecond of

1:37:51latency consuming just 8 watts of power.

1:37:57But this is not the only architecture running in the car.

1:37:59There are so many other architectures, modules and networks we need to run in the car.

1:38:05To give a sense of scale there are about billion parameters of all the networks combined producing

1:38:11around 1000 neural network signals.

1:38:13So, we need to make sure we optimize them jointly and such that we maximize the computer

1:38:21utilization throughput and minimize the latency.

1:38:24So, we built a compiler just for neural networks that shares the structure to traditional

1:38:31compilers. As you can see, it takes the massive graph of neural nets with 150 k nodes and

1:38:38375 k connection takes this thing, partitions them into independent sub graphs and compels

1:38:45each of the sub graphs natively for the inference devices. Then we have a neural network linker

1:38:52which shares the structure to traditional linker where we perform this link time optimization

1:38:57there we solve an offline optimization problem with compute memory and memory bandwidth constraints

1:39:05so that it comes with an optimized schedule that gets executed in the car.

1:39:10On the run time we designed a hybrid scheduling system which basically does heterogeneous scheduling

1:39:16on one SOC and distributed scheduling across both the SOCs to run these networks in a model

1:39:22parallel fashion.

1:39:24To get 100 tops of compute utilization, we need to optimize across all the layers of software

1:39:30right from tuning the network architecture, the compiler, all the way to implementing

1:39:36a low latency, high bandwidth, RDM link across both the SOCs and in fact going even

1:39:41deeper to understanding and optimizing the cache coherent and non coherent data path

1:39:47of the accelerator in the SOC.

1:39:49This is a lot of optimization at every level in order to make sure we get the highest

1:39:53frame rate and as every millisecond counts here.

1:39:59And this is just the visualization of the neural networks that are running in the car.

1:40:05This is our digital brain essentially.

1:40:08As you can see these operators are nothing but just the matrix multiplication convolution

1:40:12to name a few real operations running in the car.

1:40:17To train this network with a billion parameters,

1:40:20you need a lot of labeled data.

1:40:22So Aegon is going to talk about how

1:40:25do we achieve this with the auto labeling pipeline.

1:40:36Thank you, sir.

1:40:37Hi, everyone.

1:40:38I'm Yae-gun Zhang, and I'm leaving geometric vision

1:40:41at autopilot.

1:40:42So yeah, let's talk about auto labeling.

1:40:48So we have several kinds of all labeling frameworks

1:40:51to support various types of networks.

1:40:54But today, I'd like to focus on the awesome lanes net here.

1:40:59So to successfully train and generalize this network

1:41:02to everywhere, we think we went tens of millions of trips

1:41:07from probably one million intersection or even more.

1:41:12So then how to do that?

1:41:15So it is certainly achievable to source sufficient amount

1:41:20of trips, because we already have, as Tim explained earlier,

1:41:23we already have like 500,000 trips per day cash rate.

1:41:28However, comparing all those data into a training form

1:41:32is a very challenging technical problem.

1:41:36To solve this challenge, we've tried various ways

1:41:39of manual and auto labeling.

1:41:41So from the first column to the second,

1:41:44from the second to the third, each advance provided us nearly 100

1:41:48decks implement in throughput.

1:41:50But still, we run an even better auto labeling machine

1:41:54that can provide us good quality, diversity, and scalability.

1:42:02To meet all these requirements, despite the huge amount

1:42:06of engineering effort required here,

1:42:08we've developed a new auto labeling machine powered

1:42:12by multi-tribby construction.

1:42:15So these can replace 5 million hours of manual labeling

1:42:19with just 12 hours of cluster for labeling 10,000 trips.

1:42:24So how we solved?

1:42:25There are three big steps.

1:42:27The first step is high precision trajectory

1:42:29and structural recovery by multi-camera, visual,

1:42:32inertial, or geometry.

1:42:34So here, all the features, including ground surface,

1:42:37are inferred from videos by neural networks, then tracked

1:42:40and reconstructed in the vector space.

1:42:44So the typical drift rate of this trajectory in car

1:42:48is like 1.3 centimeter per meter and 0.45 milli-redium

1:42:52per meter, which is pretty decent considering

1:42:55its compact compute requirement.

1:42:57Then the recovery surface and road details

1:43:00are also used as a strong guidance for the later manual

1:43:03verification stuff.

1:43:05This is also enabled in every FSD vehicle,

1:43:08so we get pre-processed trajectories and structures

1:43:11along with the trip data.

1:43:16The second step is multi-trib reconstruction, which is the big and core piece of this machine.

1:43:22So the video shows how the previously shown trip is reconstructed and aligned with other

1:43:27trips, basically other trips from different people, not the same vehicle.

1:43:32So this is done by multiple intern steps like course alignment, pairwise matching, join optimization,

1:43:38then further surface refinement.

1:43:41In the end, the human analyst comes in and finalizes the label.

1:43:46So each heavy steps are already fully parallelized on the cluster.

1:43:51So the entire process usually takes just a couple of hours.

1:43:57The last step is actually auto labeling the new trips.

1:44:01So here, we use the same multi-trib alignment engine,

1:44:06but only between pre-built reconstruction and each new trip.

1:44:10So it's much, much simpler than fully reconstructing

1:44:13all the clips altogether.

1:44:15That's why it only takes 30 minutes per trip to auto label

1:44:19instead of several hours of manual labeling.

1:44:24And this is also the key of scalability of this machine.

1:44:31This machine easily scales as long as we have available compute

1:44:36and treat data.

1:44:38So about 50 trips were newly auto labeled from this scene,

1:44:41and some of them are shown here.

1:44:43So 53 from Deepramp vehicles.

1:44:47So this is how we capture and transform the space time

1:44:51slices of the world into the network's supervision.

1:44:54Yeah, one thing I'd like to note is that Jagan just talked

1:44:57about how we auto-label our lanes.

1:45:00But we have auto-labors for almost every task

1:45:02that we do including our planner.

1:45:04And many of these are fully automatic.

1:45:06There's no humans involved.

1:45:07For example, for objects, all the kinematics, the shapes,

1:45:10their futures, everything just comes from auto labeling.

1:45:13And the same is true for occupancy too.

1:45:15And we have really just built a machine not on this.

1:45:17Yeah, so if you can go back one slide.

1:45:21One more.

1:45:22It says, parallelized on cluster.

1:45:26So that sounds pretty straightforward, but really wasn't.

1:45:29Maybe it's fun to share how something like this comes about.

1:45:33So a while ago, we didn't have any auto labeling at all.

1:45:37And then someone makes a script.

1:45:39It starts to work.

1:45:40It starts working better until you reach a volume that's

1:45:42pretty high and we clearly need a solution.

1:45:45And so there were two other engineers in our team who were

1:45:48like, that's an interesting thing.

1:45:51What we needed to do was build a whole graph of essentially

1:45:55Python functions that we need to run one after the other.

1:45:57First you pull the clip, then you do some

1:45:58fleeting, then you do some network inference, then another

1:46:01network inference, and so you finally get this.

1:46:04But so you need to do this at a large scale.

1:46:06So I tell them, we probably need to shoot for 100,000

1:46:10clips per day or like 100,000 items, that seems good.

1:46:14And so the engineers said, well, we can do a bit of post-press

1:46:19and a bit of elbow grease.

1:46:20We can do it.

1:46:21Meanwhile, we are a bit later, and we're

1:46:23doing 20 million of these functions every single day.

1:46:28Again, we pull in around half a million clips.

1:46:30And on those, we run and done the functions

1:46:32each of these in a streaming fashion.

1:46:34And so that's kind of the back end

1:46:36info that's also needed to not just run training,

1:46:38but also auto labeling.

1:46:40It really is like a factory that produces labels,

1:46:42and it's like production lines, yield, quality, inventory.

1:46:46Like, all of the same concepts applied to this label factory

1:46:50that applies for the factory for our cars.

1:46:52That's right.

1:46:55OK.

1:46:56Thanks, Tim and Ashok.

1:46:58So, yeah.

1:46:59So concluding this section, I'd like

1:47:01to share a few more challenging and interesting examples

1:47:04for network, for sure, and even for humans, probably.

1:47:08So from the top, there's examples

1:47:10for lack of lights, case, or fogging night, or roundabout,

1:47:15and have your occlusions by parked cars,

1:47:18and even rainy night with the raindrops on camera lenses.

1:47:22These are challenging, but once their original scenes

1:47:24are fully reconstructed by other clips,

1:47:27all of them can be auto-labeled so that our cars can drive

1:47:30even better through these challenging scenarios.

1:47:34So now, let me pass the mic through David

1:47:36to learn more about how it's creating the new world

1:47:38on top of these labels.

1:47:39Thank you.

1:47:47Thank you, Yagan.

1:47:48My name is David, and I'm going to talk about simulation.

1:47:52So simulation plays a critical role in providing data that is difficult to source and or hard

1:47:57to label.

1:47:58However, 3D scenes are notoriously slow to produce.

1:48:03Take for example the simulated scene playing behind me.

1:48:06A complex intersection from Market Street in San Francisco, it would take two weeks for

1:48:12artists to complete.

1:48:13And for us, that is painfully slow.

1:48:16However, I'm going to talk about using Yagan's automated

1:48:19ground truth labels, along with some brand new tooling that

1:48:22allows us to procedurally generate this scene, and many

1:48:25like it, in just five minutes.

1:48:27That's an amazing 1,000 times faster than before.

1:48:31So let's dive in to how a scene like this is created.

1:48:36We start by piping the automated ground truth labels into our

1:48:39simulated world creator tooling inside the software Houdini.

1:48:43Starting with road boundary labels,

1:48:45we can generate a solid road mesh

1:48:47and re-topologize it with the lane graph labels.

1:48:50This helps inform important road details

1:48:52like crossroad slope and detailed material blending.

1:48:57Next, we can use the line data

1:48:58and sweep geometry across its surface

1:49:01and project it to the road,

1:49:02creating lane paint decals.

1:49:07Next, using median edges,

1:49:09we can spawned island geometry and populate it

1:49:11with randomized foliage.

1:49:13This drastically changes the visibility of the scene.

1:49:16Now, the outside world can be generated

1:49:18through a series of randomized heuristics.

1:49:21Modular building generators create visual obstructions

1:49:24while randomly placed objects like hydrants

1:49:26can change the color of the curbs

1:49:28while trees can drop leaves below it,

1:49:30obscuring lines or edges.

1:49:33Next, we can bring in map data to inform positions

1:49:36of things like traffic lights or stop signs.

1:49:39We can trace along its normal

1:49:41to collect important information like number of lanes

1:49:44and even get accurate street names on the signs themselves.

1:49:48Next, using Langraph, we can determine lane connectivity

1:49:51and spawn directional road markings on the road

1:49:54and there are accompanying road signs.

1:49:57And finally, with Langraph itself,

1:49:59we can determine lane adjacency and other useful metrics

1:50:03to spawn randomized traffic permutations

1:50:05inside our simulator.

1:50:07And again, this is all automatic,

1:50:09No artists in the loop and happens within minutes.

1:50:12And now this sets us up to do some pretty cool things.

1:50:16Since everything is based on data and heuristics,

1:50:18we can start to fuzz parameters to create visual variations

1:50:21of the single ground truth.

1:50:23It can be a subtle as object placement and random material

1:50:26swapping to more drastic changes,

1:50:29like entirely new biomes or locations of environment,

1:50:32like urban, suburban, or rural.

1:50:35This allows us to create infinite, targeted permutations

1:50:38for specific ground truth that we need more ground truth for.

1:50:43And all this happens within a click of a button.

1:50:48And we can even take this one step further

1:50:49by altering our ground truth itself.

1:50:52Say John wants his network to pay more attention

1:50:55the directional road markings to better detect

1:50:57an upcoming captive left-turn lane.

1:50:59We can start to procedurally alter our lane graph

1:51:02inside the simulator to help to create entirely new flows

1:51:06through this intersection, to help focus the networks'

1:51:09attention to the road markings to create more accurate predictions.

1:51:13And this is a great example of how this tooling allows us to create new data that can never

1:51:17be collected from the real world.

1:51:20And the true power of this tool is in its architecture and how we can run all tasks in parallel to

1:51:26infinitely scale.

1:51:27So you solve the tile creator tool in action, converting the ground truth labels into their

1:51:33counterparts.

1:51:34Next, we can use our tile extractor tool to divide this data into geo-hash tiles about

1:51:40150 meters square in size.

1:51:43We then save out that data into separate geometry and instance files.

1:51:48This gives us a clean source of data that's easy to load and allows us to be rendering

1:51:52engine agnostic for the future.

1:51:56Then using a tile loader tool, we can summon any number of those cache tiles using a geo-hash

1:52:01ID.

1:52:02Currently, we're doing about these 5x5 tiles or 3x3,

1:52:06usually centered around fleet hotspots

1:52:08or interesting land-graft locations.

1:52:11And the tile loader also converts these tile sets

1:52:14into U-assets for consumption by the Unreal Engine

1:52:17and gives you a finished project from what you saw

1:52:20in the first slide.

1:52:23And this really sets us up for size and scale.

1:52:26And as you can see on the map behind us,

1:52:28we can easily generate most of San Francisco city streets.

1:52:32And this didn't take years or even months of work,

1:52:35but rather two weeks by one person.

1:52:38We can continue to manage and grow all this data

1:52:41using our PDG network inside of the tooling.

1:52:44This allows us to throw compute at it

1:52:46and regenerate all these tile sets overnight.

1:52:50This ensures all environments are of consistent quality

1:52:53and features, which is super important for training

1:52:56since new ontologies and signals are constantly released.

1:53:03And now to come full circle, because we generated all these

1:53:06tile sets from ground truth data that contain all the weird

1:53:10intricacies from the real world.

1:53:11And we can combine that with the procedural, visual, and

1:53:14traffic variety to create limitless, targeted data for the

1:53:18network to learn from.

1:53:20And that concludes the Sim section.

1:53:21I'll pass it to Kate to talk about how we can use all this

1:53:24data to improve Autopilot.

1:53:26Thank you.

1:53:36Thanks, David.

1:53:37Hi, everyone.

1:53:38My name is Kate Park.

1:53:39And I'm here to talk about the data engine, which is the

1:53:42processed by which we improve our neural networks via data.

1:53:46We're going to show you how we deterministically solve

1:53:49interventions via data and walk you

1:53:52through the life of this particular clip.

1:53:54In this scenario, Autopilot is approaching a turn

1:53:58and incorrectly predicts that crossing vehicle

1:54:00has stopped for traffic and thus a vehicle

1:54:03that we would slow down for.

1:54:04In reality, there's nobody in the car.

1:54:07It's just awkwardly parked.

1:54:09We built this tooling to identify the mispredictions,

1:54:12correct the label, and categorize this clip

1:54:15into an evaluation set.

1:54:17This particular clip happens to be one of 126

1:54:20that we've diagnosed as challenging parked cars at turns.

1:54:24Because of this infra, we can curate this evaluation set

1:54:28without any engineering resources

1:54:30custom to this particular challenge case.

1:54:35To actually solve that challenge case

1:54:36requires mining thousands of examples like it.

1:54:39And it's something Tesla can trivially do.

1:54:42We simply use our data sourcing info, request data,

1:54:46and use the tooling shown previously to correct the labels.

1:54:50By surgically targeting the mispredictions

1:54:52of the current model, we're only adding the most valuable examples

1:54:56to our training set.

1:54:58We surgically fix 13,900 clips.

1:55:02And because those were examples where

1:55:04the current model struggles, we don't even

1:55:06need to change the model architecture.

1:55:09A simple way update with this new, valuable data

1:55:12is enough to solve the challenge case.

1:55:14So you see we no longer predict that crossing vehicle

1:55:17as stopped as shown in orange, but parked as shown in red.

1:55:21In academia, we often see that people keep data constant,

1:55:25but at Tesla it's very much the opposite.

1:55:28We see time and time and again that data is one of the best,

1:55:32if not the most deterministic lever

1:55:34to solving these interventions.

1:55:37We just showed you the data engine loop for one challenge

1:55:40case, namely these parked cars at turns.

1:55:42But there are many challenge cases,

1:55:44even for one signal of vehicle movement.

1:55:47We apply this data engine loop to every single challenge

1:55:50case we've diagnosed, whether it's buses, curvy roads,

1:55:53stopped vehicles, parking lots.

1:55:55And we don't just add data once.

1:55:57We do this again and again to perfect the semantic.

1:56:01In fact, this year, we updated our vehicle movement signal

1:56:04five times, and with every way update trained on the new data,

1:56:08we push our vehicle movement accuracy up and up.

1:56:13This data engine framework applies to all our signals,

1:56:17whether they're 3D multi-cam video,

1:56:19whether the data is human labeled, auto labeled,

1:56:22or simulated, whether it's an offline model

1:56:24or an online model.

1:56:26And Tesla's able to do this at scale

1:56:29because of the fleet advantage,

1:56:31the info that our end team has built,

1:56:33and the labeling resources that feed our networks.

1:56:36To train on all this data, we need a massive amount of compute.

1:56:40So I'll hand it off to Pete and Ganesh

1:56:42to talk about the Dojo Supercomputing Platform.

1:56:45Thank you.

1:56:46Thank you, Katie.

1:56:55Thanks, everybody.

1:56:55Thanks for hanging in there.

1:56:56We're almost there.

1:56:59My name is Pete Bannon.

1:57:00I run the custom silicon and low voltage teams at Tesla.

1:57:05And my name is Ganesh Wrenke.

1:57:06I run the Dojo program.

1:57:14Thank you.

1:57:15I'm frequently asked, why is a car company building a supercomputer for training?

1:57:21And this question fundamentally misunderstands the nature of Tesla.

1:57:27At its heart, Tesla is a hardcore technology company.

1:57:31All across the company, people are working hard in science and

1:57:35engineering to advance the fundamental understanding and

1:57:40methods that we have available to build cars, energy solutions, robots,

1:57:45than anything else, so can we can do

1:57:48to improve the human condition around the world.

1:57:51It's a super exciting thing to be a part of,

1:57:53and it's a privilege to run a very small piece of it

1:57:56in the semiconductor group.

1:57:58Tonight we're gonna talk a little bit about Dojo

1:58:01and give you an update on what we've been able to do

1:58:03over the last year, but before we do that,

1:58:05I wanted to give a little bit of background

1:58:07on the initial design that we started a few years ago.

1:58:10When we got started, the goal was to provide

1:58:12a substantial improvement to the training

1:58:15latency for our autopilot team.

1:58:17Some of the largest neural networks they train today

1:58:19run for over a month, which inhibits their ability

1:58:23to rapidly explore alternatives and evaluate them.

1:58:27So a 30x speed up would be really nice

1:58:30if we could provide it at a cost competitive

1:58:33and energy competitive way.

1:58:35To do that, we wanted to build a chip

1:58:38with a lot of arithmetic units that we could utilize

1:58:42at a very high efficiency.

1:58:43And we spent a lot of time studying whether we could do

1:58:46that using DRAM, various packaging ideas, all of which

1:58:51failed.

1:58:52And in the end, even though it felt like an unnatural act,

1:58:54we decided to reject DRAM as the primary storage medium for

1:58:58this system, and instead focus on SRAM embedded in the chip.

1:59:03SRAM provides, unfortunately, a modest amount of capacity,

1:59:06but extremely high bandwidth and very low latency.

1:59:09And that enables us to achieve high utilization with the

1:59:12with medic units. Those choices, that particular choice led to a whole bunch of other choices.

1:59:20For example, if you want to have virtual memory and you need page tables, they take a lot of

1:59:24space. We didn't have space, so no virtual memory. We also don't have interrupts. The accelerator

1:59:31is a bare-bones, raw piece of hardware that's presented to a compiler and the compiler is

1:59:37responsible for scheduling everything that happens in a deterministic way. So there's

1:59:41no need or even desire for interrupts in the system.

1:59:45We also chose to pursue model parallelism as a training methodology, which is not the

1:59:51typical situation.

1:59:53Most machines today use data parallelism, which consumes additional memory capacity, which

1:59:59we obviously don't have.

2:00:01So all of those choices led us to build a machine that is pretty radically different from what's

2:00:08available today.

2:00:10We also had a whole bunch of other goals.

2:00:12One of the most important ones was no limits.

2:00:14So we wanted to build a compute fabric that would scale

2:00:18in an unbounded way, for the most part.

2:00:20I mean, obviously, there's physical limits now and then.

2:00:23But pretty much if your model was too big for the computer,

2:00:26you just had to go buy a bigger computer.

2:00:29That's what we were looking for.

2:00:31Today, the way machines are packaged.

2:00:33There's a pretty fixed ratio of, for example, GPUs, CPUs,

2:00:37and DRAM capacity and network capacity.

2:00:40And we really wanted to disaggregate all that

2:00:43so that as models evolved, we could vary the ratios

2:00:47of those various elements and make the system more flexible

2:00:50to meet the needs of the autopilot team.

2:00:54Yeah, and it's so true, Pete, like no limits philosophy

2:00:58was our guiding star all the way.

2:01:00All of our choices were centered around that.

2:01:04And to the point that we didn't want traditional data center

2:01:08infrastructure to limit our capacity

2:01:11to execute these programs at speed.

2:01:15So that's why we integrated vertically our data center.

2:01:25The entire data center, by doing a vertical integration

2:01:29of the data center, we could extract new levels of efficiency.

2:01:33we could optimize power delivery, cooling,

2:01:37and as well as system management across the whole data

2:01:41center stack, rather than doing box by box

2:01:44and integrating that those boxes into data centers.

2:01:49And to do this, we also wanted to integrate early

2:01:55to figure out limits of scale for our software workloads.

2:01:59So we integrated Dojo environment

2:02:01into our autopilot software very early.

2:02:04And we learned a lot of lessons.

2:02:06And today, Bill Chang will go over our hardware update,

2:02:11as well as some of the challenges that we faced along the way.

2:02:15And Rajeev Kurian will give you a glimpse

2:02:18of our compiler technology, as well as go over some

2:02:23of our cool results.

2:02:24Great.

2:02:25Thank you.

2:02:31Thanks, Pete.

2:02:32Thanks, Ganesh.

2:02:34I'll start tonight with a high-level vision of our system.

2:02:38That will help set the stage for the challenges

2:02:42and the problems we're solving.

2:02:44And then also how software will then

2:02:46leverage this for performance.

2:02:49Now, our vision for Dojo is to build a single unified

2:02:52accelerant, a very large one.

2:02:55Software would see a seamless compute plane

2:02:58with globally addressable, very fast memory,

2:03:01all connected together with uniform, high bandwidth, and low latency.

2:03:09Now to realize this, we need to use density to achieve performance.

2:03:14Now we leverage technology to get this density in order to break levels of hierarchy all

2:03:19the way from the chip to the scale-out systems.

2:03:24Now, Silicon technology has done this for decades.

2:03:28ships had followed Moore's law for density integration

2:03:33to get performance scaling.

2:03:37Now, a key step in realizing that vision

2:03:39was our training tile.

2:03:41Not only can we integrate 25 dies at extremely high bandwidth,

2:03:46but we can scale that to any number of additional tiles

2:03:49by just connecting them together.

2:03:54Now, last year, we showcased our first functional training

2:03:57tile.

2:03:58And at that time, we already had workloads running on it.

2:04:03And since then, the team here has been working hard and

2:04:07diligently to deploy this at scale.

2:04:11Now, we've made amazing progress and had a lot of milestones

2:04:14along the way.

2:04:15And of course, we've had a lot of unexpected challenges.

2:04:19But this is where our fail-fast philosophy has allowed us

2:04:23to push our boundaries.

2:04:27Now, pushing density for performance presents all new

2:04:30challenges.

2:04:32One area is power delivery.

2:04:35Here, we need to deliver the power to our compute die,

2:04:38and this directly impacts our top line compute performance.

2:04:43But we need to do this at unprecedented density.

2:04:46We need to be able to mash our die pitch

2:04:49with a power density of almost 1 amp per millimeter squared.

2:04:54And because of the extreme integration,

2:04:56this needs to be a multi-tiered vertical power solution.

2:05:01And because there's a complex heterogeneous material stack up,

2:05:04we have to carefully manage the material transition,

2:05:08especially CTE.

2:05:12Now why does the coefficient of thermal expansion

2:05:14matter in this case?

2:05:16CTE is a fundamental material property.

2:05:20And if it's not carefully managed,

2:05:22that stack up would literally rip itself apart.

2:05:27So we started this effort by working with vendors

2:05:30to develop this power solution.

2:05:34but we realized that we actually had to develop this in-house.

2:05:38Now to balance schedule and risk, we built quick iterations

2:05:43to support both our system bring up and software development,

2:05:47and also to find the optimal design and stack up

2:05:50that would meet our final production goals.

2:05:53And in the end, we were able to reduce CTE over 50%

2:05:58and meet our performance by 3x over our initial version.

2:06:03Now needless to say, finding this optimal material stack up

2:06:07while maximizing performance at density

2:06:10is extremely difficult.

2:06:14Now, we did have unexpected challenges along the way.

2:06:18Here's an example where we pushed the boundaries

2:06:20of integration that led to component failures.

2:06:25This started when we scaled up to larger and longer workloads

2:06:29and then intermittently, a single site on a tile would fail.

2:06:35Now they started out as recoverable failures,

2:06:38but as we pushed some much higher and higher power,

2:06:41these would become permanent failures.

2:06:45Now to understand this failure,

2:06:47you have to understand why and how we build our power modules.

2:06:52Solving density at every level

2:06:55is the cornerstone of actually achieving

2:06:58our system performance.

2:07:00Now because our X, Y plane is used

2:07:02for high bandwidth communication,

2:07:04Everything else must be stacked vertically.

2:07:08This means all other components other than our die

2:07:11must be integrated into our power modules.

2:07:14Now that includes our clock and our power supplies,

2:07:17and also our system controllers.

2:07:21Now in this case, the fairies were

2:07:22due to losing clock output from our oscillators.

2:07:27And after an extensive debug, we found

2:07:30that the root cause was due to vibrations on the module

2:07:33from piezoelectric effects are nearby capacitors.

2:07:39Now, singing caps are not a new phenomenon,

2:07:42and in fact, very common in power design.

2:07:45But normally, clock chips are placed

2:07:47in a very quiet area of the board,

2:07:49and often not affected by power circuits.

2:07:52But because we needed to achieve this level of integration,

2:07:56these oscillators need to be placed in very close proximity.

2:08:00Now, due to our switching frequency

2:08:02And then the vibration resonance created,

2:08:05it caused out-of-plane vibration on our MEMS oscillator

2:08:09that caused it to crack.

2:08:13Now, the solution to this problem is a multi-prong approach.

2:08:16We can reduce the vibration by using soft terminal caps.

2:08:22We can update our MEMS part with a lower Q factor

2:08:26for the out-of-plane direction.

2:08:30And we can also update our switching frequency

2:08:33to push the resonance further away

2:08:35from these sensitive bands.

2:08:40Now, addition to the density at the system level,

2:08:44we've been making a lot of progress

2:08:46at the infrastructure level.

2:08:49We knew that we had to read, examine every aspect

2:08:52of the data center infrastructure in order

2:08:54to support our unprecedented power and cooling density.

2:08:59We brought in a fully custom design CDU

2:09:02to support Dojo's dense cooling requirements.

2:09:06And the amazing part is we're able to do this

2:09:08at a fraction of the cost versus buying off the shelf

2:09:11and modifying it.

2:09:13And since our Dojo Cabinet integrates enough power and cooling

2:09:16to match an entire row of standard IT racks,

2:09:21we need to carefully design our Cabinet and infrastructure

2:09:24together.

2:09:26And we've already gone through several iterations

2:09:28of this Cabinet to optimize this.

2:09:32And earlier this year, we started

2:09:33load testing our power and cooling infrastructure.

2:09:36and we were able to push it over two megawatts

2:09:39before we tripped our substation

2:09:40and got a call from the city.

2:09:46Now last year, we introduced only a couple

2:09:48of components of our system.

2:09:50The custom D1 die and the training tile,

2:09:53but we teased the exit pod as our end goal.

2:09:57We'll walk through the remaining parts of our system

2:09:59that are required to build out this exit pod.

2:10:04Now the system tray is a key part of realizing our vision

2:10:07of a single accelerator.

2:10:09It enables us to seamlessly connect tiles together, not only within the cabinet, but

2:10:15between cabinets.

2:10:17We can connect these tiles at very tight spacing across the entire accelerator, and

2:10:23this is how we achieve our uniform communication.

2:10:27This is a laminated bus bar that allows us to integrate very high power, mechanical and

2:10:32thermal support, and an extremely dense integration.

2:10:37It's 75 millimeters in height and supports six tiles at 135 kilograms.

2:10:43This is the equivalent of three to four fully loaded high performance racks.

2:10:49Next, we need to feed data to the training tiles.

2:10:55This is where we've developed the Dojo interface processor.

2:10:59It provides our system with high bandwidth DRAM to stage our training data.

2:11:05And it provides full memory bandwidth to our training tiles using TTP, our custom protocol

2:11:11that we can use to communicate across our entire accelerator.

2:11:15It also has high speed ethernet that helps us extend this custom protocol over standard

2:11:20ethernet.

2:11:22And we provide native hardware support for this with little to no software overhead.

2:11:27And lastly, we can connect to it through a standard Gen4 PCIE interface.

2:11:36Now we pair 20 of these cards per tray,

2:11:39and that gives us 640 gigabytes of high bandwidth DRAM.

2:11:43And this provides our disaggregated memory layer

2:11:46for our training tiles.

2:11:48These cards are a high bandwidth ingest path,

2:11:51both through PCIe and ethernet.

2:11:55They also provide a high-rate XZ connectivity path

2:11:58that allows shortcuts across our large dojo accelerator.

2:12:04Now we actually integrate the host directly

2:12:07underneath our system tray.

2:12:09These hosts provide our ingest processing and connect to

2:12:13our interface processors through PCIe.

2:12:16These hosts can provide hardware video decoder support

2:12:21for video-based training.

2:12:24And our user applications land on these hosts.

2:12:28So we can provide them with a standard x86 Linux environment.

2:12:35Now we can put two of these assemblies into one cabinet

2:12:39and pair it with redundant power supplies

2:12:41that do direct conversion of three phase 480 volt AC power

2:12:46to 52 volt DC power.

2:12:52Now by focusing on density at every level,

2:12:56we can realize the vision of a single accelerator.

2:13:00Now starting with the uniform nodes on our custom D1 die,

2:13:04we can connect them together in our fully integrated training

2:13:07tile, and then finally seamlessly connecting them

2:13:12across cabinet boundaries to form our dojo accelerator.

2:13:17And altogether, we can house two full accelerators

2:13:21in our exit pod for a combined one exit

2:13:23flop of ML Compute.

2:13:26Now, all can altogether this amount of technology

2:13:29and integration has only ever been done a couple of times

2:13:33in the history of Compute.

2:13:35Next, we'll see how software can leverage this

2:13:38to accelerate their performance.

2:13:48Thanks, Bill.

2:13:49My name is Rajiv, and I'm going to talk some numbers.

2:13:53So our software stack begins with the PyTorch extension

2:13:56that speaks to our commitment to one's standard PyTorch

2:13:59models out of the box.

2:14:01We're going to talk more about our JIT compiler

2:14:03and the Injast pipeline that feeds the hardware with data.

2:14:07Abstractly, performance is tops times utilization times

2:14:11accelerator occupancy.

2:14:14We've seen how the hardware provides peak performance.

2:14:16It's the job of the compiler to extract utilization

2:14:18from the hardware while code is running on it.

2:14:22And it's the job of the ingest pipeline

2:14:23to make sure that data can be fed at a throughput high enough

2:14:26for the hardware to not ever start.

2:14:30So let's talk about why communication bound models are

2:14:32difficult to scale.

2:14:34But before that, let's look at why

2:14:35resonant 50-like models are easier to scale.

2:14:38You start off with a single accelerator,

2:14:40run the forward and backward passes, followed by the optimizer.

2:14:44Then to scale this up, we run multiple copies of this

2:14:47on multiple accelerators.

2:14:48And while the gradients produced by the backward pass

2:14:50do need to be reduced and this introduces some communication,

2:14:54this can be done pipeline with the backward pass.

2:15:00This setup scales fairly well, almost lenient.

2:15:05For models with much larger activations,

2:15:07we run into a problem as soon as we

2:15:09want to run the forward pass.

2:15:10The batch size that fits in a single accelerator

2:15:13is often smaller than the batch norm surface.

2:15:15So to get around this, researchers

2:15:17typically want this set up on multiple accelerators

2:15:20in sync batch norm mode.

2:15:21This introduces latency bound communication

2:15:23to the critical path of the forward pass,

2:15:26and we already have a communication bottleneck.

2:15:29And while there are ways to get around this,

2:15:31they usually involve tedious manual work best suited

2:15:34for a compiler.

2:15:36And ultimately, there's no skirting around the fact

2:15:38that if your state does not fit in a single accelerator,

2:15:41you can be communication about.

2:15:45And even with significant efforts from our ML engineers,

2:15:48we see such models don't scale linearly.

2:15:52The Dojo system was built to make such models work

2:15:55at high utilization.

2:15:57The high density integration was built

2:15:59to not only accelerate the compute bond portions

2:16:01of a model, but also the latency bound portions

2:16:05like a batch norm or the bandwidth bound portions

2:16:08like a gradient all reduced or a parameter all gathered.

2:16:11A slice of the dojo mesh can be carved out to run any model.

2:16:17The only thing users need to do is to make the slice large enough to fit a bashroom surface

2:16:22for their particular model.

2:16:25After that, the partition presents itself as one large accelerator, freeing the users

2:16:30from having to worry about the internal details of execution, and as the job of the compiler

2:16:36to maintain this abstraction.

2:16:39membrane synchronization primitives in uniform low latency, makes it easy to accelerate all

2:16:44forms of parallelism across integration boundaries. Tensors are usually stored sharded in SRAM

2:16:49and replicated just in time for a Layers execution. We depend on the high dojo bandwidth to

2:16:55hide this replication time. Tensor replication and other data transfers are overlap with compute

2:17:01and the compiler can also recompute Layers when it's profitable to do so. We expect most

2:17:08models to work out of the box.

2:17:10As an example, we took the recently released stable diffusion

2:17:13model and got it running on Dojo in minutes.

2:17:16Out of the box, the compiler was able to map it in a model

2:17:19parallel manner on 25 Dojo dies.

2:17:22Here are some pictures of a Cybertruck on Mars generated

2:17:26by a stable diffusion running on Dojo.

2:17:28It looks like it still has some ways

2:17:37to go before matching the Tesla Design Studio team.

2:17:42So we've talked about how communication bottlenecks

2:17:44can hamper scalability.

2:17:46Perhaps an asset test of a compiler

2:17:48and the underlying hardware is executing

2:17:50a cross-di-bashram layer.

2:17:52Like mentioned before, this can be a serial bottleneck.

2:17:55The communication phase of a bashram

2:17:57begins with nodes computing their local mean

2:18:00and standard deviations.

2:18:01Then coordinating to reduce these values,

2:18:03then broadcasting these values back,

2:18:06and then they resume their work in parallel.

2:18:09So what would an ideal bashram look like on 25 dojo dots?

2:18:13Let's say the previous list activations are already split

2:18:16across dyes.

2:18:19We would expect the 350 nodes on each diode to coordinate

2:18:22and produce dilocle mean and standard deviation values.

2:18:26Ideally, these would get further reduced with the final value

2:18:29ending somewhere towards the middle of the tile.

2:18:33We would then hope to see a broadcast of this value

2:18:35radiating from the center.

2:18:38Let's see how the compiler actually

2:18:39executes a real bash-room operation across 25 dyes.

2:18:43The communication trees were extracted from the compiler,

2:18:46and the timing is from a real hardware one.

2:18:49We're about to see 8,750 nodes on 25 dies coordinating

2:18:53to reduce and then broadcast the bash drum mean

2:18:56and standard deviation values.

2:18:59Dialocal reduction followed by global reduction

2:19:03towards the middle of the tie.

2:19:05Then the reduce value broadcast radiating from the middle

2:19:10accelerated by the hardware's broadcast facility.

2:19:15This operation takes only five microseconds on 25 dojo dice.

2:19:20The same operation takes 150 microseconds on 24 GPUs.

2:19:24This is an orders of magnitude improvement over GPUs.

2:19:28And while we talked about an already

2:19:30this operation in the context of a batch norm,

2:19:32it's important to reiterate that the same advantages apply

2:19:35to all other communication primitives.

2:19:38And these primitives are essential for large scale training.

2:19:43So how about full model performance?

2:19:45So while we think that the ResNet 50 is not a good representation of real world Tesla workloads,

2:19:50it is a standard benchmark, so let's start there.

2:19:54We are already able to match the A100 die for die.

2:19:57However, perhaps a hint of Dojo's capabilities is that we're able to hit this number with

2:20:02just a batch of eight per die.

2:20:04But Dojo was really built to tackle larger complex models.

2:20:09So when we set out to tackle real world workloads,

2:20:11we looked at the usage patterns of our current GPU cluster.

2:20:15And two models stood out, the auto labeling networks,

2:20:17a class of offline models that are used to generate ground

2:20:20truth, and the occupancy networks that you heard about.

2:20:24The auto labeling networks are large models

2:20:26that have high arithmetic intensity,

2:20:28while the occupancy networks can be in just balance.

2:20:31We chose these models because together they

2:20:33account for a large chunk of our current GPU cluster usage,

2:20:36and they would challenge the system in different ways.

2:20:42So how do we do on these two networks?

2:20:44The results we're about to see were measured on multi-die systems

2:20:47for both the GPU and dojo, but normalized to per-die numbers.

2:20:52On our auto labeling network, we're already

2:20:55able to surpass the performance of an A100

2:20:57with our current hardware running on our older generation VRMs.

2:21:01On our production hardware with our newer VRMs,

2:21:04that translates to doubling the throughput of an A100.

2:21:08And our model showed that with some key compiler optimizations,

2:21:11we could get to more than 3x the performance of an A100.

2:21:15We see even bigger leaps on the occupancy network.

2:21:20Almost 3x with our production hardware with room for more.

2:21:33So what does that mean for Tesla?

2:21:35With a current level of compiler performance,

2:21:37we could replace the ML compute of 1, 2, 3, 4, 5,

2:21:43in six GPU boxes with just a single dojo tile.

2:21:55And this dojo tile costs less than one of these GPU boxes.

2:21:59Woo!

2:22:04What it really means is that networks that

2:22:06took more than a month to train now take less than a week.

2:22:13Alas, when we measured things, it did not turn out so well.

2:22:17At the PyTorch level, we did not see our expected performance

2:22:19out of the gate.

2:22:21And this timeline chart shows our problem.

2:22:23The teeny, tiny little green bars,

2:22:25that's the compile code running on the accelerator.

2:22:29The row is mostly white space where the hardware is just

2:22:31waiting for data.

2:22:36With our dense ML compute, Dojo hosts effectively

2:22:39have 10x more ML compute than the GPU hosts.

2:22:42The data loaders running on this one host simply

2:22:44can keep up with all that ML hardware.

2:22:48So to solve our data loader scalability issues,

2:22:51we knew we had to get over the limit of this single host.

2:22:54The Tesla transport protocol moves data seamlessly

2:22:57across host, tiles, and ingest processors.

2:23:00So we extended the Tesla transport protocol

2:23:02to work over ethernet.

2:23:04We then built the Dojo network interface

2:23:06card, the DINIC, to leverage TTP over ethernet.

2:23:09This allows any host with a DINIC card

2:23:12to be able to DM it to and from other TTP endpoints.

2:23:17So we started with the Dojo mesh.

2:23:19Then we added a tier of data loading

2:23:21host equipped with the DINIC card.

2:23:26We connected these hosts to the mesh via an Ethernet switch.

2:23:29Now, every host in this data loading tier

2:23:31is capable of reaching all TTPN points in the Dojo mesh

2:23:34via hardware accelerated DMA.

2:23:39After these optimizations went in, our occupancy

2:23:42went from 4% to 97%.

2:23:46So the data loading sections have reduced.

2:23:53The data loading sections have reduced drastically,

2:23:56and the ML hardware is kept busy.

2:23:58We actually expect this number to go to 100% pretty soon.

2:24:01After these changes went in, we saw the full expected speed up

2:24:05from the PyTorch layer, and we were back in business.

2:24:10So we started with hardware design that breaks through

2:24:12traditional integration boundaries in service of our vision

2:24:15of a single giant accelerator.

2:24:17We've seen how the compiler and Injust layers build on top

2:24:20of that hardware.

2:24:22So after proving our performance on these complex real world

2:24:24networks, we knew what our first large-scale deployment

2:24:27would target.

2:24:28are high-rhythmic intensity auto labeling networks.

2:24:33Today, that occupies 4,000 GPUs over 72 GPU racks.

2:24:37With our dense computer and our high performance,

2:24:40we expect to provide the same throughput with just

2:24:43four Dojo cabinets.

2:24:54And these four Dojo cabinets will

2:24:55be part of our first exapod that we

2:24:57plan to build by quarter one of 2023.

2:25:00This will more than double Tesla's auto labeling capacity.

2:25:11The first exapod is part of a total of seven exapods that we plan to build in Palo Alto

2:25:16right here across the wall.

2:25:21We have a display cabinet from one of these exapods for everyone to look at.

2:25:28Six tiles, density packed on a tray, 54 petaflops of compute, 640 gigabytes of high bandwidth

2:25:34memory with power and host defeated.

2:25:39A lot of computing.

2:25:44We're building out new versions of all our cluster components and constantly improving

2:25:49our software to hit new limits of skill. We believe that we can get another 10X improvement

2:25:54with our next generation hardware. And to realize our ambitious goals, we need the best

2:26:01software and hardware engineers. So please come talk to us or visit tesla.com. Thank

2:26:06you.

2:26:26All right. So hopefully that was enough detail. And now we can move to questions. And guys,

2:26:39guys, like I think the team come out on stage.

2:26:43And we really wanted to show the depth and breadth of Tesla

2:26:50and artificial intelligence, compute hardware, robotics

2:26:56actuators, and try to really shift the perception of the

2:27:01company away from a lot of people think we're just a car

2:27:06company, or we make cool cars, whatever.

2:27:08But they don't have, most people have no idea that Tesla is

2:27:14arguably the leader in real world AI, hardware, and software.

2:27:20And that we're building what is arguably the most radical

2:27:28computer architecture since the query one supercomputer.

2:27:32And I think if you're interested in developing some of

2:27:36the most advanced technology in the world, that's going to

2:27:38really affect the world in a positive way,

2:27:41tells us the place to be.

2:27:44So yeah, let's fire away with some questions.

2:27:48I think there's a mic at the front and a mic at the back.

2:27:56Or on this side.

2:27:57Just throw a mic to people.

2:28:00Jump ball for the mic.

2:28:03Yeah, hi, thank you very much.

2:28:05I was impressed here.

2:28:08Yeah, I was impressed very much by Optimos,

2:28:11but I wonder why Ted on driven the hunt.

2:28:14Why did he choose a tendled driven approach for the hunt?

2:28:18Because tendons are not very durable.

2:28:21And why spring loaded?

2:28:24Yeah.

2:28:26Hello?

2:28:26Is this great?

2:28:27Cool.

2:28:27Awesome.

2:28:28Yes, that's a great question.

2:28:30When it comes to any type of actuation scheme,

2:28:32there's trade-offs between whether or not

2:28:34it's a tendent driven system or some type of linkage-based

2:28:37system.

2:28:38Just keep the mic close to your mouth.

2:28:39A little bit closer.

2:28:40You hear me?

2:28:41Cool.

2:28:42So, yeah, the main reason why we went for a tendon-based system is that, you know, first

2:28:48we actually investigated some synthetic tendons, but we found that metallic boating cables

2:28:52are, you know, a lot stronger.

2:28:56One of the advantages of these cables is that it's very good for part reduction.

2:29:01We do want to make a lot of these hands, so having a bunch of parts, a bunch of small

2:29:04linkages ends up being, you know, a problem when you're making a lot of something.

2:29:10One of the big reasons that tendons are better than linkages in a sense is that you can be

2:29:16anti-backlash.

2:29:18Anti-backlash essentially allows you to not have any gaps or stuttery motion in your

2:29:24fingers.

2:29:26Spring-loaded, mainly what spring-loaded allows us to do is it allows us to have active opening.

2:29:33Instead of having to have two actuators to drive the fingers closed and then open, we

2:29:38We have the ability to have the tendon drive them close, and then the springs passively

2:29:42extend.

2:29:44And this is something that's seen in our hands as well, right?

2:29:46We have the ability to actively flex, and then we also have the ability to extend.

2:29:50Yeah.

2:29:51I mean, our goal with Optimus is to have a robot that is maximally useful as quickly

2:29:57as possible.

2:29:59So there's a lot of ways to solve the various problems of a humanoid robot.

2:30:04And we're probably not barking up the right tree on all the technical solutions.

2:30:09And I should say that we're open to evolving the technical solutions that you see here over

2:30:14time.

2:30:15They're not locked in stone.

2:30:17But we have to pick something.

2:30:20And we want to pick something that's going to allow us to produce the robot as quickly

2:30:25as possible and have it, like I said, be useful as quickly as possible.

2:30:29We're trying to follow the goal of fastest path to a useful robot that can be made at

2:30:35volume.

2:30:36And we're going to test the robot internally at Tesla in our factory.

2:30:41And just see, like how useful is it?

2:30:44Because you have to have a, you're going to close the loop on reality to confirm that

2:30:48the robot is in fact useful.

2:30:52And yeah, so we're just going to use it to build things.

2:30:57And we're confident we can do that with the hand that we have currently designed.

2:31:01But for sure, there'll be hand version two, version three.

2:31:04And we may change the architecture quite significantly over time.

2:31:15The Optimus robot is really impressive.

2:31:17You did a great job.

2:31:19Bipedal robots are really difficult.

2:31:22But what I noticed might be missing from your plan is to acknowledge the utility of the human

2:31:29spirit.

2:31:30wondering if Optimus will ever get a personality and be able to laugh at our jokes while it

2:31:36folds our clothes.

2:31:38Yeah, absolutely.

2:31:40I think we want to have really fun versions of Optimus, so that Optimus can both do be

2:31:49utilitarian and do tasks, but can also be kind of like a friend and a buddy and hang out

2:31:57with you and I'm sure people will think of all sorts of creative uses for this robot.

2:32:05And you know, the thing, once you have the core intelligence and actuators figured out,

2:32:12then you can actually, you know, put all sorts of costumes, I guess, on the robot.

2:32:21I mean, you can make the robot look, you can skin the robot

2:32:26in many different ways.

2:32:29And I'm sure people will find very interesting ways

2:32:32to versions of Optimus.

2:32:40Thanks for the great presentation.

2:32:42I wanted to know if there was an equivalent to interventions

2:32:46in Optimus.

2:32:47It seems like labeling through moments where humans disagree

2:32:50with what's going on is important.

2:32:51And in a humanoid robot, that might

2:32:54be also a desirable source of information.

2:33:01That's what I was saying.

2:33:05Yeah, I think we will have ways to remote operate the robot

2:33:09and intervene when it does something bad,

2:33:12especially when we are training the robot and bringing it up.

2:33:15And hopefully, we designed it in a way

2:33:17that we can stop the robot from, if it's going to hit something,

2:33:21we can just hold it and then it will stop.

2:33:22don't like you know crush your hand or something and those are all intervention

2:33:25data yeah and we can learn a lot from our simulation systems to where we can

2:33:30check for collisions and super ways that those are bad actions yeah I mean so

2:33:35optimist we want over time to for it to be you know an android kind of android

2:33:41that you've seen in in sci-fi movies like Star Trek the next generation like

2:33:46data but obviously we could program the robot to be less robot-like and more

2:33:51friendly and it can obviously learn to emulate humans and feel very natural.

2:33:58So as AI in general improves, we can add that to the robot and it should be obviously able

2:34:07to do simple instructions or even intuit what it is that you want.

2:34:13So you could give it a high level instruction and then it can break that down into a series

2:34:17of actions and take those actions.

2:34:20Hi, yeah, it's exciting to think that with the optimus you will think that you can achieve

2:34:31orders of magnitude of improvement in economic output.

2:34:36That's really exciting.

2:34:38And when Tesla started, the mission was to accelerate the advent of renewable energy

2:34:43or sustainable transport.

2:34:45So with the optimus, do you still see that mission being the mission statement of Tesla,

2:34:51Or is it going to be updated with mission to accelerate the advent of infinite abundance

2:34:58or limitless economy?

2:35:01Yeah, it is not strictly speaking.

2:35:07Optimists is not strictly speaking directly in line with accelerating sustainable energy.

2:35:17To the degree that it is more efficient at getting things done than a person, it does,

2:35:22I guess, help with sustainable energy.

2:35:25But I think the mission effectively does somewhat broaden with the advent of Optimus to, you

2:35:32know, I don't know, make in the future awesome.

2:35:35So, you know, I think you look at Optimus and I know about you, but I'm excited to see

2:35:41what Optimus will become.

2:35:43And this is like, if you could tell like any given technology, do you want to see what

2:35:53it's like in a year, two years, three years, four years, five years, ten?

2:35:58I'd say for sure.

2:35:59You definitely want to see what's happening with Optimus.

2:36:03Whereas a bunch of other technologies are sort of plateaued.

2:36:07I've got name names here, but I...

2:36:12You know, so I think often this is going to be incredible in like five years, ten years

2:36:22like mind-blowing.

2:36:24And I'm really interested to see that happen and I hope you are too.

2:36:29Thank you.

2:36:30I have a quick question here.

2:36:32Justin, and I was wondering, like, are you planning to extend like conversational capabilities

2:36:40for the robot.

2:36:41And my second follow-up question to that is,

2:36:44what's the end goal?

2:36:45What's the end goal with Optimus?

2:36:49Yeah, Optimus would definitely have

2:36:50conversational capabilities.

2:36:52So you'd be able to talk to it and have a conversation,

2:36:58and it would feel quite natural.

2:37:00So from an end goal standpoint,

2:37:04I don't know, I think it's gonna keep evolving,

2:37:07And I'm not sure where it ends up, but some place interesting

2:37:13for sure.

2:37:16We always have to be careful about the, don't go down the

2:37:19Terminator path.

2:37:20That's a, I thought maybe we should start off with a video

2:37:25of the Terminator starting off with this skull crushing,

2:37:29but it might be people might not get too seriously.

2:37:32So we do want Optimus to be safe, so we are designing

2:37:38in safeguards where you can locally stop the robot.

2:37:45And with basically a localized control ROM

2:37:50that you can't update over the internet,

2:37:52which I think that's quite important, essential, frankly.

2:37:59So like a localized stop button or remote control,

2:38:05something like that that cannot be changed.

2:38:12But I mean, it's definitely gonna be interesting.

2:38:15It won't be boring.

2:38:22Okay, yeah.

2:38:23I see you today, you have very attractive product

2:38:25with Dojo and its applications.

2:38:28So I'm wondering what's the future for the Dojo platform?

2:38:30We would like provide like infrastructure and service,

2:38:35like AWS or you would like still achieve like the NVIDIA.

2:38:39So basically what's the future?

2:38:40because I say you use a 7 nanometer,

2:38:43so you develop a cost easily over 10 million dollars.

2:38:47How do you make the penis wise?

2:38:51Yeah, I mean, Dojo is a very big computer

2:38:56and actually will use a lot of power

2:38:59and needs a lot of cooling.

2:39:00So I think it's probably gonna make more sense

2:39:02to have Dojo operate in like Amazon Web Services manner

2:39:06than to try to sell it to someone else.

2:39:09So the most, that would be the most efficient way to operate Dojo is just have it be a service

2:39:17that you can use that's available online and that where you can train your models way

2:39:23faster and for less money.

2:39:26And as the world transitions to software 2.0.

2:39:31And that's on the Bingo card.

2:39:36Someone I know it has to know to drink five tequila's.

2:39:41So let's see.

2:39:46Software 2.0 will use a lot of neural net training.

2:39:54So it kind of makes sense that over time as there's more neural net stuff, people will

2:40:01people want to use the fastest, lowest cost neural net training system.

2:40:07So I think there's a lot of opportunity in that direction.

2:40:14Hi.

2:40:15My name is Ali Jahanyan.

2:40:17Thank you for this event.

2:40:19It's very inspirational.

2:40:21My question is, I'm wondering what is your vision for humanity robots that understand

2:40:31our emotions and art and can contribute to our creativity?

2:40:40Well, I think there's, you're already seeing robots

2:40:43that at least are able to generate very interesting art

2:40:48with like Dolly and Dolly II.

2:40:53And I think we'll start seeing AI that can actually

2:40:57generate even movies that have a, that have coherence.

2:41:01like interesting movies and tell jokes.

2:41:03So it's quite remarkable how fast AI is advancing

2:41:10at many companies besides Tesla.

2:41:14We're headed for a very interesting future.

2:41:17And yeah, so you guys want to comment on that?

2:41:21Yeah, I guess the optimal store work

2:41:23can come up with physical art, not just digital art.

2:41:28You can ask for some dance moves in text or voice,

2:41:31and then you can produce those in the future.

2:41:33So it's a lot of physical heart, not just digital art.

2:41:38Oh, yeah.

2:41:38Yeah, computers can absolutely make a physical art.

2:41:40Yeah.

2:41:41A hundred percent.

2:41:42Yeah, like dance, play, soccer, or whatever you.

2:41:45I mean, it needs to get more agile, but over time, for sure.

2:41:51Thanks so much for the presentation.

2:41:53For the Tesla autopilot slides, I noticed

2:41:56that the models that you were using

2:41:58were heavily motivated by language models.

2:42:00And I was wondering what the history of that was and how

2:42:03much of an improvement it gave.

2:42:05I thought that that was a really interesting curious choice

2:42:07to use language models for the lane transitioning.

2:42:11So there are sort of two aspects for why we

2:42:13transitioned to language modeling.

2:42:14So the first talk loud and close to it.

2:42:17OK.

2:42:17It's not time to go right here.

2:42:19OK, got it.

2:42:21Yeah, so the language models help us in two ways.

2:42:24The first way is that it lets us predict lanes that we

2:42:25couldn't have otherwise.

2:42:27As As Schok mentioned earlier, basically, when we predicted

2:42:30lanes in sort of a dense 3D fashion, you can only model certain kinds of lanes.

2:42:35But we want to get those crisscrossing connections inside of intersections.

2:42:38It's just not possible to do that without making it a graph prediction.

2:42:41If you try to do this with dense segmentation, it just doesn't work.

2:42:45Also, the lane prediction is a multimodal problem.

2:42:48Sometimes you just don't have sufficient visual information to know precisely how things look

2:42:52on the other side of the intersection.

2:42:54So you need a method that can generalize and produce coherent predictions.

2:42:59You don't want to be predicting two lanes and three lanes

2:43:01at the same time.

2:43:02You want to commit to one in a general model like these language

2:43:04models provides that.

2:43:10Hi.

2:43:12Hi.

2:43:12My name is Giovanni.

2:43:15Yeah, thanks for the presentation.

2:43:17It's really nice.

2:43:18I have a question for FSD team.

2:43:21So for the neural networks, how do you test,

2:43:27how do you do unit tests, software units tests on that?

2:43:31Do you have like a punch or I don't know,

2:43:33mid-thousands or yes, cases where the neural network

2:43:40that after you train it, you have to pass it

2:43:42before you release it to as a product, right?

2:43:46Yeah, what's your software unit testing strategies

2:43:49for this, basically?

2:43:50Yeah, glad you asked us like a series of tests

2:43:53that we have defined starting from unit test

2:43:55for software itself, but then for the neural network models,

2:43:58we have VAP sets defined where, you know,

2:44:00You can define, if you just have a large test set,

2:44:04that's not enough what we find.

2:44:05We need sophisticated VIP sets for different failure modes.

2:44:10And then we queue it them and grow them

2:44:11over the time of the product.

2:44:12So over the years, we have hundreds of thousands of examples

2:44:17where we have been failing in the past that we have curated.

2:44:20And so for any new model, we test

2:44:22against the entire history of these failures

2:44:25and then keep adding to this test set.

2:44:28On top of this, we have shadow modes

2:44:30where we ship these models in silent to the car,

2:44:32and we get data back on where they are failing or succeeding.

2:44:36And there's an extensive QA program.

2:44:39It's very hard to ship a regression.

2:44:41There's like nine levels of filters before it hits customers.

2:44:44But then we have really good info to make this all efficient.

2:44:49I'm one of the QA testers.

2:44:50So I have QA the car like a.

2:44:53QA tester.

2:44:54Yeah.

2:44:55So I'm constantly in the car just being

2:44:58queuing whatever the latest alpha-bulb is that doesn't totally crash.

2:45:03It finds a lot of bugs.

2:45:05Yeah.

2:45:06Hi.

2:45:07Great event.

2:45:08I have a question about foundational models for autonomous driving.

2:45:13We have all seen that big models that really can, when you scale up with data and model

2:45:20parameter, right, from GPT-3 to Palm, it can actually now do reasoning.

2:45:25So you see that it's essential scaling up

2:45:29foundational models with data and size.

2:45:32And then at least you can get a teacher model, right?

2:45:36That potentially can solve all the problems.

2:45:38And then you distill to a student model.

2:45:42Is that how you see foundational models relevant

2:45:45for a time of training?

2:45:46That's quite similar to our auto labeling model.

2:45:48So we don't just have models that run in the car.

2:45:51We train models that are entirely offline.

2:45:53that are extremely large that can't run in real time on the car.

2:45:57So we just run those offline on the servers,

2:45:59producing really good labels that can then

2:46:03train the online networks.

2:46:05So that's one form of the installation of these teacher

2:46:08student models.

2:46:10In terms of foundation models, we are building some really,

2:46:12really large data sets that are multiple petabytes.

2:46:16And we are seeing that some of these tasks work really well

2:46:19when we have these large data sets, like the kinematics,

2:46:21like I mentioned, we go in all the kinematics out

2:46:24of all the objects and up to the fourth derivative.

2:46:27And people thought we couldn't do detection with cameras.

2:46:29And detection, depth, velocity, acceleration.

2:46:32And imagine how precise these have

2:46:34to be for these high-order derivatives to be accurate.

2:46:37And these all comes from these kind of large data sets

2:46:39and large models.

2:46:42So we're seeing the equivalent of foundation models

2:46:43in our own way for geometry and kinematics and things

2:46:47like those.

2:46:49You want to have anything, John?

2:46:52Yeah, I'll keep it brief.

2:46:53Basically, whenever we train on a larger data set,

2:46:56we see big group of log.

2:46:58OK.

2:46:59Basically, whenever we train on a larger data set,

2:47:01we see big improvements in our model performance.

2:47:03And basically, whenever we initialize our networks

2:47:05with some pre-training step from some other auxiliary task,

2:47:08we basically see improvements.

2:47:10The self-supervised or supervised with large data sets

2:47:13both help a lot.

2:47:17Hi.

2:47:18So at the beginning, Elon said that Tesla was potentially

2:47:21interested in building artificial general intelligence

2:47:24systems given the potentially transformative impact

2:47:27of technology like that, it seems prudent to invest

2:47:30in technical AGI safety expertise specifically.

2:47:34I know Tesla does a lot of technical narrow AI safety

2:47:38research.

2:47:39I was curious if Tesla was intending

2:47:41to try to build expertise in technical, artificial, general

2:47:45intelligence safety specifically.

2:47:48Well, I mean, if we're still looking

2:47:50like we're getting the making a significant contribution

2:47:54to artificial general intelligence,

2:47:56then we'll for sure invest in safety,

2:47:59I'm a big believer in AI safety.

2:48:01I think there should be an AI regulatory authority

2:48:05at the government level, just as there

2:48:08is a regulatory authority for anything

2:48:11that affects public safety.

2:48:12So we have regulatory authority for aircraft and cars

2:48:15and food and drugs because they affect public safety.

2:48:20And AI also affects public safety.

2:48:23So I think, and this is not really something

2:48:25that government understands yet,

2:48:27but I think there should be a referee that is ensuring,

2:48:32or trying to ensure public safety for AGI.

2:48:39And you think of like, well,

2:48:41what are the elements that are necessary to create AGI?

2:48:44Like the accessible data set is extremely important.

2:48:49important. And if you've got a large number of cars and humanoid robots processing petabytes

2:48:59of video data and audio data from the real world, just like humans, that might be the

2:49:07biggest data set. It probably is the biggest data set. Because in addition to that, you

2:49:13can obviously incrementally scan the internet.

2:49:17But what the internet can't quite do

2:49:18is have millions or hundreds of millions

2:49:21of cameras in the real world, and with, like I said,

2:49:26with audio and other senses as well.

2:49:29So I think we probably will have the most amount of data.

2:49:35And probably the most amount of training power,

2:49:39Therefore, probably we will make a contribution to AGI.

2:49:48Hey, I noticed the semi was back there,

2:49:51but we haven't talked about it too much.

2:49:53I was just wondering for the semi truck,

2:49:55what are the changes you're thinking about

2:49:57from a sensing perspective?

2:49:59I imagine there's very different requirements obviously

2:50:01than just a car, and if you don't think that's true,

2:50:04why is that true?

2:50:06I know I think basically you can drive a car,

2:50:10I mean, think about what drives any vehicle.

2:50:12It's a biological neural net with eyes, with cameras,

2:50:18essentially.

2:50:18So what is your primary sensors are two cameras

2:50:27on a slow gimbal, a very slow gimbal.

2:50:30That's your head.

2:50:32So if a biological neural net with two cameras on a slow gimbal

2:50:38can drive a semi-truck, then if you've

2:50:40got like eight cameras with continuous 360 degree vision

2:50:45operating at a higher frame rate and much higher reaction rate,

2:50:48then I think it is obvious that you should be able to drive

2:50:51a semi or any vehicle much better than human.

2:50:56Hi, my name is Akshay.

2:50:58Thank you for the event.

2:51:00Assuming, you know, optimus would be used for different use

2:51:03cases and would evolve at different pace for these use

2:51:07cases, would it be possible to sort of develop and deploy

2:51:12different software and hardware components independently

2:51:16and deploy them in the optimus so that the overall feature

2:51:22development is faster for optimus.

2:51:25I have some questions.

2:51:31We did not comprehend.

2:51:33Unfortunately, our neural net did not

2:51:35comprehend the question.

2:51:39So next question.

2:51:44Hi, I want to switch a gear to the other parrots.

2:51:46So when you guys plan to roll out the FSD beta

2:51:50to countries other than US and Canada,

2:51:53and also my next question is,

2:51:55what's the biggest bottleneck or the technological barrier

2:51:58you think in the current auto-parrotic stack,

2:52:00and how you envision to solve that,

2:52:02to make the auto-parrotic is considerably better

2:52:05than human in terms of performance matrix safety assurance

2:52:09and the human confidence.

2:52:11I think you also mentioned for the FSD B11,

2:52:14you are guys going to combine the highway and the city

2:52:17as a single stack and some architectural big improvement.

2:52:21Can you maybe explain a bit on that?

2:52:23Thank you.

2:52:24Well, that's a whole bunch of questions.

2:52:26Well, we're hopeful to be able to,

2:52:30I think from a technical standpoint,

2:52:33FSD beta should be possible to roll out FSD beta worldwide

2:52:39by the end of this year.

2:52:40But we, you know, for a lot of countries we need regulatory approval and so we are somewhat

2:52:48gated by the regulatory approval in other countries.

2:52:54But I think from a technical standpoint it will be ready to go to a worldwide beta by

2:53:01the end of this year.

2:53:03And there's quite a big improvement that we're expecting to release next month.

2:53:07will be especially good at assessing the velocity of fast-moving cross-traffic and a bunch of

2:53:15other things. So, anyone want to elaborate? Yeah, I guess so. There used to be a lot of

2:53:24differences between production autopilot and the pulse-out driving beta, but those differences

2:53:28have been getting smaller and smaller over time. I think just a few months ago, we now use the

2:53:33same vision-only object detection stack in both FSD and in the production autopilot on

2:53:38on all vehicles.

2:53:40There's still a few differences.

2:53:42The primary one being the way that we predict lanes right now.

2:53:45So we upgraded the modeling of lanes

2:53:46so that it could handle these more complex geometries,

2:53:48like I mentioned in the talk.

2:53:50In production on up high that we still use a simpler lane

2:53:53model, but we're extending our current FSD beta models

2:53:57to work in all sort of highway scenarios as well.

2:54:01Yeah, and the version of FSD beta that I drive actually

2:54:04does have the integrated stack.

2:54:06So it uses the FSD stack both in city streets and highway.

2:54:11And it works quite well for me.

2:54:14We need to validate it in all kinds of weather,

2:54:17like heavy rain, snow, dust.

2:54:20And just make sure it's working better than the production

2:54:25stack in across a wide range of environments.

2:54:30But it went pretty close to that.

2:54:32I mean, I think it's maybe, I'll definitely

2:54:37be before the end of the year and maybe November.

2:54:41Yeah, in our personal drives, the FSD Stack on Highway

2:54:44drives already way better than the production stack we have.

2:54:47And we do expect to also include the parking lot stack

2:54:50as a part of the FSD stack before the end of this year.

2:54:53So that will basically bring us to you,

2:54:56sit in the car in the parking lot,

2:54:58and drive till the end of the parking lot

2:54:59at a parking spot before the end of this year.

2:55:02And in terms of the fundamental metric to optimize against

2:55:07is how many miles per, between a necessary intervention?

2:55:12So just massively improving the how many miles the car can

2:55:18drive in full autonomy before an intervention is required.

2:55:22That is safety critical.

2:55:25So yeah, that's the fundamental metric

2:55:29that we're measuring every week.

2:55:31And we're making radical improvements on that.

2:55:37Hi.

2:55:37Thank you so much for the presentation.

2:55:40Very inspiring.

2:55:42My name is Daisy.

2:55:43I actually have a non-technical question for you.

2:55:46I'm curious if you're back to your 20s, what

2:55:50are some of the things you wish you knew back then?

2:55:52What are some advice you would give to your younger self?

2:56:07Well, I'm trying to figure out something useful to say.

2:56:14Yeah, joined Tesla would be one thing.

2:56:16I think just trying to try to expose yourself to as many smart people as possible.

2:56:28I don't read a lot of books.

2:56:34Now I did do that though.

2:56:37So I think there's some merit to just also not being necessarily too intense and enjoying

2:56:49the moment a bit more, I would say, to 2020 something me.

2:56:54Just to stop and smell the roses occasionally would probably be a good idea.

2:57:02like when we were developing the Falco 1 rocket and on the Kwajaleen Atoll and we had this

2:57:11beautiful little island that we were developing the rocket on and not once during that entire

2:57:16time that I even have a drink on the beach.

2:57:18I'm like, well, I should have had a drink on the beach.

2:57:21That would have been fine.

2:57:26Thank you very much.

2:57:27I think you have excited all of the robotics people with the optimus.

2:57:33This feels very much like 10 years ago in driving, but as driving has proved to be harder

2:57:38than it actually looked 10 years ago, what do we know now that we didn't 10 years ago

2:57:43that would make, for example, AGI on a humanoid come faster?

2:57:49Well, I mean, it seems to me that AGI is advancing very quickly.

2:57:55Hardly a week goes by without some significant announcement.

2:58:00And yeah, I mean, at this point, AI seems to be able to win at almost any rule-based game.

2:58:11It's able to create extremely impressive art, engage in conversations that are very sophisticated,

2:58:25write essays.

2:58:27And these just keep improving.

2:58:31And there's so much more talented people working on AI.

2:58:37And the hardware is getting better.

2:58:38I think AI is on a super, like, a strong exponential curve

2:58:43of improvement, independent of what we do at Tesla.

2:58:49And obviously, we'll benefit somewhat

2:58:51from that exponential curve of improvement with AI.

2:58:57Like Tesla just also happens to be very good at actuators,

2:58:59that motors gearboxes, controllers, power electronics,

2:59:03batteries, sensors.

2:59:07And really, I'd say that the biggest difference

2:59:12between the robot on four wheels and the robot with ons

2:59:15and legs is getting the actuators right.

2:59:19It's an actuators and sensors problem.

2:59:23And obviously, how you control those actuators and sensors.

2:59:26But actuators and sensors and how you control actuators.

2:59:33I don't know, we have to have the ingredients necessary

2:59:36to create a compelling robot, and we're doing it.

2:59:42Hi, Elon.

2:59:45You are actually bringing the humanity to the next level.

2:59:47Literally Tesla and you are bringing the humanity

2:59:50to the next level.

2:59:51So you said optimus prime.

2:59:54Optimus will be used in next Tesla factory.

2:59:57My question is, will a new Tesla factory

3:00:00will be fully run by Optimus program?

3:00:04And when can general public order a humanoid?

3:00:10Yeah, I think it'll start optimists

3:00:13with very simple tests in the factory,

3:00:17like maybe just loading apart, like you saw in the video,

3:00:19loading apart, carrying apart from one place to another,

3:00:25or loading apart into one of our more conventional robot

3:00:30cells that welds the body together.

3:00:35So we'll start just trying to, how

3:00:38do we make it useful at all?

3:00:40And then gradually expand the number of situations

3:00:43where it's useful.

3:00:45And I think that the number of situations

3:00:48where optimists is useful will grow exponentially,

3:00:53really, really fast.

3:00:56In terms of when people can order one, I don't know.

3:00:59I think it's not that far away.

3:01:02Well, I think you mean when can people receive one?

3:01:07So I don't know.

3:01:09I'd say probably within three years, not more than five

3:01:14years, within three to five years,

3:01:16you could probably receive an optimist.

3:01:23I feel the best way to make the progress for AGIs to involve

3:01:26as many smart people across the world as possible.

3:01:29And given the size and resource of Tesla compared

3:01:32to robot companies, and given the state of human order research

3:01:36at the moment, wouldn't make sense for the kind of Tesla

3:01:39to sort of open source form of the assimilation,

3:01:43hardware parts, I think Tesla can still

3:01:45be the dominant platformer, where it can be something

3:01:48like Android OS or like iOS stuff for the entire human

3:01:52or research.

3:01:53Would that be something that, rather than keeping

3:01:55the optimus to just test the researchers or the factory itself, can you open it and let

3:02:01the whole world explore human research?

3:02:10I think we have to be careful about optimus being potentially used in ways that are bad,

3:02:16because that is one of the possible things to do.

3:02:19So I think we would provide optimus where you can provide instructions to optimus, but

3:02:30where those instructions are governed by some laws of robotics

3:02:36that you cannot overcome.

3:02:40So not doing harm to others.

3:02:46It would, I think, probably quite a few safety-related things

3:02:50with the optimist.

3:02:51Yeah.

3:02:53All right, well, we'll just take maybe a few more questions

3:02:55and then thank you all for coming.

3:02:59questions one deep and one broad on the deep for optimists.

3:03:05What's the current and what's the ideal controller bandwidth?

3:03:09And then in the broader question, there's

3:03:11this big advertisement for the depth and breadth

3:03:14of the company.

3:03:15What is it uniquely about Tesla that enables that?

3:03:21Anyone want to tackle the bandwidth question?

3:03:25So the technical bandwidth of the?

3:03:28Close to your mouth and mouth.

3:03:29OK.

3:03:30For the bandwidth question, you have to understand or figure out

3:03:33what is the task that you wanted to do,

3:03:36and what is the frequency transform of that task,

3:03:39what is it that you want your limbs to do?

3:03:41And that's where you get your bandwidth from.

3:03:43It's not a number that you can specifically just say,

3:03:45you need to understand your use case.

3:03:46And that's where the bandwidth comes from.

3:03:50What is the broad question?

3:03:52I don't quite remember.

3:03:54The breadth and depth thing.

3:03:55I can answer the breadth and depth.

3:03:57We'll just be considering.

3:04:05I mean, seriously, on the ban the question,

3:04:07I think we probably will just end up

3:04:09increasing the bandwidth or, you know,

3:04:11it was translates to the effect of dexterity

3:04:16and reaction time of the robot.

3:04:19Like, you could say, it's not 1 hertz.

3:04:23And it's maybe you don't need to go all the way to 100 hertz.

3:04:27But I don't know, maybe 10, 25.

3:04:30I don't know.

3:04:31Over time, I think the bandwidth will increase quite a bit, or translated to dexterity and

3:04:37latency.

3:04:39You'd want to minimize that over time.

3:04:42Yeah.

3:04:43Minimize latency, maximize dexterity.

3:04:48In terms of breadth and depth, I guess we've got, we're a pretty big company at this point.

3:04:55So we've got a lot of different areas of expertise that we necessarily have to develop

3:04:58in order to make electric cars, and then in order to make autonomous electric cars.

3:05:06We've just, I mean Tesla is like a whole series of startups basically.

3:05:11And so far they've almost all been quite successful.

3:05:17So we must be doing something right.

3:05:20And I consider one of my core responsibilities around the company is to have an environment

3:05:26where great engineers can flourish.

3:05:30And I think in a lot of companies,

3:05:33I don't know, maybe most companies,

3:05:35if somebody's a really talented driven engineer,

3:05:38they're unable to actually,

3:05:42their talents are suppressed, at a lot of companies.

3:05:46And it's, you know, and some of the companies

3:05:49that the engineering talent is suppressed

3:05:51in a way that is maybe not obviously bad,

3:05:55but where it's just so comfortable and you paid so much money,

3:05:59and the output you actually have to produce is so low,

3:06:04that it's like a honey trap.

3:06:06So there's a few honey trap places in Silicon Valley.

3:06:10They don't necessarily don't seem like bad places

3:06:13for engineers, but you have to say a good engineer went in,

3:06:17and what did they get out?

3:06:19And the output of that engineering talent

3:06:22It seems very low, even though there

3:06:27seem to be enjoying themselves.

3:06:29That's why I call it the fuel honey trap companies

3:06:31in Silicon Valley.

3:06:32Tesla is not a honey trap.

3:06:34We're demanding, and it's like, you're

3:06:36going to get a lot of shit done, and it's going to be really cool.

3:06:42And it's not going to be easy.

3:06:44But if you are a super talented engineer,

3:06:49your talents will be used, I think, to a greater degree than anywhere else.

3:06:59You know, SpaceX also that way.

3:07:04Hi, I have two questions. So, both to the autopilot team. So, the thing is like,

3:07:10I have been following your progress for the past few years. So, today you have made changes on,

3:07:14like, the lane detection. You said that, like, previously you were doing instance

3:07:18And semantic segmentation now you guys are built transform

3:07:20models for building the lanes.

3:07:23So what are some other common challenges which you guys

3:07:26are facing right now, which you are solving in future

3:07:29as a curious engineer so that we as a researcher

3:07:32can work on those, start working on those.

3:07:34And the second question is, I'm really

3:07:35curious about the data engine.

3:07:38You guys have told a case like where the car is stopped.

3:07:42So how are you finding cases which

3:07:44is very much similar to that from the data which you have?

3:07:47So a little bit more on the data engine would be great.

3:07:49So that's it.

3:07:50OK.

3:07:52I'll start.

3:07:52I'll answer the first question using occupancy network

3:07:55as an example.

3:07:56So what you saw in the presentation

3:07:59did not exist a year ago.

3:08:01So we only spent one year of one time

3:08:04where we shipped more than 12 occupancy network.

3:08:06And you have a one foundation model, actually,

3:08:10to represent the entire physical world around everywhere.

3:08:14And you all were at the condition.

3:08:15It's actually really, really challenging.

3:08:17So only over a year ago, we're kind of like driving a 2D

3:08:21where if there's a wall and then if there's a curb,

3:08:24we kind of represent with the same static edge, which

3:08:27is obviously not ideal.

3:08:30There's a big difference between a curb and a wall.

3:08:32When you drive, you make different choices.

3:08:34So after we realized that we have to go to 3D,

3:08:37we have to basically rethink the entire problem

3:08:40and think about how we address that.

3:08:42So this will be one example of challenges.

3:08:45we have a conquer in the past year.

3:08:51Yeah, to answer the question about how we actually source

3:08:53examples of the tricky stopped cars,

3:08:56there's a few ways to go about this.

3:08:58But two examples are one, we can trigger for disagreements

3:09:01within our signals.

3:09:03So let's say that parked bit flickers between parked

3:09:06and driving.

3:09:07We'll trigger that back.

3:09:08And the second is we can leverage more of the shadow

3:09:10mode logic.

3:09:11So if the customer ignores the car

3:09:13but we think we should stop for it,

3:09:15we'll get that data back too.

3:09:16So these are just different like various trigger logic that allows us to get those data campaigns

3:09:21back.

3:09:25Hi.

3:09:26Sorry, go ahead.

3:09:27Thank you for the amazing presentation.

3:09:29Thanks so much.

3:09:31So there are a lot of companies that are focusing on the AGI problem.

3:09:35And one of the reasons why it's such a hard problem is because the problem itself is so

3:09:39hard to define.

3:09:40Several companies have several different definitions.

3:09:42They focus on different things.

3:09:44So what is Tesla, how is Tesla defining the AGI problem?

3:09:47And what are you focusing on specifically?

3:09:51Well, we're not actually specifically focused on AGI.

3:09:55I'm simply saying that AGI seems likely

3:09:59to be an emergent property of what we're doing,

3:10:04because we're creating the old these autonomous cars

3:10:06and autonomous humanoids that are actually

3:10:11within a truly gigantic data stream that's coming in

3:10:16and being processed, it's by far the most amount of real world

3:10:20data.

3:10:21And, Ade, you can't get by just searching the internet,

3:10:24because you have to be out there in the world.

3:10:26And interacting with people and interacting with the roads

3:10:29and just, Earth is a big place.

3:10:33And reality is messy and complicated.

3:10:36So I think it's sort of like likely to just,

3:10:39it just seems likely to be an emergent property

3:10:41of if you've got tens or hundreds of millions

3:10:45autonomous vehicles and maybe even a comparable number of humanoids, maybe more than that on

3:10:50the humanoid front.

3:10:52Well that's just the most amount of data.

3:10:55And if that video is being processed, it just seems likely that the cars will definitely

3:11:04get way better than human drivers and the humanoid robots will become increasingly indistinguishable

3:11:12from humans, perhaps.

3:11:16And so then, like I said, you have this emergent property of AGI.

3:11:27And arguably, humans collectively are sort of a super intelligence as well, especially

3:11:32as we improve the data rate between humans.

3:11:36I mean, the thing like, that seemed way back in the early days, the internet was like the

3:11:40The internet was like humanity acquiring a nervous system,

3:11:44where now all of a sudden, any one element of humanity

3:11:48could know all of the knowledge of humans

3:11:51by connecting to the internet, almost all the knowledge,

3:11:54certainly a huge part of it, whereas previously,

3:11:57we would exchange information by osmosis.

3:12:00By, like, in order to transfer data,

3:12:03so you would have to write a letter.

3:12:05Someone would have to carry the letter by person

3:12:07to another person and then a whole bunch of things in between and then it was like, yeah,

3:12:15I mean insanely slow when you think about it and even if you were in the Library of Congress

3:12:21you still didn't have access to all the world's information and you certainly could insert

3:12:25it and obviously very few people are in the Library of Congress. So I mean one of the

3:12:32I mean, one of the great sort of equality elements, like the internet has been the most, the biggest

3:12:41equalizer in history in terms of access to information and knowledge.

3:12:48And any student of history, I think, would agree with this because, you know, you go back

3:12:52a thousand years, there were very few books, like, and books would be incredibly expensive,

3:12:58But only a few people knew how to read, and even a small number of people even had a book.

3:13:03Now, look at it, you can access any book instantly, you can learn anything basically for free.

3:13:11It's pretty incredible.

3:13:13So I was asked recently what period of history would I prefer to be at the most?

3:13:26And my answer was right now.

3:13:28This is the most interesting time in history, and I read a lot of history.

3:13:34So let's do our best to keep that going.

3:13:39To go back to one of the other questions I would ask, the thing that's happened over time with

3:13:45respect to Tesla autopilot is that the neural nets have gradually absorbed more and more

3:13:55software. And in the limit, of course, you could simply take the videos as seen by the car

3:14:03and compare those to the steering inputs from the steering wheel and pedals, which are very

3:14:08simple inputs. And in principle, you could train with nothing in between, because that's

3:14:15what humans are doing with a biological neural net. You could train based on video and what

3:14:22trains the video is the moving of the steering wheel in the pedals with no other software

3:14:28in between. We're not there yet, but it's gradually going in that direction. All right,

3:14:37but we have a last question. I think we got a question at the front here. Hello, they're

3:14:45right there. I will do two questions. Fine.

3:14:48Hi, thanks for such a great presentation.

3:14:52Will the question last?

3:14:54OK, cool.

3:14:55With FSD being used by so many people,

3:14:57do you think how do you evaluate the company's risk tolerance

3:15:01in terms of performance statistics?

3:15:03And do you think there needs to be more transparency

3:15:05or regulation from third parties as to what's good enough

3:15:09and defining thresholds for performance

3:15:14across some many miles?

3:15:16Sure.

3:15:17The number one design requirement at Tesla is safety.

3:15:26So, and that goes across the board.

3:15:29So in terms of the mechanical safety of the car, we have the lowest probability of injury

3:15:34of any cars ever tested by the government for just a passive mechanical safety,

3:15:39essentially crash structure and airbags and whatnot.

3:15:44We have the highest rating for active safety as well.

3:15:51I think it's going to get to the point where active safety is so ridiculously good.

3:15:57It's just absurdly better than a human.

3:16:01And then with respect to autopilot, we do publish this,

3:16:06broadly speaking, the statistics on miles driven with cars that have no autonomy

3:16:13Tesla cars with no autonomy with kind of hardware one, hardware two, hardware three, and then

3:16:21the ones that are in FSD beta.

3:16:23And we see steady improvements all along the way.

3:16:27And sometimes there's this dichotomy of, you know, should you wait until the car is like

3:16:34on earth three times safer than a person before deploying any technology.

3:16:39But I think that's actually morally wrong.

3:16:42The point of which you believe that adding autonomy reduces injury and death.

3:16:51I think you have a moral obligation to deploy it.

3:16:54Even though you're going to get sued and blamed by a lot of people, because the people whose

3:16:59lives you saved don't know that their lives are saved.

3:17:03And the people who occasionally die or get injured, they definitely know, or their state

3:17:08does, that it was, you know, whatever there was a problem with autopilot. That's why you

3:17:14have to look at the numbers into total miles driven, how many accidents occurred, how many

3:17:20accidents were serious, how many fatalities, and you know, we've got well over three million

3:17:25cars on the road, so that's a lot of miles driven every day. It's not going to be perfect,

3:17:31What what matters it is that is that it is very clearly safer than not to playing it

3:17:38Yeah, so I think last question. I think yes, so thanks. Well the last question here

3:17:55Okay, hi, so I

3:17:59Do not work on hardware, so maybe the hardware team and you guys can enlighten me

3:18:04why is it required that there be symmetry in the design of optimus? Because humans, we

3:18:13have handedness, right? We use some set of muscles more than others, or a time there is

3:18:19where in terror, right? So maybe you'll start to see some joint failures or some actuator

3:18:25failures more, or a time, I understand that this is extremely pre-stage. Also, we as humans

3:18:32We as humans have based so much fantasy and fiction

3:18:37or superhuman capabilities.

3:18:39Like all of us don't want to walk right over there.

3:18:42We want to extend our arms and we have all these,

3:18:45you know, a lot of fantasy, fantastical designs.

3:18:49So considering everything else that is going on

3:18:53in terms of batteries and intensity of compute,

3:18:57maybe you can leverage all those aspects

3:19:00to coming up with something, well, I don't know,

3:19:04more interesting in terms of the robot that you're building.

3:19:08And I'm hoping you're able to explore those directions.

3:19:13Yeah, I think it would be cool to have, like,

3:19:15make Inspector Gadget real.

3:19:17That would be pretty sweet.

3:19:19So yeah, I mean, right now we just

3:19:22want to make basic humanoid work well.

3:19:25And our goal is fastest path to a useful humanoid robot.

3:19:30I think this will ground us in reality, literally,

3:19:34and ensure that we are doing something useful.

3:19:39Like, one of the hardest things to do

3:19:41is to be useful, to actually, and then

3:19:45to have high utility out of the curve.

3:19:48Like, how many people did you help?

3:19:50And how much help did you provide to each person on average?

3:19:56And then how many people did you help?

3:19:58The total utility.

3:20:00like trying to actually ship useful product that people like to a large number of people

3:20:06is so insanely hard.

3:20:08It boggles the mind.

3:20:09You notice why I can say like, man, there's a hell of a difference between a company that

3:20:13is ship product and one has not ship product.

3:20:15It's a game.

3:20:16This is night and day.

3:20:18And then even once you ship product, can you make the cost, the value of the output worth

3:20:23more than the cost of the input, which is again, insanely difficult, especially with hardware.

3:20:28So but I think over time, I think it's cool to do creative things and have like eight

3:20:34arms and whatever and have different versions and maybe there'll be some hardware like companies

3:20:44that are able to add things to an optimist like maybe we add a power port or something

3:20:51like that or attach them, you can add attachments to your optimist like you can add them to

3:20:55your phone.

3:20:56There could be a lot of cool things that could be done over time and there could be maybe an ecosystem of small companies that or free companies that

3:21:04make add-ons for optimists. So with that,

3:21:11I'd like to thank the team for their hard work. You guys are awesome.

3:21:18And thank

3:21:21Thank you all for coming and for everyone online. Thanks for tuning in.

3:21:25And I think this will be one of those great videos where you can like if you you can fast-forward to the bits that you find most interesting

3:21:33But we try to give you a tremendous amount of detail

3:21:37Literally so you can look at the video at your leisure and you can focus on the parts that you find interesting and skip the other parts

3:21:44So thank you all it's and we'll do this I try to do this every year and we might do a part a monthly podcast even

3:21:51So, but I think it would be great to sort of bring you along for the ride and show you

3:22:00what cool things are happening.

3:22:02And yeah, thank you.

3:22:04All right, thanks.