Robot guidance with cameras: how it works, when to choose 2D or 3D

A robot is excellent at repeating a path, but only when the part is exactly where it is expected to be. A tray that shifts, a conveyor that vibrates or a part that rotates to a different angle is enough to cause a missed pick. A robot guidance camera (vision guided robotics) solves this: the camera tells the robot where the part is and how it is oriented, and the robot adjusts its motion on every cycle. This article explains how a robot guidance system works, when a 2D camera is enough, when you need 3D, and why hand-eye calibration decides the quality of the whole system.

How a robot guidance system works

A robot guidance system has four connected blocks: the camera with its lens and lighting, the image processor (a smart camera or a vision PC), the robot with its controller, and the calibration that lets both sides share one coordinate system. The processing chain repeats on every cycle:

1. Trigger: the robot or PLC signals that a part has entered the camera's field of view.
2. Image capture: a 2D camera captures a flat image; a 3D camera captures a point cloud or a depth map.
3. Part search: the vision software recognises the part from its shape or a trained pattern and calculates position and rotation.
4. Coordinate transform: the result is converted from camera coordinates to robot coordinates using the calibration data.
5. Send to robot: the pick pose is passed over Ethernet, then the robot moves, picks and places.

The whole chain must be stable in time. Communication delay and arm vibration at the moment of capture directly affect pick accuracy, so they belong in the design from the start rather than in commissioning.

2D cameras: enough when the part sits on one plane

A 2D camera produces a flat image and determines X, Y and rotation on that plane. It is compact, fast and cheap to integrate, and it fits when:

• parts sit on a conveyor, tray or fixture at a nearly constant height;
• parts lie flat and do not overlap;
• the contrast between part and background is clear, or can be controlled with lighting;
• the robot needs position and orientation but not height.

Typical examples are picking components from a loosely positioned tray, or picking products that travel on a conveyor tracked by an encoder. For this kind of task a 2D camera with well-designed lighting is usually more stable than a complex 3D setup. Lighting matters as much as the camera: bad lighting destabilises even the best algorithm.

3D cameras: needed when height and pose are unknowns

A 3D camera gives the robot height and tilt as well. Choose 3D when:

• parts are stacked or piled in a bin (bin picking) and the robot has to select the top part;
• parts are tilted or lie in many poses, so the robot needs all six degrees of freedom (X, Y, Z and three rotations) to pick;
• pallet or layer height varies, as in depalletizing;
• surfaces have little texture or match the background colour, so 2D struggles to separate the part but the shape is clear;
• volume, flatness or height must be checked during the pick.

In exchange, 3D needs more data, more careful setup and collision checking between the gripper, the bin and neighbouring parts. To go deeper, see the 3D camera page and the article on the Cognex In-Sight L38 industrial 3D vision system.

Choosing 2D or 3D by pick-and-place task

A practical rule: if 2D solves the task, use 2D, because simpler is more stable. Move to 3D only when height or pose is a variable the robot cannot ignore.

Hand-eye calibration: the step that decides pick accuracy

The camera sees the part in camera coordinates, while the robot works in robot coordinates. Hand-eye calibration creates the transform between the two. If this step is off, a correctly detected part is still picked in the wrong place.

There are two ways to mount the camera, each with its own calibration method:

• Fixed camera (eye-to-hand): the camera is mounted on a frame looking down at the work area. It is stable, less affected by arm vibration, and common for conveyors and bin picking.
• Camera on the robot arm (eye-in-hand): the camera moves with the robot, sees closer and is more flexible across positions, but accuracy depends more on arm vibration and repeatability.

A few principles keep calibration robust:

• use a flat, rigid calibration plate, not a warped printout;
• collect data at many robot poses spread across the real working area;
• fix the camera bracket firmly, because even a slight shift forces recalibration;
• measure the error by touching a known point with the robot before starting production;
• recalibrate after any change of camera, lens or gripper, or after moving the robot.

Integration with the robot: what to agree before commissioning

Vision software can be excellent and the system can still fail if integration is unclear. Agree on the following between the vision, robot and PLC teams from the start:

• Protocol: camera and robot exchange data over Ethernet, using a protocol the robot controller supports. Fix the data format: coordinates, angle, part ID and error status.
• Handshake: who triggers the capture, when the camera returns the result, how the robot acknowledges it, and what happens when no part is found.
• Gripper: the gripper design must match the poses the camera can recognise and the clearance around the part.
• Lighting and environment: ambient light, dust and metal reflections must be handled with suitable lighting, filters and shielding.
• Processing platform: small tasks can run on a smart camera, while heavy 3D tasks usually need a more powerful PC; see smart camera vs PC-based vision.

Before designing, it helps to understand the basics in what is machine vision, since field of view (FOV), resolution and lighting all affect pick accuracy.

Common mistakes in deployment

• Choosing 3D for a task that 2D already covers, which adds complexity and cycle time.
• Choosing 2D for parts whose height varies, which makes picking unstable.
• Ignoring lighting and letting the image change with the time of day.
• Mounting the camera loosely, so calibration error grows over time.
• Testing with clean lab samples instead of real parts with mixed batches, dust and line variation.

POC: validate on real samples before investing

The safest way to decide between 2D and 3D is a POC (proof of concept) on real parts. Prepare:

• representative samples, including difficult ones such as shiny, black, multicolour or reflective parts;
• information about the trays, bins, conveyor or pallets in use;
• robot brand, controller generation and the cycle time you want;
• pick and place tolerance requirements;
• photos or video of the installation area, with the space available for the camera.

BeeVision engineers test your samples, assess whether 2D or 3D fits, and recommend the camera, lens, lighting and robot integration. Results depend on each application, so a POC on real samples is always the first step worth taking. For an overview of the solution, see BeeVision's robot guidance page.

Frequently asked questions

How is camera-guided robotics different from a fixed-path robot?

A fixed-path robot repeats taught coordinates, so the product must always be in the same place. A vision-guided robot reads the real part position on every cycle and adjusts the pick point.

Can I start with 2D and upgrade to 3D later?

Yes, if the robot interface is designed to be open from the beginning. But if height or pose is clearly unknown, assess 3D during the POC to avoid rework.

Is a fixed camera or an arm-mounted camera better?

There is no universal answer. A fixed camera is stable and easier to calibrate, while an arm-mounted camera is more flexible when many positions must be viewed. The choice depends on machine layout and the required accuracy.

When must hand-eye calibration be repeated?

Repeat it when the camera, lens or gripper changes, when the robot is moved, or when pick error starts to grow.

Follow BeeVision on LinkedIn and YouTube for more 3D vision videos. If you are considering robot guidance for your line, send samples for a POC and our engineers will recommend the right setup.