A Deep Dive into End-of-Line Camera Extrinsic Calibration for Autonomous Vehicles

This article develops end-of-line (EOL) camera extrinsic calibration and calibration-board placement for autonomous-vehicle camera rigs, including bird’s-eye-view applications.

There are two linked optimization problems: estimate camera and board poses from measurements, then choose physical board positions and orientations that make the camera estimates more precise. Camera intrinsics are assumed known, and absolute extrinsics require a world-frame anchor.

The presentation follows this order:

1. General calibration problem formulation: variables, coordinate conventions, measurements, priors, and identifiability.
2. Solver: sparse factor-graph bundle adjustment for the pose estimates.
3. Board-position and orientation optimization formulation: camera uncertainty, design variables, constraints, and optimization methods.
4. Experiments from simple to complex: one camera and one board, two-camera cases, a multi-camera workspace, and a vehicle rig.

Geometric preprocessing, ray-plane derivations, and optional design formulations are collected in appendices. The detailed ADMM formulation and solver benchmarks remain in the independent comparison.

1. General Calibration Problem Formulation

For multiple boards and cameras, the observation structure should be defined at the marker-point level. Since each camera may see only a subset of the markers on a board, the primitive observation is a camera-point edge:

A board-camera graph is only a compressed graph obtained after grouping all observed points on the same board. The general model uses known board-frame marker coordinates, measured world-frame marker coordinates when available, and image observations. Reprojection residuals share the corresponding physical board pose; the primitive measurements remain camera-point observations.

First define the estimation problem independently of its solver or the eventual board-placement design.

1.1 Sets and Observation Graph

Let there be $B$ boards and $C$ cameras.

Boards:

Cameras:

Board $b$ has known marker points:

Marker $i$ on board $b$ has known board-frame coordinate:

For a planar checkerboard or marker board, usually

Define the global point set:

For point $p=(b,i)$, define

as the board containing point $p$, and define

The camera-point observation graph is

where

For each observation edge $(c,p)$, the measured image point is

Camera $c$ has known intrinsic matrix

Define the neighborhoods:

1.2 Pose Variables and Coordinate Convention

Use a uniform local-to-world pose convention for both boards and cameras. $T^B_b$ denotes the pose of board $b$, and $T^C_c$ denotes the pose of camera $c$.

For board $b$:

maps board-local coordinates to world coordinates:

Here

Define the Euclidean extraction matrix

Equivalently,

For camera $c$:

maps camera-local coordinates to world coordinates. Therefore, to project a world point into camera $c$, use the inverse camera pose:

where $\bar X_p^W=[(X_p^W)^\top,1]^\top$.

Explicitly,

and

This is consistent with treating both the board and camera variables as local-to-world poses. The usual computer-vision world-to-camera extrinsic still appears, but now it is written explicitly as $(T^C_c)^{-1}$.

1.3 Projection Model

For point $p=(b,i)$, the homogeneous world coordinate is

The camera-coordinate point is

and its Euclidean coordinate is

The projected pixel is

If

then, for a standard pinhole model,

These pinhole equations assume undistorted image coordinates. For fisheye or strongly distorted automotive cameras, replace $\Pi(K_cX)$ by the calibrated camera projection model.

It is useful to define the shorthand

where $\bar X^W=[(X^W)^\top,1]^\top$. The reprojection residual is

If the image measurement covariance is

the whitened residual is

1.4 Board and World Measurement Priors

In the original EOL setup, some board marker coordinates may be measured in the world frame. Let

be the measured world coordinate of point $p$. Define the set of points with world measurements as

The board/world residual is

With covariance

the board prior term is

You may also have camera pose priors, rig priors, or a fixed reference camera. Write those directly on the camera pose. Let $\bar T^C_c$ be the prior or nominal camera pose. The operator

is the Lie-group logarithm that converts a relative pose into a 6D tangent-vector error, usually ordered as three translation components and three rotation components. For example:

1.5 Centralized Calibration Objective

The cleanest problem is:

This is the point-level multi-board, multi-camera bundle-adjustment problem.

The graph edge is $(c,p)$, but the residual depends on

So the raw observation graph is camera-point, while the variable graph is camera-board because each point is rigidly attached to a board.

This shared board-generated point is how multi-camera consistency enters the bundle-adjustment problem. If two cameras $c_1$ and $c_2$ observe the same point $p$, both reprojection residuals use the same world point

Thus the two camera observations are coupled through the same board pose and the same physical marker point, rather than through two independent 3D point variables or an extra pairwise cross-camera residual.

With the uniform local-to-world convention, the board-to-camera relative transform is

This maps board $b$’s local coordinates into camera $c$’s local coordinates, so the projection can also be written as

1.6 Gauge Freedom and Identifiability

Without an absolute anchor, each connected component has an $SE(3)$ gauge freedom.

For any

the transformation

leaves every relative camera-board projection unchanged:

Therefore, every connected component needs at least one of the following:

or

Connectivity alone is not enough. The observations must also be geometrically nondegenerate: enough non-collinear board points, reasonable viewing angles, positive depth, and no nearly rank-deficient configurations.

2. Solver: Sparse Factor-Graph Bundle Adjustment

With the measurement model and gauge fixed, the solver estimates the poses for a given set of observations. It does not yet decide where to move the physical boards.

2.1 Solver Choice

Sparse factor-graph bundle adjustment is the recommended solver for this centralized EOL calibration problem. The detailed ADMM derivation, including point-copy and pose-copy consensus variants, has been moved to ADMM vs. Factor-Graph Bundle Adjustment for Multi-Camera Calibration.

2.2 Factor-Graph Structure

The centralized problem is also a factor graph.

Variables:

Factors:

for every observation edge $(c,p)\in\mathcal E$.

Board priors:

Camera priors:

GTSAM describes factor graphs as bipartite graphs connecting factors to the variables they depend on, and emphasizes that such sparse structure is exploited by nonlinear optimization methods.

This is arguably the cleanest implementation model: one factor per camera-point observation, with the point’s board membership determining which board pose variable the factor touches.

2.3 Sparse Updates and Schur Elimination

Solve the centralized objective using sparse nonlinear least squares over camera and board pose increments.

The problem is exactly:

Use pose increments:

Linearize every residual:

Stack all residuals and solve the damped normal equations:

Each residual touches only one board pose and one camera pose. Therefore the normal matrix has sparse block structure:

If there are no camera-camera priors, $H_{CC}$ is block diagonal over cameras before Schur fill-in. If there are no board-board priors, $H_{BB}$ is block diagonal over boards before Schur fill-in.

You can eliminate boards:

Then back-substitute:

Or eliminate cameras instead, depending on whether there are fewer cameras or fewer boards.

This is the same structural idea behind Schur methods in bundle adjustment. Ceres’ documentation describes bundle adjustment as minimizing reprojection error over cameras and 3D points, and its solver documentation explains that bundle adjustment has a special block structure that can be exploited with Schur complement methods.

2.4 Initialization and Engineering Pipeline

Use this hierarchy:

Best if centralized

Use sparse factor-graph bundle adjustment:

This gives the best local convergence and avoids ADMM penalty tuning. In a centralized EOL station, it is better than ADMM as the production solver; the detailed rationale and benchmark comparison are in ADMM vs. Factor-Graph Bundle Adjustment for Multi-Camera Calibration.

If distribution is mandatory

Use ADMM only when camera-local ownership, communication limits, or distributed computation prevent a centralized solve. The independent comparison linked above contains the complete point-copy and pose-copy formulations.

Best engineering pipeline

1. Initialize board poses from world measurements, if available.
2. Initialize cameras using PnP against current board-generated world points.
3. Run sparse centralized BA if possible.
4. Use ADMM only if the system must be distributed, parallelized by camera/board, or too large for a single optimizer.
5. Optionally use robust losses or outlier rejection on image residuals.

The estimation model can be summarized as follows:

Use this convention consistently:

Everything else, including ADMM, sparse Schur bundle adjustment, and factor-graph optimization, is a solver strategy for this same constrained geometric estimation problem.

3. Board-Position and Orientation Optimization Formulation

The outer optimization chooses the physical position and orientation of each board. Its score comes from the reprojection measurements and priors in (P), not from numerical damping or solver-specific penalties.

The objective value itself is not a good placement guide. For a hypothetical board placement with perfectly predicted observations, the expected reprojection residual is zero almost everywhere. The useful placement information is in the local curvature of the objective: the reprojection Jacobian and the normal matrix or information matrix.

3.1 Design Variables and Lower-Level Calibration

Board placement is an outer design problem around the calibration estimator. The important distinction is between the commanded physical placement of a board and the pose estimated inside calibration:

Let

be the board-placement design, and let

be the lower-level calibration variables. For a fixed placement design $s$, the measurement graph, visibility, image uncertainty, and marker projections all depend on where the boards are placed. Write the visible camera-marker edge set as $\mathcal E(s)$.

The lower-level calibration problem is the same joint bundle-adjustment problem used throughout this article:

with

Here $\beta(p)$ maps marker point $p$ to its board, and $\bar q_p$ is the homogeneous board-frame marker coordinate. The term $\phi_b(T_b^B;S_b)$ connects the estimated board pose to the commanded physical placement. A pose-prior model is

If board marker corners are surveyed in the world frame, the same idea can be written as a world-point prior:

3.2 Placement-Dependent Measurements

The placement design $s$ affects the calibration problem through the visibility graph, the projected corner locations, the corner uncertainty, and the observed measurement graph. For a candidate placement $s$, define the predicted projection

The image measurement model is

The covariance can depend on placement:

The observation edge set is also placement-dependent:

A typical deterministic visibility condition is

where $a_{cp}(s)$ is projected marker size and $\alpha_{cp}(s)$ is the viewing angle between the camera ray and the board normal.

3.3 Normal Matrix and Camera-Pose Information

For the calibration least-squares problem, the matrix used below is specifically the Gauss-Newton normal matrix. If

with residual vector $r(x)$ and Jacobian $J=\partial r/\partial x$, then the exact Hessian is

The normal matrix is

This is the Gauss-Newton approximation of the Hessian. It is the right object for placement scoring for three reasons. First, after calibration, residuals are small, so the second term is usually negligible. Second, for a hypothetical candidate board, the predicted measurements are generated from the candidate itself, so the expected residual is zero and the second-order residual term vanishes in the local model. Third, with whitened Gaussian residuals, $J^\top J$ is the Fisher information matrix. Throughout the placement formulation and experiments, “Hessian” means this normal-matrix/information approximation unless stated otherwise.

What the Hessian Says About Each Component

For a small parameter perturbation $\delta$, the second-order Taylor expansion is

Near a stationary point, $\nabla F(x)\approx0$, so the local loss increase is approximately

If only parameter component $x_j$ is perturbed, then

Therefore, $H_{jj}$ measures the local curvature with respect to $x_j$ while the other components are held fixed. A large value means that the loss changes rapidly in that coordinate; a small value means that the loss is locally insensitive to it. This comparison is meaningful only after rotation, translation, and other parameter units have been scaled consistently.

The off-diagonal entry $H_{jk}$ measures coupling between components $x_j$ and $x_k$. For a joint perturbation,

A large off-diagonal term means that the effects of the two parameters are difficult to separate, or that a change in one can be partly compensated by a change in the other. In the calibration problem, the same interpretation applies to $6\times6$ camera and board pose blocks: diagonal blocks describe local pose curvature, while off-diagonal blocks describe camera-board or camera-camera coupling.

The normal matrix also separates by measurement factor. If the whitened least-squares objective is

then

For a proposed perturbation $\delta$, factor $i$ contributes the local quadratic increase

This provides a direct way to measure how much a particular corner observation, camera, or board constrains a chosen parameter direction. Contributions from multiple independent factors add through $\sum_iH_i$.

The Hessian diagonal alone does not give the marginal uncertainty of a parameter because it ignores compensation through the other parameters. After gauge freedoms have been removed, the joint covariance is approximately

and $(H_{\mathrm{GN}}^{-1})_{jj}$ describes the marginal uncertainty of component $x_j$ after accounting for all parameter correlations. Thus, $H_{jj}$ is fixed-others sensitivity, whereas $(H^{-1})_{jj}$ is all-others-adjusted uncertainty.

The Hessian does not, by itself, say how much a component contributes to the current scalar loss. The residual values determine the current loss and the gradient gives its first-order change. The Hessian describes how that contribution changes locally as the parameters move.

3.4 Linearized Measurement Information

For point $p$ on board $b$, define the predicted camera-coordinate point

The weighted reprojection residual is

For a candidate board pose $T^B_b$, linearize the residual around the current camera and board estimates:

where

For the pinhole model,

For this placement-scoring derivation, use local right perturbations

Then the camera-pose Jacobian is

The board-pose Jacobian is

These Jacobians convert a candidate physical board placement into expected camera-pose information.

One convention detail matters here. The right-perturbation Jacobians above produce information in right-tangent coordinates. A current normal matrix computed with left updates,

cannot be added directly to candidate information computed with right updates,

The perturbations are related by

Therefore, either compute both the current and candidate Jacobians with the same perturbation convention, or transform each pose block before adding information matrices. If $A=\operatorname{blkdiag}(\operatorname{Ad}_{T_1},\dots,\operatorname{Ad}_{T_C})$ maps right camera perturbations to left camera perturbations, then a left-coordinate information matrix is converted to right coordinates by

3.5 Marginalizing Uncertain Board Poses

For one camera, $A_{cp}$ can be stacked directly. For multiple cameras, each local camera Jacobian must first be embedded into the global camera tangent vector. Let

Let

select camera $c$’s 6D block, so that

The global camera Jacobian block for edge $(c,p)$ is

Stack these embedded residual Jacobians for the candidate board pose:

Here $m$ is the number of predicted camera-point observation edges for this candidate pose.

Let $\Lambda_b^{\mathrm{prior}}$ denote the information supplied by surveyed world coordinates, a board-pose measurement, or another board prior. The candidate’s joint information contribution is

For camera-pose guidance, the board pose is usually a nuisance variable. Marginalizing it gives the effective information contributed to the camera pose:

This expression has three useful interpretations.

1. If the board pose is accurately known, then $\Lambda_b^{\mathrm{prior}}\rightarrow\infty$, and $\Delta\Lambda_c\approx A_b^\top A_b$. Almost all image information contributes to the camera pose.
2. If the board pose is poorly known, the second term becomes large. Much of the apparent image information is spent estimating the board pose.
3. With no world anchor, a simultaneous global transformation of all camera and board poses leaves projections unchanged. Absolute camera pose is unobservable; only relative board-to-camera pose is constrained.

For the complete multi-camera system, append the hypothetical candidate information to the current normal matrix:

Partition the result into camera poses $C$ and nuisance variables $N$, including board poses:

The marginalized camera-pose information is

The predicted camera-pose covariance is approximately

This inverse assumes the gauge has been fixed and $\Lambda_C$ is nonsingular. If the system still has gauge freedom or near-null directions, use a constrained inverse or pseudoinverse over the observable subspace.

This is the next-best-pose principle: append the expected measurements of a hypothetical pose, compute the resulting information, and select the pose that gives the largest uncertainty reduction.

3.6 Upper-Level Camera-Uncertainty Objective

The upper-level problem chooses $s$ so that the lower-level camera poses are accurate and well conditioned. Optimized reprojection residual is not the best design objective, because a weak or degenerate layout can still fit its own noisy observations. A more useful objective is camera-pose uncertainty.

Linearize the lower-level residuals at the nominal solution and split the perturbation into camera-pose variables and nuisance variables:

where $\theta$ contains camera-pose perturbations and $\zeta$ contains board-pose perturbations. The Gauss-Newton or Fisher information matrix is

partitioned as

After marginalizing the board poses and other nuisance variables, the effective camera-pose information is the Schur complement:

The predicted camera-pose covariance is

The A-optimal board-placement problem is therefore

The matrix $D$ normalizes rotation and translation units, for example by dividing translations by a millimeter tolerance and rotations by a degree tolerance. The feasible set $\mathcal S$ contains the physical and production constraints:

Typical constraints are board mountable regions, collision avoidance with the vehicle, minimum spacing between boards, image-bound margins, minimum visible marker count, acceptable viewing angle, acceptable projected marker size, and takt-time or robot-motion limits.

3.7 Feasible Physical Placements

The upper-level feasible set should encode physical, visibility, quality, and production constraints:

Typical physical constraints include

Typical visibility and quality constraints include

For an EOL station, takt-time or robot-motion constraints can be added as

Corner-Quality and Detection Constraints

Pure geometric information can favor an excessively oblique board. In practice, strong foreshortening makes corners elongated, blurred, or difficult to detect. Candidate scoring should therefore use pose-dependent image covariance,

or an expected detection probability:

This balances geometric strength against actual feature-localization accuracy. Otherwise, the placement optimizer can prefer grazing views that look informative geometrically but produce anisotropic and unreliable corner measurements.

The feasible set $\mathcal F$ should therefore require:

enough visible markers, image points inside the image with a safety margin, sufficient projected square size, acceptable focus and illumination, and a non-grazing incidence angle.

3.8 Continuous Board-Pose Optimization

For continuous placement variables, optimize the uncertainty objective subject to the feasible set, using derivatives of the predicted information matrix. A numerical iteration proposes a new board pose; it does not require collecting an image or physically moving the board at every step. The first experiment instantiates this formulation with SLSQP.

The derivative below applies locally where the measurement model is differentiable and the visibility set is fixed. Visibility changes and geometric constraints must also be handled by the placement optimizer.

For the uncertainty objective

the differential is

The Schur-complement differential is

This gives a path to gradient-based board-placement optimization when the placement variables are continuous.

3.9 Sequential Next-Board Placement

If boards are placed one at a time, the upper level becomes a next-best-placement problem. Suppose the current calibration data provide camera-pose information

A candidate new board placement $s_{\mathrm{new}}$ contributes expected information

Then choose

For $M$ future placements, use

This formulation directly answers the operational question: where should the next board be placed to reduce camera-pose uncertainty the most?

Interpreting the Information Geometrically

The information matrix translates into concrete board-placement guidance:

Therefore, repeatedly placing a small, fronto-parallel board at the image center is usually inefficient. It generates nearly redundant Jacobian rows. A better sequence contains:

1. A large, well-centered observation for robust initialization.
2. A laterally displaced observation with tilt about one board axis.
3. An observation on the opposite side with tilt about the other axis.
4. A different board-to-camera distance.
5. For multi-camera calibration, placements in camera-overlap regions so the same board points are observed by multiple cameras.

The exact next placement should be determined by the weakest eigenvector of the current marginalized information matrix. For example, if the largest covariance direction is mostly a combination of camera $t_z$ and pitch, the next-best score will usually favor a laterally displaced and tilted board rather than another centered, fronto-parallel observation.

Practical Placement-Optimization Loop

The resulting EOL procedure is:

1. Solve centralized problem (P) using all measurements currently available.
2. Compute the current sparse normal matrix and camera-pose covariance.
3. Parameterize the board’s continuous position and orientation and define workspace, visibility, and corner-quality constraints.
4. At each numerical trial pose, predict visibility, pixel positions, corner covariance, and reprojection Jacobians.
5. For an additional observation, add its predicted information to the current normal matrix; for a replacement, rebuild the information for the revised layout without double-counting the old observation.
6. Marginalize board poses and other nuisance variables with the Schur complement.
7. Optimize the normalized camera-pose uncertainty over the feasible pose variables. If the station allows only a discrete set of mounts, solve the corresponding candidate-selection problem instead.
8. Place and survey the board, capture the observations, resolve (P), and repeat.

The central idea is:

3.10 Complete Board-Placement Formulation

For the method statement, the most useful final form is

In words, the lower level estimates camera and board poses, while the upper level chooses physical board placements that make the lower-level camera-pose estimate maximally informative. The final objective should not merely minimize the lower-level reprojection error. It should minimize the predicted camera-pose uncertainty induced by that lower-level optimization.

The experiments now instantiate this formulation with increasing system complexity. Alternative criteria, residual-based objectives, and the fixed-budget discrete subset formulation are retained in Appendix C.

4. Experiments from Simple to Complex

The first experiment directly optimizes a continuous six-coordinate board pose. The subsequent experiments introduce a second camera, uncertain board poses, partial multi-camera visibility, and a vehicle-scale layout. Discrete candidate searches are labeled separately: they address restricted layout choices rather than replace the continuous optimization method.

4.1 One Camera, One Board: Continuous Pose Optimization

Start with one fixed camera and one movable board. The task is to optimize the board’s position and orientation so that a subsequent image estimates the camera’s six-degree-of-freedom extrinsic pose as accurately as possible. The camera intrinsics are already known. We start from one board pose and use continuous constrained optimization; we do not choose among a few manually prescribed placements.

The board’s world pose must be known after moving it, for example from an accurately surveyed fixture. In this first example, that pose is treated as exact. If both the camera and board poses were completely unknown, an image would determine only their relative pose, not the camera’s absolute world pose. Finite board-survey uncertainty is handled by the marginalization in Section 3.5; this first experiment isolates the exactly known-board case.

Geometry and Design Variables

Use the nominal camera frame as the world frame: $x$ points right, $y$ down, and $z$ forward. The fixed camera has a $1280\times960$ image, $f_x=f_y=800$ pixels, principal point $(640,480)$, and no distortion. The board has $7\times5$ corners with $0.10$ m spacing, spanning $0.60\times0.40$ m.

The simulation’s true camera pose equals the nominal pose. In a real station, evaluate the design using the current approximate camera pose; the purpose of the new image is to refine that estimate.

The six continuous design variables are

Translation is in metres and the reported angles are in degrees. Define the board-to-world rotation and world corner positions by

At zero angles the printed board normal faces the camera. These are fixed-axis $x,y,z$ Euler angles, matching SciPy’s lowercase xyz convention. The initial board pose is

Turn Camera Uncertainty into the Placement Objective

For each proposed board pose, predict the corner locations and the camera-pose Jacobian. At the nominal camera pose $T_C=I$, use camera perturbations ordered as $\delta\theta=(\delta t,\delta\omega)$, with translation in metres and rotation in radians:

Here $J_\pi$ differentiates the pixel projection, including the focal lengths, and $[X]_\times v=X\times v$. The factor $1/\sigma_i$ whitens the two pixel-coordinate residuals for corner $i$.

To represent poorer corner localization at grazing angles, assume the illustrative noise model

Here $n_B=R_B(0,0,1)^\top$ is the printed board normal. The feasibility constraints below keep $\nu_i$ positive and away from zero. This incidence-dependent noise is an assumed model, not a measured property of a particular camera. For camera-pose estimation, its values are fixed at the nominal design geometry.

With independent Gaussian image noise and a full-rank whitened Jacobian, the local camera-pose covariance is approximately

This is a local, linearized covariance prediction, not a guarantee of global pose-estimation accuracy; see the Ceres covariance documentation.

Normalize translation by a $5$ mm tolerance and rotation by a $0.1^\circ$ tolerance:

The board-placement optimization is

Thus we minimize over the physical board pose $s$, not the camera pose. The objective is the predicted sum of squared, tolerance-normalized camera-pose errors. The tolerance ratio specifies the relative importance of translation and rotation.

The feasible set $\mathcal S$ requires:

• Board centre: $x_B\in[-0.6,0.6]$ m, $y_B\in[-0.4,0.4]$ m, and $z_B\in[1,3]$ m.
• Board angles: $\alpha,\beta\in[-60^\circ,60^\circ]$ and $\gamma\in[-45^\circ,45^\circ]$.
• Every corner remains at least $40$ pixels inside the image and at least $0.6$ m in front of the camera.
• Every corner’s incidence angle is at most $70^\circ$, and adjacent grid corners remain at least $12$ pixels apart.

How the Optimizer Decides Which Way to Move

Moving the board changes $X_i$, the noise weights, and therefore $J$ and $H$. The derivative of the uncertainty objective with respect to board coordinate $s_j$ is

This derivative tells us whether increasing a board coordinate improves or worsens predicted camera precision. It accounts for both projection geometry and the change in corner noise.

The implementation uses SciPy’s SLSQP constrained optimizer with numerical derivatives, starting only at $s_0$. At each iteration it evaluates $U$ and the constraints near the current pose, builds a local quadratic subproblem, and updates the six board coordinates together. Coordinates are scaled internally so that metres and degrees have comparable numerical ranges.

In schematic form, with inequality constraints $c(s)\ge0$, the update is

Here $B_k$ approximates the Hessian of the constrained optimization Lagrangian, and $a_k$ is a line-search step length. This calculation produces a new continuous pose; it is not a lookup in a candidate-pose grid. All of these evaluations are numerical predictions, so the board does not need to be physically moved at every optimizer iteration.

Optimized Pose and Physical Adjustment

The optimizer converges successfully in 35 iterations:

For this coordinate convention, the instruction is: move the board centre 0.791 m left, 0.186 m down, and 1.500 m toward the camera along the optical axis, then set its orientation to the final Euler angles above. The final $z_B=1$ m is optical-axis depth, not Euclidean camera-to-board distance.

The optimized board is closer, occupies more of the image, and has a different viewing direction and tilt. Together these changes reduce the normalized covariance trace by 98.9%. The optimum reaches the allowed minimum centre depth and the left and bottom image margins. Its largest corner incidence angle is $56.9^\circ$, within the $70^\circ$ limit.

This is a constrained local optimum under the stated model, not a proof of the globally best board pose. In particular, the ideal camera model has no additional penalty for uncertain intrinsics, lens distortion, or peripheral detection errors. Adding those effects, or board-survey uncertainty, can change the optimized pose. The result is not a general instruction to put every calibration board in an image corner.

Board Position in the Camera View During Optimization

The following snapshots select iterations 0, 1, 2, 7, 14, 21, 22, 28, and 35 from the same optimization history. They are intermediate states of one continuous optimization, not separate experiments or candidate layouts used to choose a winner. Read left to right, then top to bottom.

Each panel projects the same board through the fixed camera, with identical image dimensions and pixel axes. The checker pattern spans the known corner grid; its coloured outline and dots show the projected footprint and all 35 corners. The dashed rectangle marks the required 40-pixel image margin. Each title gives the iteration and camera-uncertainty objective $U$.

Initially the board appears small, near the image centre. By iteration 2 it is substantially larger; later updates shift and rotate its projection toward the lower-left region. Iterations 1 and 21 violate constraints and are shown explicitly rather than hidden: iteration 1 even puts part of the board outside the image. These are numerical optimizer states, not instructions to physically move the board through those poses. At iteration 35 the projected board satisfies the constraints and touches the allowed left and bottom margins.

The selected-iteration data records the exact six-coordinate board poses, uncertainty scores, and feasibility flags used for this figure. The full optimization history includes every recorded iteration.

Why Not Place the Board Symmetrically at the Image Centre?

A centred board is an intuitive choice, but visual symmetry does not automatically minimize the uncertainty of all six camera-pose parameters. The off-centre result needs a precise interpretation: the centre is competitive, and the lower-left corner is not uniquely optimal.

To check the effect of centring, fix the physical board centre on the optical axis, $x_B=y_B=0$, and continuously optimize the remaining four coordinates:

This uses the same objective, noise model, and feasibility constraints as the six-coordinate optimization. SLSQP starts from the previous solution’s depth and orientation, with its lateral position reset to zero. It converges in 16 iterations to a local centred solution with $z_B=1$ m and approximately $(\alpha,\beta,\gamma)=(0^\circ,49.57^\circ,-10.52^\circ)$. Thus, “centred” does not mean “front-facing”: the board centre projects to the principal point, but its orientation remains free.

The centred solution actually has better predicted translation precision. The off-centre solution sacrifices some translation precision for better rotation precision, and the chosen $5$ mm / $0.1^\circ$ normalization favors that trade-off. Its objective is only 12.6% lower than this optimized centred solution. The earlier 98.9% reduction is relative to the distant initial pose and combines changes in depth, orientation, and lateral position; it is not a 98.9% advantage of off-centre placement over centred placement.

This is a diagnostic constrained optimization, not a selection among manually prescribed experiments. Both reported solutions are local numerical solutions, not proofs of global optimality.

The geometric reason is that different camera motions can produce similar image motions. For a camera-frame point $(X,Y,Z)$, the unwhitened horizontal pixel coordinate has the following derivatives under the camera-perturbation convention used above:

Here $t_x$ and $\omega_y$ refer to camera translation and rotation, not board-design coordinates. For a nearly front-facing board near the image centre, $X/Z$ is small and corner depths are similar. Both a sideways camera translation and a small rotation about the camera’s $y$ axis then produce approximately uniform horizontal image shifts. Their Jacobian columns are nearly proportional, making those camera parameters difficult to distinguish.

Tilting the board introduces depth variation across corners, while moving it off-axis changes the spatial pattern of rotational image motion. Either can help separate the effects of different pose parameters. Symmetry can reduce some parameter correlations, but does not eliminate this translation-rotation coupling. The objective accounts for the joint correlations through $C=(J^\top J)^{-1}$, rather than scoring only the diagonal of $J^\top J$. These are local covariance predictions under the assumed measurement noise, not measured real-camera accuracies; see the Ceres covariance documentation.

Symmetry also does not imply that the minimum must occur at the centre. In this model, reflecting both the board position and orientation horizontally or vertically preserves the score. All four mirrored versions of the off-centre solution were checked: they satisfy the constraints and have the same $U=0.42765855$. The lower-left placement is therefore one member of a symmetry-related set, not a uniquely preferred image region.

For the fixed-axis xyz Euler convention used here, horizontal reflection changes the signs of x, beta, and gamma; vertical reflection changes the signs of y, alpha, and gamma. Applying both gives the fourth pose. Reflecting only the board centre while retaining its original orientation is not this symmetry check.

Finally, the model assumes exact intrinsics, no lens distortion, and an exactly known board pose. Its noise model penalizes grazing incidence but includes no additional penalty for peripheral-image errors. Uncertainty in intrinsic calibration, residual distortion, actual corner-detection quality, and board surveying can change the optimum. The conclusion is a model-specific translation-versus-rotation trade-off, not a general recommendation to move calibration boards to image corners.

Reproduce the Centred-Placement and Symmetry Checks

From the repository root, run the simulation source with NumPy, SciPy, and Matplotlib installed:

python content/posts/2025-02-EOL-calib/simulations/one_camera_one_board_placement.py --centred-checks-only

This command first reruns the original six-coordinate optimization from the stated initial pose. It then fixes the lateral coordinates to zero, initializes depth and orientation from that computed solution, and optimizes the remaining four coordinates. Finally, it constructs and checks all four mirrored off-centre poses. It does not read saved solutions, generate noisy observations, or use Monte Carlo errors to select a placement.

Both optimizations use scaled variables, SciPy’s default numerical derivatives, and a maximum of 200 SLSQP iterations. The objective stopping tolerance ftol is 1e-10 for the original optimization and 1e-11 for the centred diagnostic. The script rejects unsuccessful optimization, final poses violating a nonlinear constraint or coordinate bound beyond a slack tolerance of 1e-7, and failed camera-Jacobian finite-difference checks. For the mirrors, it checks equality of the objective and both predicted RMS errors with relative tolerance 1e-10 and absolute tolerance 1e-12.

The command regenerates these files in the article’s resources/ directory; add --output-dir /tmp/ocob-centred-checks to write elsewhere:

• Centred/off-centre comparison: exact poses, predicted errors, objective values, and iteration counts behind the table above.
• Centred optimization history: the starting pose and every recorded iterate, with both lateral coordinates fixed to zero.
• Mirror-symmetry checks: all four poses, feasibility flags, uncertainty metrics, and score differences from the original pose.
• Reproducibility record: starting poses, variable scales, tolerances, solver outcomes, package versions, and the calculated 12.6% score reduction.

The saved results used Python 3.10.9, NumPy 1.23.5, SciPy 1.10.0, and Matplotlib 3.7.0. Other numerical builds may change the last digits or iteration counts. The accompanying regression tests verify the reported values within numerical tolerances, the translation-versus-rotation trade-off, and the reflected board geometry. The full simulation command below also regenerates these diagnostics before running its separate noisy-fit validation.

Estimate the Camera After Moving the Board

Once the optimized pose is chosen, move and re-survey the board, acquire an image, and estimate the camera pose by minimizing the whitened reprojection loss over $T_C$:

Here $\pi_K$ is pixel projection and $u_i$ is the measured corner. Unlike the outer optimization over $s$, this inner least-squares problem adjusts the camera estimate while holding the surveyed board fixed.

For validation only, the script generates 500 independent noisy images at the initial pose and another 500 at the optimized pose, and refines the camera estimate for each image. These trials are not inputs to placement optimization, and observations from different images are not accumulated. The reported errors are RMS magnitudes of the full three-dimensional translation or rotation error, not average per-axis standard deviations.

The camera refinements start near the true pose and test the local positive-depth solution branch. They do not test global initialization or planar pose ambiguities; the OpenCV PnP documentation describes methods that can return multiple pose solutions. Likewise, SLSQP’s numerical trajectory is not a collision-checked robot trajectory.

The reproducible simulation source requires NumPy, SciPy, and Matplotlib:

python content/posts/2025-02-EOL-calib/simulations/one_camera_one_board_placement.py

The script checks the camera Jacobian against finite differences and saves the optimization history, placement and uncertainty summary, and individual noisy-fit errors. The example’s key sequence is: predict camera uncertainty, optimize the board pose, then collect measurements and estimate the camera.

4.2 Two Cameras, One Board: Corner Refinement and Calibration

This two-camera, one-board experiment checks board-survey refinement, image-corner localization, and pose estimation at a fixed board placement; it is not a second placement-search algorithm. The geometric preprocessing is derived in Appendix A, and the two-camera objective is written explicitly in Appendix D.

We construct an 8x8 chessboard with 1 m squares. We set the the origin as the center of the board. We place the board in the world coordinate with extrinsic:

which means the normal of the board plane is $[0, 0, 1]^\top$.

We add Gaussian noise on the corners coordinates with noise zero mean and $0.005$ standard deviation.

With the noisy corners’ coordinates, we can get the optimized board’s extrinsic using the rigid board-fit procedure in Appendix A. The optimized normal is $[1.38*10^{-3}, -2.16*10^{-5}, 9.999*10^{-1}]^\top$. That is already very close the ground truth.

Then we simulate two cameras with the same intrinsic matrix: $f_x =800$, $c_x = 500$, $f_y = 800$, $c_y=400$, and $H=800$, $W=1000$. We rotate the first camera along the z-axis $10$ degrees and the second $-10$ degrees, and translate the first camera along the x-axis $2$ meters and the second $-2$ meters. Then the images of the chessboard in the two cameras are shown in the figure below.

Then, we apply Gaussian blur on the observed images and add Gaussian noise on them. The new images are shown in the figure below.

After that, we use opencv “findChessboardCorners” and find the rough corners and then use “cornerSubPix” to refine the corners’ coordinates. The refined corners are shown in the figure below.

Pay attention to the sequence of the detected corners. (The detected corners start from the blue one and end with the green one in the above figure.) Rearrange the detected corners according to the refined chessboard corners. Using the “solvePnP” function with method “SQPNP”, which is a global optimization method, we can find the estimated extrinsic of the two cameras. We record the errors of the estimation using the original measurements of the chessboard corners and the refined ones.

The translation error is the norm of the difference between the estimated and the ground truth translation. The rotation error is

converted from radians to degrees.

As shown in the table, the refined measurements result in lower errors for both rotation and translation, demonstrating the effectiveness of the corner refinement process.

In the final step, we do jointly optimization. We warp the image in camera 1 with parameters before and after jointly optimization. The results are shown in the figure below.

In the above figure, we can hardly tell which one is better. The translation error is $0.0298$ and $0.0257$ meters for camera 1 and camera 2 respectively. The rotation error is $0.204$ and $0.199$ degrees for camera 1 and camera 2 respectively. That is expected increased error for single camera calibration however more consistent for the two cameras.

Detect Corners on the Warped Image

Corner detection often becomes challenging on original images due to severe distortion or extreme viewing angles. To address this limitation, we can create a virtual camera with an optimized viewpoint. By warping the original image onto this virtual camera’s image plane, we can simulate a more frontal view of the chessboard. This transformation adjusts the perspective so that the chessboard appears as if viewed head-on, significantly improving corner detection reliability. We can then accurately detect the corners on this warped image rather than struggling with the difficult perspective of the original.

The figure below illustrates this approach. On the left is the original image where the severe viewing angle prevents the standard algorithm from successfully detecting the corners. On the right is the perspective-corrected warped image, where we can successfully identify and locate all corners.

4.3 Two Cameras, Two Boards: Uncertain Boards and Discrete Placement

Building on the one-camera placement optimization in Section 4.1, consider a calibration station with two cameras and two physical calibration boards. Here the lower level estimates both camera poses and both uncertain board poses from noisy image corners and noisy surveyed board corners. This separate experiment uses a discrete candidate set to illustrate several upper-level search strategies; Section 4.1 demonstrates direct continuous board-pose optimization from one starting pose.

The experiment is generated by:

python chess-board-simulate/two_camera_two_board_placement.py

The two cameras are fixed and look into a shared workspace. Each board has a $7\times5$ marker grid with $0.12$ m spacing. Candidate board poses are sampled over position, height, yaw offset, and tilt. A candidate is feasible only if both cameras see at least 20 corners, the projected corners are inside the image margin, the projected marker size is large enough, and the board is not too grazing.

Noise Model

The simulated image noise follows the placement-dependent idea from the bilevel formulation. For each visible corner, the pixel standard deviation is

Here $a_{cp}$ is the projected marker size in pixels, $\iota_{cp}$ is the incidence between the board normal and the camera ray, and $\rho_{cp}$ is normalized distance from the image center. This makes small, oblique, or edge-of-image corners noisier. Surveyed world marker positions use $\sigma_z=2.5$ mm.

Upper-Level Solvers

For each feasible candidate board pose $g$, the script predicts visibility and computes the Schur-marginalized camera-pose information contribution $\Delta\Lambda_g$. For a selected two-board set $\mathcal A=\{g_1,g_2\}$, the A-optimal score is

The simulation compares four placement strategies:

1. Manual initial: two nearby, front-workspace boards.
2. Greedy next-best: choose the first board with the best single-board score, then choose the second board that gives the best updated two-board score.
3. Simulated annealing: start from the manual pair, randomly replace one board at a time, and accept or reject moves using a decreasing temperature.
4. Exhaustive optimum: enumerate all valid two-board pairs in the discrete candidate set and choose the pair with the lowest A-optimal trace.

This mirrors the reference strategies in the placement-guidance literature: sample a feasible pose set, reject degenerate candidates, score by parameter uncertainty rather than residual alone, and use either direct search, sequential next-best selection, or simulated annealing to search the pose space.

The feasible candidate set and selected layouts are:

The single-board candidate score map is:

The selected board placements are:

The optimized placements are not chosen by reprojection residual. They spread the two boards in depth, lateral position, height, and tilt so that the camera-pose Jacobians span stronger independent directions.

Noisy Lower-Level Validation

After selecting the placements, each layout is validated by 24 Monte Carlo trials. Each trial re-samples image noise and surveyed board-corner noise, then solves the lower-level bundle-adjustment problem from perturbed initial camera poses.

The important observation is that the manual layout has the lowest pixel RMSE, but the worst camera-pose accuracy. It fits its own observations well while leaving weak pose directions. The optimized layouts increase the pixel RMSE slightly, because the views are more oblique and noisy, but they reduce the predicted camera covariance trace by about $4\times$ and reduce the Monte Carlo camera-pose error from $0.128^\circ$ and $4.93$ mm to about $0.05^\circ$ and $2.1$-$2.3$ mm.

This illustrates the same bilevel principle: the upper level should not optimize expected residual. It should optimize the information geometry of the lower-level calibration problem.

4.4 Multi-Board and Multi-Camera Calibration

The simulation code for the two-camera chessboard experiment in Section 4.2 is in livey/chess-board-simulate. A companion script, multi_board_multi_camera.py, now runs a scaling study for the multi-board and multi-camera formulation in this post.

The synthetic scenarios contain:

1. Four to eight calibrated cameras with known intrinsics.
2. Six to twenty-four target boards, each carrying a $7\times5$ grid of marker points.
3. Noisy camera-point observations $u_{cp}$.
4. Noisy surveyed world measurements of board marker points.
5. Partial visibility: each camera sees only a subset of boards and marker points.

This section reports centralized bundle adjustment, which directly solves (P) over the camera and board poses. The solver-comparison tables, ADMM convergence plots, and communication results are collected in the independent ADMM comparison.

The simulation can be reproduced with:

cd chess-board-simulate
python multi_board_multi_camera.py

The script writes figures and CSV tables into logs/multi_board_multi_camera/. The detailed figures below use the largest case, with eight cameras and twenty-four boards. The scaling tables compare all scenarios from the same deterministic run.

Problem Size

The sweep changes the number of camera poses, board poses, observed camera-point edges, and surveyed board points while keeping the same formulation.

Simulated Observation Graph

The detailed scene has eight cameras arranged around twenty-four calibration boards. The colored board points are the physical marker locations. The faint lines indicate observed camera-board connections.

The visibility matrix shows how many marker points each camera observes on each board.

This is intentionally not a complete bipartite graph. In the 8-camera, 24-board case, only 82 of the 192 possible camera-board pairs are visible. The coupling comes from shared boards and world-frame board priors, not from every camera seeing every board. The full $8\times24$ visibility matrix is written to resources/mbmc_visibility_counts.csv.

Image Observations and Reprojection Residuals

The next figure shows the noisy image observations in each camera. The small black residual vectors connect the measured points to the centralized BA predictions.

Bundle Adjustment Results

For the 8-camera, 24-board case, the initialization perturbs camera poses by several degrees and tens of centimeters, while board poses are initialized from noisy surveyed world marker measurements. Centralized BA reduces reprojection error from 63.0991 px to 1.0486 px.

The sparse nonlinear least-squares solve takes 1.4180 s and 11 outer iterations in this Python implementation. These are implementation timings for this experiment.

The board error is already small at initialization because the boards are directly anchored by surveyed world points. The camera error drops much more strongly because the image residuals provide most of the camera-pose information. The full per-camera and per-board error tables are written to resources/mbmc_camera_pose_errors.csv, resources/mbmc_board_pose_errors.csv, resources/mbmc_scaling_camera_pose_errors.csv, and resources/mbmc_scaling_board_pose_errors.csv.

Bundle Adjustment Scaling

Centralized BA uses a sparse Jacobian: each image residual touches one camera pose and one board pose. The following results show its accuracy and runtime as the calibration graph grows.

The timings are from the serial Python reference implementation. Reprojection RMSE remains near the pixel noise floor across the four scenarios. The full comparison with the distributed solvers is in the independent comparison.

Board-Placement Guidance Experiment

The same script also runs a next-board-placement experiment. Starting from the solved 8_cameras_6_boards case, it computes the current marginalized camera-pose information from the centralized BA Jacobian. It then generates feasible candidate poses for one additional board, predicts the visible image measurements for each candidate, appends the candidate’s expected Jacobian information, marginalizes the candidate board pose with the Schur complement, and scores the candidate by normalized camera covariance reduction.

The first diagnostic is the actual optimization matrix. The left plot below is the sparsity pattern of the whitened Jacobian $J$ for the current centralized BA problem. Each reprojection row touches one camera pose and one board pose. Each surveyed world-point row touches only one board pose. The right plot is the Gauss-Newton normal matrix $H_{\mathrm{GN}}=J^\top J$, not the full exact Hessian. This is intentional: near the solved calibration state, and especially for zero-expected-residual candidate measurements, $J^\top J$ is the local curvature/information matrix that determines pose covariance. Its camera-board off-diagonal blocks are exactly the couplings that later become camera-camera information after board poses are marginalized.

The best candidate is not selected by the expected residual value; the expected residual is zero for all candidates. It is selected by the curvature that the candidate observations add to the camera-pose variables.

For each candidate board pose, the script computes the candidate’s predicted normal matrix contribution, marginalizes the candidate board pose, and adds the resulting camera-pose information to the current $\Lambda_C$. The next figure shows $\Lambda_C$ before adding a board, after adding the low-score feasible board, after adding the recommended board, and the information gain from the recommended board alone. Each $6\times6$ diagonal block is one camera pose. Off-diagonal blocks indicate camera-camera coupling induced by shared board observations.

The candidate map below shows all feasible candidate board centers. Color is predicted normalized camera covariance reduction, and marker size increases with the number of predicted marker observations.

Board placement is not only a center position; it is a pose. The arrows in the next plot show the horizontal component of the board normal. In this experiment, candidate normals are generated to face outward toward the surrounding cameras, and the optimizer selects among these full poses.

The selected candidates are:

In this current calibration state, the optimizer prefers P17 because it is visible from five cameras and lies in an overlap region. It contributes information to more camera-pose blocks and creates stronger camera-to-camera coupling through one physical board. P10 is still feasible, but its information is more local and therefore reduces the marginalized camera uncertainty much less.

The per-camera uncertainty plot shows where that improvement goes. P17 reduces both rotation and translation uncertainty for more cameras than P10, especially for cameras that are weakly constrained by the existing six-board layout.

To check that this is not only a symbolic score, the script then adds either P10 or P17 as the next board, simulates noisy image and surveyed world measurements, re-solves centralized BA, and repeats this for eight noise draws. The table reports mean actual errors; the figure shows the same comparison with error bars.

The reprojection RMSE stays near the pixel noise floor for both candidates, so it is not a sensitive placement metric. The camera-pose covariance and ground-truth pose error are more informative: the recommended board gives a larger predicted uncertainty reduction and a lower average camera-pose error after re-solving.

What This Demonstrates

The primitive observation is the camera-point edge $(c,p)$, while the optimized variables are camera and board poses. Centralized BA exploits this sparse structure and combines image observations with surveyed board priors to estimate both sets of poses. It is the default solver for an EOL station that can process all observations on one machine.

4.5 Vehicle Rig: Board Count and Placement Refinement

The previous synthetic scene places cameras around a workspace. A more EOL-like setup is a vehicle rig. In this experiment, the cameras are mounted on a car-like body with seven cameras: front wide, front narrow, left-front, right-front, left-rear, right-rear, and rear. The side cameras are crossed in position: the cameras named left-front and right-front are physically closer to the rear, while the cameras named left-rear and right-rear are physically closer to the front. This creates overlapping side views. Boards are sampled from uniform rectangular perimeter rings around the vehicle, then the number of selected boards is increased.

The visibility matrix for the 40-board case is intentionally sparse. Each board is visible to a subset of cameras, and each camera sees the boards in its sector. The side overlap is present: in this run, 7 boards are seen by both left-side cameras and 7 boards are seen by both right-side cameras.

The centralized bundle-adjustment results for the board-count sweep are:

As the number of boards grows, camera error improves strongly at first. In this run, centralized BA goes from $0.1033^\circ$ and $7.0$ mm at 8 boards to $0.0412^\circ$ and $2.5$ mm at 32 boards. Additional boards mainly add redundancy, so the error plateaus near the pixel-noise and world-measurement limits.

This sweep also tells us how to use the placement-guidance method. The useful signal is not only the reprojection RMSE, because all layouts are near the pixel noise floor. The useful signal is that the 40-board uniform layout has more measurements than the 32-board layout, but worse camera-pose accuracy:

So the 40-board case is not limited by board count. It is limited by where those boards put information into the camera graph. The replacement experiment therefore keeps the board budget fixed at 40 and uses the solved 40-board state to ask a more targeted question: which candidate board poses would reduce the remaining camera-pose covariance the most?

The 40-board solution also identifies which camera directions are still weak. In the centralized BA result, the rear camera and the wide front camera remain larger-error poses than most side cameras:

This is why the guidance step does not simply spread boards more uniformly. It uses the solved uniform layout to score how each candidate board would couple the remaining uncertain cameras to the rest of the rig after board poses are marginalized.

Jacobian-Guided Refinement of the 40-Board Layout

The 40-board uniform layout can also be treated as the starting point for the board-placement guidance method. Here the goal is not to add one more board, but to replace the 40 boards with a better 40-board set using the solved uniform layout as the current calibration state.

The adjustment loop is:

1. Solve the original uniform 40-board layout with centralized BA.
2. Build the Schur-marginalized camera-pose information matrix from that solved state.
3. Generate the same perimeter-ring candidate board poses used by the vehicle experiment.
4. For each candidate, predict visible marker observations using the solved camera poses, build the candidate Jacobian, marginalize the candidate board pose, and score the candidate by camera covariance reduction.
5. Start from the exact 40 initial board positions and evaluate one-for-one swaps: remove one existing board, add one candidate, and keep the swap only if the predicted camera covariance improves.
6. Repeat the swap search until no candidate swap improves the Schur-marginalized camera-pose trace.
7. Re-simulate noisy image and surveyed-world measurements for the refined board placement, then solve the calibration again.

For one candidate board pose $g$, the method linearizes the image residuals at the solved camera poses from the initial 40-board layout. Its camera-pose information contribution is computed after marginalizing the candidate board pose:

For a one-for-one replacement, the predicted score is the reduction in the normalized camera covariance trace after replacing an existing board contribution $\Lambda_r$ with $\Lambda_g$. This means the method is not just counting visible corners. It asks whether the new board’s Jacobian directions reduce the weak camera-pose modes left by the solved uniform layout.

As a concrete example, start from the blue-square 40-board top-down layout above. The initial board B5 corresponds to candidate G033 at $(-2.77,-3.25)$ m. It is visible from only one camera in the initial solution, and adding that pose alone would reduce the current camera covariance trace by only $0.25\%$. The highest-scoring unused candidate is G021 at $(-3.25,-0.82)$ m. It is visible from two cameras and, more importantly, its Jacobian couples a weak side/rear direction to the rest of the rig, giving a $23.7\%$ one-board trace reduction. The first greedy replacement therefore moves B5 to G021, while keeping the board normal pointed toward the vehicle. The predicted normalized trace drops from $4.1914$ to $3.2031$ after this one swap.

The score map below is the direct analog of the earlier board-placement guidance plot. Color shows the single-board trace reduction if that candidate is added to the solved uniform layout. Red squares are the final 40-board refined placement after the swap search. Some red squares have modest single-board scores because the final set is produced by cumulative swaps, not by independently taking the top 40 one-board candidates.

The highest single-board candidate scores are concentrated near side-overlap regions, where a board couples multiple cameras strongly:

The actual refinement performs 14 beneficial swaps and keeps 26 of the original 40 board positions:

The predicted information metric improves before any noisy re-simulation:

The matrix view shows why the refinement helps: the refined placement increases the Schur-marginalized camera-pose information and strengthens several camera-camera coupling blocks after board poses are marginalized.

The per-camera uncertainty view shows that the improvement is not only global trace reduction. The front and rear cameras receive the largest predicted uncertainty reduction, while the side cameras also improve.

The new board positions are shown below. Gray hollow squares are the original uniform 40-board positions; red squares are the refined positions, with arrows showing board normals.

After re-simulating noisy image and surveyed-world measurements with the refined board set, centralized BA reduces mean camera rotation error by 58.9% and translation error by 47.5%. Reprojection RMSE stays near the pixel noise floor, so camera-pose error is the more informative metric here.

The exact refined board centers and horizontal normal directions are:

4.6 What the Experiments Establish

The one-board experiment demonstrates continuous physical pose optimization and exposes the translation-rotation trade-off behind an off-centre result. The two-camera experiments add shared geometry and uncertain board poses. The workspace and vehicle experiments then test sparse calibration, partial visibility, board count, and fixed-budget placement refinement.

Across these cases, the calibration loss estimates poses, while the marginalized information predicts which physical observations will improve those estimates. Reprojection RMSE alone is not a reliable placement objective. The numerical optima remain conditional on the assumed world anchor, camera model, corner noise, and feasible workspace.

Appendix A. Geometric Preprocessing and Board Survey Refinement

These derivations support initialization and the fixed-board image-processing experiment in Section 4.2. Ray-plane reconstruction is an optional geometric route, not a required residual in the reprojection-based solver of Section 2.

A.1 Optional Geometric Pipeline

1. Do single camera intrinsic calibration

2. Measure each corner in the world coordinate

3. Refine the corners’ coordinates according to board constraints (plane, parallel, equally spaced)

4. Find the plane equation in the world coordinate

5. Initialize each camera’s extrinsic parameters by solving the perspective-n-point (PnP) problem

6. Estimate the 3D coordinates by the intersection of the image ray and board plane

7. Construct the objective loss function from pixel reprojection residuals that share board-generated marker points, plus optional world-point or pose priors

A.2 Refine Surveyed Corners Using Board Constraints

Assume the center of the board is located at the origin of a local coordinate, $(0,0,0)^\top$, the corner coordinate is located at $\{\mathbf{x}_i\}^N_{i=1}$, then the board normal is $(0,0,1)^\top$. Assume the board is transformed to the world coordinate by rotation, $\mathbf{R}$, and translation, $\mathbf{t}$. We have the observed coordinates $\{\hat{\mathbf{y}}_i\}$. Then we minimize the objective:

Because the board-corner correspondences are already known, this is an orthogonal Procrustes/Kabsch rigid-alignment problem. ICP is needed only when the correspondences also have to be estimated.

If the board-frame origin is at the board center, then the center of the board is $\mathbf{x}_0=\mathbf{t}^*$, and the normal of the board is $\mathbf{R}^*[0, 0, 1]^\top$.

Now we have refined board corners in the world coordinates, i.e., $\{\mathbf{y}_i^*\}^N_{i=1}$.

Unknown Board Dimension

If we do not have the board dimension, then we can directly fit the plane using the measured corners, $\{\mathbf{x}_i\}^N_{i=1}$ (with a slight abuse of notation). Use the plane convention $\mathbf{n}^\top \mathbf{x} + c = 0$ (with unit normal vector, i.e., $\|\mathbf{n}\| = 1$). Then we can solve for $\mathbf{n}$ and $c$ by minimizing the following objective:

For any fixed $\mathbf{n}$, taking derivative with respect to $c$ and setting it to zero, we have:

where $\bar{\mathbf{x}} = \frac{1}{N} \sum_i \mathbf{x}_i$ is the mean of the measured corners.

So, the objective function becomes:

where $\mathbf{\Sigma}_x = \frac{1}{N} \sum_i (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top$ is the biased sample covariance matrix of the measured corners.

Considering $\|\mathbf{n}\| = 1$, $\mathbf{n}$ is the eigenvector of $\mathbf{\Sigma}_x$ corresponding to its smallest eigenvalue.

The refined corners’ coordinates are the projection of the measured corners on the plane, which can be written as:

which has clear geometric meaning: the measured corners minus the signed distance along the plane normal. The sign of $\mathbf n$ remains arbitrary as long as $c$ is changed consistently.

Appendix B. Ray-Plane Geometry and Optional Residuals

The first two derivations use world-to-camera extrinsics $(R,t)$: $X^C=RX^W+t$. Relative to the main article’s camera-to-world pose, $R=(R^C)^\top$ and $t=-(R^C)^\top t^C$. The coordinate conversion is explicit here so that the main formulation retains one consistent local-to-world convention.

B.1 Closed-Form Ray-Plane Intersection

We are given a 3D plane:

and a camera model with the projection equation:

where:

• $\mathbf{u} \in \mathbb{R}^3$ is the homogeneous image coordinate (usually $\mathbf{u} = [u, v, 1]^\top$)
• $d$ is an unknown scale (depth) factor
• $\mathbf{K}$ is the camera intrinsic matrix
• $\mathbf{R}$ and $\mathbf{t}$ are the rotation and translation (extrinsic parameters) of the camera
• $\mathbf{x} \in \mathbb{R}^3$ is the 3D point we wish to recover

Our goal is to find $\mathbf{x}$ in terms of the known quantities $\mathbf{u}, \mathbf{K}, \mathbf{R}, \mathbf{t}, \mathbf{n},$ and $c$.

Rewrite the Projection Equation

However, the scale factor $d$ is still unknown.

Use the Plane Constraint to Solve for $d$

Since $\mathbf{x}$ lies on the plane, it must satisfy:

Substitute the expression we just found for $\mathbf{x}$:

Expanding the scalar equation gives

Now solve for $d$:

Note: It is important that $\mathbf{n}^\top \mathbf{R}^\top \mathbf{K}^{-1} \mathbf{u} \neq 0$ so that the division is valid (this means that the back-projected ray is not parallel to the plane).

Substitute the expression for $d$ back into the equation for $\mathbf{x}$:

B.2 Ray-Plane Intersection as an Optimization Problem

If we write the ray-plane intersection as an optimization, the pixel-space objective should be dimensionally consistent:

subject to the plane constraint

where $\mathbf{u}\in\mathbb R^2$ is a pixel coordinate and

If $\mathbf{u}$ instead denotes normalized undistorted image coordinates, then $\mathbf{K}$ can be omitted and the ray direction is $\mathbf{R}^{-1}[\mathbf{u}^\top,1]^\top$.

Even when the measured image coordinate $\mathbf{u}$ is noisy, the pinhole camera model combined with the infinite plane constraint induces a homography from the plane to the image. For any image coordinate, there is a back-projected ray, and if that ray intersects the plane, the corresponding 3D point projects exactly back to the same coordinate. Therefore, Eq. (1) has minimum value zero whenever the ray intersects the plane. By itself, ray-plane intersection does not correct noisy image measurements; board-marker geometry or other multi-view observations must provide the denoising constraint.

1. Parameterizing the Back‐Projection Ray

For normalized image coordinates, any 3D point $\mathbf{x}$ whose projection is $\mathbf{u}$ must satisfy

Solving for $\mathbf{x}$ gives

It is convenient to define the camera center in world coordinates as

and to introduce the direction vector

Thus, the 3D point $\mathbf{x}$ can be written as

All points of this form project exactly to $\mathbf{u}$, which means the optimal value of Eq. (1) is zero

2. Enforcing the Plane Constraint

Since $\mathbf{x}$ must lie on the plane, we impose

Substitute $\mathbf{x} = \mathbf{x}_c + \lambda\mathbf{d}$ into the plane equation:

Expanding, we obtain

Recalling that $\mathbf{x}_c = -\mathbf{R}^{-1}\mathbf{t}$, the equation becomes

Solving for $\lambda$, we have

or equivalently,

provided that $\mathbf{n}^\top \mathbf{d} \neq 0$.

A physically visible intersection also requires positive camera depth.

3. The Final Solution

Substitute the value of $\lambda$ back into the expression for $\mathbf{x}$:

This is the unique 3D point on the plane $\mathbf{n}^\top \mathbf{x} + c = 0$ selected by the measured ray, provided the ray is not parallel to the plane and the depth is positive.

4. Summary

a. Back‐Projection: The projection model

is a homography when restricted to the plane. Hence, every point along the ray

projects to $\mathbf{u}$.

b. Plane Intersection: The plane constraint $\mathbf{n}^\top \mathbf{x} + c = 0$ uniquely determines the scalar $\lambda$, and hence the unique point $\mathbf{x}$.

Thus, the final answer is

B.3 Optional Board-Plane Residual

The main formulation uses reprojection error. As an optional alternative, inverse ray-plane intersection gives the point where a back-projected image ray intersects a known plane, provided the ray is not parallel to the plane. This subsection returns to the local-to-world camera pose convention used in the main article.

For board $b$, its world-frame plane is

where

Camera $c$’s center in world coordinates is now directly:

For homogeneous image coordinate

the ray direction in camera coordinates is

The same direction in world coordinates is

The ray-plane intersection is

The board-generated world point is

The post-style residual would be

where

is the tangent basis of the board plane.

I recommend using the tangent-plane residual, not the full 3D residual, because both points are supposed to lie on the board plane and the meaningful error is along the board surface.

However, the reprojection objective is usually preferable because pixel noise is naturally modeled in image coordinates. The ray-plane residual requires converting pixel uncertainty into board-plane uncertainty, which depends on viewing angle and can become ill-conditioned when

Appendix C. Alternative Design Formulations

The main article uses camera-uncertainty minimization. This appendix retains alternative criteria and derivations for readers who need a different design objective or a discrete placement budget.

C.1 Single-Board Objective and Tolerance Scaling

A direct next-board-placement problem is

where $\mathcal F$ is the feasible set of board poses, and $D$ normalizes translation and rotation by required tolerances. For a single camera pose, one example is

For multiple cameras, use the corresponding block-diagonal normalization.

Other useful criteria are the D-optimal score

which reduces the overall uncertainty volume, and the E-optimal score

which improves the least observable camera-pose direction. For extrinsic calibration, the E-optimal criterion is often the more practical one, because a calibration can have a small average error while still containing one poorly constrained combination, such as camera height and pitch.

C.2 Overview of Alternative Criteria

Other experimental-design objectives can be substituted:

or a worst-camera objective:

C.3 Detailed Alternative Upper-Level Criteria

The A-optimal objective in (BL-A) minimizes the normalized trace of the predicted camera covariance. Other criteria are useful in different production settings.

D-optimal placement minimizes the uncertainty volume:

Equivalently,

E-optimal placement improves the weakest observable camera-pose direction:

This is often useful for camera extrinsic calibration because the failure mode is usually not the average error, but one poorly constrained direction, such as height-pitch coupling or lateral-yaw coupling.

If every camera must satisfy a production tolerance, use a worst-camera objective:

This avoids a placement that helps some cameras but leaves one camera poorly calibrated.

C.4 Abstract Bilevel Form and Objective Choices

Putting the lower and upper levels together gives the clean abstract formulation:

where

and $\mathcal L(x;s)$ is the joint calibration objective. The upper objective can be chosen according to the available information.

If the goal is truly optimized residual, use

If the study is a simulation and the ground-truth camera poses are known, use the actual camera-pose error:

For a real calibration station, the most practical design objective is information-based:

For EOL board-placement design, $\mathcal U_{\mathrm{info}}(s)$ is preferable to optimized residual because it directly measures predicted camera-pose uncertainty.

C.5 Why Optimized Residual Is Not the Placement Objective

If the upper level literally minimizes the final optimized residual, the bilevel problem is

The normalized version is usually better:

The normalization prevents a placement with fewer visible markers from appearing artificially better only because it has fewer residual terms. However, residual minimization is not the best placement-design objective. In the noise-free case, many feasible board placements can achieve nearly zero residual. A low optimized reprojection error does not necessarily mean that the camera pose is accurately estimated. A degenerate placement can fit its own observations while leaving some camera-pose directions weakly observable.

This is why the more useful second-level problem is an optimal experimental design problem: choose board placements that minimize predicted camera-pose uncertainty.

C.6 Residual Value Function and Envelope Derivative

The residual-minimization bilevel problem can also be written with the value function

Then

If the lower-level solution is unique and smooth, the envelope theorem gives

This is computationally convenient because the derivative of $x^\star(s)$ is not needed when the upper objective is exactly the optimized lower-level loss. For calibration placement design, however, $V(s)$ is usually not the best objective because minimized residual can be small even when the pose estimate is uncertain.

C.7 Fixed-Budget Discrete Board Selection

The 40-board replacement experiment in Section 4.5 is a discrete, fixed-budget approximation of the board-placement design problem. Let $\mathcal G$ be the finite set of candidate perimeter-board poses, let $\mathcal A\subset\mathcal G$ be the selected board set, and let $|\mathcal A|=40$. For each candidate $g$, compute its marginalized camera-pose information contribution

Then the fixed-budget subset problem is

subject to the same spacing, visibility, and placement constraints. The greedy swap procedure in Section 4.5 starts from the uniform set $\mathcal A_0$, evaluates replacements

and accepts the swap if

where $U(\mathcal A)$ is the A-optimal trace objective in (SUB). This is why the experiment uses the solution of the initial 40-board calibration: the solved state gives the Jacobian, the Schur-marginalized camera information, and the covariance metric used to decide which physical board placements should be replaced.

Appendix D. Two-Camera Bundle-Adjustment Special Case

Suppose the board marker coordinates in the board frame are $\{ \mathbf{x}_i^g \}_{i=1}^N$. The measured marker points in the world frame are $\{ \mathbf{x}_i^o \}_{i=1}^N$. The observed image markers in camera $c$ are $\{\mathbf{u}_{ci}\}$.

A standard bundle-adjustment objective should minimize pixel reprojection error, not a 3D inverse-ray-plane error. Let the board pose be

so the board-generated world point is

Let camera $c$ have world-to-camera extrinsic $(\mathbf{R}_c,\mathbf{t}_c)$ and intrinsic matrix $\mathbf{K}_c$. A clean two-camera objective is

The shared board pose already couples the two cameras: if both cameras observe marker $i$, both residuals use the same physical point $\mathbf{R}_0\mathbf{x}_i^g+\mathbf{t}_0$. Therefore a separate term such as

where $\phi_c$ is ray-plane intersection, is not an independent measurement. It is algebraically dependent on the same image observations and shared board or plane geometry, and it can double-count the same observations. The general multi-board formulation in Section 1 includes this two-camera reprojection objective as a special case.