In this post, we will discuss the end-of-line (EOL) camera calibration, especially for camera bird’s-eye view (BEV) extrinsic calibration.

Suggested Pipeline

1. Do single camera intrinsic calibration

2. Measure each corner in the world coordinate

3. Refine the corners’ coordinates according to board constraints (plane, parallel, equally spaced)

4. Find the plane equation in the world coordinate

5. Initialize each camera’s extrinsic parameters by solving the perspective-n-point (PnP) problem

6. Estimate the 3D coordinates by the intersection of the image ray and board plane

7. Construct the objective loss function (consistency of different cameras with common seen points and single camera’s observation)

Refine the Corner’s Coordinates by the Board Constraints

Assume the center of the board is located at the origin of a local coordinate, $(0,0,0)^\top$, the corner coordinate is located at $\{\mathbf{x}_i\}^N_{i=1}$, then the board normal is $(0,0,1)^\top$. Assume the board is transformed to the world coordinate by rotation, $\mathbf{R}$, and translation, $\mathbf{t}$. We have the observed coordinates $\{\hat{\mathbf{y}}_i\}$. Then we minimize the objectives:

This problem can be easily solved since it is a simple Iterative Closest Point (ICP) problem.

Then we can get the center of the board as $\mathbf{x}_0=\mathbf{t}^*$, and the normal of the board is $\mathbf{R}^*[0, 0, 1]^\top$.

Now we have refined board corners in the world coordinates, i.e., $\{\mathbf{y}_i^*\}^N_{i=1}$.

Unknown Board Dimension

If we do not have the board dimension, then we can directly fit the plane using the measured corners, $\{\mathbf{x}_i\}^N_{i=1}$ (with a slight abuse of notation). Assume the plane equation is $\mathbf{n}^\top \mathbf{x} - c = 0$ (with unit normal vector, i.e., $\|\mathbf{n}\| = 1$). Then we can solve for $\mathbf{n}$ and $c$ by minimizing the following objective:

For any fixed $\mathbf{n}$, taking derivative with respect to $c$ and setting it to zero, we have:

where $\bar{\mathbf{x}} = \frac{1}{N} \sum_i \mathbf{x}_i$ is the mean of the measured corners.

So, the objective function becomes:

where $\mathbf{\Sigma}_x = \frac{1}{N} \sum_i (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top$ is the biased sample covariance matrix of the measured corners.

Considering $\|\mathbf{n}\| = 1$, $\mathbf{n}$ is the eigenvector of $\mathbf{\Sigma}_x$ corresponding to its smallest eigenvalue.

The refined corners’ coordinates are the projection of the measured corners on the plane, which can be written as:

which has clear geometric meaning: the measured corners minus the projection on the plane normal.

The Intersection of the Image Ray and Board Plane

We are given a 3D plane:

and a camera model with the projection equation:

where:

• $\mathbf{u} \in \mathbb{R}^3$ is the homogeneous image coordinate (usually $\mathbf{u} = [u, v, 1]^\top$)
• $d$ is an unknown scale (depth) factor
• $\mathbf{K}$ is the camera intrinsic matrix
• $\mathbf{R}$ and $\mathbf{t}$ are the rotation and translation (extrinsic parameters) of the camera
• $\mathbf{x} \in \mathbb{R}^3$ is the 3D point we wish to recover

Our goal is to find $\mathbf{x}$ in terms of the known quantities $\mathbf{u}, \mathbf{K}, \mathbf{R}, \mathbf{t}, \mathbf{n},$ and $c$.

Rewrite the Projection Equation

However, the scale factor $d$ is still unknown.

Use the Plane Constraint to Solve for $d$

Since $\mathbf{x}$ lies on the plane, it must satisfy:

Substitute the expression we just found for $\mathbf{x}$:

Because the scalar product is invariant under transposition, we can write:

Now, solve for $d$:

Note: It is important that $\mathbf{n}^\top \mathbf{R}^\top \mathbf{K}^{-1} \mathbf{u} \neq 0$ so that the division is valid (this means that the back-projected ray is not parallel to the plane).

Substitute the expression for $d$ back into the equation for $\mathbf{x}$:

Solve the Intersection Problem as an Optimization Problem

We wish to solve

subject to the plane constraint

where: • $\mathbf{R}$ is the rotation matrix, • $\mathbf{t}$ is the translation vector, • $\mathbf{n}$ is the normal of the plane, • $c$ is a constant, and • $\mathbf{e} = \begin{pmatrix} 0 \ 0 \ 1 \end{pmatrix}$ (which selects the depth coordinate).

Even when the measured image coordinate $\mathbf{u} \in \mathbb{R}^2$ is noisy, the pinhole camera model combined with the plane constraint induces a projective mapping (a homography) from the plane to the image. In other words, for any $\mathbf{u}$ there exists an entire ray of points $\mathbf{x}$ that project exactly to $\mathbf{u}$. The plane constraint then selects one unique point on that ray.

1. Parameterizing the Back‐Projection Ray

Any 3D point $\mathbf{x}$ whose projection is $\mathbf{u}$ must satisfy

This implies that there exists a scalar $\lambda$ such that

Solving for $\mathbf{x}$ gives

It is convenient to define the camera center in world coordinates as

and to introduce the direction vector

Thus, the 3D point $\mathbf{x}$ can be written as

All points of this form project exactly to $\mathbf{u}$, which means the optimal value of Eq. (1) is zero

2. Enforcing the Plane Constraint

Since $\mathbf{x}$ must lie on the plane, we impose

Substitute $\mathbf{x} = \mathbf{x}_c + \lambda\mathbf{d}$ into the plane equation:

Expanding, we obtain

Recalling that $\mathbf{x}_c = -\mathbf{R}^{-1}\mathbf{t}$, the equation becomes

Solving for $\lambda$, we have

or equivalently,

provided that $\mathbf{n}^\top \mathbf{d} \neq 0$.

3. The Final Solution

Substitute the value of $\lambda$ back into the expression for $\mathbf{x}$:

This is the unique 3D point on the plane $\mathbf{n}^\top \mathbf{x} + c = 0$ that minimizes the reprojection error, even when $\mathbf{u}$ is noisy.

4. Summary

a. Back‐Projection: The projection model

is a homography when restricted to the plane. Hence, every point along the ray

projects to $\mathbf{u}$.

b. Plane Intersection: The plane constraint $\mathbf{n}^\top \mathbf{x} + c = 0$ uniquely determines the scalar $\lambda$, and hence the unique point $\mathbf{x}$.

Thus, the final answer is

Solve Step 3 and 6 Jointly as a Bundle Adjustment Problem

Suppose the board’s markers coordinate in the board’s frame are $\{ \mathbf{x}_i^g \}_{i=1}^N$ in the board’s frame. The measured marker points in the world frame are $\{ \mathbf{x}_i^o \}_{i=1}^N$. The observed image markers in camera 1 are $\{ \mathbf{u}_i^1 \}_{i=1}^{N_1}$ and the observed image markers in the second camera are $\{ \mathbf{u}_i^2 \}_{i=1}^{N_2}$.

We construct the loss as:

Where:

• the transformation from the local board to the world board, $\mathbf{T}_0$;
• The world coordinates of the board, $\{\mathbf{x}_i\}$;
• The transformation from the cameras to the world board, $\mathbf{T}_i$, where $i = 1, 2$;
• $C_{1,2}$ is the marker set commonly seen by camera 1 and camera 2;
• $\pi(\cdot)$ is the intersection of the camera ray and the world plane.

In the first part, we put a hard constraint that the true 3D position of the points should be a rigid transformation of the board.

where $I_\infty(x)= \begin{cases} 0, \text{ if } x=0, \\ \infty, \text{otherwise} \end{cases}$.

The second part of the loss is the difference between the observations of 3D points and the true position.

The third part is the perspective-n-point loss

The last one is the consistency loss. Here, we can simply put the $\ell_2$ loss on the points. Other losses are possible, e.g., image similarity loss, etc.

Simulations

We construct a 8x8 chessboard with each square’s size be of 1m. We set the the origin as the center of the board. We place the board in the world coordinate with extrinsic:

which means the normal of the board plane is $[0, 0, 1]^\top$.

We add Gaussian noise on the corners coordinates with noise zero mean and $0.005$ standard deviation.

With the noisy corners’ coordinates, we can get the optimized board’s extrinsic according to step 3. The optimized normal is $[1.38*10^{-3}, -2.16*10^{-5}, 9.999*10^{-1}]^\top$. That is already very close the ground truth.

Then we simulate two cameras with the same intrinsic matrix: $f_x =800$, $c_x = 500$, $f_y = 800$, $c_y=400$, and $H=800$, $W=1000$. We rotate the first camera along the z-axis $10$ degrees and the second $-10$ degrees, and translate the first camera along the x-axis $2$ meters and the second $-2$ meters. Then the images of the chessboard in the two cameras are shown in the figure below.

Then, we apply Gaussian blur on the observed images and add Gaussian noise on them. The new images are shown in the figure below.

After that, we use opencv “findChessboardCorners” and find the rough corners and then use “cornerSubPix” to refine the corners’ coordinates. The refined corners are shown in the figure below.

Pay attention to the sequence of the detected corners. (The detected corners start from the blue one and end with the green one in the above figure.) Rearrange the detected corners according to the refined chessboard corners. Using the “solvePnP” function with method “SQPNP”, which is a global optimization method, we can find the estimated extrinsic of the two cameras. We record the errors of the estimation using the original measurements of the chessboard corners and the refined ones.

The translation error is the norm of the difference between the estimated and the ground truth translation. The rotation error is defined as the norm of $\text{Log}(\mathbf{R}_1 \boxminus \mathbf{R}_2)$.

As shown in the table, the refined measurements result in lower errors for both rotation and translation, demonstrating the effectiveness of the corner refinement process.

In the final step, we do jointly optimization. We warp the image in camera 1 with parameters before and after jointly optimization. The results are shown in the figure below.

In the above figure, we can hardly tell which one is better. The translation error is $0.0298$ and $0.0257$ meters for camera 1 and camera 2 respectively. The rotation error is $0.204$ and $0.199$ degrees for camera 1 and camera 2 respectively. That is expected increased error for single camera calibration however more consistent for the two cameras.

Detect Corners on the Warped Image

Corner detection often becomes challenging on original images due to severe distortion or extreme viewing angles. To address this limitation, we can create a virtual camera with an optimized viewpoint. By warping the original image onto this virtual camera’s image plane, we can simulate a more frontal view of the chessboard. This transformation adjusts the perspective so that the chessboard appears as if viewed head-on, significantly improving corner detection reliability. We can then accurately detect the corners on this warped image rather than struggling with the difficult perspective of the original.

The figure below illustrates this approach. On the left is the original image where the severe viewing angle prevents the standard algorithm from successfully detecting the corners. On the right is the perspective-corrected warped image, where we can successfully identify and locate all corners.

Multi-Board and Multi-Camera Calibration Formulation

For multiple boards and cameras, the observation structure should be defined at the marker-point level. Since each camera may see only a subset of the markers on a board, the primitive observation is a camera-point edge:

A board-camera graph is only a compressed graph obtained after grouping all observed points on the same board. The post’s original two-camera formulation uses known board-frame marker coordinates, measured world-frame marker coordinates, image observations, a hard board-rigidity constraint, ray-plane inverse projection, and a cross-camera consistency term. The multi-board/multi-camera generalization should preserve the point-level observation structure.

Below is a careful formulation and two solvers: a point-level ADMM solver and a centralized sparse bundle-adjustment solver, which is what I would use when distribution is not required.

1. Sets and Graph

Let there be $B$ boards and $C$ cameras.

Boards:

Cameras:

Board $b$ has known marker points:

Marker $i$ on board $b$ has known board-frame coordinate:

For a planar checkerboard or marker board, usually

Define the global point set:

For point $p=(b,i)$, define

as the board containing point $p$, and define

The camera-point observation graph is

where

For each observation edge $(c,p)$, the measured image point is

Camera $c$ has known intrinsic matrix

Define the neighborhoods:

2. Pose Variables and Coordinate Convention

Use a uniform local-to-world pose convention for both boards and cameras. The leading world-frame symbol is omitted because local-to-world is the default: $T^B_b$ denotes the pose of board $b$, and $T^C_c$ denotes the pose of camera $c$.

For board $b$:

$$ \boxed{ T^B_b

\begin{bmatrix} R^B_b & t^B_b\ 0 & 1 \end{bmatrix} \in SE(3) } $$

maps board-local coordinates to world coordinates:

$$ \boxed{ X_p^W

T^B_{\beta(p)}\bar q_p. } $$

Here

Equivalently,

$$ \boxed{ X_p^W

R^B_{\beta(p)} q_p+t^B_{\beta(p)}. } $$

For camera $c$:

$$ \boxed{ T^C_c

\begin{bmatrix} R^C_c & t^C_c\ 0 & 1 \end{bmatrix} \in SE(3) } $$

maps camera-local coordinates to world coordinates. Therefore, to project a world point into camera $c$, use the inverse camera pose:

$$ \boxed{ \bar X_{cp}^C

(T^C_c)^{-1}\bar X_p^W. } $$

where $\bar X_p^W=[(X_p^W)^\top,1]^\top$.

Explicitly,

$$ (T^C_c)^{-1}

\begin{bmatrix} (R^C_c)^{\top} & -(R^C_c)^{\top}t^C_c\ 0 & 1 \end{bmatrix}, $$

and

$$ \boxed{ X_{cp}^C

(R^C_c)^{\top} \left( X_p^W-t^C_c \right). } $$

This is consistent with treating both the board and camera variables as local-to-world poses. The usual computer-vision world-to-camera extrinsic still appears, but now it is written explicitly as $(T^C_c)^{-1}$.

3. Projection Model

For point $p=(b,i)$, the world coordinate is

$$ \boxed{ X_p^W

T^B_{\beta(p)}\bar q_p. } $$

The camera-coordinate point is

$$ \boxed{ \bar X_{cp}^C

(T^C_c)^{-1}T^B_{\beta(p)}\bar q_p. } $$

The projected pixel is

$$ \boxed{ \hat u_{cp}

\Pi\left( K_c (T^C_c)^{-1} T^B_{\beta(p)} \bar q_p \right). } $$

If

then, for a standard pinhole model,

$$ \boxed{ \hat u_{cp}

\begin{bmatrix} f_{x,c}X/Z+c_{x,c}\ f_{y,c}Y/Z+c_{y,c} \end{bmatrix}. } $$

It is useful to define the shorthand

$$ \pi_c(X^W;T^C_c)

\Pi\left(K_c(T^C_c)^{-1}\bar X^W\right), $$

where $\bar X^W=[(X^W)^\top,1]^\top$. The reprojection residual is

$$ \boxed{ r_{cp}(T^C_c,T^B_{\beta(p)})

\Pi\left( K_c (T^C_c)^{-1} T^B_{\beta(p)} \bar q_p \right)-u_{cp}. } $$

If the image measurement covariance is

the whitened residual is

$$ \boxed{ e_{cp}(T^C_c,T^B_{\beta(p)})

\Sigma_{u,cp}^{-1/2} \left[ \Pi\left( K_c (T^C_c)^{-1} T^B_{\beta(p)} \bar q_p \right) -u_{cp} \right]. } $$

4. Board/World Measurement Priors

In the original EOL setup, some board marker coordinates may be measured in the world frame. Let

be the measured world coordinate of point $p$. Define the set of points with world measurements as

The board/world residual is

$$ \boxed{ r^z_p(T^B_{\beta(p)})

R^B_{\beta(p)} q_p+t^B_{\beta(p)}-z_p. } $$

With covariance

the board prior term is

$$ \boxed{ \phi_b(T^B_b)

\frac12 \sum_{p\in\mathcal P_b\cap\mathcal Z} \left| R^B_b q_p+t^B_b-z_p \right|{\Sigma{z,p}^{-1}}^2. } $$

You may also have camera pose priors, rig priors, or a fixed reference camera. Write those directly on the camera pose. Let $\bar T^C_c$ be the prior or nominal camera pose. The operator

is the Lie-group logarithm that converts a relative pose into a 6D tangent-vector error, usually ordered as three translation components and three rotation components. For example:

$$ \boxed{ \psi_c(T^C_c)

\frac12 \left| \operatorname{Log} \left( (\bar T^C_c)^{-1}T^C_c \right) \right|_{\Sigma_c^{-1}}^2. } $$

5. Correct Centralized Problem Formulation

The cleanest problem is:

This is the point-level multi-board, multi-camera bundle-adjustment problem.

The graph edge is $(c,p)$, but the residual depends on

So the raw observation graph is camera-point, while the variable graph is camera-board because each point is rigidly attached to a board.

This shared board-generated point is also the multi-camera consistency constraint. If two cameras $c_1$ and $c_2$ observe the same point $p$, both reprojection residuals use the same world point

Thus the two camera observations are coupled through the same board pose and the same physical marker point, rather than through two independent 3D point variables.

With the uniform local-to-world convention, the board-to-camera relative transform is

$$ \boxed{ T_{CB}^{(c,b)}

(T^C_c)^{-1}T^B_b. } $$

This maps board $b$’s local coordinates into camera $c$’s local coordinates, so the projection can also be written as

$$ \boxed{ \hat u_{cp}

\Pi \left( K_c T_{CB}^{(c,\beta(p))} \bar q_p \right). } $$

6. Gauge Freedom and Identifiability

Without an absolute anchor, each connected component has an $SE(3)$ gauge freedom.

For any

the transformation

leaves every relative camera-board projection unchanged:

$$ ((T^C_c)’)^{-1}(T^B_b)'

(HT^C_c)^{-1}(HT^B_b)

(T^C_c)^{-1}H^{-1}HT^B_b

(T^C_c)^{-1}T^B_b. $$

Therefore, every connected component needs at least one of the following:

or

Connectivity alone is not enough. The observations must also be geometrically nondegenerate: enough non-collinear board points, reasonable viewing angles, positive depth, and no nearly rank-deficient configurations.

7. ADMM Formulation Using Camera-Local Point Copies

ADMM is useful if you want a distributed or modular solver. Standard ADMM is a decomposition method well suited to distributed optimization, although the strongest convergence guarantees are for convex problems; this calibration problem is nonconvex because of $SE(3)$ and perspective projection.

Introduce a camera-local world-frame copy of each observed point:

These are still world-frame points. At consensus, each copy must equal the world point generated by the board pose:

$$ \boxed{ Y_{cp}

R^B_{\beta(p)} q_p+t^B_{\beta(p)}. } $$

This is the explicit consistency constraint across cameras. If cameras $c_1$ and $c_2$ both observe the same point $p$, then at ADMM consensus

$$ Y_{c_1p}

R^B_{\beta(p)} q_p+t^B_{\beta(p)}

Y_{c_2p}. $$

So a separate pairwise term such as $\|Y_{c_1p}-Y_{c_2p}\|^2$ is not necessary; all camera-local copies are tied to the same board-generated world point.

where $\bar Y_{cp}=[Y_{cp}^\top,1]^\top$.

Define the local reprojection loss:

$$ \boxed{ \ell_{cp}(T^C_c,Y_{cp})

\frac12 \left| \Sigma_{u,cp}^{-1/2} \left[ \Pi\left( K_c (T^C_c)^{-1} \bar Y_{cp} \right) -u_{cp} \right] \right|^2. } $$

Then the ADMM-split problem is:

$$ \boxed{ \begin{aligned} \min_{{T^C_c},{T^B_b},{Y_{cp}}} &\quad \sum_{(c,p)\in\mathcal E} \ell_{cp}(T^C_c,Y_{cp}) + \sum_b\phi_b(T^B_b) + \sum_c\psi_c(T^C_c) \ \text{s.t.} &\quad Y_{cp}

R^B_{\beta(p)} q_p+t^B_{\beta(p)}, \qquad (c,p)\in\mathcal E. \end{aligned} } \tag{A} $$

When the constraints are satisfied, this is equivalent to the centralized problem (P).

8. Scaled Augmented Lagrangian

For each observation edge, introduce a scaled dual variable

Let the ADMM penalty be

The scaled augmented Lagrangian is:

$$ \boxed{ \begin{aligned} \mathcal L_\mu &= \sum_{(c,p)\in\mathcal E} \ell_{cp}(T^C_c,Y_{cp}) + \sum_b\phi_b(T^B_b) + \sum_c\psi_c(T^C_c) \ &\quad+ \sum_{(c,p)\in\mathcal E} \frac{\mu_{cp}}{2} \left| Y_{cp}

R^B_{\beta(p)} q_p

t^B_{\beta(p)} + \lambda_{cp} \right|^2. \end{aligned} } $$

Constants such as

are omitted because they do not affect the primal minimizers.

9. ADMM Algorithm

At iteration $k$, perform three steps.

Step 1: Camera-Local Update

For each camera $c$, solve independently:

$$ \boxed{ \begin{aligned} (T^C_c)^{k+1}, {Y_{cp}^{k+1}}_{p\in\mathcal N(c)}

\arg\min_{T^C_c,{Y_{cp}}} &\quad \psi_c(T^C_c) \ &+ \sum_{p\in\mathcal N(c)} \frac12 \left| \Sigma_{u,cp}^{-1/2} \left[ \Pi\left( K_c(T^C_c)^{-1}\bar Y_{cp} \right) -u_{cp} \right] \right|^2 \ &+ \sum_{p\in\mathcal N(c)} \frac{\mu_{cp}}{2} \left| Y_{cp}

X_p^k + \lambda_{cp}^k \right|^2, \end{aligned} } \tag{ADMM-1} $$

where

$$ \boxed{ X_p^k

(R^B_{\beta(p)})^k q_p+(t^B_{\beta(p)})^k. } $$

This step is parallel over cameras.

The residuals used inside a Gauss-Newton or Levenberg-Marquardt solve are:

$$ r_{cp}^{\text{img}}

\Sigma_{u,cp}^{-1/2} \left[ \Pi\left( K_c(T^C_c)^{-1}\bar Y_{cp} \right) -u_{cp} \right], $$

and

$$ r_{cp}^{\text{cons}}

\sqrt{\mu_{cp}} \left[ Y_{cp}

X_p^k + \lambda_{cp}^k \right]. $$

Use Lie updates for the camera pose:

The projection internally uses the inverse $(T^C_c)^{-1}$. Use ordinary Euclidean updates for the local point copies:

This subproblem is still nonlinear because of projection, but it is local to one camera.

Step 2: Board Update

For each board $b$, solve independently:

$$ \boxed{ \begin{aligned} (T^B_b)^{k+1}

\arg\min_{T^B_b\in SE(3)} &\quad \phi_b(T^B_b) \ &+ \sum_{p\in\mathcal P_b} \sum_{c\in\mathcal N(p)} \frac{\mu_{cp}}{2} \left| Y_{cp}^{k+1}

R^B_b q_p

t^B_b + \lambda_{cp}^k \right|^2. \end{aligned} } \tag{ADMM-2} $$

This step is parallel over boards.

Rewrite the ADMM target as:

Define

$$ \tau_{cp}^{k+1}

Y_{cp}^{k+1}+\lambda_{cp}^k. $$

If $\phi_b$ consists only of isotropic point measurements,

$$ \phi_b(T^B_b)

\sum_{p\in\mathcal P_b\cap\mathcal Z} \frac{w_{z,p}}{2} |R^B_b q_p+t^B_b-z_p|^2, $$

then the board update is a weighted rigid alignment problem:

The target set contains two types of targets.

For every camera observation $(c,p)$:

For every measured world point $p\in\mathcal P_b\cap\mathcal Z$:

Compute weighted centroids:

$$ \bar q

\frac{\sum_j\alpha_jq_j}{\sum_j\alpha_j}, $$

$$ \bar\tau

\frac{\sum_j\alpha_j\tau_j}{\sum_j\alpha_j}. $$

Compute weighted covariance:

$$ H

\sum_j \alpha_j (q_j-\bar q)(\tau_j-\bar\tau)^\top. $$

Take the SVD:

Then

$$ \boxed{ (R^B_b)^{k+1}

V \operatorname{diag} \left( 1,1,\det(VU^\top) \right) U^\top, } $$

and

$$ \boxed{ (t^B_b)^{k+1}

\bar\tau-(R^B_b)^{k+1}\bar q. } $$

So

$$ \boxed{ (T^B_b)^{k+1}

\begin{bmatrix} (R^B_b)^{k+1}&(t^B_b)^{k+1}\ 0&1 \end{bmatrix}. } $$

If $\phi_b$ has anisotropic covariances or a general pose prior, solve (ADMM-2) by a small 6-DoF LM solve instead of closed-form Kabsch.

Step 3: Dual Update

For every observation edge $(c,p)\in\mathcal E$:

$$ \boxed{ \lambda_{cp}^{k+1}

\lambda_{cp}^{k} + Y_{cp}^{k+1}

(R^B_{\beta(p)})^{k+1} q_p

(t^B_{\beta(p)})^{k+1}. } \tag{ADMM-3} $$

The primal residual is

$$ \boxed{ r_{p,cp}^{k+1}

Y_{cp}^{k+1}

(R^B_{\beta(p)})^{k+1} q_p

(t^B_{\beta(p)})^{k+1}. } $$

A practical dual residual is

$$ \boxed{ r_{d,cp}^{k+1}

\mu_{cp} \left[ \left((R^B_{\beta(p)})^{k+1} q_p+(t^B_{\beta(p)})^{k+1}\right)

\left((R^B_{\beta(p)})^{k} q_p+(t^B_{\beta(p)})^{k}\right) \right]. } $$

Stop when

and

are below chosen tolerances.

10. ADMM Pseudocode

Input:
  Boards b = 1..B
  Cameras c = 1..C
  Points p = (b, i)
  Observation graph E = {(c, p)}
  Board-frame coordinates q[p]
  Pixel observations u[c, p]
  Intrinsics K[c]
  Optional world measurements z[p]
  Optional camera/rig priors
Initialize:
  Board poses (T^B_b)^0
  Camera poses (T^C_c)^0
  Local copies:
      Y_cp^0 = (R^B_beta(p))^0 q_p + (t^B_beta(p))^0
  Dual variables:
      lambda_cp^0 = 0
  Penalties:
      mu_cp > 0
Repeat until convergence:
  # Camera-local step: parallel over cameras
  for each camera c:
      solve nonlinear least squares over T^C_c and {Y_cp}_{p in N(c)}:
          image reprojection residuals using (T^C_c)^-1
          + ADMM consensus residuals to current board points
  # Board step: parallel over boards
  for each board b:
      solve rigid alignment:
          board-generated points R^B_b q_p + t^B_b
          should match Y_cp + lambda_cp
          plus measured world points z_p, if available
  # Dual step: parallel over observation edges
  for each observation edge (c, p):
      lambda_cp <- lambda_cp
                   + Y_cp
                   - R^B_beta(p) q_p
                   - t^B_beta(p)
  Check primal and dual residuals.

11. Initialization

ADMM will be sensitive to initialization because the problem is nonconvex.

Good initialization options:

1. If world marker measurements exist: initialize each board pose $(T^B_b)^0$ by weighted Kabsch from $q_p\mapsto z_p$.
2. Initialize cameras from board/world points: once board poses are initialized, compute

and solve PnP for each camera using observed correspondences

1. If no world measurements exist: fix one board pose or one camera pose to remove gauge, estimate local board-camera relative poses where possible, and propagate poses through the observation graph.
2. Initialize local point copies with the current board predictions:

1. Initialize duals to zero:

12. Penalty Choice

Use $\mu_{cp}$ in world-coordinate units. If local point copies are in meters, $\mu_{cp}$ has units like inverse square meters.

A reasonable initial choice is to relate $\mu_{cp}$ to expected 3D point uncertainty:

For example, if the board/world point uncertainty is about $5\text{ mm}$,

But this can be too stiff early in optimization. A practical strategy is to start with a smaller $\mu$, then increase it as the primal residual decreases.

Use residual balancing:

When changing $\mu$, rescale the scaled dual variable appropriately.

13. What This ADMM Split Buys You

This ADMM formulation respects the camera-point graph.

It gives:

This is better than a board-camera relative-pose ADMM when a camera may observe only a few points on a board. A board-camera relative pose

is only meaningful as a local measurement if camera $c$ sees enough non-collinear points on board $b$ to estimate the board pose. The point-level ADMM does not require that.

14. Optional: Post-Style Ray-Plane Residual

The formulation above uses reprojection error. That is the statistically cleaner choice.

The post instead discusses inverse ray-plane intersection. It derives the unique point where the back-projected image ray intersects a known plane, provided the ray is not parallel to the plane.

For board $b$, its world-frame plane is

where

Camera $c$’s center in world coordinates is now directly:

For homogeneous image coordinate

the ray direction in camera coordinates is

$$ d_{cp}^{C}

K_c^{-1}\tilde u_{cp}. $$

The same direction in world coordinates is

$$ \boxed{ d_{cp}^{W}

R^C_c d_{cp}^{C}. } $$

The ray-plane intersection is

$$ \boxed{ \phi_{cp}(T^C_c,T^B_b)

o_c

\frac{n_b^\top o_c+c_b} {n_b^\top d_{cp}^{W}} d_{cp}^{W}. } $$

The board-generated world point is

The post-style residual would be

$$ \boxed{ r_{cp}^{\text{ray}}

P_b^\top \left[ \phi_{cp}(T^C_c,T^B_{\beta(p)})

\left( R^B_{\beta(p)} q_p+t^B_{\beta(p)} \right) \right], } $$

where

is the tangent basis of the board plane.

I recommend using the tangent-plane residual, not the full 3D residual, because both points are supposed to lie on the board plane and the meaningful error is along the board surface.

However, the reprojection objective is usually preferable because pixel noise is naturally modeled in image coordinates. The ray-plane residual requires converting pixel uncertainty into board-plane uncertainty, which depends on viewing angle and can become ill-conditioned when

15. Better Centralized Method: Sparse Bundle Adjustment

If the calibration can be solved on one machine, I would use sparse nonlinear least squares instead of ADMM.

The problem is exactly:

Use pose increments:

Linearize every residual:

Stack all residuals and solve the damped normal equations:

$$ \boxed{ \left(J^\top J+\lambda D\right)\delta

-J^\top r. } $$

Each residual touches only one board pose and one camera pose. Therefore the normal matrix has sparse block structure:

$$ \begin{bmatrix} H_{CC} & H_{CB}\ H_{BC} & H_{BB} \end{bmatrix} \begin{bmatrix} \delta C\ \delta B \end{bmatrix}

\begin{bmatrix} g_C\ g_B \end{bmatrix}. $$

If there are no camera-camera priors, $H_{CC}$ is block diagonal over cameras before Schur fill-in. If there are no board-board priors, $H_{BB}$ is block diagonal over boards before Schur fill-in.

You can eliminate boards:

$$ \boxed{ \left( H_{CC}

H_{CB}H_{BB}^{-1}H_{BC} \right) \delta C

-g_C + H_{CB}H_{BB}^{-1}g_B. } $$

Then back-substitute:

$$ \boxed{ \delta B

-H_{BB}^{-1} \left( g_B+H_{BC}\delta C \right). } $$

Or eliminate cameras instead, depending on whether there are fewer cameras or fewer boards.

This is the same structural idea behind Schur methods in bundle adjustment. Ceres’ documentation describes bundle adjustment as minimizing reprojection error over cameras and 3D points, and its solver documentation explains that bundle adjustment has a special block structure that can be exploited with Schur complement methods.

16. Factor-Graph View

The centralized problem is also a factor graph.

Variables:

Factors:

for every observation edge $(c,p)\in\mathcal E$.

Board priors:

Camera priors:

GTSAM describes factor graphs as bipartite graphs connecting factors to the variables they depend on, and emphasizes that such sparse structure is exploited by nonlinear optimization methods.

This is arguably the cleanest implementation model: one factor per camera-point observation, with the point’s board membership determining which board pose variable the factor touches.

17. Practical Recommendation

Use this hierarchy:

Best if centralized

Use sparse factor-graph bundle adjustment:

This gives the best local convergence and avoids ADMM penalty tuning.

Best if distributed

Use point-level ADMM:

with:

This respects the actual camera-point observation graph and supports partial board visibility.

Best engineering pipeline

1. Initialize board poses from world measurements, if available.
2. Initialize cameras using PnP against current board-generated world points.
3. Run sparse centralized BA if possible.
4. Use ADMM only if the system must be distributed, parallelized by camera/board, or too large for a single optimizer.
5. Optionally use robust losses or outlier rejection on image residuals.

The main final formulation is:

Use this convention consistently:

$$ \boxed{ T_{CB}^{(c,b)}

(T^C_c)^{-1}T^B_b: \text{board } b \rightarrow \text{camera } c. } $$

Everything else, including ADMM, sparse Schur bundle adjustment, and factor-graph optimization, is a solver strategy for this same constrained geometric estimation problem.

18. How the Formulation Guides Board Placement

The centralized formulation (P), not the ADMM split (A), is the useful object for deciding where to put the target board. The ADMM penalty, dual variables, and local point copies are solver artifacts. Board-placement decisions should be based on the physical reprojection terms and any measurement priors in (P).

The objective value itself is not a good placement guide. For a hypothetical board placement with perfectly predicted observations, the expected reprojection residual is zero almost everywhere. The useful placement information is in the local curvature of the objective: the reprojection Jacobian and the Hessian or information matrix.

Linearized Measurement Information

For point $p$ on board $b$, define the predicted camera-coordinate point

$$ x_{cp}

\begin{bmatrix} X_{cp}\Y_{cp}\Z_{cp} \end{bmatrix}

(R^C_c)^\top \left( R^B_b q_p+t^B_b-t^C_c \right). $$

The weighted reprojection residual is

$$ r_{cp}

\Sigma_{u,cp}^{-1/2} \left[ \Pi(K_cx_{cp})-u_{cp} \right]. $$

For a candidate board pose $T^B_b$, linearize the residual around the current camera and board estimates:

where

$$ A_{cp}

\frac{\partial r_{cp}}{\partial\delta\eta_c}, \qquad B_{cp}

\frac{\partial r_{cp}}{\partial\delta\xi_b}. $$

For the pinhole model,

$$ J_{\pi}(x_{cp})

\begin{bmatrix} f_{x,c}/Z_{cp} & 0 & -f_{x,c}X_{cp}/Z_{cp}^{2} \ 0 & f_{y,c}/Z_{cp} & -f_{y,c}Y_{cp}/Z_{cp}^{2} \end{bmatrix}. $$

For this placement-scoring derivation, use local right perturbations

Then the camera-pose Jacobian is

$$ \boxed{ A_{cp}

\Sigma_{u,cp}^{-1/2} J_\pi(x_{cp}) \begin{bmatrix} -I & [x_{cp}]_\times \end{bmatrix}. } $$

The board-pose Jacobian is

$$ \boxed{ B_{cp}

\Sigma_{u,cp}^{-1/2} J_\pi(x_{cp}) (R^C_c)^\top R^B_b \begin{bmatrix} I & -[q_p]_\times \end{bmatrix}. } $$

These Jacobians convert a candidate physical board placement into expected camera-pose information.

Marginalized Camera Information

Stack all residual Jacobians for the candidate board pose:

Here $m$ is the number of predicted camera-point observation edges for this candidate pose.

Let $\Lambda_b^{\mathrm{prior}}$ denote the information supplied by surveyed world coordinates, a board-pose measurement, or another board prior. The candidate’s joint information contribution is

$$ \Delta H_b

\begin{bmatrix} A_b^\top A_b & A_b^\top B_b\ B_b^\top A_b & B_b^\top B_b+\Lambda_b^{\mathrm{prior}} \end{bmatrix}. $$

For camera-pose guidance, the board pose is usually a nuisance variable. Marginalizing it gives the effective information contributed to the camera pose:

$$ \boxed{ \Delta\Lambda_c(T^B_b)

A_b^\top A_b

A_b^\top B_b \left( B_b^\top B_b+\Lambda_b^{\mathrm{prior}} \right)^{-1} B_b^\top A_b. } $$

This expression has three useful interpretations.

1. If the board pose is accurately known, then $\Lambda_b^{\mathrm{prior}}\rightarrow\infty$, and $\Delta\Lambda_c\approx A_b^\top A_b$. Almost all image information contributes to the camera pose.
2. If the board pose is poorly known, the second term becomes large. Much of the apparent image information is spent estimating the board pose.
3. With no world anchor, a simultaneous global transformation of all camera and board poses leaves projections unchanged. Absolute camera pose is unobservable; only relative board-to-camera pose is constrained.

For the complete multi-camera system, append the hypothetical candidate information to the current normal matrix:

$$ H(T^B_b)

H_{\mathrm{current}} + \Delta H(T^B_b). $$

Partition the result into camera poses $C$ and nuisance variables $N$, including board poses:

$$ H

\begin{bmatrix} H_{CC} & H_{CN}\ H_{NC} & H_{NN} \end{bmatrix}. $$

The marginalized camera-pose information is

$$ \boxed{ \Lambda_C(T^B_b)

H_{CC}

H_{CN}H_{NN}^{-1}H_{NC}. } $$

The predicted camera-pose covariance is approximately

This is the next-best-pose principle: append the expected measurements of a hypothetical pose, compute the resulting information, and select the pose that gives the largest uncertainty reduction.

Board-Placement Objective

A direct next-board-placement problem is

$$ \boxed{ (T^B_b)^*

\underset{T^B_b\in\mathcal F}{\arg\min} ; \operatorname{tr} \left( D\Sigma_C(T^B_b)D^\top \right), } $$

where $\mathcal F$ is the feasible set of board poses, and $D$ normalizes translation and rotation by required tolerances. For a single camera pose, one example is

$$ D

\operatorname{diag} \left( 1/\tau_x,, 1/\tau_y,, 1/\tau_z,, 1/\tau_{\mathrm{roll}},, 1/\tau_{\mathrm{pitch}},, 1/\tau_{\mathrm{yaw}} \right). $$

For multiple cameras, use the corresponding block-diagonal normalization.

Other useful criteria are the D-optimal score

which reduces the overall uncertainty volume, and the E-optimal score

which improves the least observable camera-pose direction. For extrinsic calibration, the E-optimal criterion is often the more practical one, because a calibration can have a small average error while still containing one poorly constrained combination, such as camera height and pitch.

Physical Placement Preferences

The information matrix translates into concrete board-placement guidance:

Therefore, repeatedly placing a small, fronto-parallel board at the image center is usually inefficient. It generates nearly redundant Jacobian rows. A better sequence contains:

1. A large, well-centered observation for robust initialization.
2. A laterally displaced observation with tilt about one board axis.
3. An observation on the opposite side with tilt about the other axis.
4. A different board-to-camera distance.
5. For multi-camera calibration, placements in camera-overlap regions so the same board points are observed by multiple cameras.

The exact next placement should be determined by the weakest eigenvector of the current marginalized information matrix. For example, if the largest covariance direction is mostly a combination of camera $t_z$ and pitch, the next-best score will usually favor a laterally displaced and tilted board rather than another centered, fronto-parallel observation.

Visibility and Corner-Quality Constraints

Pure geometric information can favor an excessively oblique board. In practice, strong foreshortening makes corners elongated, blurred, or difficult to detect. Candidate scoring should therefore use pose-dependent image covariance,

$$ \Sigma_{u,cp}

\Sigma_{u,cp}(T^B_b), $$

or an expected detection probability:

$$ \Delta H(T^B_b)

\sum_{(c,p)\in\mathcal E(T^B_b)} p_{\mathrm{det},cp}(T^B_b) J_{cp}^{\top} \Sigma_{u,cp}^{-1}(T^B_b) J_{cp}. $$

This balances geometric strength against actual feature-localization accuracy. Otherwise, the placement optimizer can prefer grazing views that look informative geometrically but produce anisotropic and unreliable corner measurements.

The feasible set $\mathcal F$ should therefore require:

enough visible markers, image points inside the image with a safety margin, sufficient projected square size, acceptable focus and illumination, and a non-grazing incidence angle.

Practical Next-Board-Placement Loop

The resulting EOL procedure is:

1. Solve centralized problem (P) using all measurements currently available.
2. Compute the current sparse Hessian and camera-pose covariance.
3. Generate feasible candidate board poses in the calibration workspace.
4. For each candidate, predict visibility, pixel positions, corner covariance, and reprojection Jacobians.
5. Add its predicted information to the current Hessian.
6. Marginalize board poses and other nuisance variables with the Schur complement.
7. Select the board pose giving the largest normalized camera-pose uncertainty reduction.
8. Place and survey the board, capture the observations, resolve (P), and repeat.

The central idea is: