Vision-Guided Brick Sorting with a Franka Arm

robo.innovate Hackathon 2026 · Munich · Track 08 by Marmon

One brick at
a time.

Marmon’s example kiln car held 28,800 bricks. In six days, our team built a tabletop prototype that could find a reachable brick, judge its colour, and tell a Franka arm where to pick it.

My role
Perception lead · grasp testing · final pitch
Team
KalmanKrew, 5 people
Hardware
Franka Research 3 + RealSense D435
Timeline
6 days, 23–28 Oct 2026
Follow one camera frame
Live on the Franka Research 3

From kiln car to lab table

After firing, every brick on a kiln car has to be inspected, sorted by shade, and restacked onto pallets. Marmon challenged us to automate this unloading step, known as dehacking.

On our lab table, a woven stack of LEGO bricks stood in for a kiln car. The first decision was the same at either scale. Which brick can the robot pick without disturbing the stack?

In the brickyardOn our table
A kiln car of fired bricksA woven LEGO stack on a baseplate
Unload layer by layerPick only the exposed top layer
Inspect and grade by shadeClassify 5 LEGO colours
Palletize and scrap rejectsOne pile per good colour, plus a scrap bin
A real brick sample labelled Bad, with LEGO bricks in front of it
A real "bad" sample from the sponsor next to the LEGO bricks that stood in for it.

My role on KalmanKrew

Shiva explaining the perception pipeline to teammates at the Franka robot cell

I led perception for the LEGO demo, taking the pipeline from raw camera frames to 3D targets for the motion planner.

  • Perception: top-layer detection, size-based mask cleanup, colour classification and 3D pose export.
  • Tooling: RGB-D capture, live detection and a modular runner for replaying recorded frames.
  • On the robot: tuned grasps with the manipulation team and tested the full sorting loop.

From 61 masks to 22 pick targets

The camera sees the top layer, parts of lower layers, and loose bricks around the stack. Select a stage below to see how one RealSense frame becomes a set of 3D pick targets. You can also use the arrow keys after focusing a stage.

Step 1 — Fit the table, then read the scene 1 RGB-D frame
Step 2 — Turn depth into height height map
Step 3 — Ask SAM 3 for “lego brick” 61 masks
Step 4 — Keep only what the gripper can reach 23 masks
Step 5 — Classify colour, fit a 3D pick target 22 targets · 18 ✓ · 4 ✗
  1. The wrist-mounted D435 captures an aligned colour + depth frame. RANSAC fits the dominant plane (green tint) — that is the table. Those same table pixels later serve as the white reference for colour.

  2. Every depth pixel is back-projected and measured against the table plane. Warmer = higher, so the top layer of the stack stands out before any learning is involved.

  3. A single text prompt returns instance masks for every visible brick — top layer, lower layers and loose ones alike. Colours here are just instance IDs.

  4. A height histogram finds the highest substantial surface; only masks within one brick height (≈ 9.6 mm) of it survive. Lower layers and loose table bricks drop out.

  5. A brick-size prior splits merged masks and rejects partial ones. Each brick is colour-classified in CIELAB and gets a fitted centre and yaw: green = accept, red = scrap, yellow dot = grasp centre.

61 SAM 3 masks 23 top layer 22 pick targets 18 accept 4 scrap

One real capture from the RealSense D435, top-down over the LEGO stack. Stage images from the team's annotated pipeline figure.

Which bricks are reachable?

SAM 3 is very good at finding brick-shaped regions, but it has no notion of physical size. Two touching bricks can come back as one mask. A brick half-hidden under the layer above still looks like a brick, and so does the side face of an edge brick.

To fix this, I project each top-layer mask onto the table plane and measure its footprint in millimetres. I then compare that against a known brick size: L × W × H ≈ 30 × 15 × 10 mm. The prior then makes one of four calls:

≈ L × W

One clean brick

keep

W × n·L

Bricks SAM merged

split into n

W × < L

Mostly hidden by the layer above

reject

< W wide

Side face of an edge brick

reject

Every brick that survives gets a fitted L × W footprint. That gives its centre in the camera frame, its yaw on the table and its layer index, which is exactly what a grasp planner needs.

Which pile takes each brick?

The lab had bright desk lamps, glare on glossy plastic and shifting exposure. Matching raw RGB values would have mislabelled bricks constantly. The classifier corrects for that in three steps:

  1. White-balance on the table. The plane fit already says which pixels are table, so the colour model uses them as a grey reference in linear RGB.
  2. Measure the clean interior. Colour is sampled inside the fitted footprint, away from the edges. The darkest 15 % (shadows) and brightest 10 % (glare) are dropped.
  3. Match in CIELAB with a shared offset. One bounded Lab offset is fitted jointly across all bricks in the frame to absorb exposure drift. Then each brick takes its nearest reference colour, with a confidence from distance and margin.
Red×7Pile A
Dark red×5Pile B
Dark orange×6Pile C
Reddish brown×3Scrap
Medium nougat×1Scrap

Counts from the capture above.

Where should the gripper go?

Perception ends at a clean, documented contract. For every brick, the pipeline writes a 3D box in the camera optical frame (x right, y down, z forward, metres). The robotics side applies the hand-eye transform and plans the motion.

0123467box centreyawxyz
# per brick → bricks.json, poses.csv
id, colour, quality, confidence
layer           # index in the stack
centre_xyz      # box centre [m]
top_centre_xyz  # top face   [m]
yaw             # on the table
size_lwh        # fitted size
corners[8]      # top 0–3, bottom 4–7

I also wrapped the stages in a modular pipeline runner. It can capture a fresh frame or replay a recording, run segmentation, classification and pose export as separate processes, and give every run its own timestamped folder. If any stage fails, it stops and writes an explicit INCOMPLETE.json, so stale results are never silently reused.

Pick, place, look again

Even a successful pick can nudge the next brick out of position. The Franka picks a few bricks, returns to its observation pose, and rescans before committing to the next set of targets.

Observation pose Perceivepipeline above Pick ≤ N brickstop layer only Placecolour pile or scrap
The Franka arm above the LEGO stack, with sorted bricks on the tabletop
The working cell: stack on the left, sorted colour piles on the right.

I helped tune and test the grasping side alongside our manipulation teammates. Most of the work went into details that only show up on real hardware:

  • Grasp width and force for small, glossy 2×4 bricks
  • Pre-grasp clearance and offsets so the fingers don't hit neighbouring bricks
  • 3D-printed gripper fingers
  • Varying layer heights as the stack shrinks
  • A camera-to-gripper transform taken from CAD and refined on the robot

We demonstrated sorting on the Franka. Measuring pick success across repeated trials is the next step.

The final pitch

On day six, I presented three parts of the same idea to the jury. The Franka showed how to unload a stack, the real-brick inspector showed how to grade fired bricks, and the simulation showed how those steps could fit into a factory line.

Beyond the lab table

My teammates built two companion prototypes.

Real-brick inspection

Grading real fired bricks

SAM 3 masks, hand-crafted colour and texture features, and DINOv2 patch features, fused through two linear SVMs. On 258 photos of the 10 sponsor samples, 9 of 10 bricks were graded correctly: all 4 good bricks accepted, 5 of 6 bad ones rejected.

Factory simulation

Scaling the idea up

A simulated line covering dehacking, inspection, reject removal and palletizing, with a nominal throughput of ≈ 6,400 bricks per hour. The throughput estimate comes from scripted motion and simulated shade labels.

Six days in Munich

What I’d take into the next project

Geometry first

Use depth to narrow the search

A plane fit and a height histogram narrowed the candidate masks to the working layer. That made it easier to trace failed picks back to the geometry or the mask.

Calibrate to the scene

Use the world as a reference

The table doubled as a white reference, and one Lab offset per frame absorbed exposure drift. Both were more reliable than tuning camera settings for every lighting change.

Close the loop

Rescan beats predict

Rescanning after a few picks handled disturbed bricks and failed motions far better than trying to plan the whole stack from one frame.

Next step

Validate on unseen bricks

The inspector was evaluated on ten physical samples. Testing on unseen bricks and measuring pick success across repeated runs would show how well the system generalises.

Team KalmanKrew: Shiva Rudra (perception, pitch), Varun Teja (perception and real-brick classification), Tanay Ranjan (perception–manipulation integration, pick-and-rescan loop), Ameed Nazhurudeen (manipulation), and Ahmet Sinan Erdem (vision and simulation). Mentored by Peter So; challenge hosted by Jeff Garascia and Michael Ostrow from Marmon.

Shiva Rudra Lolla
Shiva Rudra Lolla
Robotics Researcher

My research interests include autonomous systems and perception. Drop me a message if you want to connect!