<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Computer Vision | Portfolio</title><link>https://lshivarudra.github.io/tag/computer-vision/</link><atom:link href="https://lshivarudra.github.io/tag/computer-vision/index.xml" rel="self" type="application/rss+xml"/><description>Computer Vision</description><generator>Hugo Blox Builder (https://hugoblox.com)</generator><language>en-us</language><lastBuildDate>Wed, 28 Oct 2026 00:00:00 +0000</lastBuildDate><image><url>https://lshivarudra.github.io/media/icon_hu0b7a4cb9992c9ac0e91bd28ffd38dd00_9727_512x512_fill_lanczos_center_3.png</url><title>Computer Vision</title><link>https://lshivarudra.github.io/tag/computer-vision/</link></image><item><title>Vision-Guided Brick Sorting with a Franka Arm</title><link>https://lshivarudra.github.io/project/brickpalletizing/</link><pubDate>Wed, 28 Oct 2026 00:00:00 +0000</pubDate><guid>https://lshivarudra.github.io/project/brickpalletizing/</guid><description>&lt;div class="bp-story">
&lt;section class="bp-hero" aria-label="Project introduction">
&lt;div class="bp-hero-copy">
&lt;span class="bp-eyebrow">robo.innovate Hackathon 2026 · Munich · Track 08 by Marmon&lt;/span>
&lt;p class="bp-lede">One brick at&lt;br> &lt;em>a time.&lt;/em>&lt;/p>
&lt;p class="bp-deck">Marmon’s example kiln car held 28,800 bricks. In six days, our team built a tabletop prototype that could find a reachable brick, judge its colour, and tell a Franka arm where to pick it.&lt;/p>
&lt;dl class="bp-meta">
&lt;div>&lt;dt>My role&lt;/dt>&lt;dd>Perception lead · grasp testing · final pitch&lt;/dd>&lt;/div>
&lt;div>&lt;dt>Team&lt;/dt>&lt;dd>KalmanKrew, 5 people&lt;/dd>&lt;/div>
&lt;div>&lt;dt>Hardware&lt;/dt>&lt;dd>Franka Research 3 + RealSense D435&lt;/dd>&lt;/div>
&lt;div>&lt;dt>Timeline&lt;/dt>&lt;dd>6 days, 23–28 Oct 2026&lt;/dd>&lt;/div>
&lt;/dl>
&lt;a class="bp-hero-link" href="#one-frame">Follow one camera frame &lt;span aria-hidden="true">↓&lt;/span>&lt;/a>
&lt;/div>
&lt;figure class="bp-hero-video">
&lt;video controls muted playsinline preload="metadata" poster="/project/brickpalletizing/robot-poster_hu90ce59d7d5e707887047f8f6fb13ca5a_99928_960x0_resize_q80_h2_lanczos.webp" aria-label="Franka arm picking and sorting LEGO bricks">
&lt;source src="https://lshivarudra.github.io/project/brickpalletizing/robot-demo.mp4" type="video/mp4">
&lt;a href="https://lshivarudra.github.io/project/brickpalletizing/robot-demo.mp4">Download the video&lt;/a>
&lt;/video>
&lt;figcaption>Live on the Franka Research 3&lt;/figcaption>
&lt;/figure>
&lt;/section>
&lt;nav class="bp-chapter-nav" aria-label="Case study chapters">
&lt;a href="#the-challenge">The challenge&lt;/a>
&lt;a href="#one-frame">The perception pipeline&lt;/a>
&lt;a href="#on-the-robot">The robot demo&lt;/a>
&lt;a href="#the-final-pitch">The final pitch&lt;/a>
&lt;/nav>
&lt;section class="bp-chapter" aria-labelledby="the-challenge">
&lt;h2 id="the-challenge">From kiln car to lab table&lt;/h2>
&lt;p>After firing, every brick on a kiln car has to be inspected, sorted by shade, and restacked onto pallets. Marmon challenged us to automate this unloading step, known as dehacking.&lt;/p>
&lt;p>On our lab table, a woven stack of LEGO bricks stood in for a kiln car. The first decision was the same at either scale. &lt;strong>Which brick can the robot pick without disturbing the stack?&lt;/strong>&lt;/p>
&lt;div class="bp-map" role="table" aria-label="How the factory problem maps to our tabletop demo">
&lt;div class="bp-map-head" role="row">&lt;span role="columnheader">In the brickyard&lt;/span>&lt;span aria-hidden="true">&lt;/span>&lt;span role="columnheader">On our table&lt;/span>&lt;/div>
&lt;div class="bp-map-row" role="row">&lt;span role="cell">A kiln car of fired bricks&lt;/span>&lt;i aria-hidden="true">→&lt;/i>&lt;span role="cell">A woven LEGO stack on a baseplate&lt;/span>&lt;/div>
&lt;div class="bp-map-row" role="row">&lt;span role="cell">Unload layer by layer&lt;/span>&lt;i aria-hidden="true">→&lt;/i>&lt;span role="cell">Pick &lt;b>only the exposed top layer&lt;/b>&lt;/span>&lt;/div>
&lt;div class="bp-map-row" role="row">&lt;span role="cell">Inspect and grade by shade&lt;/span>&lt;i aria-hidden="true">→&lt;/i>&lt;span role="cell">Classify &lt;b>5 LEGO colours&lt;/b>&lt;/span>&lt;/div>
&lt;div class="bp-map-row" role="row">&lt;span role="cell">Palletize and scrap rejects&lt;/span>&lt;i aria-hidden="true">→&lt;/i>&lt;span role="cell">One pile per good colour, plus a scrap bin&lt;/span>&lt;/div>
&lt;/div>
&lt;figure class="bp-figure">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/real-brick-sample_hue0a8ac063744461c25975f9d85d5b77e_170081_1200x0_resize_q82_h2_lanczos.webp" width="1200" height="800" alt="A real brick sample labelled Bad, with LEGO bricks in front of it" loading="lazy" decoding="async">
&lt;figcaption>A real "bad" sample from the sponsor next to the LEGO bricks that stood in for it.&lt;/figcaption>
&lt;/figure>
&lt;h3 id="my-role-on-kalmankrew">My role on KalmanKrew&lt;/h3>
&lt;div class="bp-role">
&lt;figure>
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/featured-team-at-robot_huc46a70e939b39281352258b741c3a862_212926_900x0_resize_q82_h2_lanczos.webp" width="900" height="600" alt="Shiva explaining the perception pipeline to teammates at the Franka robot cell" loading="lazy" decoding="async">
&lt;/figure>
&lt;div>
&lt;p class="bp-role-lead">I led perception for the LEGO demo, taking the pipeline from raw camera frames to 3D targets for the motion planner.&lt;/p>
&lt;ul class="bp-checks">
&lt;li>&lt;b>Perception:&lt;/b> top-layer detection, size-based mask cleanup, colour classification and 3D pose export.&lt;/li>
&lt;li>&lt;b>Tooling:&lt;/b> RGB-D capture, live detection and a modular runner for replaying recorded frames.&lt;/li>
&lt;li>&lt;b>On the robot:&lt;/b> tuned grasps with the manipulation team and tested the full sorting loop.&lt;/li>
&lt;/ul>
&lt;/div>
&lt;/div>
&lt;/section>
&lt;section class="bp-chapter" aria-labelledby="one-frame">
&lt;h2 id="one-frame">From 61 masks to 22 pick targets&lt;/h2>
&lt;p>The camera sees the top layer, parts of lower layers, and loose bricks around the stack. Select a stage below to see how one RealSense frame becomes a set of 3D pick targets. You can also use the arrow keys after focusing a stage.&lt;/p>
&lt;div class="bp-stepper" role="group" aria-label="Perception pipeline, one frame at a time">
&lt;input type="radio" name="bp-step" id="bp-step-1" class="bp-step-input" checked aria-label="Step 1: Observe">
&lt;input type="radio" name="bp-step" id="bp-step-2" class="bp-step-input" aria-label="Step 2: Measure">
&lt;input type="radio" name="bp-step" id="bp-step-3" class="bp-step-input" aria-label="Step 3: Segment">
&lt;input type="radio" name="bp-step" id="bp-step-4" class="bp-step-input" aria-label="Step 4: Filter">
&lt;input type="radio" name="bp-step" id="bp-step-5" class="bp-step-input" aria-label="Step 5: Decide">
&lt;div class="bp-stage">
&lt;figure class="bp-frame bp-frame-1">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/step1_hud207ad0dfbaf56c88c95b6c1ff3b2075_69174_513x0_resize_q85_h2_lanczos.webp" width="513" height="482" alt="Step 1 — Fit the table, then read the scene" loading="lazy" decoding="async">
&lt;span class="bp-frame-count">1 RGB-D frame&lt;/span>
&lt;/figure>
&lt;figure class="bp-frame bp-frame-2">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/step2_huf01bad41641ed29192e7d462706bd523_73388_516x0_resize_q85_h2_lanczos.webp" width="516" height="482" alt="Step 2 — Turn depth into height" loading="lazy" decoding="async">
&lt;span class="bp-frame-count">height map&lt;/span>
&lt;/figure>
&lt;figure class="bp-frame bp-frame-3">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/step3_hud57b806fef003d5a255ff0f872f81326_83082_515x0_resize_q85_h2_lanczos.webp" width="515" height="482" alt="Step 3 — Ask SAM 3 for “lego brick”" loading="lazy" decoding="async">
&lt;span class="bp-frame-count">61 masks&lt;/span>
&lt;/figure>
&lt;figure class="bp-frame bp-frame-4">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/step4_hu2dcaba9b0e2e954e08d0349e3945d7b6_77714_516x0_resize_q85_h2_lanczos.webp" width="516" height="482" alt="Step 4 — Keep only what the gripper can reach" loading="lazy" decoding="async">
&lt;span class="bp-frame-count">23 masks&lt;/span>
&lt;/figure>
&lt;figure class="bp-frame bp-frame-5">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/step5_huf3870867c3e914f1e82bc47812229a65_139138_682x0_resize_q85_h2_lanczos.webp" width="682" height="850" alt="Step 5 — Classify colour, fit a 3D pick target" loading="lazy" decoding="async">
&lt;span class="bp-frame-count">22 targets · 18 ✓ · 4 ✗&lt;/span>
&lt;ul class="bp-callouts" aria-hidden="true">
&lt;li style="top:6.5%" class="is-muted">loose · ignored&lt;/li>
&lt;li style="top:28.2%" class="is-bad">scrap&lt;/li>
&lt;li style="top:47.6%" class="is-bad">scrap&lt;/li>
&lt;li style="top:63.3%" class="is-good">accept&lt;/li>
&lt;li style="top:87.3%" class="is-target">grasp&lt;/li>
&lt;/ul>
&lt;/figure>
&lt;/div>
&lt;ol class="bp-step-list">
&lt;li class="bp-step bp-step-1">
&lt;label for="bp-step-1">
&lt;span class="bp-step-num">1&lt;/span>
&lt;span class="bp-step-head">&lt;small>Observe&lt;/small>Fit the table, then read the scene&lt;/span>
&lt;/label>
&lt;p>The wrist-mounted D435 captures an aligned colour &amp;#43; depth frame. RANSAC fits the dominant plane (green tint) — that is the table. Those same table pixels later serve as the white reference for colour.&lt;/p>
&lt;/li>
&lt;li class="bp-step bp-step-2">
&lt;label for="bp-step-2">
&lt;span class="bp-step-num">2&lt;/span>
&lt;span class="bp-step-head">&lt;small>Measure&lt;/small>Turn depth into height&lt;/span>
&lt;/label>
&lt;p>Every depth pixel is back-projected and measured against the table plane. Warmer = higher, so the top layer of the stack stands out before any learning is involved.&lt;/p>
&lt;/li>
&lt;li class="bp-step bp-step-3">
&lt;label for="bp-step-3">
&lt;span class="bp-step-num">3&lt;/span>
&lt;span class="bp-step-head">&lt;small>Segment&lt;/small>Ask SAM 3 for “lego brick”&lt;/span>
&lt;/label>
&lt;p>A single text prompt returns instance masks for every visible brick — top layer, lower layers and loose ones alike. Colours here are just instance IDs.&lt;/p>
&lt;/li>
&lt;li class="bp-step bp-step-4">
&lt;label for="bp-step-4">
&lt;span class="bp-step-num">4&lt;/span>
&lt;span class="bp-step-head">&lt;small>Filter&lt;/small>Keep only what the gripper can reach&lt;/span>
&lt;/label>
&lt;p>A height histogram finds the highest substantial surface; only masks within one brick height (≈ 9.6 mm) of it survive. Lower layers and loose table bricks drop out.&lt;/p>
&lt;/li>
&lt;li class="bp-step bp-step-5">
&lt;label for="bp-step-5">
&lt;span class="bp-step-num">5&lt;/span>
&lt;span class="bp-step-head">&lt;small>Decide&lt;/small>Classify colour, fit a 3D pick target&lt;/span>
&lt;/label>
&lt;p>A brick-size prior splits merged masks and rejects partial ones. Each brick is colour-classified in CIELAB and gets a fitted centre and yaw: green = accept, red = scrap, yellow dot = grasp centre.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;div class="bp-funnel" aria-label="Detections remaining after each stage">
&lt;span style="--n:61">&lt;b>61&lt;/b> SAM 3 masks&lt;/span>
&lt;span style="--n:23">&lt;b>23&lt;/b> top layer&lt;/span>
&lt;span style="--n:22">&lt;b>22&lt;/b> pick targets&lt;/span>
&lt;span style="--n:18" class="is-good">&lt;b>18&lt;/b> accept&lt;/span>
&lt;span style="--n:4" class="is-bad">&lt;b>4&lt;/b> scrap&lt;/span>
&lt;/div>
&lt;/div>
&lt;p class="bp-note">One real capture from the RealSense D435, top-down over the LEGO stack. Stage images from the team's annotated pipeline figure.&lt;/p>
&lt;h3 id="which-bricks-are-reachable">Which bricks are reachable?&lt;/h3>
&lt;p>SAM 3 is very good at finding brick-shaped regions, but it has no notion of physical size. Two touching bricks can come back as one mask. A brick half-hidden under the layer above still looks like a brick, and so does the side face of an edge brick.&lt;/p>
&lt;p>To fix this, I project each top-layer mask onto the table plane and measure its &lt;strong>footprint in millimetres&lt;/strong>. I then compare that against a known brick size: L × W × H ≈ 30 × 15 × 10 mm. The prior then makes one of four calls:&lt;/p>
&lt;div class="bp-prior">
&lt;div class="bp-prior-card is-keep">
&lt;div class="bp-prior-viz">&lt;span class="bp-brick" style="--l:4;--w:2;--c:#c9332b">&lt;/span>&lt;/div>
&lt;h4>≈ L × W&lt;/h4>&lt;p>One clean brick&lt;/p>&lt;span class="bp-tag">keep&lt;/span>
&lt;/div>
&lt;div class="bp-prior-card is-split">
&lt;div class="bp-prior-viz">&lt;span class="bp-brick is-split" style="--l:8;--w:2;--c:#d27a24">&lt;/span>&lt;/div>
&lt;h4>W × n·L&lt;/h4>&lt;p>Bricks SAM merged&lt;/p>&lt;span class="bp-tag">split into n&lt;/span>
&lt;/div>
&lt;div class="bp-prior-card is-reject">
&lt;div class="bp-prior-viz">&lt;span class="bp-brick is-partial" style="--l:4;--w:2;--c:#7a1f26">&lt;/span>&lt;/div>
&lt;h4>W × &amp;lt; L&lt;/h4>&lt;p>Mostly hidden by the layer above&lt;/p>&lt;span class="bp-tag">reject&lt;/span>
&lt;/div>
&lt;div class="bp-prior-card is-reject">
&lt;div class="bp-prior-viz">&lt;span class="bp-brick is-sliver" style="--l:4;--w:2;--c:#6b3a22">&lt;/span>&lt;/div>
&lt;h4>&amp;lt; W wide&lt;/h4>&lt;p>Side face of an edge brick&lt;/p>&lt;span class="bp-tag">reject&lt;/span>
&lt;/div>
&lt;/div>
&lt;p>Every brick that survives gets a fitted L × W footprint. That gives its &lt;strong>centre in the camera frame, its yaw on the table and its layer index&lt;/strong>, which is exactly what a grasp planner needs.&lt;/p>
&lt;h3 id="which-pile-takes-each-brick">Which pile takes each brick?&lt;/h3>
&lt;p>The lab had bright desk lamps, glare on glossy plastic and shifting exposure. Matching raw RGB values would have mislabelled bricks constantly. The classifier corrects for that in three steps:&lt;/p>
&lt;div class="bp-colours">
&lt;ol class="bp-colour-steps">
&lt;li>&lt;b>White-balance on the table.&lt;/b> The plane fit already says which pixels are table, so the colour model uses them as a grey reference in linear RGB.&lt;/li>
&lt;li>&lt;b>Measure the clean interior.&lt;/b> Colour is sampled inside the fitted footprint, away from the edges. The darkest 15 % (shadows) and brightest 10 % (glare) are dropped.&lt;/li>
&lt;li>&lt;b>Match in CIELAB with a shared offset.&lt;/b> One bounded Lab offset is fitted jointly across all bricks in the frame to absorb exposure drift. Then each brick takes its nearest reference colour, with a confidence from distance and margin.&lt;/li>
&lt;/ol>
&lt;div class="bp-palette" aria-label="Colour classes and destinations in the example capture">
&lt;div class="bp-palette-row">&lt;i style="--c:#c91a09">&lt;/i>&lt;span>Red&lt;/span>&lt;em>×7&lt;/em>&lt;b class="is-good">Pile A&lt;/b>&lt;/div>
&lt;div class="bp-palette-row">&lt;i style="--c:#720e0f">&lt;/i>&lt;span>Dark red&lt;/span>&lt;em>×5&lt;/em>&lt;b class="is-good">Pile B&lt;/b>&lt;/div>
&lt;div class="bp-palette-row">&lt;i style="--c:#a95500">&lt;/i>&lt;span>Dark orange&lt;/span>&lt;em>×6&lt;/em>&lt;b class="is-good">Pile C&lt;/b>&lt;/div>
&lt;div class="bp-palette-row">&lt;i style="--c:#582a12">&lt;/i>&lt;span>Reddish brown&lt;/span>&lt;em>×3&lt;/em>&lt;b class="is-bad">Scrap&lt;/b>&lt;/div>
&lt;div class="bp-palette-row">&lt;i style="--c:#cc8e69">&lt;/i>&lt;span>Medium nougat&lt;/span>&lt;em>×1&lt;/em>&lt;b class="is-bad">Scrap&lt;/b>&lt;/div>
&lt;p>Counts from the capture above.&lt;/p>
&lt;/div>
&lt;/div>
&lt;h3 id="where-should-the-gripper-go">Where should the gripper go?&lt;/h3>
&lt;p>Perception ends at a clean, documented contract. For every brick, the pipeline writes a 3D box in the &lt;strong>camera optical frame&lt;/strong> (x right, y down, z forward, metres). The robotics side applies the hand-eye transform and plans the motion.&lt;/p>
&lt;div class="bp-handoff">
&lt;svg viewBox="0 0 300 190" role="img" aria-label="A brick drawn as a 3D box with numbered corners, centre point, yaw arrow and camera axes">
&lt;g fill="none" stroke="currentColor" stroke-width="1.6" stroke-linejoin="round">
&lt;path d="M60 70 L160 40 L250 70 L150 100 Z" class="bp-svg-top"/>
&lt;path d="M60 70 L60 110 L150 140 L150 100 M150 140 L250 110 L250 70"/>
&lt;path d="M60 110 L160 80 L250 110 M160 80 L160 40" stroke-dasharray="4 4" opacity=".45"/>
&lt;/g>
&lt;path d="M159 94 L200 158" stroke="currentColor" stroke-width="1" opacity=".5" fill="none"/>
&lt;circle cx="155" cy="90" r="5" class="bp-svg-dot"/>
&lt;path d="M155 70 L205 55" class="bp-svg-yaw" stroke-width="2.4" fill="none" marker-end="url(#bp-arrow)"/>
&lt;defs>&lt;marker id="bp-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse">&lt;path d="M0 0 L10 5 L0 10 z" class="bp-svg-yaw-fill"/>&lt;/marker>&lt;/defs>
&lt;g font-size="11" font-family="inherit" fill="currentColor">
&lt;text x="44" y="68">0&lt;/text>&lt;text x="160" y="34">1&lt;/text>&lt;text x="256" y="68">2&lt;/text>&lt;text x="146" y="114">3&lt;/text>
&lt;text x="44" y="118">4&lt;/text>&lt;text x="256" y="118">6&lt;/text>&lt;text x="146" y="154">7&lt;/text>
&lt;text x="198" y="168" class="bp-svg-label">box centre&lt;/text>&lt;text x="208" y="52" class="bp-svg-label">yaw&lt;/text>
&lt;/g>
&lt;g stroke="currentColor" stroke-width="1.6" fill="currentColor" font-size="11" opacity=".75">
&lt;path d="M18 172 L58 172" marker-end="url(#bp-arrow)"/>&lt;text x="62" y="176" stroke="none">x&lt;/text>
&lt;path d="M18 172 L18 186" marker-end="url(#bp-arrow)"/>&lt;text x="24" y="188" stroke="none">y&lt;/text>
&lt;circle cx="18" cy="172" r="3.5" fill="none"/>&lt;circle cx="18" cy="172" r="1" stroke="none"/>&lt;text x="6" y="164" stroke="none">z&lt;/text>
&lt;/g>
&lt;/svg>
&lt;pre class="bp-schema">&lt;code>&lt;span># per brick → bricks.json, poses.csv&lt;/span>
id, colour, quality, confidence
layer &lt;span># index in the stack&lt;/span>
centre_xyz &lt;span># box centre [m]&lt;/span>
top_centre_xyz &lt;span># top face [m]&lt;/span>
yaw &lt;span># on the table&lt;/span>
size_lwh &lt;span># fitted size&lt;/span>
corners[8] &lt;span># top 0–3, bottom 4–7&lt;/span>&lt;/code>&lt;/pre>
&lt;/div>
&lt;p>I also wrapped the stages in a &lt;strong>modular pipeline runner&lt;/strong>. It can capture a fresh frame or replay a recording, run segmentation, classification and pose export as separate processes, and give every run its own timestamped folder. If any stage fails, it stops and writes an explicit &lt;code>INCOMPLETE.json&lt;/code>, so stale results are never silently reused.&lt;/p>
&lt;/section>
&lt;section class="bp-chapter" aria-labelledby="on-the-robot">
&lt;h2 id="on-the-robot">Pick, place, look again&lt;/h2>
&lt;p>Even a successful pick can nudge the next brick out of position. The Franka picks a few bricks, returns to its observation pose, and &lt;strong>rescans&lt;/strong> before committing to the next set of targets.&lt;/p>
&lt;div class="bp-loop" aria-label="Pick and rescan loop">
&lt;span>Observation pose&lt;/span>&lt;i aria-hidden="true">→&lt;/i>
&lt;span>Perceive&lt;small>pipeline above&lt;/small>&lt;/span>&lt;i aria-hidden="true">→&lt;/i>
&lt;span>Pick ≤ N bricks&lt;small>top layer only&lt;/small>&lt;/span>&lt;i aria-hidden="true">→&lt;/i>
&lt;span>Place&lt;small>colour pile or scrap&lt;/small>&lt;/span>
&lt;em aria-hidden="true">↺ rescan · also the recovery path after a motion failure&lt;/em>
&lt;/div>
&lt;div class="bp-split">
&lt;figure class="bp-figure bp-tall">
&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/sorting-setup_hud4839b953d02cf54763eb51f45e1e73e_200372_700x0_resize_q82_h2_lanczos.webp" width="700" height="1244" alt="The Franka arm above the LEGO stack, with sorted bricks on the tabletop" loading="lazy" decoding="async">
&lt;figcaption>The working cell: stack on the left, sorted colour piles on the right.&lt;/figcaption>
&lt;/figure>
&lt;div>
&lt;p>I helped tune and test the grasping side alongside our manipulation teammates. Most of the work went into details that only show up on real hardware:&lt;/p>
&lt;ul class="bp-checks">
&lt;li>Grasp width and force for small, glossy 2×4 bricks&lt;/li>
&lt;li>Pre-grasp clearance and offsets so the fingers don't hit neighbouring bricks&lt;/li>
&lt;li>3D-printed gripper fingers&lt;/li>
&lt;li>Varying layer heights as the stack shrinks&lt;/li>
&lt;li>A camera-to-gripper transform taken from CAD and refined on the robot&lt;/li>
&lt;/ul>
&lt;p class="bp-honest">We demonstrated sorting on the Franka. Measuring pick success across repeated trials is the next step.&lt;/p>
&lt;/div>
&lt;/div>
&lt;/section>
&lt;section class="bp-chapter" aria-labelledby="the-final-pitch">
&lt;h2 id="the-final-pitch">The final pitch&lt;/h2>
&lt;p>On day six, I presented three parts of the same idea to the jury. The Franka showed how to unload a stack, the real-brick inspector showed how to grade fired bricks, and the simulation showed how those steps could fit into a factory line.&lt;/p>
&lt;div class="bp-embed">
&lt;iframe src="https://www.youtube-nocookie.com/embed/UwtO5sTqreg?rel=0" title="Team KalmanKrew final hackathon presentation demo" loading="lazy" referrerpolicy="strict-origin-when-cross-origin" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen>&lt;/iframe>
&lt;/div>
&lt;h3 id="beyond-the-lab-table">Beyond the lab table&lt;/h3>
&lt;p>My teammates built two companion prototypes.&lt;/p>
&lt;div class="bp-cards">
&lt;article>
&lt;video controls muted playsinline preload="metadata" poster="/project/brickpalletizing/real-brick-classifier-poster_hu21d0d3c1410613aa0a0e5eeee658cf49_24694_960x0_resize_q80_h2_lanczos.webp" aria-label="Live real-brick inspection: good and bad predictions">
&lt;source src="https://lshivarudra.github.io/project/brickpalletizing/real-brick-classifier.mp4" type="video/mp4">
&lt;a href="https://lshivarudra.github.io/project/brickpalletizing/real-brick-classifier.mp4">Download the video&lt;/a>
&lt;/video>
&lt;span class="bp-card-kicker">Real-brick inspection&lt;/span>
&lt;h3>Grading real fired bricks&lt;/h3>
&lt;p>SAM 3 masks, hand-crafted colour and texture features, and DINOv2 patch features, fused through two linear SVMs. On 258 photos of the 10 sponsor samples, &lt;b>9 of 10 bricks&lt;/b> were graded correctly: all 4 good bricks accepted, 5 of 6 bad ones rejected.&lt;/p>
&lt;/article>
&lt;article>
&lt;div class="bp-embed">
&lt;iframe src="https://www.youtube-nocookie.com/embed/kOvg-SudFjg?rel=0" title="Factory simulation: dehacking, inspection, reject removal and palletizing" loading="lazy" referrerpolicy="strict-origin-when-cross-origin" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen>&lt;/iframe>
&lt;/div>
&lt;span class="bp-card-kicker">Factory simulation&lt;/span>
&lt;h3>Scaling the idea up&lt;/h3>
&lt;p>A simulated line covering dehacking, inspection, reject removal and palletizing, with a nominal throughput of &lt;b>≈ 6,400 bricks per hour&lt;/b>. The throughput estimate comes from scripted motion and simulated shade labels.&lt;/p>
&lt;/article>
&lt;/div>
&lt;h3 id="six-days-in-munich">Six days in Munich&lt;/h3>
&lt;div class="bp-gallery">
&lt;figure class="is-wide">&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/final-presentation_hu129587b46ef9ad5554f97e74e8296372_367593_1400x0_resize_q82_h2_lanczos.webp" width="1400" height="934" alt="Shiva presenting with a microphone during the final pitch, team behind him" loading="lazy" decoding="async">
&lt;figcaption>Pitching on the final day.&lt;/figcaption>&lt;/figure>
&lt;figure>&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/team-introduction_hu5b0d4fedbcaa95d3a012b449bf77180f_312844_900x0_resize_q82_h2_lanczos.webp" width="900" height="600" alt="Team KalmanKrew at the opening-day introductions" loading="lazy" decoding="async">
&lt;figcaption>Day 1: meeting the team.&lt;/figcaption>&lt;/figure>
&lt;figure>&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/team-discussion_hu2b942e656fafe86a973ae90dd4099e77_217562_900x0_resize_q82_h2_lanczos.webp" width="900" height="600" alt="Shiva explaining an idea to teammates over a real brick sample" loading="lazy" decoding="async">
&lt;figcaption>Working out the approach over a real brick.&lt;/figcaption>&lt;/figure>
&lt;figure>&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/answering-questions_huf946165c65c7636ab0b8b5710a25318e_325464_900x0_resize_q82_h2_lanczos.webp" width="900" height="600" alt="Shiva answering jury questions with a microphone, Team KalmanKrew standing beside him" loading="lazy" decoding="async">
&lt;figcaption>Questions from the jury.&lt;/figcaption>&lt;/figure>
&lt;figure>&lt;img src="https://lshivarudra.github.io/project/brickpalletizing/teams-and-marmon_hu4555a3a5ffdd63e9e343d7d2f3de934b_322435_900x0_resize_q82_h2_lanczos.webp" width="900" height="675" alt="Teams 8A and 8B with Jeff Garascia and Michael Ostrow from Marmon" loading="lazy" decoding="async">
&lt;figcaption>Both Marmon teams with our challenge hosts.&lt;/figcaption>&lt;/figure>
&lt;/div>
&lt;/section>
&lt;section class="bp-chapter" aria-labelledby="what-comes-next">
&lt;h2 id="what-comes-next">What I’d take into the next project&lt;/h2>
&lt;div class="project-roadmap bp-lessons">
&lt;section>
&lt;span class="roadmap-label">Geometry first&lt;/span>
&lt;h3>Use depth to narrow the search&lt;/h3>
&lt;p>A plane fit and a height histogram narrowed the candidate masks to the working layer. That made it easier to trace failed picks back to the geometry or the mask.&lt;/p>
&lt;/section>
&lt;section>
&lt;span class="roadmap-label">Calibrate to the scene&lt;/span>
&lt;h3>Use the world as a reference&lt;/h3>
&lt;p>The table doubled as a white reference, and one Lab offset per frame absorbed exposure drift. Both were more reliable than tuning camera settings for every lighting change.&lt;/p>
&lt;/section>
&lt;section>
&lt;span class="roadmap-label">Close the loop&lt;/span>
&lt;h3>Rescan beats predict&lt;/h3>
&lt;p>Rescanning after a few picks handled disturbed bricks and failed motions far better than trying to plan the whole stack from one frame.&lt;/p>
&lt;/section>
&lt;section>
&lt;span class="roadmap-label">Next step&lt;/span>
&lt;h3>Validate on unseen bricks&lt;/h3>
&lt;p>The inspector was evaluated on ten physical samples. Testing on unseen bricks and measuring pick success across repeated runs would show how well the system generalises.&lt;/p>
&lt;/section>
&lt;/div>
&lt;div class="bp-team">
&lt;p>&lt;b>Team KalmanKrew&lt;/b>: Shiva Rudra (perception, pitch), &lt;a href="https://www.linkedin.com/in/varun-teja-988359142/">Varun Teja&lt;/a> (perception and real-brick classification), &lt;a href="https://tanayranjan.github.io/">Tanay Ranjan&lt;/a> (perception–manipulation integration, pick-and-rescan loop), &lt;a href="https://www.linkedin.com/in/ameed-robotics/">Ameed Nazhurudeen&lt;/a> (manipulation), and &lt;a href="https://www.linkedin.com/in/ahmet-sinan-erdem/">Ahmet Sinan Erdem&lt;/a> (vision and simulation). Mentored by &lt;a href="https://www.petermjso.com/">Peter So&lt;/a>; challenge hosted by &lt;a href="https://www.marmon.com/about-us/the-marmon-team/jeff-garascia/">Jeff Garascia&lt;/a> and Michael Ostrow from Marmon.&lt;/p>
&lt;/div>
&lt;/section>
&lt;/div></description></item></channel></rss>