From Blueprint to Bill of Materials: Automating Quantity Takeoff Verification with a Multi-Model ML Pipeline

Project Overview

Z-App is a full-stack application built to automate quantity takeoff verification for Japanese residential construction projects. While the platform includes a secure web interface for project submission, authentication, and review workflows, its core innovation lies in a machine learning pipeline that automatically verifies manual quantity takeoff sheets against architectural floor plans. Instead of requiring estimators to manually inspect every drawing and recalculate material quantities, the system performs the verification process automatically and highlights only the discrepancies that require human attention. In a typical Japanese homebuilding workflow, estimators create a manual quantity takeoff (積算/割り付け, *waritsuke*) in Microsoft Excel by interpreting architectural floor plans and calculating the required construction materials, including studs (スタッド), runners (ランナー), fastening clips, and other structural components. This manual process is both labor intensive and susceptible to counting mistakes, inconsistent measurements, and transcription errors, especially when handling large project volumes. Z-App addresses this challenge by reconstructing the same bill of materials directly from the architectural drawings and comparing the results with the existing Excel workbook, allowing reviewers to focus only on genuine inconsistencies rather than repeating the entire estimation process. The backend processes each project through a sequence of specialized computer vision and data processing stages. After receiving the floor plan drawings and corresponding Excel takeoff files, the system extracts the architectural drawings, identifies the drawing style, and dynamically selects the most suitable segmentation model for that layout. Individual rooms are isolated before multiple vision models detect structural elements such as studs, runners, openings, and fastening symbols. A dedicated OCR pipeline interprets dimension labels associated with each structural element and validates extracted measurements against the values recorded in the Excel sheet, reducing errors caused by optical character recognition. The resulting detections are grouped into room-level material quantities before being reconciled with the manually prepared takeoff, producing a structured comparison that identifies missing items, incorrect counts, and measurement mismatches. To improve the reliability of the verification process, the pipeline incorporates a configurable rule-based validation layer that filters known false-positive scenarios before results reach the user. Rather than presenting raw model predictions, the application generates an annotated Excel report containing discrepancy labels, confidence scores, supporting references, and review metadata, enabling estimators to quickly validate flagged items while maintaining complete traceability from each reported issue back to the original architectural drawing. Every machine learning component within the system is domain-specific and fine-tuned using an internally curated dataset of annotated Japanese residential floor plans. Separate models are optimized for tasks such as room segmentation, structural component detection, opening identification, and drawing-style classification, allowing the pipeline to adapt to the visual differences commonly found across architectural drawing standards. Model versions, confidence thresholds, and deployment configurations are managed through a centralized configuration and model registry, enabling new model releases or threshold adjustments without requiring application code changes. The platform also emphasizes production reliability and auditability throughout the inference workflow. Every pipeline execution is fully traceable, with intermediate predictions, visualization overlays, confidence scores, and processing metadata recorded for later inspection. Automated monitoring tracks prediction confidence distributions to detect potential model drift over time, while configurable guardrail rules ensure that every modification to the final output can be attributed to a specific validation rule. This combination of specialized machine learning models, configurable business logic, and comprehensive observability transforms Z-App from a collection of vision models into a production-ready verification system that accelerates quantity takeoff review while improving accuracy, transparency, and operational efficiency.

Key Features

• ✓Multi-model ML pipeline
• ✓Floor plan extraction
• ✓Drawing style classification
• ✓Room segmentation
• ✓Stud detection
• ✓Runner detection
• ✓Fastener detection
• ✓Opening detection
• ✓OCR-based dimension extraction
• ✓Excel reconciliation
• ✓Confidence scoring
• ✓Guardrail rules engine
• ✓Annotated Excel reports
• ✓FastAPI backend
• ✓Next.js frontend
• ✓Weights & Biases experiment tracking
• ✓Production drift monitoring
• ✓Audit trail generation

The Challenge

One of the most complex challenges in the pipeline was accurately associating detected stud and runner clusters with their corresponding dimension labels on architectural drawings. Since multiple structural elements can share identical length values and labels are often positioned close together, simple proximity-based matching frequently produced incorrect assignments, allowing multiple clusters to reference the same label while leaving others unused. This resulted in systematic counting errors that required a more robust spatial matching strategy. Another significant challenge involved extracting reliable dimension information from printed drawings. OCR errors, such as merged digit sequences, partially recognized text, or confusion between room identifiers and measurement values, could propagate through the pipeline and affect room grouping, material aggregation, and the final comparison against the Excel takeoff. To minimize these cascading errors, additional validation and reconciliation logic was introduced before the extracted values were consumed by downstream stages. The project also required distinguishing genuine model failures from naturally complex architectural scenarios. Overlapping symbols, multi-room layouts, window openings intersecting structural elements, and subtle discrepancies between OCR output and the Excel takeoff often appeared similar from the perspective of the detection models. Rather than relying on a single confidence threshold, the system employs multiple validation stages and configurable guardrail rules to identify the underlying cause of each discrepancy before presenting it for human review. Producing meaningful verification results also required moving beyond simple mismatch detection. A difference between the machine learning prediction and the Excel takeoff does not necessarily indicate an error, as discrepancies may arise from detection uncertainty, OCR inaccuracies, clustering artifacts, or ambiguities in the original drawings themselves. The pipeline therefore combines confidence scores from multiple stages to prioritize discrepancies according to their reliability, allowing reviewers to focus on the most credible issues first. Finally, maintaining consistent performance in production presented an ongoing engineering challenge. Because the verification workflow depends on several independently trained vision models, degradation in any single component can silently affect downstream processing and compromise the overall comparison results. To address this, the system incorporates continuous monitoring, inference tracing, and audit mechanisms that help detect model drift early while preserving complete traceability for every prediction and reconciliation decision.

Our Solution

The solution was implemented as a modular, multi-stage ML pipeline where each vision model specializes in a specific task, including floor plan extraction, drawing style classification, room segmentation, structural component detection, OCR, and quantity reconciliation. Instead of relying solely on model predictions, the pipeline combines configurable guardrail rules with composite confidence scoring to validate detections, reduce false positives, and prioritize discrepancies based on their reliability. The final results are written back in the same Excel format used by estimators, requiring no changes to their existing workflow. By automating quantity takeoff verification and surfacing only meaningful discrepancies, the system significantly reduces manual review effort, improves material estimation accuracy, minimizes ordering errors, and enables estimation teams to process more projects in less time.

Technologies Used

// Tech Stack & Tools