TakeState

TakeState: Make it Measurable Can we make a photographers instructions measurable? Product/Tabletop photography requires a lot of consideration to ensure that positioning of objects in the image is correct. A minor mistake such as placing the glass on the wrong side of the bottle will cause the photographer to have to reshoot. Usually these instructions will be checked manually which can become repetitive. TakeState attempts to make this more measurable. The creator can input the instructions in natural language e.g.,stating "the glass must be on the right of the bottle" & will then check if this has been followed in the uploaded image This uses a range of technologies to carry out different tasks. Muse Spark 1.3 is used to take the natural language & create a structured rule. SAM 3.1 is used to find & segment the objects requested. The objects are then checked using a geometry engine written in Python. This uses the decoded masks and the centroids that have been mapped back to the original image coordinates. This will return one of three outcomes: PASS: The spatial relationship has been measured correctly. FAIL: The objects are in the wrong position. A hint is provided to help correct the positioning. UNABLE TO VERIFY: The objects required cannot be found reliably, no segmentation evidence can be found or there is some ambiguity. A key consideration I had was ensuring that if there was not enough visual evidence to make a guess it would not be made. Even though there is centre points in the bounding boxes they are never used to make a decision on whether to pass or fail. If there isn’t a reliable result it will be stated. If there are multiple detections the user can select which one they want. The application has been written in Python using Streamlit. This includes the paths to integrate the models, decode the masks, calculate the geometry, limit the number of API calls, and ensure there are safeguards in place for uncertain results. I have also created a test suite to test the application offline. This includes testing the geometry calculations, mask decoding, ambiguity, uncertainty handling, API safeguards and application functionality. The latest run on my Windows machine shows 116 tests passed and 0 failed For now TakeState only checks for spatial relationships in individual images. In the future it could be extended to compare images for consistent lighting, to match reference compositions, and to ensure overall consistency. Overall TakeState was created with a simple idea, that not everything should have to be based on someone’s opinion. If there is a spatial requirement then this should be made into a measurable rule. The visual evidence can be analysed to provide an explanation as to why the image has passed, failed or couldn’t be verified.