Sketch2Code - Microsoft AI Lab

Sketch2Code uses AI to convert hand-written drawings to working HTML prototypes. Designers share ideas on a whiteboard, then changes are shown in the browser instantly.

Creative design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is usually manually translated into a working HTML wireframe. This takes time and delays the design process.

Computer Vision is a discipline inside artificial intelligence that gives an application the capability to see and understand what it is seeing. Using Microsoft Cognitive Services, we can train Custom Computer Vision with millions of images and enable object detection for a wide range of types of objects.

In this case, we trained the model to recognize hand-drawn web design elements like a textbox or button. We use the text recognition functionality present in the Computer Vision Service to extract hand-written text present in the design. By combining the design element and the extracted content, we can generate HTML snippets of the different elements in the design. We then can infer the layout of the design from the position of the identified elements and generate the final HTML code.

Part of Azure Cognitive Services, the Custom Vision and Computer Vision APIs are used for object detection, OCR including handwriting recognition. With the Azure Cloud Platform, the objects returned help to identify the layout. Layouts are passed to the Sketch2Code MVC web application to generate HTML.

Snip Insights helps users find intelligent insights from a snip or screenshot. AI services convert a captured image to translated text, automatically detecting and tagging image content.

Microsoft Math uses optical character recognition (OCR) for handwriting to extract a math equation from a student’s photo of their notes. The problem is then standardized, classified, solved, and returned with solution steps and similar problems.

Make artificial intelligence real for your business today.

Discover Azure AI—a portfolio of AI services designed for developers and data scientists. Take advantage of the decades of breakthrough research, responsible AI practices, and flexibility that Azure AI offers to build and deploy your own AI solutions.