This project's goal is to provide a suite of tools that will allow a designer to develop sketch recognition systems for new domains. The designer will draw examples of shapes that the system should recognize. The examples are used to generate a formal description of the shapes the system should recognize. This description is then compiled into efficient bottom up recognizers. Finally, these recognizers are employed by the recognition system that uses both top-down contextual recognition techniques and the bottom-up shape recognizers to generate a model of the sketch.

The information explosion has made much more knowledge available to everyone, but knowing what is available and finding what is relevant in this mass is difficult. This project seeks to improve the design of information spaces by better making use of our spatial cognition and visual perception. Examples of our approach include information spaces for journal articles, MIT EECS classes, questions and answers, and movies. We are extending this approach to visualize other domains such as object-oriented software programs.

We aim to create a tool that allows an engineer to sketch a mechanical system as she would on paper, and then allows her to interact with the design as a mechanical system, for example by seeing a simulation of her drawing. We have built an early incarnation of such a tool, called ASSIST, which allows a user to sketch simple mechanical systems and see simulations of her drawings in a two-dimensional kinematic simulator.

We created a natural sketch recognition environment for UML (Unified Modeling Language). Our system differs from graffiti-based approaches to this task, in that it recognizes objects by how they look, not by how they were drawn. Our goal is a system where the user can sketch UML diagrams on a tablet or whiteboard in the same way they would on paper, but the diagrams would then be recognized by the computer.

One of the most basic steps in understanding hand drawn sketches is converting the original digitized pen strokes in the sketch into the intended geometric objects. We have implemented a system that combines multiple sources of knowledge to provide robust early processing for freehand sketching.