Publications
Press Coverage / Journal Papers / Conference Papers / Theses / MIT Reports and Papers / Other Work / Movies
Press Coverage
AUVSI Unmanned Systems Magazine: Artificial Intelligence and Machine Learning - Pave Robotics' Future
November 2011
Smithsonian Air & Space Magazine: Pilot Not Included
July 2011
PC World: Microsoft Looks in the Crystal Ball: Research chief demonstrates Magic Paper, virtual classrooms, and other projects.
October 2003
CNN: Breathing life into messy sketches
October 2003
New Scientist: Smart software makes sense of rough sketches
September 2003
Mass HighTech: MITs tablet tech gets a look-see from Microsoft
July 2003
Journal Articles
2012
Continuous Body and Hand Gesture Recognition for Natural Human-Computer Interaction. ACM Transactions on Interactive Intelligent Systems (TiiS). In Press.
2008
Sketch recognition in interspersed drawings using time-based graphical models. Computers and Graphics 32(5), pp.500-510. October 2008.
[BibTeX][PDF]2007
Book Review: Gesture in Human-Computer Interaction and Simulation. Gesture 7(1). 2007.
[BibTeX][RTF]Magic Paper: Sketch-Understanding Research. Computer 40(9), pp.34--41. Los Alamitos, CA, USA, 2007.
[BibTeX][PDF]Sketch Interpretation Using Multiscale Models of Temporal Patterns. IEEE Computer Graphics and Applications 27(1), pp.28-37. January 2007.
[BibTeX][PDF]2005
LADDER, a sketching language for user interface developers. Elsevier, Computers and Graphics 28, pp.518-532. 2005.
[BibTeX][PDF]Conference Papers
2011
ChemInk: A Natural Real-Time Recognition System for Chemical Drawings. In Proceedings of the: 2011 International Conference on Intelligent User Interfaces. 2011. [BibTex ][PDF]
Tracking Body and Hands For Gesture Recognition: NATOPS Aircraft Handling Signals Database. In Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition (FG 2011) , pp.500-506. Santa Barbara, CA, March 2011.
[BibTeX][PDF]Multi-Signal Gesture Recognition Using Temporal Smoothing Hidden Conditional Random Fields. In Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition (FG 2011), pp.388-393. Santa Barbara, CA, March 2011.
[BibTeX][PDF]2010
Multimodal Interaction with an Autonomous Forklift. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp.243-250. Osaka, Japan, March 2010.
[BibTeX][PDF]A Voice-Commandable Robotic Forklift Working Alongside Humans in Minimally-Prepared Outdoor Environments. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Anchorage, AK, May 2010.
[BibTeX][PDF]Toward Natural Interaction in the Real World: Real-time Gesture Recognition. In Proceedings of the International Conference on Multimodal Interfaces (to appear). Beijing, China, November 2010.
[BibTeX][PDF]2009
Symmetric Multimodal Interaction in a Dynamic Dialogue. In 2009 Intelligent User Interfaces Workshop on Sketch Recognition. February 2009.
[BibTeX][PDF]Combining Speech and Sketch to Interpret Unconstrained Descriptions of Mechanical Devices. In Proceedings of the 2009 International Joint Conference on Artificial Intelligence (IJCAI), pp.1401-1406. Pasadena, California, July 2009.
[BibTeX][PDF]Recognizing Interspersed Sketches Quickly. In Graphics Interface Conference 2009, pp.157-166. New York, NY, USA, May 25-27 2009.
[BibTeX][PDF]A Visual Approach to Sketched Symbol Recognition. In Proceedings of the 2009 International Joint Conference on Artificial Intelligence (IJCAI), pp.1463-1468. 2009.
[BibTeX][PDF]Learning from Neighboring Strokes: Combining Appearance and Context for Multi-Domain Sketch Recognition. In Advances in Neural Information Processing Systems 22, pp.1401--1409. 2009.
[BibTeX][PDF]Visual Recognition of Sketched Symbols. In 2009 Intelligent User Interfaces Workshop on Sketch Recognition. 2009.
[BibTeX][PDF]Emergency Response System On a Pen-Based Tabletop Display. In ACM Conference on Interactive Tabletops and Surfaces (ITS 2009) Poster. Banff, Canada, Nov 2009.
[BibTeX][PDF]2008
Discourse Topic and Gestural Form. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pp.1-6. 2008.
[BibTeX][PDF]Gestural Cohesion for Topic Segmentation. In Proceeings of ACL-08, pp.852-860. Columbus, Ohio, June 2008.
[BibTeX][PDF]Modeling Gesture Salience as a Hidden Variable for Coreference Resolution and Keyframe Extraction. In Journal of Artificial Intelligence Research, pp.353-398. February 2008.
[BibTeX][PDF]2007
Speech and Sketching: An Empirical Study of Multimodal Interaction. In SBIM '07: Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling, pp.83-90. New York, NY, USA, August 2-3 2007.
[BibTeX][PDF]Conditional Modality Fusion for Coreference Resolution. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp.352 - 359. 2007.
[BibTeX][PDF]Turning Lectures into Comic Books with Linguistically Salient Gestures. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada. AAAI Press 2007, pp.877 - 882. 2007.
[BibTeX][PDF]Recognition of Hand Drawn Chemical Diagrams. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada., pp.846-851. 2007.
[BibTeX][PDF]2006
Gesture Features for Coreference Resolution. In Machine Learning for Multimodal Interaction: Third International Workshop, MLMI 2006, Bethesda, MD, USA, May 1-4, 2006, Revised Selected Papers (Lecture Notes in Computer Science), pp.154-165. Secaucus, NJ, USA, 2006.
[BibTeX][PDF]Gesture Improves Coreference Resolution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pp.37-40. New York City, USA, June 2006.
[BibTeX][PDF][JPEG]Interactive Learning of Structural Shape Descriptions from Automatically Generated Near-miss Examples. In Intelligent User Interfaces (IUI), pp.37-40. 2006.
[BibTeX][PDF]Sketch Interpretation Using Multiscale Models of Temporal Patterns. In Proceedings of NESCAI 2006, pp.28-37. Ithaca, New York, April 28-29 2006.
[BibTeX]2005
Dynamically Constructed Bayes Nets for Multi-Domain Sketch Understanding. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI)2005, pp.1407-1412. San Francisco, California, August 1 2005.
[BibTeX][PDF]Gestural Features for Sentence Segmentation. In Proceedings of 6th European Gesture Workshop, pp.1-2. 2005.
[BibTeX][PDF]HMM-Based Efficient Sketch Recognition. In Proceedings of the International Conference on Intelligent User Interfaces (IUI'05), pp.281-283. New York, New York, January 9-12 2005.
[BibTeX][Extended PDF][Extended PS][PDF][PS][PPT]2004
Building the Design Studio of the Future. In AAAI 2004 Symposium on Making Pen-Based Interaction Intelligent and Natural, pp.1-7. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF][PPS]Speech and Sketching for Multimodal Design. In Proceedings of the 9th International Conference on Intelligent User Interfaces, pp.214-216. 2004.
[BibTeX][PDF][PS]Sketch Recognition User Interfaces: Guidelines for Design and Development. In AAAI 2004 Symposium on Making Pen-Based Interaction Intelligent and Natural, pp.8-14. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF]SketchREAD: A Multi-Domain Sketch Recognition Engine. In Proceedings of UIST 2004, pp.23-32. New York, New York, October 24-27 2004.
[BibTeX][PDF][PS]New Approach to Early Sketch Processing. In AAAI 2004 Symposium on Making Pen-Based Interaction Intelligent and Natural, pp.29-34. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF]A Salience-Based Approach to Gesture-Speech Alignment. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pp.25-32. East Stroudsburg, Pennsylvania, May 2-7 2004.
[BibTeX][PDF]Visual and Linguistic Information in Gesture Classification. In International Conference on Multimodal Interfaces (ICMI'04), pp.113-120. New York, New York, October 14-15 2004.
[BibTeX][PDF]Automatically Transforming Symbolic Shape Descriptions for Use in Sketch Recognition. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04), pp.450-456. San Jose, California, 2004.
[BibTeX][PDF]Shady: A Shape Description Debugger for Use in Sketch Recognition. In AAAI Fall Symposium on Making Pen-Based Interaction Intelligent and Natural, pp.71-77. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF]ETCHA Sketches: Lessons Learned from Collecting Sketch Data. In AAAI 2004 Symposium on Making Pen-Based Interaction Intelligent and Natural, pp.134-140. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF]Handling Overtraced Strokes in Hand-Drawn Sketches. In Making Pen-Based Interaction Intelligent and Natural, pp.141-144. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF][PS]Scale-space Based Feature Point Detection for Digital Ink. In Making Pen-Based Interaction Intelligent and Natural, pp.145-151. Menlo Park, California, October 21-24 2004.
[BibTeX][PDF][PS]Perceptually Based Learning of Shape Descriptions. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04), pp.482-487. San Jose, California, 2004.
[BibTeX][PDF]2003
Natural Gesture in Descriptive Monologues. In Supplementary Proceedings of the ACM Symposium on User Interface Software and Techology (UIST'03), pp.69-70. New York, New York, November 2-5 2003.
[BibTeX][PDF]LADDER: A Language to Describe Drawing, Display, and Editing in Sketch Recognition. In Proceedings of the 2003 International Joint Conference on Artificial Intelligence (IJCAI), pp.461-467. Acapulco, Mexico, 2003.
[BibTeX][PDF][PS]2002
A Framework for Multi-Domain Sketch Recognition. In AAAI Spring Symposium on Sketch Understanding, pp.1-8. Stanford, California, March 25-27 2002.
[BibTeX][PDF][PS]Sketch Understanding in Design: Overview of Work at the MIT AI Lab. In Sketch Understanding, Papers from the 2002 AAAI Spring Symposium, pp.24-31. Stanford, California, March 25-27 2002.
[BibTeX][PDF][PS]An Agent-Based System For Capturing and Indexing Software Design Meetings. In Proceedings of International Workshop on Agents In Design, WAID'02, pp.203-218. 2002.
[BibTeX][PDF][PPT]Tahuti: A Geometrical Sketch Recognition System for UML Class Diagrams. In AAAI Spring Symposium on Sketch Understanding, pp.59-68. Stanford, California, March 25-27 2002.
[BibTeX][PDF][PS]2001
Preserving the Freedom of Sketching to Create a Natural Computer-Based Sketch Tool. In Human Computer Interaction International Proceedings. 2001.
[BibTeX][PDF][PS]Resolving Ambiguities to Create a Natural Computer-Based Sketching Environment. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pp.1365--1371. Seattle, Washington, USA, August 2001.
[BibTeX][PDF][PS]Query By Attention: Visually Searchable Information Maps. In Proceedings of Fifth International Conference on Information Visualization (InfoVis 2001), pp.85-93. 2001.
[BibTeX][PDF][PS]Naturally Conveyed Explanations of Device Behavior. In Workshop on Perceptive User Interfaces. 2001.
[BibTeX][PDF][PS]Sketch Based Interfaces: Early Processing for Sketch Understanding. In Workshop on Perceptive User Interfaces, Orlando FL, pp.1-8. 2001.
[BibTeX][PDF][PS]Theses
Multi-Signal Gesture Recognition Using Body and Hand Poses. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, September 2010.
[BibTeX][PDF]MIDOS: Multimodal Interactive DialOgue System. Ph.D Thesis for Massachusetts Institute of Technology. June 2009.
[BibTeX][PDF]Combining Representations for Improved Sketch Recognition. Ph.D Thesis for Massachusetts Institute of Technology. September 2009.
[BibTeX][PDF]Drawing on the World: Sketch in Context. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, September 2009.
[BibTeX][PDF]Gesture in Automatic Discourse Processing. Ph.D Thesis for Massachusetts Institute of Technology. Camnridge, MA, May 2008.
[BibTeX][PDF]LADDER: A Perceptually-based Language to Simplify Sketch Recognition User Interface Development. Ph.D Thesis for Massachusetts Institute of Technology. Cambridge, MA, January 2007.
[BibTeX][PDF]Envisioning Sketch Recognition: A Local Feature Based Approach to Recognizing Informal Sketches. Ph.D Thesis for Massachusetts Institute of Technology. Cambridge, MA, May 2007.
[BibTeX][PDF]Sketch Recognition for Course of Action Diagrams. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, September 2007.
[BibTeX][PDF]Using Context to Resolve Ambiguity in Sketch Understanding. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, February 2006.
[BibTeX][PDF]Sketch Interpretation Using Multiscale Stochastic Models of Temporal Patterns. Ph.D Thesis for Massachusetts Institute of Technology. May 2006.
[BibTeX][PDF]Multi-Domain Sketch Understanding. Ph.D Thesis for Massachusetts Institute of Technology. August 2004.
[BibTeX][PDF]Segmentation and Alignment of Speech and Sketching in a Design Environment. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, February 2003.
[BibTeX][PDF][PS]Dr. Jones: A software Design Explorer's Crystal Ball. Ph.D Thesis for Massachusetts Institute of Technology. August 2003.
[BibTeX][PDF]Perceptually Based Learning of Shape Descriptions. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, 2003.
[BibTeX][PDF]Feature Point Detection and Curve Approximation for Early Processing of Free-Hand Sketches. Master's Thesis. May 2001. Department of EECS, MIT.
[BibTeX][PDF][PS]A Natural Sketching Environmant: Bringing the Computer into Early Stages of Mechanical Design. Master's Thesis for MIT. 2000.
[BibTeX][PDF][PS]Understanding Naturally Conveyed Explanations of Device Behavior. Master's Thesis for Massachusetts Institute of Technology. Cambridge, MA, 2000.
[BibTeX][PDF][PS]Designing Navigable Information Spaces. Master's Thesis for Massachusetts Institute of Technology. 1998.
[BibTeX][PDF][PS]MIT Reports and Publications
2010
Deskcorder: A Recorder for What Happens at your Desk. In Sixth Annual CSAIL Student Workshop, pp.27--29. 32 Vassar St., Cambridge, MA, 02139-4307, September 2010.
[BibTeX][PDF]2006
Initial Results from Speech and Sketching User Study. In Second Annual CSAIL Student Workshop. 2006.
[BibTeX][PDF]2003
Creating a Multimodal Design Environment Using Speech and Sketching. In Third Annual MIT CSAIL Student Oxygen Workshop. 2003.
[BibTeX][PDF][PS][PPS]Dynamically Constructed Bayesian Networks for Sketch Understanding. In Third Annual MIT CSAIL Student Oxygen Workshop. 2003. Won Best Presentation.
[BibTeX][PDF][PS][PPT]Generic and HMM based approaches to freehand sketch recognition. In Proceedings of the MIT Student Oxygen Workshop. 2003.
[BibTeX][PDF][PS]Recognition efficiency issues for freehand sketches. Proceedings of the MIT Student Oxygen Workshop . 2003.
[BibTeX][PDF][PS]2002
Mutual Disambiguation of Verbal and Sketching Inputs in a Design Environment. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2002.
[BibTeX][PDF][PS]Designs for the Future. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2002.
[BibTeX][PDF][PS]Dr. Jones: A Software Design Explorer's Crystal Ball. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2002.
[BibTeX][PDF][PS]A Domain Description Language for Sketch Recognition. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2002.
[BibTeX][PDF][PS]A Domain Description Language for Sketch Recognition. Proceedings of the 2nd Annual MIT Student Oxygen Workshop . 2002.
[BibTeX][PDF][PS]Multi-Domain Sketch Recognition. Proceedings of the 2nd Annual MIT Student Oxygen Workshop . 2002.
[BibTeX][PDF][PS]Sketch Recognition in Software Design. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2002.
[BibTeX][PDF][PS]Generating Domain Specific Sketch Recognizers From Object Descriptions. In MIT Artificial Intelligence Laboratory Annual Abstract. 2002.
[BibTeX][PDF][PS]Generating Domain Specific Sketch Recognizers From Object Descriptions. Proceedings of the MIT Student Oxygen Workshop . 2002.
[BibTeX][PDF][PS]2001
A Framework for Multi-Domain Sketch Recognition. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2001.
[BibTeX][PDF][PS]Graph Exploration for Software Archeology. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2001.
[BibTeX][PDF][PS]Ligature: Gesture-Based Configuration of the E21 Intelligent Environment. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2001.
[BibTeX][PDF][PS]Ligature, Gesture-Based Configuration of the E21 Intelligent Environment. Proceedings of the MIT Student Oxygen Workshop . 2001.
[BibTeX][PDF][PS]Query by Attention: Visually Searchable Information Maps. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2001.
[BibTeX][PDF][PS]Natural Editing and Recognition of UML Class Diagrams. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2001.
[BibTeX][PDF][PS]Natural Sketch Recognition in UML Class Diagrams. Proceedings of the MIT Student Oxygen Workshop . 2001.
[BibTeX][PDF][PS]Free-Hand Stroke Approximation for Intelligent Sketching Systems. Proceedings of the MIT Student Oxygen Workshop . 2001.
[BibTeX][PDF][PS]Scale-space Based Feature Point Detection for Noisy Digital Curves. In MIT Artificial Intelligence Laboratory Annual Abstract. 2001.
[BibTeX][PDF][PS]2000
Intelligent Mechanical Engineering Design Environment: From sketching to simulation. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2000.
[BibTeX][PDF][PS]Supporting Group Brainstorming. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2000.
[BibTeX][PDF][PS]Understanding Naturally Conveyed Explanations of Device Behavior. In MIT Artificial Intelligence Laboratory Annual Abstract. September 2000.
[BibTeX][PDF][PS]Early Processing in Sketch Understanding. In MIT Artificial Intelligence Laboratory Annual Abstract. 2000.
[BibTeX][PDF][PS]1999
Behavior Annotated Designs: A multimodal approach to design rational capture and intelligent design environments. In MIT Artificial Intelligence Laboratory Annual Abstract. September 1999.
[BibTeX][PDF]
Movies
Early demonstration of the ASSIST sketch understanding system. [mpg format; 49 Meg] This is done using a Mimio/whiteboard; the current system enables a more natural sketching style and operates on either a whiteboard and a tablet computer.
A demonstration of a more complex mechanical device: a parts feeder (no sound) [23 Meg]
An early demonstration of sketch understanding applied to the domain of software diagrams. [38 Meg]
Multimodal interaction: sketching and speaking [6 Meg]
Learnng a new icon from a single example (no sound) [10 Meg]
Demo of Commander's Course of Action Tool Prototype, developed by CSAIL and Draper Laboratory [18 Meg]