:: CLUE Lab ::

Visual Languages

A visual language is a set of practices by which images can be used to communicate concepts.

In the ambit of Visual Languages, our research has focused on theoretical, methodological and applied aspects of the traditional visual languages and has deepened on the study of the sketched visual languages, that is, the languages defined on free-hand drawn sketched visual languages.

As for the theoretical aspects, extensions to the model of picturel languages to the colored, drawn and pixel pictures case have been defined, studying some of their computability and decidability aspects.
As for the methodological and implementing aspects, a technique, based on positional extended grammars, for the creation of the compilers and the visual languages and then of the visual modeling environment through the use of technologies tradiotionally employed for the textual programming languages, has been define. Some applications focus on the use of visual languages both in the e-learning and in the software engineering fields.

In the ambit of sketched visual languages, our research has brought to the definition of a new grammar formalism, called Sketch Grammar, capable of describing both the free hand drawn symbols and the syntax of the languages they compose. On the sketch grammars several procedures have been defined, similar to those defined for LR Parsing, for the automatic generation of recognizers for the described sketch languages.
In order to improve the efficiency and the accuracy of the built recognizers, several strategies have been defined, such as, the introduction of a training phase, the use of a framework based on agents for the low level recognition of symbols, the use of the ambiguity resolution context during symbol recognition, the use of techniques for error recovery to capture information on missing or erroneously recognized strokes.
The results gained in this ambit and further studies on the definition of incremental parsers based on GLR parsing have been used for the automatic generation of syntax-aware visual language editors, which allow the user to easily construct a hand drawn visual phrase. In these editors, the user is interactively aided in the correct building of the phrase with suggestions on the possible extensions of the phrase with techniques of automatic completion of incomplete symbols.
An initial experimental prototype has been integrated in the VLDesk system, already used for the generation of classical visual environments. This has allow the development of empirical studies on real languages, such as UML, to verify the effectiveness of the proposed approaches.

Publications

Our recent publications in Visual Languages research

2010

• G. COSTAGLIOLA; HAMMOND T; PLIMMER B. Editorial of the JVLC Special Issue on Sketch Computation JOURNAL OF VISUAL LANGUAGES AND COMPUTING Vol. 21-2. Pp. 67-68

2009

• Costagliola, Gennaro; Fuccella, Vittorio. A Visual System for Analyzing User Behaviour in Web Tasks VL/HCC'09. Pp. 101-104 (download draft)
• Costagliola, Gennaro; Fuccella, Vittorio. Fine-Grained Analysis of Web Tasks through Data Visualization Lecture Notes in Computer Science - Web Engineering Vol. 5648. Pp. 1-15 (download draft)

2008

• G., Casella; Deufemia, Vincenzo; V., Mascardi; Costagliola, Gennaro; M., Martelli. An agent-based framework for sketched symbol interpretation JOURNAL OF VISUAL LANGUAGES AND COMPUTING Vol. 19. Pp. 225-257
• COSTAGLIOLA G.; GRECO A. Towards Semantic Role Labeling of Hand-drawn Sketches Proceedings of VLHCC 08 - Workshop on Sketch Tools for Diagramming. Pp. 29-36
• COSTAGLIOLA G.; GRECO A. On Semantic Labeling of Hand-Drawn Sketches Proceedings of the 14th International Conference on Distributed Multimedia Systems, DMS-VLC 2008. Pp. 288-291
• COSTAGLIOLA G; DEUFEMIA V; RISI M.. Using Error Recovery Techniques to Improve Sketch Recognition Accuracy Graphics Recognition. Recent Advances and New Opportunities Vol. 5046. Pp. 157-168

2007

• COSTAGLIOLA G; DEUFEMIA V.; POLESE G. Visual Language Implementation through Standard Compiler-Compiler Techniques JOURNAL OF VISUAL LANGUAGES AND COMPUTING Vol. 18(2). Pp. 165-226
• COSTAGLIOLA G; DEUFEMIA V; RISI M.. Improving the Robustness of Diagrammatic Sketch Recognizers Proceedings of the 14th International Conference on Distributed Multimedia Systems (Visual Language Computation), DMS-VLC 2007. Pp. 241-246
• COSTAGLIOLA G; DEUFEMIA V; RISI M.. Towards Robust Sketch Recognizers through Error Recovery Techniques Extended Abstracts of the Seventh IAPR International Workshop on Graphics Recognition GREC 2007. Pp. 59-60
• Costagliola, Gennaro; Deufemia, Vincenzo; Risi, Michele. Using Grammar-based Recognizers for Symbol Completion in Diagrammatic Sketches Proceedings of 9th International Conference on Document Analysis and Recognition Vol. 2. Pp. 1078-1082
• Gennaro Costagliola, Vincenzo Deufemia, Giuseppe Polese. Modeling and Implementing Visual Languages with Extended Positional Grammars Visual Languages For Interactive Computing: Definitions And Formalization. Pp. --

2006

• Costagliola, Gennaro; Deufemia, Vincenzo; Ferrucci, Filomena; Gravino, Carmine. Constructing Meta-CASE Workbenches by Exploiting Visual Language Generators IEEE TRANSACTIONS ON SOFTWARE ENGINEERING Vol. 32. Pp. 156-175
• COSTAGLIOLA GENNARO; DEUFEMIA V.; RISI MICHELE. A Multi-layer Parsing Strategy for On-line Recognition of Hand-drawn Diagrams Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing. Pp. 103-110
• Costagliola, G.; De Lucia, A.; Deufemia, V.; Gravino, C.; Risi, M.. Case Studies of Visual Language Based Design Patterns Recovery Proceedings of 10th European Conference on Software Maintenance and Reengineering. Pp. 165-174
• COSTAGLIOLA GENNARO; DEUFEMIA V.; RISI MICHELE. A Recognition Framework based on LR Parsing for Hand-drawn Diagrams Proceedings of 12th International Conference on Distributed Multimedia Systems ( Visual Languages and Computing -DMS-VLC). Pp. 123-128
• CASELLA GIOVANNI; COSTAGLIOLA GENNARO; DEUFEMIA V.; MARTELLI MAURIZIO; MASCARDI VIVIANA. An Agent-Based Framework for Context-Driven Interpretation of Symbols in Diagrammatic Sketches Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing. Pp. 73-80

2005

• COSTAGLIOLA G.; FERRUCCI F; GRAVINO C. Adding symbolic information to picture models: definitions and properties THEORETICAL COMPUTER SCIENCE Vol. 337(1-3). Pp. 51-104
• Gennaro Costagliola, Filomena Ferrucci, Giuseppe Polese, Giuseppe Scanniello. A Visual Language Based System for Designing and Presenting E-learning Courses International Journal Of Distance Education Technologies, Vol. 3. Pp. 43466
• COSTAGLIOLA G; DEUFEMIA V; POLESE G; RISI M.. Building Syntax-Aware Editors for Visual Languages JOURNAL OF VISUAL LANGUAGES AND COMPUTING Vol. 16(6). Pp. 508-540
• COSTAGLIOLA GENNARO; DEUFEMIA V.; POLESE GIUSEPPE. Towards Syntax-Aware Editors for Visual Languages ELECTRONIC NOTES IN THEORETICAL COMPUTER SCIENCE Vol. 127. Pp. 107-125
• COSTAGLIOLA G; DEUFEMIA V; RISI M.. Sketch Grammars: A Formalism for Describing and Recognizing Diagrammatic Sketch Language Eighth International Conference on Document Analysis and Recognition (ICDAR 2005). Pp. 1226-1230
• COSTAGLIOLA G.; DEUFEMIA V; RISI M. A Trainable System for Recognizing Diagrammatic Sketch Languages Proceedings. 2005 IEEE Symposium on Visual Languages and Human-Centric Computing. Pp. 281-283
• G. COSTAGLIOLA; DE LUCIA A.; F. FERRUCCI; G. SCANNIELLO. Visual Languages for Non Expert Instructional Designers: A Usability Study Proceedings of the 11th International Conference on Distributed Multimedia Systems, DMS 2005, September 5-7, 2005, Fairmont Banff Springs Hotel, Banff, Alberta, Canada. Pp. 46-51
• G. COSTAGLIOLA; DE LUCIA A.; V. DEUFEMIA; C. GRAVINO; M. RISI. Design Pattern Recovery by Visual Language Parsing Proceedings of 9th European Conference on Software Maintenance and Reengineering. Pp. 102-111

2004

• Costagliola, Gennaro; Deufemia, Vincenzo; Ferrucci, Filomena; Gravino, Carmine. A user-centered methodology to generate visual modeling environments ICEIS 2004 - Proceedings of the Sixth International Conference on Enterprise Information Systems Vol. 3. Pp. 147-154