IMP Users Guide

This document provides guidance on using IMP to define a customized IDE for a new language.  It addresses the following topics:

The sections that are relevant to all IMP users are the prerequisites, identifying a project for the language, creating a language description, and creating a parsing service, and using you IMP IDE.  The other services indicated are more or less optiona.  Additional IMP services are under development and will be addressed here when they are ready for general use.

The discussion of the parsing service mainly addresses the case in which LPG will be used to define the language grammar and generate a lexer and parser.  However, other approaches to grammar specification and parser development can be accommodated in IMP, and the treatment of most other topics here is independent of the manner in which the parser is provided.

A note on procedure:  The steps through "creating a parsing service" are required for any IMP IDE.  The subsequent optional steps may be pursued in any order.

A note on terminology:  The text below will refer to several of the classes generated by IMP by simplified "generic" names, such as the "Plugin" class or the "ParseController" class.  As these classes are actually generated, their names will typically be prefixed by the name of the language to which they apply.  Thus, if your language is called "MyLanguage", IMP will generate classes for you that are called "MyLanguagePlugin", "MyLanguageParseController", and so on.

A note on "SAFARI" versus "IMP"  "SAFARI" is the former name of the Eclipse IMP.  Many of the Java or Eclipse software entities that IMP comprises used to be named with reference to SAFARI.  The names of those entities are being updated to refer instead to IMP.  Similarly, this document used to refer to many software entities by names based on SAFARI.  The document is likewise being updated to refer to these entities by names based on IMP.  Until the transition in names is complete, this document may refer to renamed entities by their old names and to entities that have not been renamed by their new names.  As far as we know, though, the old and new names should be directly analogous, with "imp" substituted for "safari."  We apologize for any confusion this may cause. 

Prerequisites

You must have a IMP installed in your workspace.  For information on installing IMP, please look at the installation guide (which also lists IMP prerequisites).

Since we assume here that LPG will be used to generate a parser for the language, LPG must also be available in your workspace.  (Actually, you can't install IMP without installing LPG.)  Information on how to install LPG is also available in the installation guide.

The IMP release includes some IMP-based IDEs, including those for LPG, X10, and a preferences specification language.  These may serve as examples for how you might build your own IDE.              

Identifying a project for the language

The goal of this step is to set up an Eclipse project for the development of IMP IDE for your new language.

A IMP IDE must be defined in a Eclipse plugin project.  Each IMP IDE must be defined in a separate project.  If you do not have a suitable plugin project available as a home for the new IDE then you should create one.

A note about plugin classes:  When you use the Eclipse New Project wizard to create a new plugin project, the wizard by default creates a package with the name of the project.  In that package the wizard creates a plugin class with a name of the form "<ProjectName>Plugin.java."  The Safari New Programming Language wizard likewise creates a new package and plugin class, but one that by default is named based on the programming language rather than on the project.  If you use the default values in each wizard, you can end up with two plugin classes, each in a different package.

There are two ways to avoid this.  One is just to give your project and language the same name.  The other is, during plugin-project creation, to override the default values and give your plugin class and package a name that matches that of your language.  (With the latter approach it is possible to have the name of the language be different from the name of the project in which the language IDE is defined.)

(Also note:  There is a IMP wizard called "New Project Wizard" but this is a wizard for creating project wizards, not a wizard for creating projects.)

Creating a language description

The goal of this step is to provide basic information about the identity and characteristics of the new language.

This step is accomplished entirely within a IMP wizard; to run this wizard invoke:

"File" -> "New" -> "IDE Language Support" -> "Programming Language"

A note on navigating these Eclipse menus:  Under the "New" menu you may need to select "Other ..." in order to get to the next item.  Instead of following the menus, you can type "Ctrl-N" to open the "New--Select a wizard" dialog from which you may be able to select a desired item directly.  These comments apply to all of the menu-navigation instructions below.

In the "Programming Language" wizard, you must provide information for the following required fields:

You may also provide information in the following optional fields:

Hit “Finish” when you are done editing the various fields in the wizard.  A IMP language description extension will be created using the above information and added to the project’s plugin.xml file.  This step also updates the projects Plugin class (to extend IMPPluginBase, among other modifications), or it will create a Plugin class if one does not exist.

Creating a parsing service

The goal of this step is to create a lexer and parser for the language.  These can be generated from lexer and parser grammars using LPG (which is provided with the IMP release).

A IMP wizard generates skeleton files for the lexer and parser grammars, which then need to be completed by the user with specific language details.

Given the grammar files, the lexer and parser can be generated automatically or manually using LPG (in the form of lpg).  Generation may occur by means of a LPG builder associated with the language project or by means of lpg run as an Eciplse external tool.

To run the wizard to generate the skeletons for the lexer and parser grammars, invoke
"File" -> "New" -> "IDE Language Support" -> "Parser Services" -> "LPG Grammar and Parser for UIDE".

 In the wizard:

Finishing the “LPG Grammar and Parser of UIDE” wizard has the effect of adding a parser extension to the plugin.xml file in your language project.  The wizard also creates two packages in the source folder of the language project:

You may also discover that this package contains an empty Parser class.  This may be generated automatically if the LPG builder is running automatically in your project (in which case it will be triggered by the appearance of the grammar files).

Additionally upon finishing the wizard, the LPG grammar-template file for your language will open in the editor, and the structure of the file should appear in the "Outline" view.

The grammar files contain some generally useful elements (including directives to LPG regarding generation) and some simple instructions for completing the files.  The files also include some example entries for a simple expression language.  You will need to edit these file as appropriate for your language.  A quickstart guide to LPG, as well as links to additional documentation on LPG, is found here.  As noted there, the LPG grammar files make ample provision for the inclusion of custom code in a generated parser.

The grammar files may be edited in any order.  However, for purposes of generation, the general lexer depends on they keyword lexer and the parser depends on the geneal lexer, so they should be generated in that order.

Your language project should be configured to include the LPG Grammar File Builder.  This should happen automatically when you run the parsing-service wizard.  If for some reason your language project is not configured to include the LPG builder, then you can enable the builder by selecting the "Enable LPG Builder" in the context menu for the project.  If for some reason you want to run LPG manually (which shouldn't usually be neecessary), you can find information about that here.

The LPG Grammar File Builder

If your language project is configured to include the LPG Grammar File Builder, then the generation process should occur automatically whenever a project build is triggered (which may be automatically or manually, according to the configuration and use of Eclipse).   The builder may also be triggered whenever the grammar file is changed.  (Note:  At this time, changes to the ".g" file will trigger the LPG builder but changes to the ".gi" files will not generally trigger the builder.  If you change the ".gi" files then you will have to run lpg manually.)

Notes about compilation

The generated parser classes will depend on lpg to compile, so lpg.jar should be in the build path for your language project.  This file can be found in the Eclipse "plugins" folder, under "lpg_1.0....."

If your grammar will define a type with a name that duplicates one of the Java type names (such as “Number”), then you will need to disambiguate references to this type in the parser (which can be done by explicitly importing the corresponding AST node types for your language).  The Java compiler should make this evident to you.

Other approaches

If you have your own parser:  Since the goal of this step is to produce a lexer and parser, if you already have a lexer and parser that meet IMP requirements then these may be used directly.  The "Hand-written Parser" wizard allows you to select an existing parser and creates a corresponding parser extension in the plugin.xml file.  To work within IMP, any "hand written" or non-LPG-generated parser should, at a minimum, implement the ILexer, IParser, and IParseController interfaces in the org.eclipse.uide.parser package in the org.eclipse.uide.runtime project.  This will entail providing implementations for several additional types defined by IMP or LPG.  (Some of these, unfortunately, are currently defined by concrete types, although we expect to provide interfaces for these in the future.)  Thus, at this time we recommend the use of LPG to generate parsers for use in IMP.  NOTE:  If you have a parser that was generated by LPG but outside of IMP, it should be relatively straightforward to adapt it to IMP.

If you have your own grammar:  If LPG is to be used to generate the lexer and parser, but you already have grammars for the language to be parsed, then these may be substituted into, or in place of, the IMP-generated templates.  Of course, the resulting grammar files must be compatible with LPG.

Creating a token-coloring service

The goal of this step is to create a class that will support coloring of tokens of different sorts when their text appears in the editor (for example, keyword highlighting).

IMP provides a wizard to generate a skeleton class for this purpose; this class must be specialized by the user according to details of the language and goals for the editor.  To run this wizard: invoke

"File" ->; "New" -> "IDE Language Support" -> "Editor Services"  ->  "Token Colorer"

As before, in the wizard assure that the names of your project and language have been correctly entered into the appropriate fields.  If desired, edit the remaining fields. Hit "Finish" when done.

The Token Colorer wizard creates a safari.tokenColorer package containing a TokenColorer class.  The wizard adds a tokenColorer extension to the plugin.xml file for the language project and opens the TokenColorer class in an editor.

The TokenColorer class has two important methods:  the constructor, in which a set of TextAttribut's are created (which are later used to specify the text presentation for the various token kinds), and the getColoring(..) method, which returns the TextAttribute to be used for the given token kind.

As you can see, getColoring() calls IToken.getKind() to determine the token kind (represented by an integer) and uses that to compute the correct text attributes. The set of valid token kinds is defined by the lexical analyzer; they are typically located in the LPG-generated "Parsersym" interface (in the parser package) and generally start with a prefix like "TK_".

The generated TokenColorer class includes a few examples of text attributes and token colorings based on the example expression language introduced with the grammar-file templates.  These may not compile as generated (due to inconsistencies with the generated parser and AST) and will generally have to be tailored to the language being defined.

The generated TokenColorer class is somewhat dependent on LPG:

As noted elsewhere, it is possible to implement these various interfaces without using LPG (although it is the intention of IMP to relieve the developer of this work by automating the implementation using LPG).

Creating an outlining service

The goal of this step is to create a service that provides an outline view of documents in your language.  As for the TokenColorer, there is a IMP wizard that creates a skeleton class that contains the generic structure of the outliner, and this must be tailored to create an outliner for the specific language being defined.  To run this wizard, invoke

"File" -> "New" -> "IDE Language Support" -> "Editor Services" -> "Outliner"

In the wizard, as before, assure that the names of your project and language are correct and, if desired, edit the remaining fields. Hit "Finish" when done.

The Outliner wizard creates a safari.outliner package containing an Outliner class (and an Images class).  The wizard also adds an outliner extension to the plugin.xml file for the language project and opens the Outliner class in an editor.

The Outliner class only has one important method:  createOutlinePresentation(), which typically retrieves the current AST (from an IParseController) and then visits the AST, populating the outline tree with various TreeItems corresponding to AST nodes of interest.  The visiting is done by an instance of the OutlineVisitor class, which is a member of Outliner and extends the AbstractVisitor class.

The AbstractVisitor class contains default visit(..) and endVisit(..) methods for all node types in the AST.  These methods will traverse the AST but otherwise have no effect.  The AbstractVisitor class is usually located in the “parser.Ast” package or—depending on parser-generator options—it may be contained as a member class within the generated parser class.

Some example visit(..) methods for the simple expression language are shown in the generated Outliner.OutlineVisitor class (which may prevent the generated file from compiling).  These must be adapted for the specific language being defined.

To add a program entity to the outline, in the OutlineVisitor class simply override the default visit(..) method corresponding to that type of AST node.  Most overriding methods will call createSubItem(..) with an appropriate label.  The visit(..) method should return true (or false) depending on whether the children of the node should be visited (or not).  When an item label is created it is pushed onto a stack.  Thus, whenever a default visit method is overridden to create an item label, the corresponding endVisit(..) method should also be overridden in order to pop the item label from the stack.  Note that items that are inherently childless do not need to be pushed onto the stack, and if these items are not pushed onto the stack then it is unnecessary to override endVisit(..) for them.

It will also be necessary to have one visit(..) method that calls createTopItem(..) to create a root for the outline.  (It will also be necessary to have a corresponding endVisit(..) method.)  This may be for some high-level node type (such as “program”) or it may be given a label that does not correspond to a specific node type.

The generated Outliner class is somewhat dependent on LPG:

As noted elsewhere, it is possible to implement these various interfaces without using LPG (although it is the intention of IMP to relieve the developer of this work by automating the implementation using LPG).

Creating a text-folding service

The goal of this step is to create a service that supports the folding of hierarchical syntactic elements in text editors for the language.  As with the outlining service, the IMP wizard for text folding creates a skeleton class that contains the generic structure to support text folding and this class must be tailored to the specifics of the language being defined.

 The IMP wizard that does this is called “NewFoldingUpdater.”  To run this wizard, invoke

"File" -> "New" -> "IDE Language Support" -> "Editor Services" -> "Source folding updater"

As before, in the wizard assure that the names of the project and language are correct and, if desired, edit the remaining fields. Hit "Finish" when done.

The NewFoldingUpdater wizard creates a safari.foldingUpdater package containing a FoldingUpdater class.  The wizard also adds a folding updater extension to the plugin.xml file for the language project and opens the FoldingUpdater class in an editor.

FoldingUpdater contains one main method, updateFoldingStructure(..).  This method obtains the current AST and sends it a new AbstractVisitor.  This visitor should contain a visit method for each AST node type for which folding is desired.  Each visit method should simply call AbstractVisitor.makeAnnotation(..) with its given node.  (An example is shown in a comment in the generated FoldingUpdater code.)  The method makeAnnotation makes an annotation that spans the text represented by the node, and it is through the annotations that folding is enabled.  The language-specific customizations required for folding are simply to provide visit(..) methods for the AST node types for which text is to be foldable. These visit(..) methods should return true if folding should be allowed for nested units and false if not. It may be necessary to add more complicated logic if more precise control of folding is desired, for example, if you wish to enable folding of top-level blocks but not nested blocks.

Note:  The NewFoldingUpdater wizard requires the package and class names for the language AST node type.  The wizard currently assumes that the AST node type has the default name and is located in the default package.  If these assumptions are not valid, then incorrect names for this package and class will be entered into the generated FoldingUpdater class.  If that happens, the class should be edited to fill in the correct names.

Creating a hyperlinking service

The goal of this step is to create a service that supports hyperlinking between regions of text in text editors for the language.  Link sources and targets typically correspond to entities defined in the underlying AST; common examples include links from references to definitions, such as from field references to field declarations or from class references to class definitions.  The link targets may be in the same source file as the link source or in a different source file, depending on the details of hyperlink semantics and implementation.

The IMP wizard for hyperlinking creates two classes.  One is a HyperlinkDetector, the other is a ReferenceResolver.  The HyperlinkDetector contains a method detectHyperlinks(..) that is called when it is necessary to check for hyperlinks.  It makes use of a ReferenceResolver to find the target, if any, for the current candidate link source.  The HyperlinkDetector is largely language-independent; its only significant language dependence is on a language-specific ReferenceResolver.  The ReferenceResolver is highly language specific, since it involves recognizing language-specific link sources and identifying language-specific link targets according to language-specific semantics.   Typically the generated HyperlinkDetector does not require user modification and can be used directly as generated.  The ReferenceResolver is essentially template that will require some user programming, possibly considerable.  (As generated it compiles but does nothing useful.)  The amount of programming required to complete the ReferenceResolver may depend on the functionality provided by avaliable language tools such as a compiler.  In other words, if available language tools already support reference resolution, then the ReferenceResolver may just need to retrieve the relevant information from these tools.  On the other hand, if the available language tools do not support reference resolution, then the ReferenceResolver may need to encode the logic necessary to perform resolution.

The IMP wizard to generate the hyperlinking service called “NewHyperlinkDetector" and is run by invoking

"File" -> "New" -> "IDE Language Support" -> "Editor Services" -> "Hyperlink Detector"

In contrast with some of the other language service implementations, the ReferenceResolver does not (at least by default) rely on a visitor to the AST.  The principal methods to be implemented are getLinkTarget(node) and getLinkText(node).  The user can also implement methods that will perform filtering on given link source nodes to eliminate nodes that aren't appropriate in that role.

Note:  The as with the NewFoldingUpdater wizard, the NewHyperlinkDetector wizard needs the name of the package that contains  AST node types.  If the value provided by default is not valid, then the class should be edited to fill in the correct name.

Creating a builder service

The goal of this step is to create a builder for the language.  Builders perform activities like generation or compilation for files in a particular source language.  Builders in Eclipse are associated with projects in a many-to-many relationship.  When a project is built, the various builders for the project are invoked, and each builder performs its respective build action on files in the project that have the appropriate type.  The IMP NewBuilder wizard generates a Builder class that must be customized to perform the desired language-specific build activity.

To invoke the NewBuilder wizard, select

"File" -> "New" -> "IDE Language Support" -> "Core Services" -> "Incremental builder"

In the wizard:

The NewBuilder wizard generates one package, two classes, and three plug-in extensions, and it brings up one of the classes, the “Builder” class, in an editor.

The package that is generated is the “safari.builders” package.  It contains the two generated classes, a “Builder” class and a “Nature” class.  The new extensions are a builder extension (extending org.eclipse.core.resources.builders), a nature extension (extending org.eclipse.core.resources.natures), and a problem-marker extension (extending org.eclipse.core.resources.markers).  All of these classes and extensions are created regardless of the values provided to the wizard (in particular, regardless of whether "hasNature" is true or false)..  The problem marker defines a type of marker annotation that the builder can attach to source files to represent problems encountered when building the files.

The Builder class is a skeleton that overrides or implements several methods defined in its abstract parent class, IMPBuilderBase:

Most of these methods as generated will do reasonable things (although you may tailor any of them if you find a need to do so).

The method isNonRootSourceFile(..) is used to identify files (like C/C++ ".h" files or LPG ".gi" files) that should not be compiled directly but that should nevertheless be processed for dependencies.

In any case you will have to edit the compile(..) method, which is really the main “build” method.  This will have to call your “compiler” in an appropriate way for a given source file, retrieve any diagnostic messages that result from the building, and create problem markers for those.  If your build activity runs in an external process, or uses java.io to create output files, you will also need to call IResource.refresh() on any folders that contain generated output files (assuming that they are somewhere in your workspace).

IMP defines an interface, IMessageHandler (org.eclipse.uide.editor.IMessageHandler), to facilitate the reporting of error (and other) messages from compilers (or other builders) to interested services.  IMP also provides an implementation of this interface, MarkerCreator (org.eclipse.uide.builder), that creates marker annotations for messages received and attaches those to a given source file.

The figure below shows the implementation of a Builder compile(..) method using a LPG-generated ParseController in the role of the compiler and using MarkerCreator to handle parser error messages and create marker annotations on the given source file.

/* * Implementation of the compile method in a SAFARI generated Builder * for a subset of JavaScript known as “Jsdiv” using a JikesPG-generated * Parse Controller and the provided MarkerCreator implementation for * IMessageHandler */ protected void compile(final IFile file, IProgressMonitor monitor) { try { // Get a Parse Controller to serve as the “compiler” IParseController parseController = new JsdivParseController(); // Get a MarkerCreator to handle error messages from the parse controller // and create corresponding marker annotations on the input file MarkerCreator markerCreator = new MarkerCreator(file, parseController); // Initialize the Parse Controller parseController.initialize( file.getProjectRelativePath().toString(), file.getProject(), markerCreator); // Get file contents for parsing String contents = ContentExtractionUtility. extractContentsToString(file.getLocation().toString()); // Parse the contents parseController.parse(contents, false, monitor); // Refresh the parent doRefresh(file.getParent()); } catch (Exception e) { JsdivPlugin.getInstance().writeErrorMsg(e.getMessage()); e.printStackTrace(); } }

The Nature class extends org.eclipse.uide.core.ProjectNatureBase and provides some simple method implementations and stubs.  These can be used as-is without further modification unless more sophisticated control over building is required for the nature.  (With a new nature this is not likely to be true initially.)

The builder extension will be generated to contain elements representing the various fields from the wizard.  The nature extension will be generated with a “builder” element that refers to the generated builder extension.  (This happens regardless of whether “hasNature” was set to true in the wizard, but the nature need not be applied to any project that contains files in the source language, and the builder for the language can be used independently of the nature).  In the marker extension a marker with id “problem” is defined with super type “org.eclipse.core.resources.problemmarker.”  (Marker types are maintained as in this way as metadata.)

A note on Annotations and Markers

When a SAFARI editor runs, the source text is parsed frequently and error messages are reported as annotations that appear in the editor.  These annotations are not associated with the source file and do not appear in the "Problems" view.  When a SAFARI Builder runs, the editor annotations are deleted and a new set of marker annotations are created.  These annotations also appear in the editor (and may seem identical to pre-build editor annotations).   However, the marker annotations are associated with the file and do appear in the "Problems" view.  Building may occur automatically, whenever a file is saved (or a project is cleaned), or building may occur manually, only when explicitly invoked by the user.  Regardless, problem markers will appear when builds occur.  When a file is opened in an editor the problem markers are deleted but the file is parsed immediately and new editor annotations are created.

Using your SAFARI IDE

To use your language and SAFARI IDE in a development project, you will need to add a corresponding builder entry to that project's ".project" file.  If you have defined a nature for your language you may also want to add a nature entry to that project's ".project" file.  Once this is done, the services defined for editing and building in the language should be automatically available for files identified with the extensions given in the language description.

Final words of advice

Be careful entering information into the wizards.  The wizards will generally not let you get away without providing information for required fields, but they (and the rest of SAFARI) are not necessarily robust with respect to errors in the provided information.  Errors in the provided information may or may not show up directly and may or may not have confusing effects.

You can update some of the SAFARI information manually.  While the wizards are intended to facilitate the entry of necessary (or at least useful) information at appropriate times and places, you can also manually edit the plugin.xml file to set (or reset) many of the attributes associated with SAFARI extensions.  So, if you want to add an icon or validator to an IDE that is already defined, you can do that without rerunning the wizard.  (This can help to avoid problems that can occur when the wizards are rerun ...)

Be careful when rerunning the wizards!
  1. Several of the wizards generate skeleton files that must be tailored by the user.  If you rerun one of these wizards, the tailored files will be clobbered and updates to them will be lost.  So backup these files where they won't be clobbered.
  2. Several of the wizards add extensions to the plugin.xml file of the IDE project.  Rerunning these wizards does not remove the extensions created by previous invocations.  So, if you want to rerun one of these wizards, you should manually delete the corresponding extension(s) from the plugin.xml file.



Get Firefox!