IMP Developer's Corner

This page contains information and links of particular interest to developers of IMP, that is, developers who are working on the IMP runtime or meta-tooling.

This page may be of some interest to developers who are using IMP to develop an IDE and who would like a deeper understanding of how the IMP runtime and meta-tooling work, but this page should not contain much information that is useful in the typical case of IMP IDE development.  (If it does, we should probably move that information into a more appropriate place elsewhere on the site!)

As should be apparent, this part of the IMP site is still under construction.  We will populate it initially with some things that we can think of off the top of our heads, based mainly on our own experience.  But we hope to be able to grow this part of the site substantially as the number of IMP developers grows and we receive their excellent feedback.        

Notes on miscellaneous topics follow:
The Right Stuff (additional IMP projects you may need to acquire separately, at least for now)
More of the Right Stuff (additional project you may need for IMP to build)
Building Polyglot (how to build Polyglot)
Building Required Parsers (for DSLs that are part of IMP)
Eclipse Version (the version of Eclipse on which IMP works)
Nonstandard Uses of Eclipse (some things for which we'd like to have better support)
What's What (an overview of IMP and related projects)

The Right Stuff

The original release of IMP to SourceForge neglected to include three projects that are needed for IMP to build:

org.eclipse.imp.smapi (recently renamed from com.ibm.watson.smapi)
org.eclipse.imp.smapifier (recently renamed from com.ibm.watson.smapifier)
org.eclipse.imp.lpg.metatooling

These projects have been available on SourceForge since about 16 Oct 2007.  If you are still missing them, we suggest that you check the IMP CVS repository again.

More of the Right Stuff

In addition to what comes with the IMP release, you may also need to get the following projects from sourceforge: 
 
1) lpg.runtime.java 
 
from :pserver:lpg.cvs.sourceforge.net:/cvsroot/lpg

This is necessary to build the LPG meta-tooling, which is necessary if you want to use LPG to define your language


2) polyglot 
 
from :pserver:gforge.cis.cornell.edu:/cvsroot/polyglot 

This is required mainly by IMP IDEs (or IDEs in progress) that make use of it in representing ASTs.  These include the X10DT, the IDE presentation-specification language (in org.eclipse.imp.metatooling) and our tiny, half-baked Java IDE (org.eclipse.imp.java.core).  Polyglot may also be referenced by a few other projects.


3) com.ibm.wala.shrike

In this case you don't need to check out the project but you do need the plug-in in your Eclipse installation.  You can obtain it from the WALA SVN repository at the following location:

   http://wala.svn.sourceforge.net/svnroot/wala/trunk/com.ibm.wala.shrike

In order to obtain the plug-in from the WALA SVN repository, you will need to install the Subversion plug-in in Eclipse (see http://subversion.tigris.org/).

This plug-in com.ibm.wala.shrike needed by the plug-in org.eclipse.imp.smapi. 


4) x10.compiler and x10.runtime

from :pserver:x10.cvs.sourceforge.net:/cvsroot/x10

If you want to build the X10DT-related projects, which comprise an IDE for the X10 programming language (http://x10.sourceforge.net).  The X10DT provides an example of an early IMP-based IDE, but you can always study the code without building and running it.

Building Polyglot

In order to build Polyglot in your workspace, you must first put the polyglot project there.  You can check it out from :pserver:gforge.cis.cornell.edu:/cvsroot/polyglot

If the file "lib/java_cup.jar" exists in the project, remove it.

Now you can build the polyglot project using Ant.  You can do this either outside or inside of Eclipse:

Outside of Eclipse:
This is the recommended approach, at least for now.  It requires that you have Ant installed somewhere on your system.  (You can obtain Ant, if necessary, from http://ant.apache.org/bindownload.cgi.)
1)   In a shell, run "ant base base-qq pth" from within the polyglot directory
2)  In Eclipse, refresh the polyglot project

Inside of Eclipse:
To build polyglot inside of Eclipse, you must have the Ant plugin installed.  Then, in principle, you can build the polyglot project by running the build file directly (right click on "build.xml" and select Run As -> Ant Build).

Some of us have had problems getting the polyglot project to build so easily.  We're still investigating why this happens and will post updated instructions, as warranted, when we know more.

Building Required Parsers

The IMP and LPG projects contain support for several language-specific IDEs that have been build using IMP and LPG.  These languages have parsers (and related types, including AST node types) that have been generated using LPG.  Since these types can be regenerated given a (much smaller) grammar specification, the IMP and LPG projects do not contain the generated types; rather, these projects only contain the necessary grammar specifications.  As a result, when one of these projects is newly checked out from CVS, IMP developers must run LPG on the included grammar specifications in order to regenerate the parsers and related types for these languages.  In the absence of the generated types the projects will fail to build.  (You will see compile errors in the language IDE types.)

The projects that contain these grammar specifications, and the name of the specification file, are listed in the following table:

Project
 Package and Grammar Template
org.eclipse.imp.lpg.runtime
 org.eclipse.imp.lpg.parser.LPGParser.g
org.eclipse.imp.java.core
 org.eclipse.imp.java.parser.JavaParser.g
org.eclipse.imp.metatooling
 org.eclipse.imp.presentation.LangSpec.g
org.eclipse.imp.prefspecs
 org.eclipse.imp.prefspecs.parser.PrefspecsParser.g
org.eclipse.imp.xform
 org.eclipse.imp.xform.pattern.parser.ASTPatternParser.g

The main grammar files have a ".g" extension.  Each of these is typically accompanied by some number of "grammar include" files that have ".gi" extensions.  (You may also find some grammar specifications in other projects, but these may be works in progress for which generation is not required.)

At this time, in order to initially regenerate the parser and related classes for each language, it is necessary to run LPG manually.  The LPG executable is contained in the project "lpg.runtime.java" in the folder "lpgexe."  However, when you download this project from SourceForge, it should create an Eclipse "external tool" that you can use to invoke LPG manually from within Eclipse.  This should show up as an item labeled "lpg" in the External Tools menu ("Run" --> "External Tools").  To run LPG manually, select the grammar specification to use as input and then select "lpg" in the External Tools menu.

When you run LPG in this way, it should generate several Java classes in the package in which the grammar specification is found.  LPG will also usually generate classes for AST node types, and typically these will appear in a package "ast" located under the package with the parser.  (In some cases, the AST node types may appear as member classes within the parser, according to a parameter set in the grammar specification file.)

Once you have run LPG manually on a project, the LPG builder should run automatically on that project whenever the grammar specification in the project is updated.  (The LPG builder is part of the IMP-based LPG IDE, which is included in the IMP release.  Why this builder only starts to run automatically after we have run LPG manually we don't understand--if anyone can offer an explanation, please let us know!)   Most IMP developers should not need to modify any of these grammar specifications.  Once the parser and related classes are generated and the containing projects are building, the IMP developer should be able to ignore the specifications.

There is, however, one small glitch that may require further intervention in the grammar files by the IMP developer.  For some reason, LPG may report a spurious error on grammar files (especially ".gi" files) that have not been modified and that are actually correct.  In these cases, it is sufficient to "touch" the problematic file, e.g., by adding and deleting a blank character.  That should trigger the LPG builder and, in our experience, that will typically lead to the removal of the problem marker.

Eclipse Version

General dependencies on Eclipse are discussed elsewhere on the site (see Installation).  The version of IMP that has been released on SourceForge runs on Eclipse 3.2.2.  We don't know much about how it will build and run on Eclipse 3.3, except that users of the X10DT have reported problems with some of the parallel code there.  We recommend developing for IMP using Eclipse 3.2.2, but if you know of any specific     problems of IMP on Eclipse 3.3 we'd love to hear about those.

Nonstandard Uses of Eclipse

The IMP code makes a couple of sorts of nonstandard uses of Eclipse.  One is that we have made use of Eclipse internal classes, mainly (we believe) related to elements of the GUI.  We're amateur GUI programmers this is typically the only way we've found to do some of the (seemingly ordinary) GUI things that we'd like to do.  We  hope that over time more of these GUI capabilities will be available in non-internal packages and also that we find more ways to program our GUIs (and other parts) without using Eclipse internal elements.

The other area of nonstandard use of Eclipse relates to the models for extensions and for plugin-dependencies.  As services are added to an IMP-based IDE, it is necessary to update these models.  It is also necessary to be able to read them in detail.  You wouldn't expect this to be a problem--but you'd be wrong.  In previous versions of IMP we actually maintained a parallel, modified version of org.eclipse.pde.core that would allow us to access and manipulate these models as we needed to.

For the current release we have done away with this parallel adaptation of org.eclipse.pde.core but we have introduced a couple of other compensating elements.  Regarding plugin dependencies, where we add a new IDE service, we simply rewrite the MANIFEST.MF file directly to include any new plug-in dependencies.  This is inelegant but effective.  Regarding the extensions model, we have introduced a parallel representation that gives us more complete access to the model.  We've done this through IMP-specific subtypes of a couple of the Eclipse extension-related classes, Extensions and WorkspaceExtensionsModel.  (See package org.eclipse.imp.extensionsmodel in project org.eclipse.imp.metatooling.)  We don't maintain any IMP-specific information about extensions, we just use these classes in several places where it is necessary to read or update details of the basic Eclipse extensions model.  We hope that over time the facilities in Eclipse will evolve to better suit our needs (indeed, there was a slight improvement in the transition from Eclipse 3.1 to Eclipse 3.2).  In the meantime, as far as we know, we need to maintain our own representation of the model.

What's What

Here are some notes about what various projects in or associated with the IMP release represent and where certain things are located.  (This is likely to be notably incomplete at first.)  It also points out where IMP developers are (and are not) likely to be working.  The organization of this list is somewhat ad hoc.  Project of more general interest are toward the front and projects of less general interest toward the end, but related projects are grouped together.  Of course, IMP developers may also find themselves working in their own projects.

org.eclipse.imp.metatooling

This project contains things used in the development of IMP IDEs.  Principal parts include

org.eclipse.imp.runtime

This project contains things that are used at runtime by an IMP IDE.  Some of these are also used by the IMP metatoling.  The principal parts are

org.eclipse.imp.lpg.metatooling

This project contains the IMP wizards that are used in generating LPG-based grammar specification files.  (An LPG builder--not part of this project--generates a parser and related components based on these specifications.)  Wizards in this project also support the generation of an IMP ParseController and related types, which may be used with or without an LPG-based grammar and parser.

A non-source folder called "templates" contains a variety of grammar- and parser-related templates including LPG templates for grammar specifications (used by IMP wizards), additional LPG templates for parser components (referenced by the LPG grammar templates), and IMP templates for the Parse Controller and related types (also used by the IMP wizards).

Actually, this project is not entirely focused on LPG.  At least one of the wizards only generates an IMP ParseController and related types with no LPG-specific components.  Thus the project is probably more accurately described as relating to grammar specification and parser generation in general.

IMP developers would not ordinarily have any reason to work in this project unless they were contributing some meta-tooling component related to making some additional grammar specification or parser generation functionality available to IMP IDE developers.

org.eclipse.imp.lpg.runtime

This contains the components that make up the IMP-based LPG IDE:  the editor, builder, parser (for grammar specifications), preferences, and so on.  IMP developers would not ordinarily have any reason to work in this project unless they were working on the LPG IDE itself.

lpg.runtime.java

This is the project in which the core LPG tooling lives.  It is not an IMP project but is obtained from the LPG project on SourceForge (:pserver:lpg.cvs.sourceforge.net:/cvsroot/lpg).  It contains the LPG executable(s), the generic (not IMP-adapted) LPG templates, and documentation.  It has a "src" folder with one Java package, lpg.runtime, which contains types referenced by LPG-generated parsers (of which there are several in the IMP release).

IMP developers will have no reason to work in this project.

org.eclipse.imp.preferences

This is a "metatooling"-like project, with wizards, templates, and cheatsheets, in support of the specification of preferences and preference pages for IMP IDEs.

This is not a project that would ordinarily be worked in by IMP developers unless they wre working on preferences specification for IMP IDEs.

org.eclipse.imp.prefspecs

This contains the grammar specification and IDE support for the "PrefSpecs" language, a domain-specific language (DSL) for specifying preferences and preference pages for IMP IDEs.

This is not a project that would ordinarily be worked in by IMP developers unless they wre working specifically on the PrefSpecs language and IDE.

x10.compiler and x10.runtime

These project contain (as you would expect) the X10 compiler and runtime support, respectively.  They are not IMP projects but come from the X10 project on SourceForge (:pserver:x10.cvs.sourceforge.net:/cvsroot/x10).

IMP developers will have no reason to work in this project.

org.eclipse.imp.x10*

These projects mainly represent the X10DT, which is the IMP-based IDE for X10.  These are IMP projects but IMP developers will not ordinarily have any reason to work in them.  The X10DT is maintained by IMP team members at IBM.

org.eclipse.imp.analysis

The project org.eclipse.imp.analysis is intended to hold various AST-based analyses, such as type inference and lightweight use/def analysis, in support of refactoring, content assist, quick fixes, and other IMP tooling. In particular, it contains a simple framework for representing and solving constraint systems that can be adapted to a variety of analysis problems. It's intended to be relatively easy to use, and rather lightweight compared to WALA, which supports much more sophisticated and precise analyses, at the expense of a much larger and more complex framework.

This project will naturally be of interest to IMP developers who are working on analyses in support of IMP IDE services.

org.eclipse.imp.xform

This is an IMP project in which components to support AST pattern-matching and transformation are being developed.  IMP developers would not ordinarily have any reason to work in this project unless they were working on some aspect of this problem.

....smapi and ....smapifier

These projects were originaly named "com.ibm.watson.smapi" and "com.ibm.watson.smapifier."  As of this writing we're planning to rename them but have not yet decided on the new names.

These projects address source mapping, that is, the mapping of source lines from a program in one language (such as an IMP-defined DSL) to source lines in a corresponding program in another language (such as Java), into which the first program has been translated.  These projects are essentially an adjunct of the IMP runtime.  (When an IMP-based IDE is used on an application development project, it is necessary for the builder in that IDE to be enabled on that development project.  This entails associating the corresponding language nature to the project.  If SMAP capabilities have been incorporated into the IDE, then this will also associate the SmapiProjectNature to the application development project.)

These projects have been quiescent for some time.  The typical IMP developer would not have reason to work in them unless working on this specific service.

org.eclipse.imp.cheatsheets

This project, as its name suggests, is devoted to materials for Eclipse "cheat sheets" on the development of an IMP IDE.  (Cheat sheets provide on-line documentation for Eclipes users, typically leading them step-by-step through some activity.)  IMP IDE developers would have a need to work in this project only if they were updating the exsiting documentation or (perhaps more likely) providing guidance on some new service or feature.




Get Firefox!