Useful Information about LPG
This page contains various sorts of useful information about LPG, the
LALR parser generator included with IMP. (LPG was formerly known
as JikesPG and you may still find references to JikesPG in the IMP
documentation and code.) LPG also has its own SourceForge project
at
http://sourceforge.net/projects/lpg/.
IMP does not require any particular parser-related technology, but it
does require support for parsing and AST representation. We like
LPG and have adapted it to support the parser- and AST-related
interfaces on which IMP depends, and we have made it easy to use LPG
through IMP. For instance, there are IMP wizards for using LPG to
generate a parser and AST, and we have provided an IMP-based IDE to
facilitate the editing of LPG grammar files.
This section addresses the following topics:
LPG Quickstart for IMP Users
Running LPG as an External Tool
LPG Documentation
Troubleshooting
The IMP installation includes a
version of LPG for generating a lexer, parser and related
components in support of the instantiation of a IMP IDE for a new
programming language. The lexer and parser are generated from
"grammar" files that specify the lexer and parser grammars for the
language. IMP uses LPG with three grammar files: a
"KeywordLexer.gi" file (".gi" stands for "grammar include"), which
specifies the keywords of the language, a "Lexer.gi" file, which
specifies the general lexical grammar (given the keyword lexer), and a
"Parser.g" file (".g" stands for "grammar), which specifies the parser
garmmar (given the general lexer).
When the New Programming Language
wizard is run on a IMP project, it installs a LPG builder in the
project, creates a parser package in the source folder of the project,
and copies in three default grammar files. The builder then
generates Java classes for the lexer, parser, and related components,
using templates provided with LPG and IMP.
A brief overview of the
sections of the LPG grammar files that IMP users may encounter
(not all sections will occur in all
files):
- The "Options" section includes options for LPG.
- The "$Define" section is used to define macro
symbols
that are used by LPG itself. The macros may be defined to
include
other symbols or source texts that will be substituted for subsequent
occurrences of the symbols. If a symbol is defined more than
once, an
earlier definition is overridden by a later one.
- The "$Globals" section is intended
for material that should appear at the head of a generated file, such
as Java "import" directives.
- The"$Headers" and "$Trailers" sections include
members that are to be incorporated (respectively) at the beginning or
end of the generated parser class (more specifically, before or
after the expansion of the action blocks).
- An "$Include" section causes material to be
interpolated into the input file at the point at which the $Include
occurs.
- The "$Export" section defines the token types
recognized by the lexer or parser (in effect, the output elements)
- The "$Terminals" section defines the terminal
symbols recognized by the lexer or parser (in effect, the input
elements)
- The "$Start" section gives the root symbol for the
processing of grammar rules. In the parser grammar it is
typically something like "program"
- The "$Rules" section defines the grammatical rules
that map the terminals in to the exported tokens
What do these mean for the
IMP
user?
- The $Export, $Terminals, and $Rules sections
contain the representation of the grammar. These will be the
focus of user customizations. The rules may include
arbitrary
Java code (default delimiters "/." and "./"). The inclusion of
such code may entail additions of supporting material to the $Globals
and $Headers sections.
- LPG is confgurable with
many different
options. The
generated grammar-file skeletons are given a basic set of options that
are sufficient for generating a IMP-ready parser and these options
should not generally need to be edited. (Once you have attained
some mastery of
LPG by other means, then you may wish to edit the generated
options if it suits your purposes.)
- IMP makes some use of the $Globals section to
add import statements for interfaces that the generated classes will
implement. IMP does not by default make any use of the
$Headers or $Trailers sections. However, if special-processing
code is included in the grammar rules, then the $Globals, $Headers, and
$Trailers sections can be used to add imports, fields, methods,
and so on in support of that code.
- IMP uses the $Define section to define macros
that are used elsewhere in the grammar files to more completely
parameterizes the templates. It is not generally necessary for
the IMP user to add definitions (although exerienced LPG users
may wish to do so).
- IMP makes some use of $Include sections to
incorporate portions of grammar specifications that have been put into
separate files. There is not any necessary reason for IMP
users to customize the $Include sections, unless this is needed to
support special-purpose language processing.
The LPG grammar templates used by IMP, along with the Java
classes generated from these, usually give some examples and brief
instructions about how the files should be completed for a new
language.
LPG can be run manually in Eclipse as an external tool. To be
able to do this, you must have the project lpg.runtime.java imported
into
your workspace. This project contains both the LPG executable (in
the
folder "lpgexe") and the Eclipse launch configuration (in the folder
".launchConfigurations). When this project is in your workspace,
the
launch configuration should be recognized automatically and should be
available through the External Tools menu. (To open the External
Tools dialog, navigate "Run" -> "External Tools" -> "External
Tools ..."; you should then find LPG under "Programs")
Important note on system compatibility:
As of 18 Sep 2007 only the Windows version of LPG is being maintained
and only the Windows version of the LPG executable is up-to-date.
You may be able to create an executable for other systems by
downloading the source code from the LPG project on SourceForge (
http://sourceforge.net/projects/lpg/)
and building it locally. Alternatively, if your non-Windows
system has a Windows emulator, you may be able to run the Windows
version of LPG under that (see
Troubleshooting below).
To run LPG as an external tool on a specific grammar specification
(".g" or ".gi" file), select the grammar file and then invoke LPG from
the
External Tools dialog. If you run LPG on a ".g" file, LPG will
generate a complete set of tools (parsre, lexer, and keyword
filter). If you run LPG on a ".gi" file, you can generate just
the lexer or keyword filter (depending on the specific file).
The default parameters of the launch configuration for use with IMP are
as follows:
- Location:
this is the location of the LPG executable:
${workspace_loc:/lpg.runtime.java/lpgexe/lpg-win32_x86.exe}
- Working directory: this is the location in
which the input source will be found and the output specifications
generated. It is set relative to the selected grammar file.
${container_loc}
- Arguments: this tells LPG where to find the
templates to use in generating the parser, lexer, and keyword filter:
-include-directory=${workspace_loc:/lpg.runtime.java/templates};
${workspace_loc:/org.eclipse.imp.lpg.metatooling/templates}
${resource_loc}
(note that the second second argument refers to an IMP project that
contains IMP-specific templates).
If your workspace is configured differently than what the default
presumes, or if you have renamed the relevant projects, then you may
need to adjust these parameters for your particular situation.
There is documentation for LPG on the LPG website at
http://sourceforge.net/projects/lpg/.
LPG build on Linux doesn't run
correctly: One LPG user tried building the LPG executable
from source on a Linux system and got the same errors when he tried to
use it as when he ran the Linux version of the executable that is
provided in the LPG release from SourceForge (see the message thread
"Problems with LPG parser", dated 9/17/2007, in the
eclipse.technology.imp newsgroup on news.eclipse.org, started by Oleg
Murk, olegm@gmail.com). He was able to get the Windows version of
LPG running on his system under the Windows emulator Wine:
The trick is to replace the contents of
file
$ECLIPSE_HOME/plugins/lpg.runtime_2.0.6/lpgexe/lpg-linux_x86
with a script
----------------------------
#/bin/bash
`echo wine $0 $* | sed "s/lpg-linux_x86/lpg-win32_x86.exe/" | sed
"s/\"-include/-include/" | sed "s/'\"/'/"`
----------------------------
and make it executable
chmod u+x
$ECLIPSE_HOME/plugins/lpg.runtime_2.0.6/lpgexe/lpg-linux_x86
