jfli - a Java Foreign Language Interface for Common Lisp

Copyright (c) Rich Hickey. All rights reserved.

The use and distribution terms for this software are covered by the Common Public License 1.0, which can be found in the file CPL.TXT at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.

Contents

• Introduction
• Download
• Setup and Configuration
• Quick Start
• API Reference
  • JVM Creation and Initialization
  • Wrapper Generation
  • Object Creation
  • Arrays
  • Proxies - Java calling back to Lisp
  • Utilities
• Summary

Introduction

My objective was to provide comprehensive, safe, dynamic and Lisp-y access to Java and Java libraries as if they were Lisp libraries, for use in Lisp programs, i.e. with an emphasis on working in Lisp rather than in Java.

The approach I took was to embed a JVM instance in the Lisp process using JNI. I was able to do this using LispWorks' own FLI and no C (or Java! *) code, which is a tribute to the LW FLI. On top of the JNI layer (essentially a wrapper around the entire JNI API), I built this user-level API using Java Reflection. This first version was built with, and contains code specific to, Xanalys LispWorks.

jfli ("jay fly") provides:

• Automatic function generation for constructors, fields and methods, either by named class, or entire package (sub)trees given a jar file.
• Java -> Lisp package and name mapping with an eye towards lack of surprise, lack of conflict, and useful editor completion.
• setf-able setter generation for fields as well as for methods that follow the JavaBeans property protocol.
• Java array creation and aref-like access to Java arrays.
• A 'new' macro that allows for keyword-style field and property initialization.
• Typed references to Java objects with an inheritance hierarchy on the Lisp side mirroring that on the Java side - allowing for Lisp methods specialized on Java class and interface types.
• Implementation of arbitrary Java interfaces in Lisp, and callbacks from Java to Lisp via those interfaces. (* this required a single 5-line dummy Java proxy stub, provided with jfli)
• Automatic lifetime maintenance of Lisp-referenced Java objects, boxing/unboxing of primitive args/returns, string conversions, Java exception handling, overload resolution etc.

I built jfli using LWM and LWW (using Apple's and Sun's JVMs respectively), and it works fine on both. Should be a trivial port to other LispWorks, and a possible port to any Common Lisp with a robust FLI. Should also work with any JVM with a conformant JNI implementation.

Download

jfli is hosted on SourceForge

Setup and Configuration

jfli is supplied in 2 Lisp files and an optional Java .jar file. The first Lisp file, jni.lisp, defines a low-level API to the Java Native Interface, and is not documented here. The second, jfli.lisp, depends upon jni.lisp, and provides the user API documented here. Simply compile and load jni.lisp, then compile and load jfli.lisp. (use-package :jfli) and you are ready to use the API. Note that prior to creating the JVM you must tell the library how to find the Java JNI library by setting *jni-lib-path*.

If you wish to allow for callbacks from Java to Lisp, you must place jfli.jar in your classpath when creating the JVM.

Quick Start

This sample session presumes you have already compiled jni.lisp and jfli.lisp into fasl files.

CL-USER 4 > (load "/lisp/jni")
; Loading fasl file C:\lisp\jni.fsl
#P"C:/lisp/jni.fsl"

CL-USER 5 > (load "/lisp/jfli")
; Loading fasl file C:\lisp\jfli.fsl
#P"C:/lisp/jfli.fsl"

;The user API is entirely in the jfli package
CL-USER 6 > (use-package :jfli)
T

;tell the library where Java is located
CL-USER 7 > (setf *jni-lib-path* "/j2sdk1.4.2_01/jre/bin/client/jvm.dll")
"/j2sdk1.4.2_01/jre/bin/client/jvm.dll"

;this starts the VM - note how you can set the classpath
CL-USER 8 > (create-jvm "-Djava.class.path=/lisp/jfli.jar")
0
#<Pointer: JNI:PVM = #x081022A0>
#<Pointer: JNI:PENV = #x0086A858>

;define wrappers for the members of Object
CL-USER 9 > (def-java-class "java.lang.Object")
NIL

;and of Properties, a Hashtable-like class
CL-USER 10 > (def-java-class "java.util.Properties")
#<STANDARD-CLASS |java.util|:PROPERTIES. 2066B964>

;the above will create these packages if they do not already exist
;use the packages for easy name access

CL-USER 11 > (use-package "java.lang")
T

CL-USER 12 > (use-package "java.util")
T

;create a Properties instance, note keyword-style member inits, string conversion etc
;also note typed return value
CL-USER 13 > (setf p (new properties. :getproperty "fred" "ethel"))
#<PROPERTIES. 20664A94>

;virtual functions work as normal
CL-USER 14 > (object.tostring p)
"{fred=ethel}"

;setter was generated for member function because it follows the JavaBeans property protocol
CL-USER 15 > (setf (properties.getproperty p "ricky") "lucy")
"lucy"

CL-USER 16 > (object.tostring p)
"{ricky=lucy, fred=ethel}"

CL-USER 17 > (properties.size p)
2

;totally dynamic access, create wrappers as you need
CL-USER 18 > (def-java-class "java.lang.Class")
#<STANDARD-CLASS CLASS. 20680EC4>

CL-USER 19 > (class.getname (object.getclass p))
"java.util.Properties"

CL-USER 20 > (def-java-class "java.util.Enumeration")
#<STANDARD-CLASS ENUMERATION. 20669274>

;no need to wait for the vendor to enhance the language - you use Lisp!
CL-USER 21 > (defmacro doenum ((e enum) &body body)
               (let ((genum (gensym)))
                 `(let ((,genum ,enum))
                    (do ()
                        ((not (enumeration.hasmoreelements ,genum)))
                      (let ((,e (enumeration.nextelement ,genum)))
                        ,@body)))))
DOENUM

;can't do this in Java yet
CL-USER 22 > (doenum (prop (properties.elements p)) (print (object.tostring prop)))

"lucy"
"ethel"
NIL

;doc strings are created giving original Java signatures and indicating
overloads
CL-USER 23 > (documentation 'properties.getproperty 'function)
"java.lang.String getProperty(java.lang.String,java.lang.String)
java.lang.String getProperty(java.lang.String)
"

CL-USER 24 > (documentation 'properties.new 'function)
"java.util.Properties()
java.util.Properties(java.util.Properties)
"
	

API Reference

JVM Creation and Initialization

• *jni-lib-path*

  Set this to point to your jvm dll prior to calling create-jvm.

• Function (create-jvm &rest option-strings) -> unspecified

  Creates/starts the JVM. This can only be done once (a Java limitation). You must call this prior to calling any other jfli function. The option strings can be used to control the JVM, esp. the classpath:

  (create-jvm "-Djava.class.path=/Lisp/jfli.jar")

  See the JNI documentation for other initialization options.

• Function (enable-java-proxies) -> unspecified

  Sets up the Java->Lisp callback support. Must be called (once) before any calls to new-proxy, and requires jfli.jar be in the classpath.

Wrapper Generation

• Macro (def-java-class full-class-name) -> unspecified

  Given the package-qualified, case-correct name of a Java class as a string, will generate wrapper functions for its public constructors, fields and methods.

  The core API for generation interfaces to Java is the def-java-class macro. This macro will, at expansion time, use Java reflection to find all of the public constructors, fields and methods of the given class and generate functions to access them.

  The Generated API

  When you e.g. (def-java-class "java.lang.ClassName ") you get several symbols/functions:
  • A package named |java.lang| (note case)
    from which the following are exported:
  • A class-symbol: classname. (note the dot is part of the name)
    which can usually be used where a typename is required. It also serves as the name of the Lisp typed reference class.
  • Every non-interface class with a public constructor will get;
    • A constructor, (classname.new &rest args) -> typed-reference, which returns a typed reference to the newly created object
    • A method defined on make-new, ultimately calling classname.new, specialized on (the value of) the class-symbol
    Note that if the constructor is overloaded, there is just one function generated, which handles overload resolution. The function documentation string describes the constructor signature(s) from the Java perspective. The same argument conversions are performed as are for fields (see below).
  • All public fields will get a getter function:
    (classname.fieldname [instance]) -> field value
    and a setter:
    (setf classname.fieldname [instance])
    Instance field wrappers take a first arg which is the instance. Static fields get a symbol-macro *classname.fieldname*

    If the type of the field is primitive, the field value will be converted to a native Lisp value. If it is a Java String, it will be converted to a Lisp string. Otherwise, a generic reference to the Java object is returned. Similarly, when setting, Lisp values will be accepted for primitives, Lisp strings for Strings, or (generic or typed) references for reference types.

  • Every public method will get a wrapper function:
    (classname.methodname &rest args) -> return-value
    As with constructors, if a method is overloaded a single wrapper is created that handles overload resolution. If a method follows the JavaBeans property protocol (i.e. it is called getSomething or isSomething and there is a corresponding setSomething), then a (setf classname.methodname) will be defined that calls the latter.

    The same argument and return value conversions are performed as are for fields. The function documentation string describes the method signature(s) from the Java perspective.

  • A Lisp class with the class-symbol as its name. It will have as its superclasses other Lisp classes corresponding to the Java superclass/superinterfaces, some of which may be forward-referenced-classes. An instance of this class will be returned by classname.new/make-new/new, at which point the entire hierarchy will consist of finalized standard-classes.
  • Note that, due to the need to reference other Java types during the definition of a class wrapper, symbols, classes, and packages relating to those other types may also be created. In all cases they will be created with names and packages as described above.
• Function (get-jar-classnames jar-file-name &rest packages) -> list-of-strings

  Returns a list of class name strings. Packages should be strings of the form "java/lang " for recursive lookup and "java/util/" (note trailing slash) for non-recursive.

• Function (dump-wrapper-defs-to-file filename classnames) -> unspecified

  Given a list of classnames (say from get-jar-classnames), writes calls to def-java-class to a file:

  (dump-wrapper-defs-to-file "/lisp/java-lang.lisp"
    (get-jar-classnames "/j2sdk1.4.2_01/jre/lib/rt.jar " "java/lang/"))
  (compile-file "/lisp/java-lang")
  (load "/lisp/java-lang")
  (use-package "java.lang")
  ;Wrappers for all of java.lang are now available
  

Object Creation

• Generic Function (make-new class-symbol &rest args) -> typed-reference

  Allows for definition of before/after methods on constructors. Calls classname.new. The new macro expands into a call to this.

• Macro (new class-spec &rest args) -> typed-reference
  

  class-spec -> class-name | (class-name this-name)
  class-name -> "package.qualified.ClassName" | classname.
  args -> [actual-arg]* [init-arg-spec]*
  init-arg-spec -> init-arg | (init-arg)
  init-arg -> :settable-field-or-method [params]* value (note keyword)
  | .method-name [args]* (note leading dot)
  

  Creates a new instance of class-name, by expanding into a call to the make-new generic function, then initializes it by setting fields or accessors and/or calling member functions. If this-name is supplied, it will be bound to the newly-allocated object and available to the init-args:

  (new (button. this) shell *SWT.CENTER*   ;the actual args
         :gettext "Call Lisp"               ;a javabean property
         (.addlistener *swt.selection*      ;a method call
           (new-proxy (listener.
                       (handleevent (event)
                         (declare (ignore event))
                         (setf (button.gettext this)   ;this is bound to new instance
                               (format nil "~A ~A"
                                     (lisp-implementation-type)
                                     (lisp-implementation-version)))
                         nil))))
        .setsize 200 100                    ;can omit parens
        (.setlocation 40 40))
  			

  Expands into:

  (LET* ((#:G598 (MAKE-NEW BUTTON. SHELL *SWT.CENTER*)) (THIS #:G598))
    (SETF (BUTTON.GETTEXT #:G598) "Call Lisp")
    (BUTTON.ADDLISTENER #:G598
                        *SWT.SELECTION*
                        (NEW-PROXY (LISTENER.
                                    (HANDLEEVENT
                                     (EVENT)
                                     (DECLARE (IGNORE EVENT))
                                     (SETF (BUTTON.GETTEXT THIS)
                                           (FORMAT NIL
                                                   "~A ~A"
                                                   (LISP-IMPLEMENTATION-TYPE)
                                                   (LISP-IMPLEMENTATION-VERSION)))
                                     NIL))))
    (BUTTON.SETSIZE #:G598 200 100)
    (BUTTON.SETLOCATION #:G598 40 40)
    #:G598)
  			

Array Support

• Generic Function (make-new-array type &rest dimensions) -> reference to new array

  Generic function with methods defined for all Java class designators:

  • A "package.qualified.ClassName" string
  • (the value of) A class-symbol - classname.
  • A primitive designator keyword - :boolean|:byte|:char|:double|:float|:int|:long|:short

  Creates a Java array of the requested type with the requested dimensions.

• Function (jlength array) -> integer

  Like length, for Java arrays

• Function (jref array &rest subscripts) -> reference

  Like aref, for Java arrays of non-primitive (reference) types, settable.

• Function (jref-xxx array &rest subscripts) -> value

  Where xxx = boolean|byte|char|double|float|int|long|short. Like jref, for Java arrays of primitive types, settable.

Proxies - Java calling back to Lisp

Proxies allow the creation of Java objects that implement one or more interfaces in Lisp, and thus callbacks from Java to Lisp. You must call enable-java-proxies before using this proxy API. A significant limitation is that LispWorks appears to not support calls back into Lisp other than from threads initiated by Lisp, so you must ensure that the proxy will not be called from an arbitrary Java thread!

• Macro (new-proxy &rest interface-defs) -> reference

  interface-def -> (interface-name method-defs+)
  interface-name -> "package.qualified.ClassName" | classname. (must name a Java interface type)
  method-def -> (method-name arg-defs* body)
  arg-def -> arg-name | (arg-name arg-type) arg-type -> "package.qualified.ClassName " | classname. | :primitive
  method-name -> symbol | string (matched case-insensitively)

  Creates, registers and returns a Java object that implements the supplied interfaces

• Function (unregister-proxy proxy) -> unspecified

  Stops handling for the proxy (which must have been created by new-proxy) and removes references from the Lisp side. Make sure it is no longer referenced from Java first!

Utilities

• Function (jeq obj1 obj2) -> boolean

  Are the 2 java objects the same object? Note that this is not the same as Object.equals()

• Function (find-java-class class-sym-or-string) -> reference to Java Class object

  Given a Java class designator, returns the Java Class object. Use this in preference to Class.forName() when using jfli.

• Function (make-typed-ref java-ref) -> typed-reference

  Given a generic Java reference, determines the full type of the object and returns an instance of a typed reference wrapper. classname.new/make-new/new always return typed references, but since Java methods might return Object or some interface type, and we don't want to always incur the cost of type determination, field and method wrapper functions return generic references. Use this function to create a typed reference corresponding to the full actual type of the object when desired.

• Function (box-xxx value) -> reference to Java primitive wrapper class

  Where xxx = boolean|byte|char|double|float|int|long|short|string. Given a compatible Lisp value, creates an instance of the corresponding Java primitive wrapper class, e.g. Integer. This should rarely be needed, but can be used to force overloading resolution

• Function (unbox-xxx ref) -> Lisp value

  Where xxx = boolean|byte|char|double|float|int|long|short|string. Given an instance of a Java primitive wrapper class, creates an instance of the corresponding compatible Lisp value. This should rarely be needed, but can be used to unbox values returned by Java Object-based APIs.

Summary

I hope you find jfli useful. It is my sincere intent that it enhance the utility and interoperability of Common Lisp, a language with which I am still becoming familiar, and grow to appreciate more every day. I welcome comments and code contributions.

Rich Hickey, July 2004