Friday, March 9, 2012

Getting Started with JNA

JNA (Java Native Access) is an awesome library similar to Python's ctypes and .NET's P/Invoke. It allows you to access native C libraries without writing any JNI code (actually JNA will write the JNI code for you). The only drawback is that it can't access C++ code without writing any C glue-code. In this blog, I'm going to show you how to get started with JNA, including how to access C++ code. Let's start with a simple example to access C stdlib.
import com.sun.jna.Library;
import com.sun.jna.Memory;
import com.sun.jna.Native;
import com.sun.jna.Platform;
import com.sun.jna.PointerType;

public interface CLibrary extends Library {
    CLibrary INSTANCE = (CLibrary) Native.loadLibrary(
        (Platform.isWindows() ? "msvcrt" : "c"),
        CLibrary.class);

    void printf(String format, Object... args);
    
    void system(String command);
    
    FILE fopen(String path, String mode);
    int fputs(String content, FILE fp);
    String fgets(Memory memory, int size, FILE fp);
    int fclose(FILE fp);
    
    public class FILE extends PointerType {
    }
}
import com.sun.jna.Memory;

import test.CLibrary.FILE;

public class Main {

    public static void main(String[] args) {
        CLibrary.INSTANCE.printf("Hello JNA\n");

        CLibrary.INSTANCE.system("ls -lrt");
        
        writeFile("test.txt");
        readFile("test.txt");
    }
    
    private static void writeFile(String path) {
        System.out.println("Writing " + path);
        FILE fp = CLibrary.INSTANCE.fopen(path, "w");
        if (fp != null) {
            CLibrary.INSTANCE.fputs("Hello World", fp);
            CLibrary.INSTANCE.fclose(fp);
        }
    }
    
    private static void readFile(String path) {
        System.out.println("Reading " + path);
        FILE fp = CLibrary.INSTANCE.fopen(path, "r");
        if (fp != null) {
            int size = 256;
            Memory buffer = new Memory(size);
            CLibrary.INSTANCE.fgets(buffer, size, fp);
            System.out.println(buffer.getString(0));
            CLibrary.INSTANCE.fclose(fp);
        }
    }
}
One important thing to note here is that fgets takes char* as a buffer. The JNA equivalent of this is com.sun.jna.Memory.

Now let's create a simple C++ application.

#ifndef HELLO_H_
#define HELLO_H_

#include <iostream>
#include <string>
using namespace std;

struct HelloRequest {
    string name;
    string message;
};

struct HelloResponse {
    string message;
};

class Hello {
public:
    void sayHello(const string& name) const;
    struct HelloResponse hello(const struct HelloRequest& request);
};

#endif /* HELLO_H_ */
#include "Hello.h"

void Hello::sayHello(const string& name) const {
    cout << "Hello, " << name << endl;
}

struct HelloResponse Hello::hello(const struct HelloRequest& req) {
    HelloResponse res;
    res.message = req.message + ", " + req.name;

    return res;
}
To access C++ libraries from JNA, we need to write C glue-code like this.
#ifndef CHELLO_H_
#define CHELLO_H_

extern "C" {

struct CHelloRequest {
    const char* name;
    const char* message;
};

struct CHelloResponse {
    const char* message;
};

void sayHello(const char* name);
struct CHelloResponse hello(const struct CHelloRequest& request);

}

#endif /* CHELLO_H_ */
#include "CHello.h"
#include "Hello.h"

void sayHello(const char* name) {
    Hello h;
    h.sayHello(string(name));
}

struct CHelloResponse hello(const struct CHelloRequest& request) {
    HelloRequest req;
    req.name = string(request.name);
    req.message = string(request.message);

    Hello h;
    HelloResponse res = h.hello(req);
    CHelloResponse response;
    response.message = res.message.c_str();

    return response;
}
Now compile our C/C++ source code and create a shared library. Let's call it libhello.so.

Create a new Java class to access hello library.

import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Structure;

public interface HelloLibrary extends Library {
    HelloLibrary INSTANCE = (HelloLibrary) Native.loadLibrary("hello",
        HelloLibrary.class);

    void sayHello(String name);
    
    CHelloResponse.ByValue hello(CHelloRequest request);
    
    public class CHelloRequest extends Structure {
        public String name;
        public String message;
    }
    
    public class CHelloResponse extends Structure {
        public static class ByValue extends CHelloResponse implements Structure.ByValue {
            public String message;
        }
    }
}
Take a note that hello() function returns CHelloResponse struct by value, therefore we need to implement Structure.ByValue.

Now modify our Main class to access hello library and don't forget to set LD_LIBRARY_PATH (for Linux) and PATH (for Windows) to point to the directory where the hello library is located.

import test.HelloLibrary.CHelloRequest;
import test.HelloLibrary.CHelloResponse;

public class Main {

    public static void main(String[] args) {
        HelloLibrary.INSTANCE.sayHello("Foo");
        
        CHelloRequest req = new CHelloRequest();
        req.name = "Bar";
        req.message = "Bye";
        CHelloResponse.ByValue res = HelloLibrary.INSTANCE.hello(req);
        System.out.println(res.message);
    }
}

Thursday, March 1, 2012

How to Create a Debian Package

In my previous blog, I've described how to create an RPM package. This time, I'm going to show you how to create a Debian package. I'm going to use the same code that's used in RPM tutorial blog. Creating a Debian package can be pretty complicated if we have to do it manually. Luckily there are some helper scripts that can make our lives easier.
  1. Install devscripts and debhelper
  2. sudo apt-get install devscripts debhelper
  3. Create a simple hello.cpp
  4. #include <iostream>
    using namespace std;
    
    int main() {
        cout << "Hello World" << endl;
        return 0;
    }
    
  5. Create a Makefile
  6. CC = g++ 
    CCFLAGS = -g -Wall
    SRC = hello.cpp
    TARGET = hello
    
    .PHONY: clean install
    
    all: hello
    
    install: $(TARGET)
        if [ -d ${DESTDIR} ]; then rm -rf ${DESTDIR}; fi
        mkdir -p ${DESTDIR}
        cp $(TARGET) ${DESTDIR} 
    
    hello: 
        $(CC) $(CCFLAGS) -o $(TARGET) $(SRC)
    
    clean:
        rm -rf $(TARGET)
    
  7. In the directory where we created the above files, create a new debian directory
  8. mkdir debian
  9. Create a debian/changelog file
  10. dch --create -v 0.1 --package hello
    We can create this changelog file without using dch script, but the changelog file requires a particular format and the dch script can help to create a changelog template. The changelog file should look like this.
    hello (0.1) UNRELEASED; urgency=low
    
      * Initial release. (Closes: #123)
    
     -- foo <foo@bar.com>  Thu, 01 Mar 2012 14:16:42 +0800
    
  11. Create a debian/compat
  12. The content of this file is very simple. It just contains a number 8. This file is used to tell debhelper to use compatibility helper version 8. The file should look like this.
    8
  13. Create a debian/control file
  14. This control file is used to describe the source and binary package and other information. This control file also has a particular format. Take a note that there's a space after the Description field. This to tell that it's an extended description. The description and extended description should be different; otherwise lintian (a tool to check the correctness of files rquired to create Debian packages) will complain. The control file should look like this.
    Source: hello
    Maintainer: foo <foo@bar.com>
    Section: misc
    Priority: optional
    Standards-Version: 3.9.1
    Build-Depends: debhelper (>= 8)
    
    Package: hello
    Architecture: any
    Depends: ${shlibs:Depends}, ${misc:Depends}
    Description: say hello
     say hello world
    
  15. Create a debian/rules file
  16. This file is basically a Makefile.
    #!/usr/bin/make -f
    %:
        dh $@
    
    override_dh_auto_install:
        $(MAKE) DESTDIR=$$(pwd)/debian/hello/usr/bin install
    
    The override_dh_auto_install is required because in our example, we have a custom make install target.

  17. Create a debian/copyright file
  18. For this tutorial, just create an empty copyright file.

  19. To summarize, we should've already created all these files
  20. /home/foo/testdeb/hello/hello.cpp
                           /Makefile
                           /debian/control
                           /copyright
                           /changelog
                           /compat
                           /rules
    
  21. Create a Debian package
  22. debuild -uc -us
    -uc means don't sign the .changes file
    -us means don't sign the source package
    For more information about the options, read man dpkg-buildpackage.

    You should now see your source and Debian packages in /home/foo/testdeb

  23. Install the newly created Debian package
  24. sudo dpkg -i hello_0.1_i386.deb
  25. Test if hello program was successfully installed
  26. hello
    You should see "Hello World" output

  27. Remove the hello package
  28. sudo dpkg -r hello

Tuesday, February 28, 2012

How to Create an RPM Package

In this blog, I'm going to teach you how to create an RPM package.
  1. Create a simple hello.cpp
  2. #include <iostream>
    using namespace std;
    
    int main() {
        cout << "Hello World" << endl;
        return 0;
    }
    
  3. Create a Makefile
  4. CC = g++ 
    CCFLAGS = -g -Wall
    SRC = hello.cpp
    TARGET = hello
    
    .PHONY: clean install
    
    all: hello
    
    install: $(TARGET)
        if [ -d ${DESTDIR} ]; then rm -rf ${DESTDIR}; fi
        mkdir -p ${DESTDIR}
        cp $(TARGET) ${DESTDIR} 
    
    hello: 
        $(CC) $(CCFLAGS) -o $(TARGET) $(SRC)
    
    clean:
        rm -rf $(TARGET)
    
  5. Put hello.cpp and the Makefile in a directory structure like this
  6. /home/foobar/hello/Makefile
                      /hello.cpp
    
  7. Create a tarball of hello project
  8. cd /home/foobar
    tar czvf hello-0.1.tar.gz hello/
    
  9. Create an rpmbuild directory
  10. mkdir -p /home/foobar/rpmbuild
    
  11. Create BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS directories inside the /home/foobar/rpmbuild directory
  12. BUILD --> the source files are unpacked and compiled here
    BUILDROOT --> files are installed here
    RPMS --> binary RPMs are created here
    SOURCES --> source tarballs
    SPECS --> spec files
    SRPMS --> source RPMs

  13. Create a spec file in the /hom/foobar/rpmbuild/SPECS/hello.spec
  14. Summary             : Hello World program
    Name                : hello
    Version             : 0.1 
    Release             : 1%{?dist}
    License             : GPLv3+
    Source              : %{name}-%{version}.tar.gz
    
    %description
    A simple Hello World program
    
    %prep
    %setup -q -n %{name}
    
    %build
    make
    
    %install
    make install DESTDIR="%{buildroot}%{_bindir}"
    
    %files
    %{_bindir}/hello
    
    %prep --> this stage calls %setup to unpack hello-0.1.tar.gz to /home/foobar/rpmbuild/BUILD/hello. %{name} is a macro variable that substitutes to hello

    %build --> this stage calls "make" in the /home/foobar/rpmbuild/BUILD/hello to create a hello executable

    %install --> this stage calls "make install" to install the files into /home/foobar/rpmbuild/BUILDROOT/usr/bin. %{buildroot} is a macro variable that substitutes to /home/foobar/rpmbuild/BUILDROOT. Make sure the Makefile supports DESTDIR argument

    %files --> lists all the files to be installed. %{_bindir} is a macro variablesubstitutes to /usr/bin. In this example, only hello executable needs to be installed into /usr/bin

  15. Build the source and binary RPMs
  16. rpmbuild -ba /home/foobar/SPECS/hello.spec
    -ba option means build all (prep, build, install)
    The binary RPM will be created in /home/foobar/rpmbuild/RPMS
    The source RPM will be created in /home/foobar/rpmbuild/SRPMS

  17. Install the newly created binary RPM
  18. sudo rpm -ivh hello-0.1-1.fc16.i686.rpm
  19. Test if hello program was successfully installed
  20. hello
    You should see "Hello World" output

  21. Remove the hello package
  22. sudo rpm -e hello

Wednesday, February 22, 2012

Building Boost with Visual C++ 2010 Express

Following my last blog about Building Boost with MinGW, here I'm going to show you step-by-step how to build Boost with Visual C++ 2010 Express. The latest Boost as of this writing is 1.48.0, so I'm going to use that version.

  1. Download Visual C++ 2010 Express from Microsoft website and install it.
  2. Download Windows SDK. This is optional, but it can be pretty useful if you plan to call any Windows API.
  3. Download Boost from Boost website.
  4. Extract Boost to any location of you preference.
  5. Modify the C:\Program Files\Microsoft Visual Studio 10.0\VC\bin\vcvars32.bat as shown below. You need to edit this file as an Administrator. Visual C++ 2010 Express seems to have a bug for not setting all these variables automatically. You need to uncomment the first 3 lines. Setting WindowsSdkDir is optional if you skipped step 2.
    :: @call :GetVSCommonToolsDir
    :: @if "%VS100COMNTOOLS%"=="" goto error_no_VS100COMNTOOLSDIR
    :: @call "%VS100COMNTOOLS%VCVarsQueryRegistry.bat" 32bit No64bit
    
    @SET WindowsSdkDir=C:\Program Files\Microsoft SDKs\Windows\v7.1\
    @SET VSINSTALLDIR=C:\Program Files\Microsoft Visual Studio 10.0\
    @SET VCINSTALLDIR=C:\Program Files\Microsoft Visual Studio 10.0\VC\
    @SET VS100COMNTOOLS=%VSINSTALLDIR%\Common7\IDE\
    @SET PATH=%PATH%;%VCINSTALLDIR%\bin;%VS100COMNTOOLS%
    @SET FrameworkDir32=C:\Windows\Microsoft.NET\Framework
    @SET FrameworkVersion32=v4.0.30319
    @SET Framework35Version=v3.5
    
    
  6. Open Visual Studio Command Prompt (2010) and go to your boost installation directory. 
  7. Build Boost build tool (b2).
    [boost_dir]\bootstrap.bat
    
    
  8. Build Boost (you can start making your coffee or do whatever you like since this process is gonna take a while). All the libraries will be created in the stage\lib folder.
  9. [boost_dir]\b2.exe toolset=msvc --build-type complete stage
  10. Open your Visual C++ 2010 Express and create a new Win32 Console project.
    
    
  11. Right click on the project --> Properties--> Configuration Properties --> C/C++ --> General and add [boost_dir] in the Additional Include Directories.
  12. In the same Properties window Configuration Properties --> Linker -->  General, add [boost_dir]\stage\lib in the Additional Library Directories.
  13. Let's now test our Boost libraries.
  14. #include "stdafx.h"
    #include <string>
    #include <iostream>
    #include <boost/shared_ptr.hpp>
    #include <boost/regex.hpp>
    using namespace std;
    
    int _tmain(int argc, _TCHAR* argv[])
    {
        boost::shared_ptr<string> ptr(new string("Hello World, Boost"));
        boost::smatch matches;
        boost::regex pattern("Hello World,\\s*(.*)");
        boost::regex_match(*ptr, matches, pattern);
        cout << matches[1] << endl;
    
        system("pause");
    
     return 0;
    }
    

Wednesday, February 8, 2012

Understanding serialVersionUID

If we ever wonder why we need to declare a serialversionUID when we implement java.util.Serializable, this article can provide a good idea why not declaring a serialVersionUID can cause a problem.
import java.io.Serializable;

public class Hello implements Serializable {
//    private static final long serialVersionUID = 1L;
    private String message;

//    public String getTheSameMessage() {
//        return message;
//    }
    
    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    @Override
    public String toString() {
        StringBuilder builder = new StringBuilder();
        builder.append("Hello [message=").append(message).append("]");
        return builder.toString();
    }
}
package org.fredy;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

public class Main {
    public static void main(String[] args) throws Exception {
        Hello h = new Hello();
        h.setMessage("Hello World");
        
        ObjectOutputStream oos = null;
        try {
            oos = new ObjectOutputStream(
                new FileOutputStream(new File("out.obj")));
            oos.writeObject(h);
        } finally{
            if (oos != null) {
                oos.close();
            }
        }
        
//        ObjectInputStream ois = null;
//        try {
//            ois = new ObjectInputStream(
//                new FileInputStream(new File("out.obj")));
//            Hello h1 = (Hello) ois.readObject();
//            System.out.println(h1.getMessage());
//        }
//        finally {
//            if (ois != null) {
//                ois.close();
//            }
//        }
    }
}
Now execute the Main class. This will create a file named out.obj.

Let's now uncomment some lines in Hello class and comment out some lines in Main class as depicted below.

import java.io.Serializable;

public class Hello implements Serializable {
//    private static final long serialVersionUID = 1L;
    private String message;

    public String getTheSameMessage() {
        return message;
    }
    
    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

    @Override
    public String toString() {
        StringBuilder builder = new StringBuilder();
        builder.append("Hello [message=").append(message).append("]");
        return builder.toString();
    }
}
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;

public class Main {
    public static void main(String[] args) throws Exception {
//        Hello h = new Hello();
//        h.setMessage("Hello World");
//        
//        ObjectOutputStream oos = null;
//        try {
//            oos = new ObjectOutputStream(
//                new FileOutputStream(new File("out.obj")));
//            oos.writeObject(h);
//        } finally{
//            if (oos != null) {
//                oos.close();
//            }
//        }
        
        ObjectInputStream ois = null;
        try {
            ois = new ObjectInputStream(
                new FileInputStream(new File("out.obj")));
            Hello h1 = (Hello) ois.readObject();
            System.out.println(h1.getMessage());
        }
        finally {
            if (ois != null) {
                ois.close();
            }
        }
    }
}
Execute the Main class again. A java.io.InvalidClassException will be thrown because the Hello class that was serialized is no longer compatible with updated Hello class. To fix this problem, we need to declare serialVersionUID in the Hello class.

Wednesday, February 1, 2012

Getting Started with OrientDB

OrientDB is a NoSQL document database (http://code.google.com/p/orient/). Here I'm gonna show you how to get started with OrientDB.
apply plugin: 'java'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
    maven { url 'http://www.orientechnologies.com/listing/m2' }
}

dependencies {
    compile 'com.orientechnologies:orient-commons:1.0rc7'
    compile 'com.orientechnologies:orientdb-core:1.0rc7'
}
The sample code below shows some basic create, retrieve, update, delete operations
package test;

import java.io.IOException;
import java.io.Serializable;
import java.util.List;

import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
import com.orientechnologies.orient.core.db.object.ODatabaseObjectTx;
import com.orientechnologies.orient.core.metadata.schema.OClass;
import com.orientechnologies.orient.core.metadata.schema.OType;
import com.orientechnologies.orient.core.record.impl.ODocument;
import com.orientechnologies.orient.core.serialization.OBase64Utils;
import com.orientechnologies.orient.core.sql.query.OSQLSynchQuery;

public class Main {
    static class Message implements Serializable {
        private static final long serialVersionUID = 1L;
        private String msg;
        
        public Message(String msg) {
            this.msg = msg;
        }
        
        public String getMessage() {
            return msg;
        }
    }
    
    public static void main(String[] args) throws Exception {
        ODatabaseDocumentTx db = new ODatabaseDocumentTx("local:testdb");
        if (!db.exists()) {
            db.create();
            
            // creating a schema is optional, but can be useful
            OClass personClass = db.getMetadata().getSchema().createClass("person");
            personClass.createProperty("id", OType.INTEGER);
            personClass.createProperty("name", OType.STRING);
            personClass.createProperty("msg", OType.STRING);
            db.getMetadata().getSchema().save(); // don't forget to save it
            
            // creating some records
            for (int i = 0; i < 10; i++) {
                ODocument doc = new ODocument(db, "person");
                doc.field("id", i + 1);
                doc.field("name", "foo" + i);
                Message msg = new Message("Hello" + i);
                doc.field("msg", OBase64Utils.encodeObject(msg));
                doc.save();
            }
        }
        else {
            // default username/password
            db.open("admin", "admin");
        }
        
        try {
            // query all records
            for (ODocument doc : db.browseClass("person")) {
                printPersonRecord(doc);
            }

            // query some records
            List<ODocument> results = db.query(new OSQLSynchQuery<ODocument>(
                "select * from person where name like 'foo%' order by id desc"));
            for (ODocument doc : results) {
                printPersonRecord(doc);
            }

            // create a new record
            ODocument doc = new ODocument(db, "person");
            doc.field("id", "11");
            doc.field("name", "foo11");
            Message msg = new Message("Hello" + 11);
            doc.field("msg", OBase64Utils.encodeObject(msg));
            doc.save();
            // verify it
            printPersonRecord(doc);
            // update it
            doc.field("name", "bar");
            doc.save();
            // verify it again
            printPersonRecord(doc);
            // delete it
            doc.delete();
            // count the number of records
            System.out.println(db.countClass("person"));
        }
        finally {
            db.close();
        }
    }
    
    private static void printPersonRecord(ODocument doc) 
        throws ClassNotFoundException, IOException {
        Message msg = (Message) OBase64Utils.decodeToObject(
            doc.field("msg").toString());
        System.out.println("id=" + doc.field("id") + ";" + 
            "name=" + doc.field("name") + ";" +
            "message=" + msg.getMessage());
    }
}
Creating a schema is useful to indicate the type of a field. If we omitted the creation of the schema, the query "order by id desc" would be sorted based on string rather than integer since by default all the fields are considered as a string in OrientDB.

OrientDB has a pretty useful utility to serialize/deserialize Java object and encode/decode it using base64 as shown in the example above.

If we look at the content of the testdb, there are two files created ls person.och and person.0.ocl for the "person" class. There is a limit of the number of classes/clusters that OrientDB can have. The default max number of classes/clusters is 32,768 (thanks Luca Garulli for the correction!).

Tuesday, January 3, 2012

Open Source C++ Command Line Argument/Option Parser

I've created a new open source project cpp-argparser for command line argument/option parser in C++.