Thursday, September 22, 2011

How to Implement a Shutdown Method in SocketServer.TCPServer for Python 2.5

Prior to Python 2.6, the SocketServer.TCPServer doesn't have a shutdown method. The SocketServer.TCPServer.serve_forever() also blocks. Here is a TCPServer implementation that works in Python 2.5, implements a shutdown method, and support non-blocking request. The implementation here is based on the TCPServer implementation in Python 2.6.
class Py25TCPServer(SocketServer.ThreadingTCPServer):
   def __init__(self, address_tuple, handler):
      SocketServer.ThreadingTCPServer.__init__(self, address_tuple, handler)
      self.__is_shut_down = threading.Event()
      self.__shutdown_request = False

   def serve_forever(self, poll_interval=0.5):
      """Handle one request at a time until shutdown.

         Polls for shutdown every poll_interval seconds. Ignores
         self.timeout. If you need to do periodic tasks, do them in
         another thread.
      """
      self.__is_shut_down.clear()
      try:
         while not self.__shutdown_request:
            r, w, e = select.select([self], [], [], poll_interval)
            if self in r:
               self._handle_request_noblock()
      finally:
         self.__shutdown_request = False
         self.__is_shut_down.set()

   def shutdown(self):
      """Stops the serve_forever loop.

         Blocks until the loop has finished. This must be called while
         serve_forever() is running in another thread, or it will
         deadlock.
      """
      self.__shutdown_request = True
      self.__is_shut_down.wait()

   def _handle_request_noblock(self):
      """Handle one request, without blocking.

         I assume that select.select has returned that the socket is
         readable before this function was called, so there should be
         no risk of blocking in get_request().
      """
      try:
         request, client_address = self.get_request()
      except socket.error:
         return
      if self.verify_request(request, client_address):
         try:
            self.process_request(request, client_address)
         except:
            self.handle_error(request, client_address)
            self.close_request(request)

To make our program work in both Python 2.5 and 2.6+, we can do something like this.
import sys

python_version = sys.version_info
server = None
if python_version[1] < 6:
    server = Py25TCPServer((self.host, self.port), MyHandler)
else:
    server = SocketServer.TCPServer((self.host, self.port), MyHandler)
server.server_forever()

Thursday, July 28, 2011

Gotcha in C++ Function Overloading

Suppose we have this code.
#include <iostream>
using namespace std;

class Base {
public:
    virtual void doSomething(double a) {
        cout << "This is double from Base: " << a << endl;
    }
    virtual void doSomething(int a) {
        cout << "This is int from Base: " << a << endl;
    }
    virtual ~Base() {}
};

class Derived : public Base {
public:
    virtual void doSomething(double a) {
        cout << "This is double from Derived: " << a << endl;
    }
    virtual ~Derived() {}
};

int main() {
    Derived d;

    int i = 2;
    double j = 0.1;
    d.doSomething(i);
    d.doSomething(j);

    return 0;
}
We would expect the output to be:
This is int from Base: 2
This is double from Derived: 0.1

Strangely, the output is:
This is double from Derived: 2
This is double from Derived: 0.1

The way overload resolution works in C++ is that the compiler will first search for function doSomething() inside the Derived class. If it finds it, it will stop there; otherwise it will look the one in the Base class for a matching function. To solve this problem, we need to add using Base::doSomething in order let it participate in the overload resolution.
#include <iostream>
using namespace std;

class Base {
public:
    virtual void doSomething(double a) {
        cout << "This is double from Base: " << a << endl;
    }
    virtual void doSomething(int a) {
        cout << "This is int from Base: " << a << endl;
    }
    virtual ~Base() {}
};

class Derived : public Base {
public:
    virtual void doSomething(double a) {
        cout << "This is double from Derived: " << a << endl;
    }
    using Base::doSomething; 
    virtual ~Derived() {}
};

int main() {
    Derived d;

    int i = 2;
    double j = 0.1;
    d.doSomething(i);
    d.doSomething(j);

    return 0;
}

The output is:
This is int from Base: 2
This is double from Derived: 0.1

Wednesday, July 27, 2011

Basic SVN Commands

Here I'm gonna show you some basic SVN commands with some examples.

Checking out
svn co http://[hostname]/trunk

Checking in
svn ci

Viewing SVN information
This will display information, such SVN URL, revision number, etc.
svn info

Viewing SVN status
This will display information, such as files modified, added, deleted, etc.
svn status

Updating the local SVN repository
svn up

Viewing the diff
svn diff

Viewing SVN logs
svn log

Adding a file into SVN
svn add new_file.txt

Moving/Renaming a file from one place to another place
svn move old_file.txt new_file.txt

Deleting a file from SVN
svn del junk_file.txt

Reverting local changes in a file
svn revert whatever_file.txt

Reverting local changes recursively
svn revert -R *

Creating a branch/tag
svn copy http://[hostname]/trunk http://[hostname]/branches/1.0
svn copy http://[hostname]/trunk http://[hostname]/tags/1.0

De-commiting changes in the trunk
This will first do a diff between HEAD and rev 123 in the trunk and then do a dry-run before performing an actual merge.
svn diff -r HEAD:1234 http:/[hostname]/trunk
svn merge -r --dry-run HEAD:1234 http:/[hostname]/trunk
svn merge -r HEAD:1234 http:/[hostname]/trunk
svn ci

Merging changes from the branch to the trunk
Assume that the current directory is where the trunk code was checked out.
svn diff -r 1234:HEAD http://[hostname]branches/1.0
svn merge -r 1234:HEAD --dry-run http://[hostname]/branches/1.0
svn merge -r 1234:HEAD http://[hostname]/branches/1.0
svn ci

Merging changes from the trunk to the branch
Assume that the current directory is where the branch code was checked out.
svn diff -r 1234:HEAD http://[hostname]/trunk
svn merge -r 1234:HEAD --dry-run http://[hostname]/trunk
svn merge -r 1234:HEAD http://[hostname]/trunk
svn ci

Viewing SVN externals
svn propset svn:externals .

Editing SVN externals
svn propedit svn:externals .

Viewing who modified the file
svn blame whosefault.txt

Wednesday, June 22, 2011

How to Convert C FILE* to C++ iostream

When we deal with a lot of C libraries in our C++ code, there may be cases where we need to convert the C FILE* into C++ iostream. Boost iostreams can help to deal with such problems easily. Here's an example how to do it.

#include <iostream>
#include <cstdio>
#include <unistd.h>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>

void write() {
    FILE* fp = fopen("whatever.txt", "w");
    if (fp == NULL) {
        perror("fopen error");
    }
    int fd = fileno(fp);
    boost::iostreams::stream_buffer<boost::iostreams::file_descriptor_sink> bis(fd);
    std::ostream os(&bis);
    os << "Hello World!" << std::endl;

    fclose(fp);
}

void read() {
    FILE* fp = fopen("whatever.txt", "r");
    if (fp == NULL) {
        perror("fopen error");
    }
    int fd = fileno(fp);
    boost::iostreams::stream_buffer<boost::iostreams::file_descriptor_source> bis(fd);
    std::istream is(&bis);
    while (is) {
        std::string line;
        std::getline(is, line);
        std::cout << line << std::endl;
    }
    fclose(fp);
}

int main() {
    write();
    read();

    return 0;
}

Tuesday, June 14, 2011

Creating a Custom Builder in SCons

Suppose we need to create a custom builder in SCons. This custom builder will read a text file and append an author name at the end of the file.

import os

def my_builder(target, source, env):
    for i in range(0, len(source)):
       txt = open(source[i].abspath).read()
       txt = txt + os.linesep + "Author: " + env['AUTHOR']
       open(target[i].abspath, "wb").write(txt)

env = Environment()
env['BUILDERS']['MyBuilder'] = Builder(action = my_builder)

env.MyBuilder(['output1.txt', 'output2.txt'], ['input1.txt', 'input2.txt'], 
              AUTHOR = "Foo")

The custom builder allows an argument to be passed in into our custom builder. How we pass the argument is by using the Python's keyword argument, e.g. AUTHOR = "Foo". The custom builder function can then read this value from the env, e.g. env["AUTHOR"].

How to Flatten a List In Python

Here is a simple algorithm to flatten a list in Python. The algorithm also works with the list that have nested lists.

def flatten1(l, newlist):
    if isinstance(l, list):
        for i in l: flatten1(i, newlist)
    else: newlist.append(l)

def flatten2(l):
    if isinstance(l, list):
        newlist = []
        for i in l: newlist = newlist + flatten2(i)
        return newlist
    else: return [l]
    
def flatten3(l):
    if not l: return l
    if not isinstance(l, list): return [l]
    return flatten3(l[0]) + flatten3(l[1:])
    
if __name__ == '__main__':
    l = [1, [2, 3], 4, [[5]], [], [[]], [6, 7, [8]]]
    newlist = []
    flatten1(l, newlist)
    print newlist
    
    print flatten2(l)
    
    print flatten3(l)

The output:
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8]

The first algorithm is a ugly since it requires a newlist parameter to be passed in. The second one is a bit better but it still requires a loop. The third one doesn't require any loop and is the shortest one.

Wednesday, June 1, 2011

Getting Started with SWIG

Creating JNI by hand can be pretty challenging especially if we intend to wrap a large C++ code base. In such scenario, SWIG can dramatically reduce the effort of creating the JNI wrapper. There is difference between using javah and swig tools. Swig is useful for creating JNI from C++ while javah is useful for creating JNI from Java.

Hello.h
#ifndef HELLO_H_
#define HELLO_H_

#include <string>
#include <stdint.h>

class Hello {
public:
    Hello();
    virtual ~Hello();
    void sayHello(const std::string& name) const;
    uint32_t getNumber() const;
    const char* getMessage() const;
};

#endif /* HELLO_H_ */

Hello.cpp
#include <iostream>
#include "Hello.h"
using namespace std;

Hello::Hello() {
}

Hello::~Hello() {
}

void Hello::sayHello(const std::string& name) const {
    cout << "Hello, " + name << endl;
}

uint32_t Hello::getNumber() const {
    return 10;
}

const char* Hello::getMessage() const {
    return "Hello";
}

SWIG requires an interface file.
Hello.i
%module JniHello
%{
#include "Hello.h"
typedef unsigned int uint32_t;
%}

%include "std_string.i"
%include "Hello.h"

typedef unsigned int uint32_t;
As we can see here, there is %module verbatim that indicates the module name. Everything in the %{..%} block will be copied to the resulting wrapper file created by SWIG. Here we use include std_string.i library, which will convert std::string to java.lang.String. Without including it, only char* will be converted to java.lang.String and std::string will be a pointer. We can also define a typedef to tell that uint32_t is basically an unsigned int. As a matter of fact, there is stdint.i library that can help to deal with stdint type mapping, but for this tutorial, let's keep it as it is.

Now let's compile the C++ programs and create shared libraries.
swig -c++ -java Hello.i

g++ -c -fpic Hello.cpp
g++ -shared Hello.o -o libhello.so

g++ -c -fpic Hello_wrap.cxx -I$JAVA_HOME/include -I$JAVA_HOME/include/linux
g++ -shared Hello_wrap.o -o libjnihello.so libhello.so

Test it out with a simple Java program. Make sure to add -Djava.library.path or set LD_LIBRARY_PATH to point to the directory where libhello.so and libjnihello.so are located.
Main.java
public class Main {
    static {
        try {
            System.loadLibrary("jnihello");
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }

    public static void main(String[] args) {
        Hello hello = new Hello();
        hello.sayHello("Foo");
        System.out.println(hello.getMessage());
        System.out.println(hello.getNumber());
    }
}

The output is:
Hello, Foo
Hello
10