How to Convert C FILE* to C++ iostream

When we deal with a lot of C libraries in our C++ code, there may be cases where we need to convert the C FILE* into C++ iostream. Boost iostreams can help to deal with such problems easily. Here's an example how to do it.

#include <iostream>
#include <cstdio>
#include <unistd.h>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>

void write() {
    FILE* fp = fopen("whatever.txt", "w");
    if (fp == NULL) {
        perror("fopen error");
    }
    int fd = fileno(fp);
    boost::iostreams::stream_buffer<boost::iostreams::file_descriptor_sink> bis(fd);
    std::ostream os(&bis);
    os << "Hello World!" << std::endl;

    fclose(fp);
}

void read() {
    FILE* fp = fopen("whatever.txt", "r");
    if (fp == NULL) {
        perror("fopen error");
    }
    int fd = fileno(fp);
    boost::iostreams::stream_buffer<boost::iostreams::file_descriptor_source> bis(fd);
    std::istream is(&bis);
    while (is) {
        std::string line;
        std::getline(is, line);
        std::cout << line << std::endl;
    }
    fclose(fp);
}

int main() {
    write();
    read();

    return 0;
}

Creating a Custom Builder in SCons

Suppose we need to create a custom builder in SCons. This custom builder will read a text file and append an author name at the end of the file.

import os

def my_builder(target, source, env):
    for i in range(0, len(source)):
       txt = open(source[i].abspath).read()
       txt = txt + os.linesep + "Author: " + env['AUTHOR']
       open(target[i].abspath, "wb").write(txt)

env = Environment()
env['BUILDERS']['MyBuilder'] = Builder(action = my_builder)

env.MyBuilder(['output1.txt', 'output2.txt'], ['input1.txt', 'input2.txt'], 
              AUTHOR = "Foo")

The custom builder allows an argument to be passed in into our custom builder. How we pass the argument is by using the Python's keyword argument, e.g. AUTHOR = "Foo". The custom builder function can then read this value from the env, e.g. env["AUTHOR"].

How to Flatten a List In Python

Here is a simple algorithm to flatten a list in Python. The algorithm also works with the list that have nested lists.

def flatten1(l, newlist):
    if isinstance(l, list):
        for i in l: flatten1(i, newlist)
    else: newlist.append(l)

def flatten2(l):
    if isinstance(l, list):
        newlist = []
        for i in l: newlist = newlist + flatten2(i)
        return newlist
    else: return [l]
    
def flatten3(l):
    if not l: return l
    if not isinstance(l, list): return [l]
    return flatten3(l[0]) + flatten3(l[1:])
    
if __name__ == '__main__':
    l = [1, [2, 3], 4, [[5]], [], [[]], [6, 7, [8]]]
    newlist = []
    flatten1(l, newlist)
    print newlist
    
    print flatten2(l)
    
    print flatten3(l)

The output:

[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8]
[1, 2, 3, 4, 5, 6, 7, 8]

The first algorithm is a ugly since it requires a newlist parameter to be passed in. The second one is a bit better but it still requires a loop. The third one doesn't require any loop and is the shortest one.

Getting Started with SWIG

Creating JNI by hand can be pretty challenging especially if we intend to wrap a large C++ code base. In such scenario, SWIG can dramatically reduce the effort of creating the JNI wrapper. There is difference between using javah and swig tools. Swig is useful for creating JNI from C++ while javah is useful for creating JNI from Java.

Hello.h

#ifndef HELLO_H_
#define HELLO_H_

#include <string>
#include <stdint.h>

class Hello {
public:
    Hello();
    virtual ~Hello();
    void sayHello(const std::string& name) const;
    uint32_t getNumber() const;
    const char* getMessage() const;
};

#endif /* HELLO_H_ */

Hello.cpp

#include <iostream>
#include "Hello.h"
using namespace std;

Hello::Hello() {
}

Hello::~Hello() {
}

void Hello::sayHello(const std::string& name) const {
    cout << "Hello, " + name << endl;
}

uint32_t Hello::getNumber() const {
    return 10;
}

const char* Hello::getMessage() const {
    return "Hello";
}

SWIG requires an interface file.

Hello.i

%module JniHello
%{
#include "Hello.h"
typedef unsigned int uint32_t;
%}

%include "std_string.i"
%include "Hello.h"

typedef unsigned int uint32_t;

As we can see here, there is %module verbatim that indicates the module name. Everything in the %{..%} block will be copied to the resulting wrapper file created by SWIG. Here we use include std_string.i library, which will convert std::string to java.lang.String. Without including it, only char* will be converted to java.lang.String and std::string will be a pointer. We can also define a typedef to tell that uint32_t is basically an unsigned int. As a matter of fact, there is stdint.i library that can help to deal with stdint type mapping, but for this tutorial, let's keep it as it is.

Now let's compile the C++ programs and create shared libraries.

swig -c++ -java Hello.i

g++ -c -fpic Hello.cpp
g++ -shared Hello.o -o libhello.so

g++ -c -fpic Hello_wrap.cxx -I$JAVA_HOME/include -I$JAVA_HOME/include/linux
g++ -shared Hello_wrap.o -o libjnihello.so libhello.so

Test it out with a simple Java program. Make sure to add -Djava.library.path or set LD_LIBRARY_PATH to point to the directory where libhello.so and libjnihello.so are located.

Main.java

public class Main {
    static {
        try {
            System.loadLibrary("jnihello");
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }

    public static void main(String[] args) {
        Hello hello = new Hello();
        hello.sayHello("Foo");
        System.out.println(hello.getMessage());
        System.out.println(hello.getNumber());
    }
}

The output is:

Hello, Foo
Hello
10

Creating Non-Blocking Client and Server with Java NIO

With Java NIO, creating non-blocking client and server is pretty straightforward. Below is a sample code how to do that.

HelloServer.java

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.nio.channels.WritableByteChannel;

public class HelloServer {
    private static final String HELLO_REPLY = "Hello World!";
    
    public static void main(String[] args) {
        ByteBuffer buffer = ByteBuffer.wrap(HELLO_REPLY.getBytes());
        ServerSocketChannel ssc = null;
        try {
            ssc = ServerSocketChannel.open();
            ssc.socket().bind(new InetSocketAddress(8888));
            ssc.configureBlocking(false);
            
            while (true) {
                SocketChannel sc = ssc.accept();
                // if sc == null, that means there is no connection yet
                // do something else
                if (sc == null) {
                    // pretend to do something useful here
                    System.out.println("Doing something useful....");
                    try {
                        Thread.sleep(3000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                } else { // received an incoming connection
                    System.out.println("Received an incoming connection from " +
                        sc.socket().getRemoteSocketAddress());
                    printRequest(sc);
                    buffer.rewind();
                    sc.write(buffer);
                    sc.close();
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (ssc != null) {
                try {
                    ssc.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
    
    private static void printRequest(SocketChannel sc) throws IOException {
        ReadableByteChannel rbc = Channels.newChannel(
            sc.socket().getInputStream());
        WritableByteChannel wbc = Channels.newChannel(System.out);
        ByteBuffer b = ByteBuffer.allocate(8); // read 8 bytes 
        while (rbc.read(b) != -1) {
            b.flip();
            while (b.hasRemaining()) {
                wbc.write(b);
            }
            b.clear();
        }
    }
}

HelloClient.java

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;

public class HelloClient {
    public static final String HELLO_REQUEST = "Hello!";
    public static void main(String[] args) {
        SocketChannel sc = null;
        try {
            sc = SocketChannel.open();
            sc.configureBlocking(false);
            // make sure to call sc.connect() or else 
            // calling sc.finishConnect() will throw 
            // java.nio.channels.NoConnectionPendingException
            sc.connect(new InetSocketAddress(8888));
            // if the socket has connected, sc.finishConnect() should 
            // return false
            while (!sc.finishConnect()) {
                // pretend to do something useful here
                System.out.println("Doing something useful...");
                try {
                    Thread.sleep(3000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
            System.out.println("Sending a request to HelloServer");
            ByteBuffer buffer = ByteBuffer.wrap(HELLO_REQUEST.getBytes());
            sc.write(buffer);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (sc != null) {
                try {
                    sc.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

YouTube Downloader

I've created an open source GUI front-end of the famous youtube-dl script.

youtube-dl-gui

Getting Started with SCons to Build C/C++ Applications

SCons is a wonderful cross-platform build tool, mostly used for building C/C++ applications similar to make. It's based on Python and Python is an excellent programming language. Since Python is a fully-fledged programming language, there are many things that can be done easily in SCons as opposed to writing a shell script. In this blog, I'm going to show some basic stuff to get started with SCons.

SCons requires a file normally called SConstruct. The file structure in the below example looks like this.

scons-project
  |--- src/Hello.cpp
  |        Main.cpp
  |--- include/Hello.h
  |--- SConstruct

Hello.h

#ifndef HELLO_H_
#define HELLO_H_

class Hello {
public:
    Hello();
    virtual ~Hello();
    void sayHello() const;
};

#endif /* HELLO_H_ */

Hello.cpp

#include "Hello.h"
#include <iostream>
using namespace std;

Hello::Hello() {
}

Hello::~Hello() {
}

void Hello::sayHello() const {
    cout << "Hello!" << endl;
}

Main.cpp

#include "Hello.h"
#include <iostream>
using namespace std;

int main() {
    Hello hello;
    hello.sayHello();

#ifdef DEBUG
    cout << "Debugging..." << endl;
#endif
}

This is the simplest SConstruct file.

SConstruct

import os

env = Environment(CCFLAGS="-g -O0 -DDEBUG", CPPPATH="./include")

env.Program("bin/Main", ["src/Hello.cpp", "src/Main.cpp"])

This will produce an executable called Main. The CPPPATH here is to tell where the header files are located. And the CCFLAGS are the compiler flags used by the GCC compiler that tells the compiler to add the debug statement and turn off the optimization.

If we need to control how the object files are created, e.g. we want to put the object files in a directory called obj. We can use the Object node in our SConstruct. The SConstruct will look like this.

import os

env = Environment(CCFLAGS="-g -O0 -DDEBUG", CPPPATH="./include")

hello_obj = env.Object("obj/Hello", "src/Hello.cpp")
main_obj = env.Object("obj/Main", "src/Main.cpp")
env.Program("bin/Main", main_obj + hello_obj, LIBPATH="lib", LIBS="hello")

If we need to build a static library, i.e. libhello.a that can be linked against Main, we can modify the SConstruct to look this.

import os

env = Environment(CCFLAGS="-g -O0 -DDEBUG", CPPPATH="./include")

env.StaticLibrary("lib/hello", ["src/Hello.cpp"])
env.Program("bin/Main", ["src/Main.cpp"], LIBPATH="lib", LIBS="hello")

To build a shared library, i.e. libhello.so, we just need to change from StaticLibrary to SharedLibrary. The SConstruct will look like this.

import os

env = Environment(CCFLAGS="-g -O0 -DDEBUG", CPPPATH="./include")

env.SharedLibrary("lib/hello", ["src/Hello.cpp"])
env.Program("bin/Main", ["src/Main.cpp"], LIBPATH="lib", LIBS="hello")