1. Introduction

In embedded environments, the cost of the hardware is an important consideration. As a consequence, the memory is often very limited. The memory as well as the CPU time are critical resources which must be used with care and as efficiently as possible not only for response time and robustness purposes but also for hardware cost reduction purposes.

Several applications need to call shell commands to trigger various tasks that would be tough to make with languages like C. Hence, to make it, the C library provides the system() service which is passed as parameter the command line to run:

int system(const char *command);

The “command” parameter may be a simple executable name or a more complex shell command line using output redirections and pipes.

system() hides a call to “/bin/sh -c” to run the command line passed as parameter.

From Linux system point of view, in the simplest case, system() triggers at least two pairs of fork()/exec() system calls: one for “sh -c” and another for the command line itself as depicted in Figure 1.

Figure 1: system() internals

Moreover, fork() triggers a duplication of some resources (memory, file descriptors...) of the calling process (the father) to make the forked process (the child) inherit them. If the calling process is big from a memory occupation point of view or the overall memory occupation is high, the system() call may fail because of a lack of free memory. Even tough Linux benefited multiple enhancements like the Copy On Write (i.e. COW) to make the fork() more efficient and less cumbersome, this may lead to a memory over consumption which triggers Linux defense mechanisms like Out Of Memory (OOM) killer.

This paper aims at addressing the problem of system() overuse with some alternate solutions to enhance existing applications in a confident way that is to say with a minimal impact on the existing source code.

2. Use of vfork()

Linux inherited vfork() from BSD as an optimization of the application using fork() directly followed by a call to exec(). The idea was to avoid copying to much resources from the parent process when the child address space is immediately replaced by a new program. Compared to fork(), vfork() creates a new process without copying the page tables of the parent process. It is useful in performance-sensitive applications. As system() is merely a call to fork() immediately followed by a call to exec(), it is possible to rewrite it by replacing the call to fork() by vfork(). § A.1. shows a source code proposal for this solution. It is inspired by the original system() source code in the GLIBC library.

Such a solution makes system() slightly more efficient: it is exactly the same behaviour except that vfork() is supposed to be more efficient than fork(). As a consequence, we save memory demand and CPU time.

It is possible to go farther by using a lightweight server which runs the command as described in the following paragraph.

3. Use of local sockets and control messages

Through the fork/exec mechanism, Linux permits a child process inherit the file descriptors of its father. This is widely used to make a process use the standard input (stdin) and outputs (stdout and stderr) of its father as depicted on Figure 2.

The dup() system call is also a smart feature to duplicate open file descriptors inside a process. This is for example used to setup pipes between a father process and a child process: in the writer process, the input of the pipe is redirected to the standard outputs; in the reader process, the output of the pipe is redirected to the standard input of the process as depicted in Figure 3.

The sharing or copying of open file descriptors as described above implies that the processes are in the same hierarchy (father/child inheritance).

In the AF_UNIX socket family (i.e. Sockets for the inter-process communication on the same machine), Linux provides the ability to send ancillary data using sendmsg() and recvmsg(). The ancillary data typed with SCM_RIGHTS transfer open file descriptors from one process to another. On receipt of those data, the destination process gets copies of the sent file descriptors as if it implicitly called dup() system call. The processes do not need to be in the same hierarchy, they merely need to be connected through a local socket to pass the open file descriptors from one process to another. Of course, the passed files descriptors get the first empty slots in the table of the destination process as dup() system call would do. For example, in Figure 4, the originator process sends its standard input and standard outputs (i.e. 0, 1 and 2 file descriptors) to the destination process. The latter already has a standard input and outputs. So, this will trigger copies of the arriving file descriptors in the first available slots: 3, 4 and 5.

So, this mechanism can be used in a client/server application where the server runs the system() service for the command line sent by the client. Along with the command line, the file descriptors for stdin, stdout and stderr of the client are passed in the ancillary data to make the server redirect the input and outputs to them. § A.2. shows an example of source code for this solution and Figure 5 depicts the principle.

It is possible to go farther by eliminating the step which consists to run and terminate a shell (i.e. “sh -c”) in order to save more CPU time as described in the following paragraph.

4. Remanent shell

As some applications need to call system() very often, it means that “sh -c” is run very often. Moreover, the execution and termination of multiple shells by several concurrent applications sucks CPU time and memory resources. It is possible to plan a solution where a shell is executed once and stays ready to use in any application needing to run commands.

The idea consists to start one (or more ?) background shell(s) at application startup. We don’t use the “-c” option which runs one command line and then makes the shell exit. The shell must live in background during the application lifetime even after command execution. Each time the application needs to run a command, it submits it to the background shell. This saves the CPU time and memory needed to start and stop the shell. Figure 6 depicts the principle.

Figure 6: Background shell

Without “-c” option, the shell is interactive. In other words, it needs to be in front of a terminal. Linux provides the pseudo-terminal (i.e. PTY) concept to manage this kind of needs. The PTY is setup between the application process (master side) and the background shell process (slave side). The latter believes that it is interacting with an operator through a real terminal whereas the operator is actually the application process: cf. Figure 7.

Figure 7: Pseudo-terminal

As the shell is in interactive mode, it displays a prompt to wait for a command. It gets the command, executes it and displays a new prompt at the end of the command to wait for another one. At first sight, the application process would need to do some tricky work to parse the displays from the shell in order to discriminate the command display from the displayed prompt at the end of the command. Moreover, the application must also get the result of the command (i.e. the exit status). To make it simple, it is possible to use PDIP (i.e. Programmed Dialogs with Interactive Programs). This is an open source (https://sourceforge.net/projects/pdip/). The package is fully documented with online manuals, html pages (http://pdip.sourceforge.net/) and examples. It is an expect-like1 tool but much more simple to use than its ancestor. It provides the ability to pilot interactive programs. It comes in two flavors: a command named pdip which is used to control interactive programs from a shell script and an C language API offered by a shared library called libpdip.so to control interactive programs from a C/C++ language program. The latter is interesting to implement the current solution.

In the source tree, the isys sub-directory contains a variant of system() using the above principle (cf. isys.c embedded in a shared library called libisys.so). § A.3. presents some details about this library.

With libisys.so, the application process calls an API named isystem() which behaves the same as system() but actually it hides the PTY and the running background shell described above (cf. Figure 8).

The solution described in this chapter saves the fork()/exec() of “sh -c” by keeping at least one running background shell per application process. Depending on the application’s behaviour, it may be useful to keep at least a running shell. But it may be cumbersome from a memory point of view if the application calls to isystem() are rare. It is possible to enhance this implementation to reduce the number of running background shells by sharing them with all the running applications as proposed in the following chapter.

5. Remote shell

To go farther in the preceding implementation, we propose to share running shells with all the application processes. The principle consists to setup a daemon process managing one or more background shells (static configuration or dynamic setting on demand for example). Let’s call it rsystemd (i.e. rsystem daemon) to comply with Unix naming scheme. It is started before any application (at system startup for example) and waits for commands to run on a named socket. It submits the command to one of the shells that it manages and reports the result to the originating application processes. To make it, rsystemd relies on libpdip.so to interact with the shells as explained in § 4. on application process side, an API named rsystem() behaves the same as system() but actually it hides the interaction with systemd through the named socket: the command line passed as argument is written into the socket to make rsystemd run it and return the displays and the command status. The principle is depicted in Figure 9.

Figure 9: rsystemd

In the source tree of the PDIP package, the rsys sub-directory contains a variant of system() using the above principle (cf. rsystem.c embedded in a shared library called librsys.so which implements rsystem() API and rsystemd.c which implements the daemon part). § A.4. presents some details about this library.

This proposal not only saves CPU time as we do not continuously fork()/exec() and terminate shell processes but it also saves memory space as the running shells are shared with several processes.

By the way, we must not forget that this solution differs from original system() service from a user interface point of view as the shells are running in separate processes which are not children of the application processes: they are childs of rsystemd. As a consequence, the father to child inheritance mechanism does not operate here (file descriptors, environment variables, signal disposition…). But most of the time it is not required by the users of system().

Another point, if rsystemd is designed with a fixed number of running background shells we may face some starvation problems as the shell command requests may not be satisfied immediately if their number is bigger than the running background shells. So, this introduces some possible latency. Moreover we may also face some deadlocks if there are dependencies between shell commands: a command waits for the setting of some resource by another command which can’t get an available background shell. But if rsystemd is designed to launch brand new background shells to satisfy pending command requests when all its configured running background shells are busy, the latter problems won’t occur.