Standard Operating Procedures for Embedded Linux Systems

Follow these procedures for the smoothest path to great embedded Linux.
Ten Downsizing Methods

The organization of the downsizing issue is displayed in the bottom of Figure 1. We divide the methods into two parts, because the software platform of an embedded system typically consists of a kernel and root filesystem. The first part is how to get a small kernel, and the second part is how to downsize each component in the root filesystem, including libraries and shells. The second part also discusses how to compress the whole root filesystem. We describe all methods in detail below, along with experimental results. After explaining all methods, we show the effect of these methods on our laboratory embedded system, called the Wall system. Table 1 presents the specification for the Wall system. The system is a network security gateway that provides application-layer content filters, such as antispam and antivirus.

Table 1. Specification of Wall

KernelX86, MMU, QoS, Ethernet, WirelessLinux 2.6.6; 1,302,362 Bytes
ConnectionLAN, DMZ, WAN, DHCP, DNS relay, Dynamic DNS, Link load balance, Bridge modeppp-2.4.1, rp-pppoe-3.5
SecurityIPSec, PPTP, L2TP, SSL-VPNfreeswan2.06, 12tpd-0.69
FirewallNAT, firewall, UPNP, traffic profiling, APP firewalliptables-1.2.9, hotplug, iproute2
MailAntispam, antivirus, POP3 proxyp3scan
WebTransparent proxy, URL, URL keyword, content keywordp3scan
IMMSN logDevelopment based on L7Filter
BW ControlTCTC
ManagementWeb, SSL, FTP, log rotationthttpd-2.21b, Openssl-0.9.7d, putre-ftpd-1.0.17a, cron
Platformi386, IXP (simple version) 
Methods for the Linux Kernel

Selecting an appropriate kernel is the first step in downsizing the kernel. If you choose an inappropriate kernel, the system may be not only large but also unable to use processor power effectively. For example, a standard Linux kernel on a hardware platform without MMU cannot work normally. Such a hardware platform requires a specific MMU-less kernel, such as uClinux. Most people use the standard Linux kernel and attempt to trim its size.

The next step is to include only the necessary modules in the standard Linux kernel by a correct configuration. In fact, the default configuration of a Linux kernel includes many unused modules, which causes you to have a big kernel. Figure 3(a) shows the experimental results on the downsizing effect of the correct configuration. In this case, a system supporting TCP/IP has a kernel image that is only 59.84% of the size of a system supporting all network protocols.

Figure 3. Effects of the Downsizing Methods on the Kernel

To downsize the kernel, the third step is to use the optimization parameters when compiling the kernel. Using parameters -O1, -O2 or -O3 can improve performance, and using -Os can reduce size. However, optimizing for both performance and size simultaneously is not possible. Therefore, we generally select -O2 to achieve a balance between size and performance. As shown in Figure 3(b), the -Os parameter reduces the size of the kernel image by 22.82% as compared with -O3, but it causes worse performance.

Besides including only the necessary modules and compiling the kernel with the optimal parameters, to downsize the kernel further, you can decrease the size of the static buffer and array allocated in the kernel, because the kernel typically declares a large buffer and array for standard PCs. To find out which buffer or array occupies large memory space, you can use the command nm. This command can list the allocated size of each variable in an object file. With that information, you can browse the corresponding source code of the object file and alter the initial size of the buffer or array. Another approach for shrinking the buffer size is to modify the options in the menuconfig of the kernel to decrease the maximum number of supported peripherals, as shown in Figure 3(c) and (d).

Methods for the Root Filesystem

As shown in Figure 1, we identify six methods for downsizing the root filesystem. First, you can adopt a tool called BusyBox, which provides a fairly complete environment for any small or embedded system. BusyBox combines tiny versions of many common UNIX utilities into a single small executable file, and it is highly modular, allowing commands to be included or excluded at compile time. The space used for BusyBox is 7.04% of that of the original tool, as demonstrated in Figure 4.

Figure 4. Effects of Downsizing Methods on the Root Filesystem

Next, we introduce three methods for removing unused libraries or downsizing required libraries. First, you can use the command ldd to identify the required shared libraries for each program, and then with this information, you can remove the unused libraries. Notably, if a shared library is not used by programs, you additionally should check whether it is used by other shared libraries. Figure 4 shows that removing redundant libraries reduces the root filesystem to 6.55% of its original size. Second, you can replace the standard C library with a small C library, such as uClibc, Newlib or diet libc. Such libraries remove the unused functions, so their size is smaller than Glibc, as shown in Table 2. This table presents the differences in functionality between the four libraries. Third, you can use a library optimizer tool named Libopt to rebuild the libraries that include the only necessary functions for the executable programs and shared libraries found in the root filesystem. This tool utilizes objdump and nm to gather information about library object files, shared libraries and executable programs.

Table 2. Comparison between Different C Libraries

 GNU C LibraryuClibcdiet libcNewlib
MMU-less supportingNoYesYesYes
Setting menuconfigonly make./configure
NoteStandard C libraryNeeds cross-compiler toolchainOften linked as static libraryManaged by Red Hat

The fifth method for downsizing the root filesystem is to remove unnecessary documents. You can eliminate some directories, such as /home, /mnt, /opt, /root, /boot and /proc, if unused. You also can remove the man, info, include and example directories to reduce the size when additionally integrating a package into the root filesystem. In general, an embedded system executes only specific programs, so users can operate it easily without the help documents or examples in these directories.

The final method is to avoid uncompressing the whole root filesystem into SDRAM. The root filesystem is compressed to save the stored space, for example, Flash RAM. However, after the filesystem is uncompressed into SDRAM, the Flash memory allocated for the filesystem is no longer necessary. For instance, if the compressed size of the root filesystem is 4MB and its compression rate is 50%, the system occupies 4MB of Flash memory and 8MB of SDRAM. Therefore, the system wastes much memory storage, because of the duplicate data. For this problem, you can use CRAMFS. CRAMFS is a read-only filesystem, designed for simplicity and space efficiency. You do not need to uncompress a CRAMFS image before mounting it. A CRAMFS image is zlib-compressed, one page at a time to enable random read access. The metadata is not compressed, but is expressed in a terse representation that is more space-efficient than in traditional filesystems, such as ext2 or FAT. However, due to the read-only property of compressed files, random write access is hard to implement for them. As shown in Figure 4, CRAMFS compresses the filesystem to 12.77% of its original size.

Now that we've covered the six methods, let's move on to the effect of these methods on the Wall Project, as shown in Figure 5. First, we used BusyBox to substitute for the multiple utility programs used in the original shell. Then, we compiled all the required packages with the parameters --strip-unneeded and -O2. Next, we used the commands strip and objcopy to remove the unnecessary contents of packages. Finally, we deleted unnecessary directories, such as man, info and example. Figure 5(a) illustrates the result of these processes. However, the size of Wall was still 139MB. Hence, we had to view the contents of /usr indepth, as shown in Figure 5(b) and (c). In the Wall Project, removing unneeded documents and files saved 20.6MB of space. About 15.9MB of space then can be saved by eliminating unused libraries. However, as you can see, Perl occupied much space in our system. Other methods may exist to solve this problem, but it is sufficient to consider only what we have done above.

We found that the optimization of package size is also useful for downsizing when integrating a new package into the root filesystem. Actually, most programs and libraries are compiled at optimizing level 2 by default (gcc options -g and -O2) and are compiled for a specific CPU. On Intel platforms, software is compiled for i386 processors by default. To minimize the package size, you should not adopt the -g option, which adds the debug info in the execution files. Additionally, remember to use -strip and --strip-all to remove all symbols. In more-advanced methods, we used the command readelf to check for any redundant sections in the execution files, and we used objcopy to remove those redundant sections. However, this approach may be not efficient for small programs.

Figure 5. Downsizing Results on the Root Filesystem of the Wall Project


Geek Guide
The DevOps Toolbox

Tools and Technologies for Scale and Reliability
by Linux Journal Editor Bill Childers

Get your free copy today

Sponsored by IBM

Upcoming Webinar
8 Signs You're Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
11am CDT, April 29th
Moderated by Linux Journal Contributor Mike Diehl

Sign up now

Sponsored by Skybot