So, what am I doing with OpenSolaris?

I’ve written more than once in the past weeks about my messing with OpenSolaris, but I haven’t explained very well why I’m doing that, and what exactly is that I’m doing.

So the first thing I have to say is that since I started getting involved in lscube I focused on getting the buildsystem in shape so that it could be more easily built, especially with out-of-tree builds, which is what I usually do since I might have to try the build with multiple compilers (say hi to the Intel C Compiler and Sun Studio Express). But since then, I only tested it under Linux, which is quite a limitation.

While FreeBSD is reducing tremendously the gap it had against GNU/Linux (here it’s in full, since I intend Linux and glibc together), OpenSolaris has quite a few differences from it, which makes it an ideal candidate to check for possible GNUisms creeping into the codebase. Having the Sun Studio compiler available too makes it also much simpler to test with non-GCC compilers.

Since the OpenSolaris package manager sucks, I installed Gentoo Prefix, and moved all the tools I needed, included GCC and binutils, in that. This made it much easier to deal with installing the needed libraries and tools for the projects, although some needed some tweaking too. Unfortunately there seems to be a bug with GNU ld from binutils, but I’ll have to check if it’s present also in the default binutils version or if it’s just Gentoo patching something wrong.

While using OpenSolaris I think I launched quite a few nasty Etruscan curses toward some Sun developers for some debatable choices, the first being, as I’ve already extensively written about, the package manager. But there has been quite a few other issues with libraries and include files, and the compiler itself.

Since feng requires FFmpeg to build, I’ve also spent quite a lot of time trying to get FFmpeg to build on OpenSolaris, first with GCC, then with Sun Studio, then again with GCC and a workaround with PIC: the bug I noted above with binutils is that GNU ld doesn’t seem to be able to create a shared object out of object not compiled with PIC, so it requires -fPIC to be forced on for them to build, otherwise the undefined symbols for some functions, like htonl() become absolute symbols with value 0 which cause obvious linking errors.

Since I’ve been using OpenSolaris from a VirtualBox virtual machine (which is quite slow even though I’m using it without Gnome, using SSH to have a login, and jumping inside the Gentoo prefix installation right away), I ended up trying to first build FFmpeg with the Sun Studio compiler taken from Donnie’s overlay under Linux, with Yamato building with 16 parallel processes. The problem here is that the Sun Studio compiler is quite a moving target, to the point that a Sun employee, Roman Shaposhnik , suggested me on ffmpeg-devel to try Sun Studio Express (which is, after all, what OpenSolaris has too), that should be more similar to GCC than the old Sun Studio 10 was. This is why dev-lang/sunstudioexpress is in portage, if you didn’t guess it earlier.

Unfortunately even with the latest version of Sun Studio compiler, building FFmpeg has been quite some trouble. I ended up fighting quite a bit with the configure script and not limited to that, but luckily, now most of the patches I have written have been sent to ffmpeg-devel (and some of them accepted, others I’ll have to rewrite or improve depending on what the FFmpeg developers think about them). The amount of work needed just to get one dependency to work is probably draining up the advantage I had in using Gentoo Prefix for those dependencies that work out of the box with OpenSolaris.

(I’ll probably write about the FFmpeg changes more extensively as they deserve a blog entry on their own , and actually the drafts for the blog entries I have to write starts to pile up just as much as the entries in my TODO list.)

While using OpenSolaris I also started understanding why many people hate Solaris this much; a lot of command spit out errors that don’t let you know at all what the real problem is (for instance if I try to mount a nfs filesystem with a simple mount nfs://yamato.local/var/portage /imports/portage, I get an error telling me that the nfs path is invalid; on the other hand, the actual error here is that I need to add -o vers=2 to request NFSv2 (why it doesn’t seem to work with v3 is something I didn’t want to investigate just yet). Also, the OpenSolaris version I’m using, albeit it’s described as “Developer Edition”, lacks a lot of man pages for the library functions (although I admit that most of those which are present are very clear).

In addition to the porting I’ve written about, I’ve also taken the time to extend the testsuite of my ruby-elf, so that Solaris ELF files are better supported; it is interesting to note that the elf.h file from OpenSolaris contain quite more definitions about that, I haven’t yet looked at the man pages to see if Sun provide any description about the Sun-specific sections, for which I’d also like to add further parsers classes. It has been interesting since neither the GNU nor the Sun linkers set the ELF ABI property to values different from SysV (even though both Linux and Sun/Solaris have an ABI value defined in the standard), and GNU and Sun Sections have some overlapping values (like the sections used for symbol versioning: glibc and Solaris have different ways to handle those, but the section type ID used for both is the same; the only way to discern between the two is the section name).

At the end, to resolve the problem, I modified ruby-elf not to load the sections at once, but just on request, so that by the time most sections are loaded, the string table containing the sections’ names is available. This allows to know the name of the section, and thus discern the extended sections by name rather than ABI. Regression tests have been added so that the sections are loaded properly for different elf types too. Unfortunately I haven’t been able to produce a static executable on Solaris with neither the Sun Studio compiler nor GCC, so the only tests for the Solaris ELF executables are for dynamic executables. Nonetheless, the testsuite for ruby-elf (which is the only part of it to take up space: out of 3.0MB of space occupied by ruby-elf, 2.8MB are for the tests) reached 72 different tests and 500 assertions!