man-pages-3.20 is released

I've uploaded man-pages-3.20 into the release directory (or view the online pages). Notable changes in man-pages-3.20 are:

• The addition of cpuid(4) and msr(4) device man pages (thanks to Andi Kleen)
• Numerous fixes to other pages.

Te whare karakia i Onuku, Ōtautahi

man-pages-3.19 is released

I've uploaded man-pages-3.19 into the release directory (or view the online pages). The most notable change in man-pages-3.19 is the addition of four pages describing the POSIX interval timers API: timer_create(2), timer_settime(2), timer_getoverrun(2), and timer_delete(2).

man-pages-3.18 is released

I've uploaded man-pages-3.18 (my one hundredth release as maintainer!) into the release directory (or view the online pages). Notable changes in man-pages-3.18 are:

• 4 new pages describing character sets: armscii-8(7), iso_8859-11(7), iso_8859-4(7), and iso _8859-6(7). Thanks once again to Lefteris Dimitroulakis for contributing!


• A new pthread_kill(3) page documents pthread_kill().


• A new pthread_kill_other_threads_np(3) page documents pthread_kill_other_threads_np().


• A new pthread_sigmask(3) page documents pthread_sigmask().


• The clock_getres(2) page was moved from Section 3 to Section 2 (since the interfaces it describes really are system calls, not library functions), and adds a description of CLOCK_MONOTONIC_RAW, new in 2.6.28.


• Some bugs were fixed in select_tut(2) and many changes and tidy-ups were made to the text.

linux.conf.au 2010 in Wellington

For it's 11th time out, linux.conf.au returns to New Zealand again in 2010. (YAY!!) Congratulations to the successful bidders from Wellington! The conference will run from Monday 18 January to Saturday 23 January. The organizers' website is http://www.penguinsvisiting.org.nz/.

Speaking at LCA 2009

Late notice (but that's all I got)... I'll be speaking at LCA 2009 in Hobart, Australia, on 22 Jan, considering what happens when kernel and userland don't talk.

Update: my presentation slides can be found here.

man-pages-3.17 is released

I've uploaded man-pages-3.17 into the release directory (or view the online pages). Notable changes in man-pages-3.17 are:

• A new endian(3) page documents functions added in glibc 2.9 that convert between host byte order and little- and big-endian order: htobe16(), htole16(), be16toh(), le16toh(), htobe32(), htole32(), be32toh(), le32toh(), htobe64(), htole64(), be64toh(), and le64toh().


• A new getifaddrs(3) page documents the getifaddrs() and freeifaddrs() library functions.


• 8 new pages describe various character sets: cp1251(7), iso-8859-10(7), iso_8859-13(7), iso_8859-14(7), iso_8859-3(7), iso_8859-5(7), iso_8859-8(7), and koi8-u(7). Thanks to Lefteris Dimitroulakis for contributing these!


• And various smaller changes in many other pages.

Tasmanian Devil

man-pages-3.16 is released

I've uploaded man-pages-3.16 into the release directory (or view the online pages). Notable changes in man-pages-3.16 are:

• A new pthread_getcpuclockid(3) page documents pthread_getcpuclockid().
• A new libc(7) page provides an overview of the standard C library implementations on Linux.
• A new rtld-audit(7) page provides an overview of the dynamic linker auditing API (modeled on the Solaris API): the functions la_version(), la_objsearch(), la_activity(), la_objopen(), la_objclose(), la_preinit(), la_symbind(), la_pltenter(), and la_pltexit().
  
• The ld.so(8) page adds documentation of the LD_AUDIT and LD_POINTER_GUARD environment variables.
  

See you at LCA 2009

It's too much fun to miss, so I finally made the booking... I'm going to LCA 2009 (19-24 Jan), in Hobart, Australia!

man-pages-3.15 is released

I've uploaded man-pages-3.15 into the release directory (or view the online pages). Notable changes in man-pages-3.15 are:

• A new makedev(3) page documents the makedev(), major(), and minor() macros used to manipulate device IDs.
• A new pthread_cleanup_push_defer_np(3) page documents pthread_cleanup_push_defer_np() and pthread_cleanup_pop_restore_np().
• The accept(2) page adds documentation of the new accept4() system call (coming in Linux 2.6.28)
• The fmemopen(3) page adds a description of open_wmemstream(3).
• The tcp(7) page adds a description of the use of the MSG_TRUNC flag for retrieving data from a TCP socket.
  
• Many updates to the atexit(3) page.
• Updates to many other pages
  

Special thanks to Petr Baudis for a lot of patches that went into this release.

Caballito del diablo ("Devil's little horse")

Linux Foundation fellowship, 6 months in

Not quite 6 months since I started the Linux Foundation fellowship, it's time to analyze and reflect on what has (or hasn't) been accomplished.

Some statistics

I took over maintainership of man-pages at the start of November 2004, with the first release being man-pages-2.00. From then until the fellowship started in the middle of May this year (a period of 185 weeks), I probably spent between 0 and 2.5 days a week on man-pages, most of it done as private, volunteer work. (For a period of around a year, I probably managed up about half day a week as part of my day job; thanks Google!) I'd guess it was a bit better than day a week on average (let's say 1.25 days), and we could roughly estimate that as the equivalent of 45 working weeks.

Since the fellowship started, I've worked for about 25 weeks on man-pages; that is, somewhat more than half of the estimated time that I spent on man-pages in the preceding 3.5 years. The first release during my tenure of the fellowship was man-pages-2.80, and since then there have been 15 more (man-pages-3.00 through man-pages-3.14).

What I'm expecting is that the limiting factor in the progress of man-pages is the availability of my time. If I get to work at around four times the rate I did before, then we should see a corresponding increase in the progress of man-pages. Very roughly, in the last 6 months, progress should have been somewhat more than 50% of what it was in the previous 3.5 years.So here's a first comparison:

Period	Number of releases
Pre-fellowship	80
During fellowship	16

Well, that doesn't look so good. But there's no question that there's more work going on for each release nowadays. Here's another simple statistic, derived from the commit logs:

Period	Number of commits
Pre-fellowship	3610
During fellowship	1852

Commits in the last 6 months were nearly 50% of the total during the previous 3.5 years. That seems roughly in line with expectations, and supports the theory that there's a lot more work going into each man-pages release nowadays. Of course, commits vary a lot in size, ranging from a spelling fix, to a complete new page, and going through to some of the enormous global formatting fixes that took place in the man-pages-2.* series, so this is a very rough measure. (One of the commits cleaning up source files layout in man-pages-2.47 had a diff size of more than 60000 lines(!). There were many other large formatting commits in the man-pages-2.*, which is why trying to compare the volume of diffs before and during the fellowship doesn't produce a useful metric.)Another rough measure is how many man pages were added to the set over time:

Period	Number of new pages added
Pre-fellowship	93
During fellowship	56

Again, that's roughly in line with expectations, with the number of pages added during the fellowship being somewhat more than 50% of the previous period.

But where did the new pages come from?

Period	By mtk	By others	By mtk + other(s)	Imports
Pre-fellowship	52	22	13	6
During fellowship	50	4	1	1

"mtk" is me. "Other(s)" is someone else. "Imports" are pages under a free license that I scooped up from some other source (e.g., found on the net, in a distro, or in BSD).

On the negative side, I wrote the vast majority of new pages that have been added so far during the fellowship. On the positive side, Paul Jackson contributed the single biggest page, cpuset(7), which became the fourth largest page in man-pages. (Also worth noting: in the man-pages-2.* releases, a total of 28 pages were deleted, mainly obsolete pages in Section 1.) In fact, I had hoped to be able to get even more pages written, but other tasks, such as testing, API review, and kernel patches have also taken up a significant fraction of my time during the fellowship. When considered as a (calendar) monthly rate, contributions of new pages by others are, unfortunately, essentially unchanged since before the fellowship.

So, progress towards improving contributions by others, at least in terms of new pages, has not been good. However, my gut feeling has been that more people are actually contributing to man-pages than before: the fact that there is a full-time maintainer means people are rather more likely to send bug reports, suggestions, and patches for existing pages. Here's a statistic that bears it out:

Period	Average contributors/week
Pre-fellowship	2.8
During fellowship	5.9

This was calculated by summing the number of contributors in each of the change logs in all of the releases over the two periods and then dividing by the number of calendar weeks in each period (185 and 28 respectively). 5.9 contributors per week is still much lower than I'd like, but my feeling is that the rate has increased steadily over the time of the fellowship, so that the current rate is already higher than 5.9, and set to increase further. (Another factor that may also have helped boost the number of reports is that in December 2007 I started adding a COLOPHON to each man page describing how to report bugs, and this change would have filtered into distribution CDs a few months later.)

Timeliness of documentation

Things have defintely got better during the fellowship. Most additions and changes to the kernel-userland interface during the time of the fellowship have been documented in man-pages pretty much as they occur. (This contrasts with earlier times, where interface changes have sometimes been followed only months (or in extreme cases years) later by man page updates.) Most notably, Ulrich Drepper's new system calls in Linux 2.6.27 saw man pages go out a few days after the release of that kernel.

Testing and bug reporting

I've done a fair bit of this over the course of the fellowship. Most new system calls and system call extensions got tested by me before they hit mainline. This uncovered a few bugs which were then fixed. The biggest single piece of work here was for the utimensat(2) system call, producing a test suite (later integrated into LTP), along with patches that fixed the 5 or so bugs in the interface (details here).

Many existing glibc functions also got tested as I updated the man pages for them. Most notably, updates to the man pages produced about 35 bug reports related to error reporting by the math functions. The addition of man pages for various pthreads functions has also been accompanied by a lot of testing, and a half dozen or so bug reports.

API design review

Most new system calls and system call extensions got reviewed before going into mainline. (My record on other kernel interfaces, such as /proc files, was a more spotty though.) Among other things, this resulted in a redesign of the proposed extension of the accept() system call (originally proposed as paccept(), with a signal set argument whose necessity was dubious, later revised to accept4(), which should appear in kernel 2.6.28).

Miscellaneous

A summary of other man-pages work that I've done during the time of the fellowship:

• man-pages moved from a private Subversion repository to a public git repository on kernel.org.

• In general, I'm blogging a bit more actively nowadays, and in addition to posts summarizing releases, there have been longer posts on topics such as: problems with kernel-userland interface design and implementation; the state of error reporting for glibc's math functions; and a few articles describing or summarizing Linux kernel-userland interface changes.

• After my presentation at LPC for the kernel-userland interface track, I finally got round to an idea I'd been considering for a while: creating the linux-api mailing list. The rationale for the list is that all patches that cause API/ABI changes should be CCed to the list, so that the many parties who are interested in API/ABI changes (e.g., man-pages, LSB, libc developers, kernel developers, testers such as the folk at LTP, and of course userland developers) can get an idea of what's going on. Most people still don't read Documentation/SubmitChecklist, to know they should be using this list, so I try to regularly chase people to use it (and some others also help in that regard), and by now at least some people do so without prompting.

• I've helped out LSB on a number of occasions, filing a few bug reports against the spec, but also helping out by writing man pages for functions that they wanted to specify that were currently undocumented (see, e.g., here and here). This resulted in new man pages such as getprotoent_r(3), getservent_r(3), getnetent_r(3), and pthread_getattr_np().
  

• I continue to respond to many bug reports in the manpages and manpages-dev components of Debian's bug tracking system. This has mutual benefits: on the one hand, although I'm not actually a member of Debian, I'm by far the most active fixer of their bug reports; on the other hand, most Debian bug reports for man pages really apply to the upstream pages (I ignore the ones that don't), and so the reports provide a valuable source of pointers to things that need fixing in man-pages. A big thank you to Debian users, who produce far more (and more useful) man-pages bug reports than all of the other distributions put together!

• Working on man-pages led me to find various deficiencies in POSIX.1 specifications, resulting in around a half dozen bug reports to the Austin group.

Mariposa

Publish Post

man-pages-3.14 is released

I've uploaded man-pages-3.14 into the release directory (or view the online pages). Notable changes in man-pages-3.14 are:

• A new CPU_SET(3) page documents the CPU_* macros used for manipulating CPU sets (the cpu_set_t data structure used by sched_setaffinity(2), pthread_attr_setaffinity_np(3), and pthread_setaffinity_np(3)).
• More man pages for POSIX threads library functions: 5 new pages documenting 8 functions
• pthread_attr_setinheritsched(3) (includes documentation of pthread_attr_getinheritsched())
• pthread_cancel(3)
• pthread_cleanup_push(3) (includes documentation of pthread_cleanup_pop())
• pthread_setcancelstate(3) (includes documentation of pthread_setcanceltype())
• pthread_testcancel(3)
  
• clone(2) adds documentation of the CLONE_NEWNET, CLONE_NEWUTS, CLONE_NEWPID, and CLONE_IPC flags.
• mmap(2) adds documentation of the MAP_STACK flag.
• arp(7) adds documentation of a few preously undocumented /proc files.
• icmp(7) adds documentation of a few previously undocumented /proc files.
• tcp(7) adds documentation of many previously undocumented /proc files.
• udp(7) adds documentation of a few previously undocumented /proc files.
• pthreads(7) adds lists of functions that POSIX.1 specifies are and may be cancellation points.

Lagarto

man-pages-3.13 is released

I've uploaded man-pages-3.13 into the release directory (or view the online pages). Notable changes in man-pages-3.13 are:

• More man pages for POSIX threads library functions: 6 new pages documenting 11 functions.
  
  • pthread_attr_setaffinity_np(3) (includes documentation of pthread_attr_getaffinity_np(3))
    
  • pthread_attr_setschedparam(3) (includes documentation of pthread_attr_getschedparam(3))
  • pthread_attr_setschedpolicy(3) (includes documentation of pthread_attr_getschedpolicy(3))
  • pthread_setaffinity_np(3) (includes documentation of pthread_getaffinity_np(3))
  • pthread_setschedparam(3) (includes documentation of pthread_getschedparam(3))
  • pthread_setschedprio(3)
• Various fixes and improvements to the example program in epoll(7).

Saltamontes, intentando evitarme

Recent changes in file descriptor system calls

In recent Linux kernels, especially 2.6.27, a number of system calls have changed, or new versions of existing system calls have been added, to allow more control over the file descriptors created by those system calls. (Most of this work has been done by Ulrich Drepper.) These changes have taken the form of either adding new bits to the flags bit-mask argument of an existing system call, if it had such an argument, or creating a new version of the system call that adds an extra flags argument. In most cases, two new flags have been added: a close-on-exec flag, and a non-blocking flag, which we describe shortly.

The changes are summarized in the table below. In this table, the Kernel column indicates the kernel version where the change occurred, and the Glibc column indicates the version of glibc that adds the corresponding wrapper functions and/or header file definitions. (Note: glibc 2.9 is not yet released.)

Interface	Changes	Kernel	Glibc	Notes
open(2)	New flag: O_CLOEXEC	2.6.23	2.7	Flag also supported for openat(2). These syscalls already supported O_NONBLOCK.
fcntl(2)	New flag: F_DUPFD_CLOEXEC	2.6.24	2.7	Performs a similar task to dup3(2)
recvmsg(2)	New flag: MSG_CMSG_CLOEXEC	2.6.23	2.7	-
dup3(2)	New syscall, like dup2(2), but adds flags argument (O_CLOEXEC)	2.6.27	2.9	Requires new glibc interface
pipe2(2)	New syscall, like pipe(2), but adds flags argument: O_CLOEXEC, O_NONBLOCK	2.6.27	2.9	Requires new glibc interface
socket(2)	New flags in type argument: SOCK_CLOEXEC, SOCK_NONBLOCK	2.6.27	2.9	-
socketpair(2)	New flags in type argument: SOCK_CLOEXEC, SOCK_NONBLOCK	2.6.27	2.9	-
epoll_create1(2)	New syscall, like epoll_create(2), but adds flags argument: EPOLL_CLOEXEC; the new system call drops epoll_create()'s obsolete size argument	2.6.27	2.9	Requires new glibc interface
inotify_init1(2)	New syscall, like inotify_init(2), but adds flags argument: IN_CLOEXEC, IN_NONBLOCK	2.6.27	2.9	Requires new glibc interface
eventfd2(2)	New syscall, like eventfd(2), but adds flags argument: EFD_CLOEXEC, EFD_NONBLOCK	2.6.27	2.9	The glibc eventfd() wrapper already allowed a flags argument, so no new wrapper is required
signalfd4(2)	New syscall, like signalfd(2), but adds flags argument: SFD_CLOEXEC, SFD_NONBLOCK	2.6.27	2.9	The glibc signalfd() wrapper already allowed a flags argument, so no new wrapper is required
timerfd_create(2)	New flags: TFD_CLOEXEC, TFD_NONBLOCK	2.6.27	2.9	-

A proposed analogous change for accept(2), paccept(), supporting flags SOCK_CLOEXEC and SOCK_NONBLOCK and treatment of a signal mask argument like pselect(2), was debated and then spent some time in limbo, but has recently re-emerged in a somewhat modified form, accept4() (which was in fact the original proposal), that will probably go into Linux 2.6.28 or 2.6.29.

Perhaps one day there might even be an analogous change for mq_notify(3), since (on Linux, but not on most other systems) a message queue descriptor is really just a file descriptor.

The close-on-exec flag (*_CLOEXEC)

The addition of a close-on-exec flag was the primary motivator for the system call changes. Specifying this flag causes the file descriptor created by the system call to automatically have its close-on-exec flag set. (This flag causes the file descriptor to automatically be closed if the process does a successful execve(2).)

Before the existence of this flag, it was possible to change the close-on-exec flag of a file descriptor after it has been created, using the fcntl(2) F_GETFL and F_SETFL operations. The fact that this required two additional system calls was not so problematic as the fact that the need for multiple (non-atomic) steps to set the flag on a new file descriptor meant that there were certain race conditions that could lead to races in multithreaded programs where one thread was trying to set a file descriptor's close-on-exec flag at the same time as another thread was performing a fork() plus execve(). Ulrich Drepper explains the resulting security issues in more detail.

The non-blocking flag (*_NONBLOCK)

The *_NONBLOCK flag causes the non-blocking flag to be set on the open file description associated with the new file descriptor. (For a discussion of the relationship of a file descriptor to an open file description, see the open(2) man page.)

Unlike the *_CLOEXEC flag, the *_NONBLOCK flag exists merely as a convenience: it saves two system call operations (fcntl(2) F_GETFL and F_SETFL) if we want to immediately set the non-blocking flag when opening a file descriptor.

Note that there deliberately is no *_NONBLOCK flag for dup3(2). This would not be sensible, since the new file descriptor shares an open file description with the old file descriptor.

There is also deliberately no *_NONBLOCK flag for epoll_create1(2), since equivalent functionality can be obtained with a zero timeout.

Other flags?

The flags argument added for the new system calls allows for other kinds of functionality to be added to these system calls in the future.

Future standards?

Ulrich Drepper already did some work on getting some of these interface changes into the POSIX.1-2008 standard, which includes specifications of the O_CLOEXEC flag for open() and the F_DUPFD_CLOEXEC operation for fcntl(). In the future, some the other changes may also make their way into the standard.

A note on the new system call names

The numbers in the names of the new system calls refer to the number of arguments that each system call has. This is an extension of a convention that was used for some existing Unix system calls, notably dup2(2), wait3(2), and wait4(2). Note that while the wrapper function for signalfd(2) has three arguments, the underlying signalfd4() system call really does have four arguments, as described in the man page. (However, this suggests that, in the end, this naming scheme might not have been the best choice.)

man-pages-3.12 is released

I've uploaded man-pages-3.12 into the release directory (or view the online pages). Notable changes in man-pages-3.12 are:

• Documentation is added for a set of new and changed system calls (which will be the subject of a future post) that extend the functionality of existing system calls that work with file descriptors. (The changes occurred in kernel 2.6.27.) The new and modified system calls add flags that allow the close-on-exec file descriptor flag to be set, and the non-blocking file status to be set on a file description, as the file is opened. The modified pages are:
• dup(2) adds a description of the new dup3() system call.
• epoll_create(2) adds a description of the new epoll_create1() system call.
• eventfd(2) adds a description of the new eventfd2() system call.
• inotify_init(2) adds a description of the new inotify_init1() system call.
• pipe(2) adds a description of the new pipe2() system call.
• signalfd(2) adds a description of the new signalfd4() system call.
• socket(2) and socketpair(2) add a description of the new SOCK_CLOEXEC and SOCK_NONBLOCK flags.
• timerfd_create(2) adds a description of the new TFD_CLOEXEC and TFD_NONBLOCK flags.
• A start has been made on providing man pages for the POSIX threads library functions: 15 new pages documenting 23 functions.
  
  • pthread_attr_init(3) (includes documentation of pthread_attr_destroy(3))
  • pthread_attr_setdetachstate(3) (includes documentation of pthread_attr_getdetachstate(3))
  • pthread_attr_setguardsize(3) (includes documentation of pthread_attr_getguardsize(3))
  • pthread_attr_setscope(3) (includes documentation of pthread_attr_getscope(3))
  • pthread_attr_setstacki(3) (includes documentation of pthread_attr_getstack(3))
  • pthread_attr_setstackaddr(3) (includes documentation of pthread_attr_getstackaddr(3))
  • pthread_attr_setstacksize(3) (includes documentation of pthread_attr_getstacksize(3))
  • pthread_create(3)
  • pthread_detach(3)
  • pthread_equal(3)
  • pthread_exit(3)
  • pthread_getattr_np(3)
  • pthread_join(3)
  • pthread_self(3)
  • pthread_tryjoin_np(3) (includes documentation of pthread_timedjoin_np(3))
• Many details were added to, or fixed in, ld.so(8).

El jardín de la casa

man-pages-3.11 is released

I've uploaded man-pages-3.11 into the release directory (or view the online pages). Notable changes in man-pages-3.11 are:

• A new umount(2) page has been created by splitting the umount() and umount2() material out of the old mount(2) page.
• The mount(2) page adds a description of per-process namespaces.
• Various fixes and improvements in getdents(2), including the addition of an example program.
• Many improvements and additions in signal(7), the page that provides an overview of signals on Linux.
• Numerous fixes to many other pages.