The underlying multi-stream container format for speech data is Ogg, developed at http://xiph.org. libogg++ is a C++ library implementing Ogg. It is designed to be independent of the specific codecs for the interleaved data streams, and to be thread-safe in a threading model in which each codec or transport runs its own thread.
libogg++ is FOSS, and is hosted at salsa.debian.org.
The ALingA format is a multi-stream format based on the Ogg format, and libalinga is a specialization of libogg++.