Category: Coding

OutputStreams 1.0 is now available at GitHub (user: MikeBrownUK, repository: OutputStreams).

Pretty much everything you should need to get going is in the repository, but if you’d like to know more about the project and how it came to be then read on and allow me to indulge myself by means of some slightly meandering notes:

About twenty years ago, when running a small development studio in the games industry, I came across a rather nice piece of code written by one of the team which specialised a Standard Library stream allowing you to send diagnostic messages to a file or console window using the operator << syntax provided by the Standard Library.

This seemed a whole lot nicer than the cumbersome variadic syntax of printfs and sprintfs I was used to.

Consider the following:

std::cout << "The result of " << float1 << GetOperationSymbol() << float2 << “ = “ << GetOperationResult( float1, float2 ) << std::endl;

It’s a little easier to follow logically, I think, than:

printf( “The result of %f %s %f = %f”, float1, GetOperationSymbol(), float2, GetOperationResult( float1, float2 ) );

If you adopt stream syntax for output in C++ you can of course go on to serialise just about any data type you want by providing a specialisation for operator <<.

Add in other benefits you get out-of-the-box with Standard Library streams: locales that you can imbue at runtime to format numeric output automatically in a fashion appropriate for a particular geographic region; manipulators to handle padding, whitespace and other custom formatting requirements – and there is a convincing case for ditching those sprintfs and printfs forever. And that’s exactly what I did because I stole carried that original idea around with me from company to company for more than twenty years, until I had the time earlier this year to finish off some wish-list items with the idea of releasing the whole thing publicly.

Throughout those twenty-something years of on-and-off development of this project in its various forms I’ve always tried to adhere to the following goals:

1. The library must be lightweight and performance-centric so it can be used in the games industry and other real-time environments.

2. The library should be capable of meeting many runtime logging requirements, not just output of developer diagnostic information.

3. I wanted each stream and channel object to compile away to zero code footprint if required. This was very relevant from my background on games platforms where standards prohibit the outputting of any debug information on the terminal for final consumer software. Fortunately, C++ 11 made this goal a whole lot easier without me having to resort to using preprocessor macros to hide the stream objects.

Standard Library streams can be used in multithreaded environments but interleaving of output is probable (source: N. Jousuttis, The C++ Standard Library, 2^nd edition). Here’s a little example of such interleaving when writing to std::cout with a Win32 console application – I created three threads running this very tight loop:

while ( system_clock::now() <= wait )
{
    std::cout << "Thread " << threadNumber_ << " writing to channel " << channelNumber_ << std::endl;
}

Here’s the output:

Thread 1 writing to channel Thread 0
3Thread Thread 2 writing to channel writing to channel 11 writing to channel
0Thread 02
writing to channel
1Thread Thread 31 writing to channel
0 writing to channel Thread 02

Fragmentation of a sentence and indeed any loss of meaning in output logs isn’t acceptable but I decided that occasional interleaving of whole entries from different threads at destination would be fine providing each could be optionally timestamped for visual or automatic sorting (hence the message prefixing system). I soon decided that using thread-local channels and buffers that synchronised with their destination streams only during a flush operation would minimise contention between threads and maximise performance.

The basic OutputStreams code was fully thread-safe a good few years ago but during some free time earlier this year I started work on new features I had been wanting to add for some time: ‘channels’ to represent the output from particular program domains, runtime output filtering controls for both OutputStreams and these new OutputChannels, automatic timestamping for both stream types (actually an extendable message prefixing system) and the ability for OutputChannels to attach to and write to multiple OutputStreams. These few items required a fairly substantial rethink of my class design and as a result the new codebase barely resembles how it looked a couple of months ago…

The idea of using channels/domains and message filtering came from a spell spent contracting at a developer in the North East of England whilst rebuilding and updating some of their Playstation 3 projects for release by another publisher. I noticed that their diagnostic output system used traditional printf-style syntax to construct messages but allowed you to name diagnostic domains and apply runtime filtering to logs, so I determined to find a way to implement such a system into my streams code. Ten years later I finally got round to this non-trivial task so, with a tip of the hat to them and to Jim, or maybe Rob- they will know who they are, anyway – whom I think wrote that initial spark of code I found all those years ago, here is my OutputStreams library for you to use free of charge.

Here’s what it can do out of the box:

1. Handle char, wchar_t, char16_t and char32_t OutputStreams – the latter two with limitations inherent in the existing Standard Library implementations upon which the code relies.

2. The example code allows you to send your output to files, std::cout and std::wcout, Windows consoles or the Windows debugger target (which is usually the Visual Studio output window) all using the provided sample OutputTarget class templates.

3. You can add and combine OutputTargets very easily by implementing a new class with just two mandatory functions. At one point I had a class that encapsulated all the other OutputTargets and sent its output to all of them ( no longer necessary as an OutputChannel can now attach to multiple streams).

4. You can use OutputStreams directly for best performance in a single thread context without any synchronisation bloat and still get the benefit of message filtering via member functions and stream manipulators.

5. Create one or more OutputChannels for use by a single or multiple threads, giving you synchronised writing to one or more OutputStreams with message filtering for each individual channel.

6. Use the provided OutputStamp classes to add a prefix to all messages sent to a stream or channel. The example templates provided will prefix each output segment sent to your stream with either a system date/time/millisecond string or the line number (actually the number of flush calls) since creation. You can expand the OutputStamp templates to do all sorts of things – perhaps output just portions of the date and time as they change, for example.

7. Convert the three common UTF encodings to the correct format for the final target as they are passed through the output pipe, albeit with some limitations and assumptions – see my more detailed notes for an explanation.

All library and example code has been tested with the Microsoft and G++ compilers on 32-bit and 64-bit Windows and Linux and also G++ ARM running on a Raspberry Pi 4.

Usage examples and further design notes can be found here.

I will do my best to fix any bugs I find and update my GitHub project accordingly, other commitments permitting. I have further plans for this codebase which I will share in due course – mostly further performance optimisations for games console and embedded architectures. A small googletest suite also exists for OutputStreams and is included in the source package – you will need to download googletest if you wish to build and run the test program.

The software is released under the MIT license which is the most minimalistic license I could find to let anyone use it as they wish providing that its copyright notice is carried forward and therefore the people who inspired it get their well earned namechecks via this little blog post.

Thanks for reading and I hope you find the code and my notes useful in some way.

Mike Brown, September 2022.

I’ve almost finished preparing my upcoming C++ library release. This is free-to-use source code that extends existing standard library stream and buffer classes to provide thread-safe stream-style output to a variety of media including std::cout, std::wcout, files, console windows and pretty much anywhere else you might care to send streams of text for debugging or logging purposes via a simple statically polymorphic interface.

Source streams can send information to several stream targets concurrently and runtime filtering allows output verbosity to be changed at any point without a recompile by using member functions or via direct injection using stream manipulators – useful for filtering customer-facing logs as well as changing debug output verbosity on the fly. Information sent can be prefixed with a time & date stamp, line number or pretty much any other piece of information you could imagine, again by using my provided example classes as an interface guide.

Because the code is based upon established Standard Library stream classes you get almost all of the benefits of those classes including format manipulators and locale imbuing.

A simple aliasing of types through a preprocessor define can reduce the code footprint of the streams and all literals sent directly to them to zero: useful if debug output needs stripping from binaries easily during compilation – for a games console release candidate, for example.

The library requires only C++’11 support and has so far been tested on Windows and Linux (PC and ARM architectures). Most Unicode text formats and integral C++ character types are catered for, albeit with limitations already inherent in the existing Standard Library and the platforms it is implemented on – more on this in my next post.

Just getting the last code tests, tidy-ups and documentation finished, so check back in a week or two for the library release.