- MacOS (pre-OS X)

"Revolution in the Valley"

https://www.amazon.com/Revolution-Valley-Insanely-Great-Stor...

http://basalgangster.macgui.com/RetroMacComputing/The_Long_V...

- NeXTSTEP / Mac OS X

Debatable about non-C non-Unixy part, but it surely isn't the focus of the whole stack.

"Mac OS X Internals: A Systems Approach"

https://www.amazon.com/Mac-OS-Internals-Approach-paperback/d...

http://www.nextop.de/NeXTstep_3.3_Developer_Documentation/

- Oberon and its derivatives (1992 and 2013 versions, System 3, Insight ETHOS and A2)

https://inf.ethz.ch/personal/wirth/

https://inf.ethz.ch/personal/wirth/ProjectOberon/index.html

http://www.projectoberon.com/

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.472...

http://www.ocp.inf.ethz.ch/wiki/Documentation/Front

https://www.research-collection.ethz.ch/handle/20.500.11850/...

- Symbian

"Symbian OS Internals: Real-time Kernel Programming"

https://www.amazon.com/Symbian-OS-Internals-Real-time-Progra...

"Symbian OS Platform Security: Software Development Using the Symbian OS Security Architecture"

https://www.amazon.com/Symbian-Platform-Security-Development...

"The Symbian OS Architecture Sourcebook: Design and Evolution of a Mobile Phone OS"

https://www.amazon.com/Symbian-OS-Architecture-Sourcebook-Ev...

- Mesa and Mesa/Cedar

http://www.digibarn.com/friends/curbow/star/XDEPaper.pdf

http://worrydream.com/refs/Teitelman%20-%20The%20Cedar%20Pro...

- JX

http://www4.cs.fau.de/Projects/JX/

- SPIN OS

http://www-spin.cs.washington.edu/external/overview.html

- Singularity,

https://www.microsoft.com/en-us/research/project/singularity...

https://channel9.msdn.com/Shows/Going+Deep/Singularity-A-res...

https://channel9.msdn.com/Shows/Going+Deep/Singularity-Revis...

- Barrelfish

https://www.microsoft.com/en-us/research/project/barrelfish/...

- Midori,

http://joeduffyblog.com/2015/11/03/blogging-about-midori/

https://www.infoq.com/presentations/csharp-systems-programmi...

"RustConf 2017 - Closing Keynote: Safe Systems Software and the Future of Computing"

https://www.youtube.com/watch?v=EVm938gMWl0

"While never reaching commercial release, at one time Midori powered all of Microsoft’s natural language search service for the West Coast and Asia."

Source: https://www.microsoft.com/en-us/research/project/singularity...

- Helios

https://www.microsoft.com/en-us/research/wp-content/uploads/...

- Muen

https://muen.codelabs.ch/

- GenodeOS

https://genode.org/documentation/index

- Inferno

http://www.vitanuova.com/inferno/docs.html

- Minix 3

https://wiki.minix3.org/doku.php?id=www:documentation:start

- BeOS

"Be Developer's Guide"

https://www.amazon.com/Be-Developers-Guide-Development-Team/...

"Be Advanced Topics"

https://www.amazon.com/Be-Advanced-Topics-Development-Team/d...

- Windows

https://www.amazon.de/Windows-Internals-Part-Pavel-Yosifovic...

https://www.amazon.de/Windows-Internals-Part-Mark-Russinovic...

https://channel9.msdn.com/Shows/Going+Deep/Rob-Short-and-ker...

https://channel9.msdn.com/Shows/Going+Deep/Doron-Holan-Kerne...

https://channel9.msdn.com/Events/Build/2012/3-005

Not everything by a long shot, plenty more to re-discover like VMS,IBM i and Z, Unisys ClearPath, mbed,...

Just keep an open mind and don't idolatrize UNIX, yes it has a couple of good ideas, but they don't make it the be all end all of OS design.