Midori/ Singularity

"pure" Managed OS

Ben Kloosterman — Sat, 24 Jul 2010 09:17:05 GMT

Sooos will not be a pure managed OS , like Cosmos SharpOS or Mosa , or even Singularity ( which has a small amount of C++ code)...

I still feel that GC technology has some way to go and at present the performance penalty is too bad for certain functions. This is especially so for writes as most GCs todate implement a write barrier which can more that half write performance , that said as we go to higher core counts it may be more efficient to remove the write barrier.

In particularly 3D and some mathemetical modelling which need to work over a large mutable space.

However it may be possible in future and we can test a native lib wrapped in a service vs a pure service to see the impact on performance and only have critical services that suffer from GC costs running native . Anyway Sooos will take a practical approach it will have as much code as it practical type safe and memory safe and will move other performance critical native code when there is comparable managed code.

Managed OS

Ben Kloosterman — Tue, 06 Apr 2010 03:10:21 GMT

Just an update on what I have been up to. I don’t have a lot of time and what little I have im spending on the kernel.

Some back ground

The key challenges to moderns OS are
1. Security ( more specifically hacking through 3rd party services likes Flash and limiting the damage from users allowing bad apps)
2. Reliability
3. Resiliency eg recovering from a critical failure

And to a lesser extent
4. Difficulty in managing change to an OS
5. Handling the Asynchronous nature of the internet
6. Handling multi cores efficiently

SOOOS ( Service Oriented Object Operating System) is a OS that scales from embedded to GP. It is not a simple research OS , the key goals are

*Scale from embedded to server ( this is easy with no MMU)
* Only allow managed user applications ( Java & .NET )
* A full capability security model with little ambient authority both improving code quality and the main security risks.
* TDD to increase reliability and ease of change
* Deeply asynchronous message passing IPC to improve security , maintenance , multi core scaling and performance
* Applications and Services are broken down into small services these services are load balanced and the OS can decide how many to run. These small services also allow greater reuse and flexibility as they are defined via an interface. Examples of these services are Crypto and XML parsing. The small services also increase parallelism.
* Any service can be made available to other machines.
* Self healing , services are monitored and terminated if they stop responding the system ensures there is always a minimum amount of services. .
* Performance at least 95% of Linux or windows , 105% is desirable.
* Kernelless. There is a TCB which I hope to keep under 100K lines.

doing I have continued documenting the design of my kernel ( 13 documents now) and discussing kernel designs with phil on #mosa and others.

I was using Cosmos but I think it will be a while before the compiler is at a decent state and that means heavy optimizations ( my guess 5 years Cosmos , 2 year mosa) After considering the above I was thinking for any OS to gain acceptance it needs to be fast and reliable which is difficult for us to achieve without a compiler in fact the first few non asm versions of UNIX failed ( based on B and C) until they had a good and fast compiler.

I then decided to write these components on Windows using a changeable HAL when mosa or Cosmos is ready I can change the Hal its ready to this end I wrote some IPC algorithms , locking mechanisms and a MM to see what they would look like . .I also don’t believe using corelib as a syslib is “safe” yet to write your own syslib for trusted components which I was doing is time consuming.

At this point I was working on shared data structures and for this would need an efficient ( non GC ) allocator so started looking at some Linux code , they have a good approach here with a Buddy page allocator for large amounts and SLAB/SLUB for lots of small objects ( just a large allocation say 2 Meg which hands out lots of small objects by marking them in use ) I then found a MMU_CONFIG option.

This is used by uCLinux designed for embedded systems , disable the MMU .
Features that are present
- A syslib ( uslib)
- Full Multi tasking
- Most of Linux works ,except calls to fork ( due to its use of copy on write paging) . This means things like the nvida driver
- Uses the standard Linux Kernel
- Support for both Position independent Code and remapping code on load
- Support for shared code
- Flat file format
- Flat memory model
- Support for many architectures including 680x0 ,arm etc

It than struck me If can get this and Mono working and then disable non .NET apps I have a managed OS which is fast , reliable and immediately useable ( eg package it up and use VS to create the managed apps and you already have an awesome embedded environment (though a bit big) ) .

We then add a new managed Asynch IPC , assembly install and compile , compile time range checks and capabilities and slowly move Linux drivers into managed user app drivers ( even a managed wrapper initially especially for the accelerated Nvidia and Radeon drivers)

Delete what is extra . Eventually there is less and less native code.

“Perfection is achieved not when you can’t add any more but when you cannot take any more away”

Result we start with a Linux derivative which is fast , stable and runs on many platforms and continually improve it to make it more secure , reliable and resilient.

Anyway that’s what im working on ( getting x86 Linux arch with no MMU and Mono running on it) .

The End of Linux ...and Windows

Ben Kloosterman — Tue, 04 Aug 2009 01:58:36 GMT

New generation of OS coming.

Operating Systems are about to enter their 4th generation with a revolutionary change. These being

Basic
TimeSharing
Hardware memory protection ( most were UNIX/MULTICS derived designs)
Managed OS. ( or OOOS)

While Singularity/Midori is a Manged OS it is still a traditional style OS. Not all 4th gen OS will be so traditional , things a operating system does like memory segmentation are not even needed.

A better name for these OS may be an Object Oriented Operating System or even Object Oriented Framework. Tradditional things like IPC (Asynch = events) and security can be done with language features (Interfaces ,Private members etc) especially using Capability OS techniques with the reference being a capability. In effect these OS are just huge OO applications with user apps being part of it.

The only thing regarding scheduling that may be required is preemption especially on single threaded CPU's. Yet this can still be done entirely in the OO framework .

Why do this ?
            Better performance than existing OS due to no IPC/memory overheads.
            More reliable , the OS can be easily tested and even proved.
            More secure. ( really more secure ,not like OSX/Linux due to fewer attacks)

All these things are especially needed in small internet linked devices ( including phones , Netbooks etc) , notice the latest iphone patch ? It will not be long before you will NEED a virus scanner and firewall on your phone as they will be attacked like windows.

I believe this is totally a revolutionary step as big as protected memory systems ( which are not needed now) were in building a stable OS remember early versions of Windows compared to NT /2000/XP .

However research is completely lacking . Consider the nameless OS im building with the Cosmos managed OS toolkit.

```
Managed OS - very little research
```

Single Address space - no modern research eg Multi core .

```
Asych IPC  - little modern research 
```

Capability Managed OS - no one has done it .

OO security for Capabilities - no one has done it.

Moving hardware access into user process and limit with capabilities - Very few people have done it most modern designs have user mode drivers but the hardware access is the HAL.

No Kernel calls - Minix is the only other OS I know of where IPC is used for Kernel calls .

No kernel - Even  "Nano" kernels have primatives and hardware in the Kernel .

How does this effect Linux ? I dont believe the open source community can move significant resources fast enough for a new completely incompatible Linux ( including all C and C++ user applications) and it needs a very dedicated individual to complete a kernel . Compare Linux ( mainly Linus initial effort ) to Gnu Herd ( an open source effort) . Note there are small open source managed OS efforts (Cosmos and MOSA) .

Why can Microsoft do it ? Because they have been working on this since 2002-2003 and have released a "Research Kernel" in 2005. They can pressure driver manufacturers and most .NET and Java applications can be fully supported and MS will move key apps like Office , IIS , Sql server to the new platform.

There is a real danger to open source where MS will sell a Midori OS to corporates like they did with NT and leave windows as a cheap alternative if Midori sells well. While this may not maximize short term revenue it will hold Microsofts dominant position and reinforce there sales for other applications eg Word for Windows 2 ( when windows was released) ,Office 2000 and SQL server for NT /win 2000.

Anyway Midori if the designers get it right will make its appearance in the next Major Windows Mobile version , it will be a great area as a lot of backward compatibility will be provided via Compact Framework.

More information on Midori

Ben Kloosterman — Mon, 27 Jul 2009 14:55:41 GMT

Very curious on Midori and where it is going ..

So based on the following information i will entertain some speculation.

The following is a subset of people who are working on Midori ( see http://www.nabble.com/Midori-in-The-Register-td18739750.html)
Ravi Pandya,
Dean Tribble,
Pavel Curtis,
Eric Northup,
Chris Brumme
Jonathon Shapiro

Recent papers

Ravi Pandya (Architect Technical Strategy) ( has some comments related to small hand held devices/ netbooks and Security / identity
                  Abstracting Operating environment from Operating System

Dean Tribble ( PrincipalArchitect work in security) Recent patents
                    Concurrent exception handling
                     Abstracting Operating environment from Operating System
                      Hierarchical reservation resource scheduling infrastructure
                      Co-ordinating Chores in muli processing

Pavel Curtis (Architect) Lots of work on social networking.

Eric Northup ( SOftware ENgineer) , worked on Coyotos (Security, Capability secure Windowing and kernel proof)

Chris Brumme (Architect/ Destinguished engineer) . Major player in the .NET CLR v interesting blog

I’m now on an incubation team, exploring evolution and revolution in operating systems. This is a fascinating area that includes devices, concurrency, scheduling, security, distribution, application model, programming model and even some aspects of user interaction (where I am totally out of my depth). And, as you might expect with my background, our effort also includes managed programming.

This is 2006 and is when the Incubation project (MIdori started) not Singularity ( which had major releases in 2006 but is a research project) . My guess is Midori is about 3-4 years under development ( eg started when the singularity papers were published) .

J Shapiro has a paper on automated device driver testing . Probably a senior architect position major player with Eros and Coyotos capability OS.

Make your own conclusion but at least 5 senior architects are on the team . Im really curious if they are considering capabilities my guess is they are at least going to try /investigate it since they are hiring people from the Coyotos project. Also from Chris 's comment it is likely to be a revolutionary change ( which could very well be a pure managed capability system) .

Midori To be or not

Ben Kloosterman — Mon, 18 May 2009 03:59:01 GMT

Just read this article and had to respond/..

http://www.computerworld.com.au/article/255954/windows_midori_doomed_failure

On a number of facts its just wrong , particularly.

1. Microsoft wants to evolve.

2. Its unnecessary as windows Kernel is light and replacing the kernel will not solve the security and DLL problems.

3. The purpose behind a new OS

4. Backward competability

Note this is based on Midori being a Singularity style kernel.

1. Microsoft wants to evolve.

Dont know about MS desire to evolve ..

DOS - Windows - NT
NT - Windows 2000 driver model
Vista Video driver model
Windows - Bob
Word 5 - Word for Windows 1 - clean break WYSIWYG
VB - VB.NET
C#
WPF
Office 2007 - tool strips
Com- Com+ -DCOM/Ent Services - Remoting - Windows Communication Foundation

All the above are major clean breaks some opportunistic ( Word for Windows) , some because the platform was getting to hard to evolve ( NT). Its no coincidence MS dropped windows after ME ( for NT ) and Vista is showing similar signs. No company liked to do a clean break but in IT its pain thats for the long term good ( looks at OSX and NT) , Linux will suffer here as well as community projects find it hard to make such a break without fragmenting.

MS need a new Windows Mobile platform and my prediction is Midori will replace WIndows Azure and Windows Mobile 8 ( note CE needs a replacement) . These platforms run mainly or only managed apps so its an easy transition.

2. Its unnecessary as windows Kernel is light and replacing the kernel will not solve the security and DLL problems.

Most importantly Midori and Windows kernel are similar and the kernel is not that important. Whats different is how user application , device drivers and systems ( the fat and insecure bits in Windows) run and talk , so your comments on the windows Kernel ( while correct ) are meaningless. ie replacing WIndows Kernel with Midori is pointless whats important is replacing all ( non Java/.net/managed) apps , device drivers and services. This is why itsa big deal.

3. The purpose behind a new OS

I also don't think you understand the purpose of Midori or what its about .

Security . This is the #1 driver and note how MS get smacked for security violations when its prob more secure than its competitors . This is not mentioned at all. Those Apple adds hurt .. prob enough for Bill and co to spend $5B to build a new OS. Ok the article mentions that it cant be achieved by the kernel , that is partially correct the Kernel can restrict applications . How many managed virus are there ?
Reliability. Native apps and device drivers are reliable. ( Note not the kernel)
Marketing. It is very important for a company to have the technological edge its why they can charge $$$. And Midori presents a marketing coup for MS , they have a new style OS and they can point at there competitors and say how old they are eg Linux 70-80 style kernel and command line interface. Midori only needs to be available and used by 1% of users for then to have this benefit and they can charge a premium. Remember Singularity kind of shocked the sleeping OS research community not for a managed OS ( as old OS sometimes only allowed managed languages , though revisting it was interesting) but by basically saying you can have security ,reliability. AND performance in a managed OS.
MS needs to see the world in 10 years or 15 years time . They need a single OS that scales from devices to cloud servers and an OS thats .net aware . They also need an OS thats super secure so it can handle this eg are you going to install a firewall on your mobile phone ?

4. Backward competability

Note MS are proposing to support windows drivers in one proposal at first as well as native Midori. This is possible via the Midori HAL . In addition Windows apps could be support with the same Virtualization technology as Vista.

Prediction Midori will replace Windows...eventually but first it will be an OS for cloud and hand helds. Then it will be a new style OS similar to NT3.51 and 4 when Windows 95/98 were the desktop standards.

These are the timeliness we are looking at . Note Ms needs to look at where WIndows will be in 10 years.
WIndows Mobile 8   ~2011
Azure              ~2011-2012
New Server ( Applications , files etc ) ~ 2013
Windows Mobile 8 light Desktop version 2013-2014?
Power workstation ??               2014
Desktop replacement ??             2015-2017 ?

The product however has to survive on its merits and WIndows Mobile 8 will be a key release.