Design & Practice

Is Sooos a new OS ? (Since its linux based)

Ben Kloosterman — Tue, 13 Jul 2010 03:49:36 GMT

The question is a good one as it leads to an interesting discussion what Sooos is trying to do..

Firstly Sooos is probably more of an independent OS than Android is which is basically linux with a new Java Run time , GUI and a package manager / security system.

Sooos uses the linux kernel and drivers / services , as time goes on more and more of them will be replaced but some Linux core will probably remain for a long time since

- Backward compatibility fo things taht are diffciult to support eg 3D drives/open GL
- As an extensive boot loader
- Support for more drivers

The key is all this backward support required a capability hence we can preffer newer and safer apps.

This gets us to the point what SOOOS is trying to achieve . Its not a new OS but the way we program . . In fact a run time on Windows would be quite possible ( and likely if we get some traction) and similar of HPs research in this direction with Polaris and others like capdesk.

So what are we trying to achieve in the programming field.

Make more fo the code base "managed" eg type safe and memory safe without loss of performance..
Increase reliability by allowing failure and restarts of parts of an app
Prevent parts of an app like an add in destroying an app
Make code more secure by using OO techniques. eg a priviate field is prvate to the whole machine and a class ( program ) can only touch what it has references ( capabilities to) .. Just like global data in OO we are trying to remove this at the OS level ( eg no ambient file authority for a user)
Simplify multi core ( 8+) programming

How does Sooos do this ?

It is interesting to note one of the most successfull programming models is rarely used by OS . This is the message based system with pump windows 3.X ran on it without any form of pre-emptive tasking it was cooperative. Nearly all GUIs use it , most middle ware systems etc .

The reason they are not used is the bad name micro kernels with bad IPC designs which used messaging got - however L4 and later QNX debunked all these. The problem was it came at a critical time when OS were comming up eg ( windows NT and Linux around 1990) .

Now we are approaching another such inflection point for several reasons

Hand held computers - Think all the new apple stuff
Massive amount of cores , the future is 100+ core CPUs on servers and desktops. Though its likely the desktop will become a home server ( driving TV , security etc ) with multiple clients of varying media formats
Existing OS are groaning under the historical strain ( see Linus comments on Linux becomming bloated and peoples oppinion on windows) . There seems to be desire outside of niches and corporate shops for change.
Reliability , mobility & simplicty seems to be paramount in peoples oppinions even if some features do not make it, ( web 2.0 only offers 1 of these 3)
Security - it wont be long before Virus start attacking phones to create bot nets .. People WANT unlocked phones which are secure. There is not much difference between Linux and Windows here - even Multics was far more secure all that time ago. Whats more people want to allow apps to run but even if tricked they do not want the app to have carte blanche on their machines.
We are the cusp of abandoing old languages like C. , Java and .NET will continue to catch up in performance with the roll of C being limited to a few critical libs just like assembly was around 1990.

Now back to how do we do this ..

- We make apps consist of multiple services . By making them more modular we increase reuse of code libs ( services) but also we get partial degredation from within applications ( eg the add in) . We also get portability as parts of apps can be moved local or remote ( eg distributed OS) as desired . eg Compilation and compression is offloaded to a cloud server or you home server.

- Apps use services instead of dlls , failure within a service will simply restart the request and / or return an error to the app ( even if the Service has a fatal error) as opposed to the app crashing which we have now. In fact in case of a service crash the OS will fire up an additional copy of a service , it will process the failed message by itself and continue the work queue with the other service(s) . If the service crashes a second time , an error is returned to the app and the service has a log of the message that causes the failure. As an example of this you could overheat or destroy a core and not just the OS but the applications would continue running as though nothing has happened ( appart from a delay)

- We make the default service single threaded and the OS handles all the multi threaded code / locking . Hence devs have the abbility to use multiple cores without the user writing any multi threaded code!. ( this probably covers 95% of cases and fits especially well with newer languages like Ocaml , Hascal , Scala etc )

- We implement capabilities for the security model .

See http://www.shanghai-software.com/blog/archive/2010/06/13/why-sooos-will-work.aspx or more details

We benefit from the fact that most GUI code is asyncronous ( even if it runs on 1 thread) and a similar programming model will be used.

Now back to the coding.

Developer Book Syndrome

Ben Kloosterman — Sun, 01 Nov 2009 13:33:07 GMT

A colleague of mine once stated that creativity is rare in technical people I had my doubts at the time as I’m fairly creative and often build unorthodox solutions tailor made to the problem but the longer I’m in the industry the more I agree.

The biggest symptom of this is something I called book syndrome. This is where famous author xyz ( eg Lowly , Fowler etc etc ) writes a strategy in a book and people use it in completely in inappropriate circumstances. When challenged on the solution they use the authority of the author to emphasize they are doing the right thing. While this technique may provide some job protection it is questionable that the famous author intended to apply it where it was applied ( eg Juval Lowly SOA 1 service per class ) and in many cases like Fowler there are many methods yet a single patterns is repeated ad nauseum without any concern for the special cases for each project..

And it is not just books, Universities have a lot to answer for drumming in OO yet for a lot of small to medium projects scripts and RAD provide better and more consistent results ( OO /RAD is Software theory relevant) . The fact is the bedrock of nearly all programming theory is based on 90% of the cost of an application being maintenance while true in the 50-80s this is clearly not true in the huge amount of small applications and scripts that never enter software studies ; in fact most of these programs never change over the life time of the program ( and are completely rewritten in a new technology after 10-15 years) is completely at odds with most of software development theory. I’m not against OO but I am against using OO in many inappropriate situations ( eg blindly following book – book syndrome) – even in an OO house RAD development will be faster , cheaper and deliver more reliable project deliveries.

The whole concept of an OO development house , an SOA house or a RAD house seems very wrong . Each has their pros and cons for each project and any competent team should be familiar with all techniques and use them where appropriate.

Every project needs to be taken on its own merits and facts like these are FAR more important whether OO is better than RAD or SOA.
- Amount of change /maintenance over the life of the project
- Fixed price contract
- Team skills (Architecting for the team)
- Breaking down the work ( eg SOA service layer and RAD front ends)

SOA 1 class per service Juval Lowy framework

Ben Kloosterman — Sun, 01 Nov 2009 13:12:25 GMT

I posted this a year ago.... I'm really against this make each class a separate service concept. It reminds me of the Microkernel and especially workplace OS but worse. The idea of microkernels was each service ( like memory) was an easily managed entity with carefully defined inputs and outputs which could be consumed.) Workplace OS took this to limit and performance was so poor that no amount of optimization would help. So they had no choice but to scrap the project ( were talking $2B 10 years ago) the bad thing is once such a design is made and implemented just like workplace OS it is very difficult and expensive to fix. Classes by nature frequently communicate with other classes ,serializing everything makes this far to slow. Think about big orders with 10,000 line items each line item could be a separate service call to add up some total and what do you gain ? I consider service to service communication and some sort of nearness analysis a vital part of SOA and grouping related classes into services will help significantly by - Increasing performance - Allow for better maintenance and less change as related changes will deploy together . - Allow for better OO/maintenance and Agile techniques within each service. ( though less outside) - Allows better integration and use of different services such as Java , CRM/ERP services etc. - Encourages chunkier cross service calls and better attention paid to the service interface ( for easier use, compatibility/maintenance and performance) - Discourages the service as a DB store proc wrapper. - Avoids having to put performance hacks in the wrong place to try to avoid cross class calls. This does not mean services should be 20 classes 3 is a good line but 1 to 9 is ok in a number of cases depending on class relationships. Basically if 2 classes communicate all the time they should be in the same service. I can see in 5-15 years hosted environments/frameworks hosting classes which can be moved to machines or be in proc though its far too early for this and you NEED the ability for classes to communicate via normal inproc ( without serialization) . I not you mention this for the future but telling people to use each class as a service now without some more comments on the hazzards ( especially when they don't have a lot of experience in SOA) will lead to a lot of frustration for some .

Obsession with new toys

Ben Kloosterman — Sat, 17 Oct 2009 16:58:24 GMT

I will be returning to eventing/EDA soon ...

There have been a number of posts lately on Windows Workflows 4s break with the past citing continual change and lack of support being a big problem.

Id like to make a few comments on this. The number one productivity killer in .NET today is undoubtedly new technologies and learning how to implement them well. While these are without doubt superior to past products eg Remoting /WCF , WIndows Forms / WPF , DataSets/LINQ for Sql people seem to be making the move because the feel its new and hence will be supported longer.

This is completely wrong thinking. There are still solutions today where remoting and Datasets are the most appropriate technical choice and likely the best choice in terms of skill levels. To this day Microsoft still provides the occasional patches to the system. It is often the case with many systems that exist and are maintained many years after the developers even stop support .. Open Source code for example comes with no guaranteed support at all and many projects die yet still provide useful tools and forums for many years . In comparison Microsoft products have legions of people who can assist and forums are full of work arounds it is far cheaper to apply these to an existing app then rewrte it to take advantage of some new code Microsoft has written. eg if you want ( Not my choice) a distributed object system with tightly bound types Remoting and writing your own security is still a better choice than Windows Communication Foundation.

Often in these cases the main driver is political and a hard excuse to change something, people need to be less afraid of using older code /libraries its very common in the non Microsoft world and has trivial implications for software development. Obviously non supported hardware and shrink wrapped software is a different issue.

Can EDA SOA eventing systems be fast

Ben Kloosterman — Fri, 10 Jul 2009 03:01:03 GMT

A lot of traditional eventing systems use C++ and are dubious about whether the Lego block style SOA can provide sufficient performance.

Again it boils down to design

WCF C# can process over 22000 small messeges per second on an average 2 year old quad server. http://msdn.microsoft.com/en-us/library/bb310550.aspx#wcfperform_topic4.

This is more than sufficient for most applications especially as with eventing systems you can add servers relatively simply , the algorithms which manage the propogation of subscriptions and the path are far more important . ( Distributed Event-Based Systems (Muhl , Fiege and Pietzuch) is a good book which covers the theory here .

The other big issue often sited is latency but EDAM ( which uses Windows Communication Foundation ) has sometimes achieved cross machine rates of 3ms again the design is critical here if you pass through 6 nodes each adding 20 ms it can become quickly unacceptable. Often people will use all the features of SOA such as queues , transactions , encryption etc yet wouldn't implement these on a typical C++ system these can be killers , don't use them unless you HAVE to and since most C++ eventing systems dont you better have a good reason. .

I would also add Network latency between companies , cities or even countries add far more significant delays. With human reaction being about 120ms (At best) the small delays these services add are insignificant.

The end result is you CAN build lego block style SOA , high speed eventing systems with a good a design and gain all the SOA benefits .

You could argue computer trading systems require better times but again i think the small differences you will get ( single figure ms at best ) are more than outweighed by a flexible design. eg instead of running the trades on the clients premises , critical trades can be uploaded to a providers machine near to the exchange this is often hard due to inflexible implementations . A strong and flexible design means more optomization can be made more easily eg message priritization , packing multiple data messeges for efficiency etc

Like premature code optomization the desire to be as fast as possible does not outweigh good progamming algorithms and design.

Ok i said it enough a flexible and strong design is far more important than premature optomization and making things as fast as possible,

Is SSL really secure/too much security

Ben Kloosterman — Sun, 12 Apr 2009 00:47:21 GMT

Is SSL really secure ? By encrypting messages on the wire people get the feeling its secure and people use it ubiquitiously.

Is encrypting on the wire something that really helps ? I don't really think so , with most networks it is VERY difficult and practically impossible to do .

eg
- In a LAN you need physical access to put a device in between reading all traffic from a PC ( in which case you have bigger problems)
- Switched mean a machine tap is not very usefull ( html is not broad cast)
- You need admin access to use a machine to tap if you have this you can install a reader in the browser which logs your actions ( without the ssl)
- Tapping internet links is not possible due to the amount of traffic. Note smaller links could be monitored by a government agency

However

- A simple browser addin via a trojan or admin access can read all SSL !

What about tapping Wireless/GSM , well wireless already has encryption . The encryption is prob harder to break then SSL due to the changing key,

So what do we get for SSL very little. Hence SSL just gives a false sense of security and wastes resources and adds complexity.

This brings me to a point i did some consulting for a number of projects for a company and the company had address information for their clients being sent to mobile phones . It was pointed out that the device only kept the clients the driver visited and it would be a LOT easier and cheaper to just follow the vehicles. We implementede it anyway as the client is always right .

Still i feel people over engineer security ....in the wrong places. I have seen systems with good SSL but laughable or no authentication ...ummm.

Is software theory relevant

Ben Kloosterman — Sun, 05 Apr 2009 08:12:57 GMT

I just realized my main objection to heavy theoretical based development stems not from poor theory but in appropriate use.

I think it comes down to the original premise that 90% of a softwares lifetime budget is in maintenance . While this is certainly true for large scale mainframe applications or software vender apps ( eg IE , Word etc) it is not for the majority of small apps used today. I have seen many of these applications not change for 90% of their lifetime and when replaced they are replaced by the latest and greatest language /technology.

Here is the thing i object to most - Using code that is very heavy in terms of re-use and extendability but is difficult to read , understand and overly complex. You can add to this the issue of developer skill level. Now if i have a team of 10 skilled developers building a 2 year system , its certainly going to be a major system that will last with a budget probably in the 1-4M range . However for the more common typical smaller app with 1-3 people for 1-3 months it is important to follow KISS and keep the initial cost as low as possible.

Considering this scenario makes up the vast amount of development projects ( and most programmers work) its surprising it is neglected by academia , there seems to be very little advice in building small systems as cheap as possible -

Commericial vendors have always embraced this market with VB , FoxPro , Clarion all the 4 GL ( Progress etc) but it seems sadly lacking in Universities and academia mainly because its not that interesting. Instead of that make a graphic rendered in your Comp science course a better project maybe complete a simple 3 form and DB app with some master detail screens , you have 24 hours.. At least they would be more prepared for most workplaces.

The slowness of modern development has also caused frustrations between IT and business specifically business will want an application to try a business venture however after they jump all the hoops and standards the whole startup is likely to not be viable. I have heard many business people talking about a quick win , while ther are many examples i clearly remember a business person being talked into an expensive product , like an Adobe Sharepoint type app where , the business could code business forms in Javascript and they would use the skills of the vendor for the first few form , laughable but you can see they want the quick win. While i knew nothing about the product I knew the organisation had no HTML or Javascript skills and know it will probably end in grief. The lesson here is we need to provide a quick and dirty option for the business more often .

Too many DB's

Ben Kloosterman — Sun, 29 Mar 2009 01:05:00 GMT

There seem to be a trend in using DBs like SQL Express for everything. Is this a good thing ?

For larger systems sure , for small systems im not sure..

The reason why people use these is obvious good tools/Apis and data protection /management.

However for small systems DB's are not that easy to manage , they have to be installed , connection string and security setup and most importantly maintained. Having a corruption in an index is a major issue when the DB is on a client laptop a lot of small apps dont provide Reindex faciities.

Considering these DB's are often less than 50 Meg for cases with a small amount of writes i much prefer , running it all out of memory and using an XML file. You get data integrity by always having a backup file ( eg copy the old file after a successful update).

I think this part of growing trend to apply things designed for large systems too small ones due to the better tools available.

Eventing with and without Topics

Ben Kloosterman — Sat, 21 Feb 2009 12:17:16 GMT

While WS-Eventing has no support for Topics ( see last post) it is almost essential for efficient processing of messages if you have a large amount of subscriptions.

Consider the case of 10,000 Subscriptions for 10,000 Notification you are looking at 10^8 expensive xPath comparisons which will bring many servers down to its knees.

With EDA we can easily solve this problem ( compare to an overloaded DB) .

Lets say we have a Topicless share pricing feed something like

<Trades>
<Trade>
<Symbol>BHP</Symbol>
<Price>38.12</Price>
<Volume>10000</Volume>
<Exchange>ASX</Exchange>
</Trade>
</Trades>

Now WS-Eventing has a really nice feature in that subscriptions can force notifications to contain a header this header can be a Topic. Even better the EndTo can be another WS-Eventing server. With EDAm this is even easier as this subscription can be loaded in the configuration of the Notification server.
So we create subscriptions and organise the data eg by exchange then by group.

What we want is

ShareData
ShareData.Exchange.Group. Symbol.

That way actual clients can subscribe to the specific information they need ( most likely Symbol or Group )

Main Server (where the original feed enters)
Topicless (Default mode)

2nd Server ( Runs in Must have topic and server always loads all subs from Topic)
Subscriptions in the config file ( or Data ) break the data down

Subscription for Exchange add topic

eg Xpath filter = /Trades/Trade/Exchange="ASX"   Header in subscription
<Topics> ShareData.ASX</Topics>

we add one subscription for each exchange
We add subscription for each group and each code.

eg Xpath filter = /Trades/Trade/Exchange/Symbol="BHP"   Header in subscription
<Topics> ShareData.ASX.MetalsAndMining</Topics>

We now have 2 services ( the 2nd one can become many services if the load justifies it ( not needed for a cloud) ) and people can make a subscription based on the Topic and get just the data they require Better yet they can make subscription based on a cheap topic filter which would be orders of magnitude faster than xPath ( and is easier for the engine to index) . .

Lets have a look at the data , lets asume all these subscriptions are group subscriptions ( 20 groups * 500 subscriptions)   , now each Group will have 500 Notifications . The engine now only has to process each of those notifications for 500 subscriptions so we have 20 * 500 * 500 = 5,000.000 operations ( a 20 fold reduction) . O ( n* m) vs O( n * m/log(n) ) . Taking into account the difference in performance between a simple topic filter and Xpath you would be looking at 2-3 orders of magnitude improvement. which could correspond to 1 server instead of 200.

This translates into a simpler , more responsive and more cost effective system. With EDA ( and especially EDAM ) this reconfiguration could be done with no changes in code.

OSLO to be or not

Ben Kloosterman — Thu, 12 Feb 2009 03:05:49 GMT

Based on the limited information so far .

Im not really sure about Oslo ( as opposed to Dublin and Azure which i think will be big hits) it reminds me a LOT of UML and associated higher end tools which manage synch the UML to code and database schemas.

For a start we have to learn a new language ( ok its similar but there are a lot of differences) that immediately has to be a barrier to entry . In return it doesn't make anything easier it just unifies a lot of different thigns under a bigger umbrella which so far i dont think will overcomes peoples natural resistance to change unless it provides some benefit.

However i wonder if Dublin will require some of the Oslo meta data im still waiting to see that.

Is it really worth it ?