WCF

WS-Eventing flaw

Ben Kloosterman — Sun, 15 Feb 2009 04:39:47 GMT

Just doing some work on eventing and having a look at a number of implementations there is a pretty annoying feature / flaw .

By default the only filter supported is Xpath and the specification specifically states that

This specification does not constrain notifications because any message MAY be a notification. from http://www.w3.org/Submission/WS-Eventing/.

I have no issues for this - sometimes you want any message , however this is a very expensive way to do this especially when you have a large amount of events with different topics all coming through a single notification service ( load balanced) .

All the implementations i have seen ( including the sample in the specification ) introduce the concept of a topic to route the data.

eg

Sample in spec uses a header

    <ow:EventTopics>weather.report weather.storms</ow:EventTopics>

and a custom filter

 <wse:Filter xmlns:ow="http://www.example.org/oceanwatch"
(43)           Dialect="http://www.example.org/topicFilter" >
(44)         weather.storms
(45)       </wse:Filter>

MSE ( Managed Service Engine from Microsoft)

Uses a custom filter http://services.microsoft.com/2006-07/ServicePlatform/MSE6/Eventing/EventFilter/ which includes topics in the body of their custom event message.

Roman kiss implementation includes a string topic in his subscribe message and has one topic per notification end point.

The end result is everyone will build there own system and filters.

Basically all the authors recognize that the default spec will have performance issues and can be easily optomized ; So why didnt the spec include an optional header on notifications and a simple canonical topic/subtopic filter ? This way the majority of eventing systems can talk to each other without resorting to XPath.

What is Azure Services ?

Ben Kloosterman — Sat, 07 Feb 2009 02:21:46 GMT

Are Azure services a cloud or are they an ESB ?

While Azure does not have this it has FAR more than most ESB in terms of connecting clients to services. Sure it has router support

Azure >> ESB (Internet hosted , with integrated internet security , scalable)
Azure >> Cloud computing ( not just Client - Service routing and hosting but also it has sender-receiver relay , multicast , firewall pass through , and enterprise-cloud security integration)

The big thing is people see it as a cloud .. however i think the biggest thing is it can redirect clients requests into your internal systems ( or partners!) in a secure manner with a global logon. Or facilitate secure communication between clients and companies ( as well as B2B)

A lot of people say the Enterprise will not put services in the cloud - i dispute this. VPNs are used to drive many branch offices which was not done earlier its a question how comfortable they are with the management and security. Like outsourcing to india.

SO Analysis

Ben Kloosterman — Thu, 05 Feb 2009 05:15:17 GMT

Lots of books and posts have been written but IMHO they fail miserably.
- They involve a new language/diagram
- They use propriatry tools
- Focus heavily on BPEL and you need services to drive the BPEL (mmm)
- They are overly complex. and hence violate the most important rule KISS.

I prefer a loose guideline like the following.

Consider the large-grained logical services your systems represent (such as Accounting, Sales, Manufacturing, Design)
Ignoring political / divisional boundaries, consider each large-grained logical service and conceptualize how it should be broken up into finer and finer grained services (Accounting could be factored into Purchasing, Banking, Operational Finance, Expenses, HR Finance, etc)
Repeat 1 and 2 until you have a fairly detailed map of the system under consideration. Remember, Services are fractal - one service decomposes into more services which decompose into more services ...
Now, (again, ignoring political / divisional boundaries) try and identify common shared services - even though they may presently be siloed duplicates. Now consider what your system would look like if they were all consolidated into a single service instance (which of course may publish one or many service interfaces and the associated methods)
After consolidating as much as makes sense, map out the kinds of message exchange patterns between services. One-way (fire and forget), Request-Reply (blocking calls), Full Duplex (bi-directional async). Don't focus on transports or protocols here, talk about the higher order semantics "this link needs to be secure", "this link needs to be reliable", etc.
Refactor services so services that communicate frequently are upscaled eg if you have a Order service and seperate service for OrderItems refactor them into the same service
Now map out the data structures that should be passed between services when actions are requested and the responses when complete.
Now consider what will be your organization's strategy for inter-service communications? Are you homogenous? What is the strategic direction of the platform(s) on which you build systems? If you have to interop, what will be your strategic direction? Remember, whatever lies INSIDE a service is irrelevant to the outside world, so let's not consider service internals here.
Once you know how your services will hang together, you can then start deciding on how to wrap/expose existing systems to the outside world, and where and with whom you want to bet on strategically to deliver the platform(s) for your future needs.
Wrap existing services before writing new ones.

based on Richart Turners original post

Is SOA a Belief system and what does that mean for Team dynamics

Ben Kloosterman — Thu, 05 Feb 2009 03:37:11 GMT

Best SOA posts on the net ( and the comments)

http://blogs.msdn.com/richardt/archive/2005/12/13/503358.aspx

This is what I believe SO is ... it truly is a belief system and a way of thinking. It is not a prescriptive architectural process or methodology. It's not a template that you can apply that results in a service oriented system. It's where art meets science. It's where aesthetics meets engineering. It's the thing adds a human touch to the things we create. It's inside of me and it's inside of you....

james governor said:

i am with anil. a call for an epiphany is kind of hard. you ever read any wittgenstein?

"My propositions are elucidatory in this way: he who understands me finally recognizes them as senseless, when he has climbed out through them, on them, over them. (He must so to speak throw away the ladder, after he has climbed up on it.)"

this kind of intellectual bootstrapping is an issue. also - how many people ever "got" objects. you are making an argument for a priesthood, not a revolution in software productivity.

i see too much of this from the vast IQs at Redmond. you guys need to understand not everyone is as smart as you are, and not everyone writes the kind of beautiful code you aspire too. You build tools for the masses, not for the geniuses eh.

Ben Kloosterman said:

I do agree you need to make the leap , if you have not build distributed systems before ( remoting , Corba/ RMI or web service) you will not know why it is needed. Too many variables to explain which means you will never convince people.

A bit Like TDD in the OO space ( and OO at first) people just have to try it .

I wrote this in 2004 ( 5 years ago now !) and i still believe this, jumping to SOA is a leap and unfortunately some people who have not done distributed apps before have to learn through pain . As James rightly points out though many people nevery made the jump to OO which ALSO requires a leap of faith. With OO the industry has different ideas how high to leap .

That being said i think the best thing is to design for the skills of the team ! A small app is probably best represented by 2 Tier database. A large distr. app or SOA environment is going to have people with all varieties of skills. People with good OO skills can create complicated clients , framework , services etc , people with distributed app experiences can write the contracts , framework and services , VB Procedural programmers can write clients that consume the services. In all cases because SOA services have a SIMPLE interface ( if designed well) and treat the services information as data all people can use it including different languages. In that sense organisations do not need to standardise as much as earlier . Most importantly people dont need to make the LEAP to SOA they can keep doing what they are comfortable with and willing to learn. With Distributed Objects you needed a lot of people to have great OO and distributed app skills a pretty big ask for most organisations. This is a big improvement.

DTO pattern

Ben Kloosterman — Thu, 05 Feb 2009 02:17:21 GMT

http://msdn.microsoft.com/en-us/library/ms978717.aspx
http://martinfowler.com/eaaCatalog/dataTransferObject.html

I still think Martin Fowlers description is best

An object that carries data between processes in order to reduce the number of method calls.

While DTOs were designed for the above they provide key benefits to distributed systems

1.Performance

Typically Objects in OO have quite a chatty interface , this is good for design purposes as these communication mirror the relationships between the objects however the performance implications for distributed systems are significant and worse the worry about performance last mentality in software development makes a significant issue as these issues are VERY expensive to fix as the design from the start is flawed. A simple example is Order and OrderItem while an order with a thousand line items may iterated the Items in memory in a microsecond ( say to addup the total) , in a distributed system with OrderItems in a different service this could take seconds (or more) . Much better to send all the Orders and OrderItems in one operation and then do the work required. While this is an obvious example most real life systems will have MANY issues with such chattiness which are not so easy to deal with.

2.Remove Leaky abstraction from your domain model and hence improve maintenance

Another benefit of DTO's is they preserve your business logic from contamination by the needs of the outside world and hence make the system more easy to maintain. The performance issue mentioned above is one. Another common one is since you maybe communicating with other systems a change may be required in your system which if you are not using DTOs will propogate through the whole system ( eg support for DateTime? instead of Datetime for a Java client) . Its best to have a DTO be responsible for serializing the class ( and deal with the limitations of the transport eg XML). Is your Order class really responsible for its wire representation so why make it so ? Windows Communication Foundation doesn't support generics why limit your domain model by not allowing them , this is a perfect example of the leak abstraction you get with distributed systems - All such frameworks leak and hence you violate the basic OO rules of responsibility. Clearly your domain objects should NOT be responsible for working with the communications framework (whether EJB , Windows Communication Foundation etc) or underlying data ( eg XML)
so why make it so? Also note communication errors can occur in the middle of you business logic code ? Is it the responsibility for this code ? Preserving your domain model as designed will make it easier to maintain your code. Leave the DTOs/Fascade to handle the real world.

3. Appropriateness of Domains

All systems have different ideas of what a "Person" or "Car" is , while a single application domain does not have this issue as much the more clients and services you get the more it becomes an issue. This is especially true of SOA systems. By treating the information from the server as Data ( and Procedures) each application can have a unique view of what a "Person" and "Car" is and hence map its solution closer to its real world model rather than trying to force the real world model for a new part of the system onto the current application. This is not really a DTO benefit but DTO's allow the more completes seperation of logic eg the sever converts the DTO to its logic and the client does likewise.

For example with a client UI by exposing Domain objects to View layer, Views become aware of structure of domain objects, which lets the view make some assumptions about how related objects are available. For example if a domain object "Person" was returned to a view to which it is "bound" and on some other view, "Address" of Person is to be bound, there would be a tendency for Application layer to use a semantic like person.getAddresse() which would fail since at that point Address domain object might have not been loaded at point. In essence ( or require more chatty communication via lazy loading) , with domain objects becoming available to View layers, views can always make assumptions about how data is made available. Also the reverse when domain objects are bound to views (more so in Thick clients), there will always be a tendency for View centric logic to creep inside these objects making them logically corrupt. (note the similarity with the above messaging framework /wire transfer logic creeping in)

Easier to Maintain , better performance so why are DTO''s rarely used in a lot of systems ?

People seem obssessed with using tools and most tools ( and a lot of design methodologies) are focused in an idealized world and like to use the same object throughout the distributed system. However real world practicalities then quickly reduce the quality of the domain model code. Such as focusing with serialization issues etc .

There are issues with use of DTO's also since use of DTO creates additional work in terms of creation of Assemblers (DTO to Domain objects and reverse), Proliferation of analogous objects like Patient domain object, Patient DTO ,PatientDTOConverter ,PatientView ( if not directly bound) . However note each of these Patient domains has a clear roll and the objects themselves are easy to create.

I think DTOs should be used in ALL services unless you can write the service in a few days. Small anemic domains do not require them.

LINQ for SQL thread affinity Data context

Ben Kloosterman — Wed, 14 Jan 2009 16:01:21 GMT

From http://www.cnblogs.com/sunrack/articles/1130866.html , for services we use mainly Thread Context ( the original allows a static context as well ).

Thread context is very usefull for services.

{

TDataContext result = (TDataContext)

SetOptions(result);

}

{

TDataContext result = (TDataContext)

SetOptions(result);

}

{

options.LoadWith<

context.LoadOptions = options;

}

{

}

{

}

{

key =

context =

{

context = GetDataContext();

context = GetDataContext(ConnectionString);

{

threadData =

}

{

}

{

}

{

}

///<summary>/// This class provides several static methods for loading DataContext objects /// in a variety of ways. You can load the data context as normal one new instance/// at a time, or you can choose to use one of the scoped factory methods that/// can scope the DataContext to a WebRequest or a Thread context (in a WinForm app/// for example)./// /// Using scoped variants can be more efficient in some scenarios and allows passing/// a DataContext across multiple otherwise unrelated components so that the change/// context can be shared. ///</summary>publicclassDataContextFactory<TDataContext> where TDataContext : DataContext, new()///<summary>/// Creates a new Data Context for a specific DataContext type with a connection string/// /// Provided merely for compleness sake here - same as new YourDataContext()///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="connectionString"></param>///<returns></returns>publicstatic TDataContext GetDataContext(string connectionString)Type t = typeof(TDataContext);Activator.CreateInstance(t, connectionString);return result; ///<summary>/// Creates a new Data Context for a specific DataContext type/// /// Provided merely for compleness sake here - same as new YourDataContext()publicstatic TDataContext GetDataContext()Type t = typeof(TDataContext);Activator.CreateInstance(t);return result; publicstaticvoid SetOptions(TDataContext context)DataLoadOptions options = newDataLoadOptions();Force>(Force => Force.Characters); //add more rows///<summary>/// Creates a Thread Scoped DataContext object that can be reused./// The DataContext is stored in Thread local storage.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<returns></returns>publicstatic TDataContext GetThreadScopedDataContext()return (TDataContext)GetThreadScopedDataContextInternal(null, null);///<summary>/// Creates a Thread Scoped DataContext object that can be reused./// The DataContext is stored in Thread local storage.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<returns></returns>publicstatic TDataContext GetThreadScopedDataContext(string key)return (TDataContext)GetThreadScopedDataContextInternal(key, null);///<summary>/// Creates a Thread Scoped DataContext object that can be reused./// The DataContext is stored in Thread local storage.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<returns></returns>staticobject GetThreadScopedDataContextInternal( string key, string ConnectionString)if (key == null)"__WRSCDC_" + Thread.CurrentContext.ContextID.ToString();LocalDataStoreSlot threadData = Thread.GetNamedDataSlot(key);object context = null;if (threadData != null)Thread.GetData(threadData);if (context == null)if (ConnectionString == null)elseif (context != null)if (threadData == null)Thread.AllocateNamedDataSlot(key);Thread.SetData(threadData, context);return context;///<summary>/// Returns either Web Request scoped DataContext or a Thread scoped/// request object if not running a Web request (ie. HttpContext.Current)/// is not available.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<param name="ConnectionString"></param>///<returns></returns>publicstatic TDataContext GetScopedDataContext(string key, string connectionString)//if (HttpContext.Current != null)// return (TDataContext)GetWebRequestScopedDataContextInternal(typeof(TDataContext), key, connectionString);return (TDataContext)GetThreadScopedDataContextInternal( key, connectionString);///<summary>/// Returns either Web Request scoped DataContext or a Thread scoped/// request object if not running a Web request (ie. HttpContext.Current)/// is not available.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<returns></returns>publicstatic TDataContext GetScopedDataContext(string key)//if (HttpContext.Current != null)// return (TDataContext)GetWebRequestScopedDataContextInternal(typeof(TDataContext), key, null);return (TDataContext)GetThreadScopedDataContextInternal( key, null);///<summary>/// Returns either Web Request scoped DataContext or a Thread scoped/// request object if not running a Web request (ie. HttpContext.Current)/// is not available.///</summary>///<typeparam name="TDataContext"></typeparam>///<param name="key"></param>///<returns></returns>publicstatic TDataContext GetScopedDataContext()//if (HttpContext.Current != null)// return (TDataContext)GetWebRequestScopedDataContextInternal(typeof(TDataContext), null, null);return (TDataContext)GetThreadScopedDataContextInternal( null, null);

OO with services

Ben Kloosterman — Wed, 14 Jan 2009 15:33:15 GMT

Are OO code and SOA diametrically opposed ? In a lot of ways this is true services tend to produce very anemic object models however this merely reduces reuse which is a myth in OO for the majority anyway .

However services have a unique but familiar problem . The contract of the service is critical it needs to be designed for

- The ease of use of the client

- For performance ( ie chunky calls , many small calls can bring many services down to its needs)

- To be upgradeable and allow backward compatibility.

- Compatible with an XML schema ( eg generics and interfaces create problems)

- Best to be compatible with WS standards and non Windows Communication Foundation clients ( eg nullable datetime , no char , long etc )

- Only sub object and collections needed for this call need to be propagated which is not all the objects and collections a object may contain. eg Order GetOrder(..) may contain OrderItems which contain Products but no more information about the product such as supplier etc . This is best described as truncated trees.

( see how to write a good contract)

This is dramatically different from an OO structure where ease of use is not high on the list else we would all be using statics , however its very similar to a relational database and its structure.

Does this mean services are similar to databases ( eg Procedural Programming) ? From the outside yes and smaller services probably should remain Procedural and non OO . (A good example is in the LINQ and services article)

However larger services should retain a significant OO core with a focus on code quality , testability and maintainability however this requires a reverse ORM which maps from the messages to the internal logic. Microsofts P&P Web service factory includes converterts exactly for this purposes and its a good pattern to follow.

Even in small services when mapping from a DB to messages there is a case for such a conversion layer ( and DTOs) . Take the case of LINQ for sql if you add a member to a DB table for some new functionality meant to be visible to a select group do you want every client to now have this information in the data sent down ? If custom message contracts were used only the new operation and contract used by the select group would get the new information however all the logic would be common ( just the privileged data would not be copied to contract) . If custom message contracts are not used the team would be forced to create a new schema copy of the data for the same table which would be identical except for the one member as can be seen over time this would reduce maintainability of the service .

How to write a good service contracts

Ben Kloosterman — Wed, 14 Jan 2009 15:10:47 GMT

... not hand craft wsdl

Custom messages reduce churn and change in the contract

Lets say you have this datacontract

[DataContract]

public class WorkEntry

{

[DataMember]

public DateTime StartTime;

[DataMember]

public TimeSpan Duration;

[DataMember]

public string User;

[DataMember]

public string CostCentre;

[DataMember]

string Comments;

}

What can go wrong here ? Using public fields is good , its tight and if you need to do custom things you can refactor it easily. I think there is nothing really wrong here however if this is a business object also i would be concerned.

The big gotcha maybe DateTime , lets say you have some other systems in your company that use Java . Now Java systems use a nullable DateTime and can send null ( and DateTime.Min can become null if emit default values is false) causing all sorts of issues . Management decides YOU must change however you have used this as a business object through your app . You can change the DateTime to DateTime? However there are a large amount of implication to your application in now having to deal with possible null values - including the SQL server and DB and hence what is a trivial change requires pretty a lot of work and risk and hence testing ( note TDD or unit testing here do not help and make it worse as the unit tests also have to change) .

The problem here comes from using your DTO ( data transfer object eg message /DataContract) as your business object. I think this should be avoided in all cases where the service is non trivial. In Trivial services its easy enough to write a new service or make the changes and hence using the business object as the message is ok.

Avoid .NET specifics especially

non int integer types
char
generics ( not that it works)
serializing interfaces
inheritance

Ultimately messages are serialized as XML and forcing a object model onto it will always result in difficulties and leaky abstractions for the domain. The solution is simple DONT let WCF and your communications contaminate your object models use the DTO(Data Transfer) pattern.

see DTO pattern.

LINQ for SQL and reattaching objects in nTier scenarios

Ben Kloosterman — Wed, 14 Jan 2009 15:00:23 GMT

One of the issues that keeps cropping up is LINQ for SQLs support for nTier scenarios and having to reattach the object to the DataContext.

However this problem is exhibited by all ORMs in nTier scenarios ( though not in web sites) and you basically have 2 choices

- refetch the object from cache/DB and replay all changes to see what has changed then you can craft a custom Update to just change the items changed.

- update all fields in the DB.

While the 2nd method is easier the first is more efficient especially when you consider that it is very likely that the object is still in a cache.

In both cases a row version/timestamp is normally required for concurrency .

ProcessOrder ( Order order)

{

var orderDB = ThreadDataContext.Context; //get dataContext
//Products is the table we draged and dropped in the dbml designer
var _oldOrder = (from _orders in order.Orders

select _order where _order.id = order.id).Single();

//note as you have the old order and new order you can apply some logic , this is much more efficient than doing it in a db

if ( _oldOrder.IsFinalized())

throw new InvalidOperationException("Finalized order cant be changed");

if ( _OldOrder.RowVer != _order.RowVer)

throw new ConcurrencyException("Order already Changed") ;

//update the fields

_oldOrder .Number = order.Number;
_oldOrder .CustomerId = order.CustomerId;

// etc ...

new BLL.OrderProcessor().Process(_oldOrder); //process

}

You could use reflection to automate the copying and then the new order would just overwrite the previous order but this has 2 issues

It would be slow
It would allow the client to trash the servers object by serializing on top of it .

It is also worth noting for medium and large services you are likely to have an internal object module and code to convert from messages to internal code anyway ( see OO with services article)

I think fetching the old order is good business practice anyway and stops issues like a client bug updatting a service and trashing a fulfilled order with a new one . Fetching the old order allows you to uphold one of the SOA tennats/rules that a service should never trust a client .

It is worth noting the same thing needs to be done for member objects or collections and hence its usefull to provide some simple converter classes for each type . The result of this is that services ( that write to a DB) using an ORM tend to have an external message structure different from the internal OO representation even when the OO model and service is quite small. Even though this is a burden it does not introduce a lot of time or any complexity and is a good practice to follow for all but the smallest services anyway. ( Again see OO with services article)

Simple services and Linq for SQL - match made in heaven

Ben Kloosterman — Wed, 14 Jan 2009 12:22:29 GMT

Im a pretty big fan of Linq for SQL and it works so well with services especially smaller ones ( especially when there are no updates) .

The fact services are normally responsible for about 3 tables /conceptual domain classes mean you can quickly drag the 3 tables onto the designer and generate the code. If the structure changes you just recreate the data.

Lightweight data
Light weight service
Light weight testing
Light weight procedural code. .

These services are quick to write and easy to maintain,