EDABlog

Queues love them or hate them seem to be pretty popular.  I'm definetly on the hate side especially persistant queues

Lets look at some recommendations

• Use private queues. Very few applications need public queues and the performance hit is substantial.
• Use non-transactional queues. Again, very few applications need transactional queues.
• System.Messaging is messed up in a lot of ways, but you can work around that.
• Use the asynchronous methods of System.Messaging (BeginReceive and BeginPeek) where possible.
• Learn to build your own formatter for System.Messaging.

I agree with all these but is this how queues are used ? No.

Queues tend to be used in 3 scenarios

• To capture a critical piece of information ( persistant queues)
  
• To handle a slow legacy system.
• Integrate with non .NET systems

Integration is better handled by SOAP or REST services.

Slow legacy systems can be handled better with Workflows and Windows Communication Foundation throttling. 

Capturing a critical piece of information is better handled by  Workflows or SQL . eg since you need to record the data in a DB anyway, create the data and workflow and if it fails notify the sender. Note Transactions may be used here.

Why not just use queues ?

• Its not simple , it looks simple but a real implementation has a number of complex issues ( Poison message and  performance/scaling especially)
• in scenarios where the the client cant be notified it still not a good idea to use a queue as the code that adds the item to the queue may also fail.
  
• Poison messages / Journalling . This is probably the worst thing - especially when receiving messages from different systems.  A single message can block the queue and your system as effectively as hitting the power switch.  The solution is to move the message to a new queue increasing complexity and the idea of a simple queue. However you now have a message that the system failed to  understand so you need to handle it  and remember the client thinks you have handled it . Often custom systems or manual intervention needs to be done to handle these.
• Persistant message Queues use local disk that needs to be managed. Run out of space/quota due to a large number of transactions or poison messages over time and your queue stops. ie More infrastructure to manage , more things that can go wrong.
• Scaling . Queues do not really scale past the single machine / processor. Most people use persistant/transaction queues which are bound by disk performance of a single machine.  Multi-threaded processing of a queue is possible but requires threadsafe code (which most web or Windows Communication Foundation services do not) . You can use a load balancer and write to multiple services each with their own queue but you break the concept of a single queue ( eg a legacy system  can now receive requests from n services and can become overloaded )  you now have n queues and the queues just become the transport for communicating to each service . Each machine needs to have the queues a and local storage for persistance configured. Lastly poison messages need to be dealt with at each machine.  
  
• Clustering works, but it is difficult to configure correctly and malfunctions often ( well Microsofts one does) . In addition, to run Microsoft's Cluster Server you must also run Windows Server Enterprise Edition which is costly. Note this tends to be a fallover cluster.
  
• Default serialization is custom.
• Many people try to write local and read remote from queues.

Here is a great article on using MSMQ tips http://www.devx.com/enterprise/Article/22314/0/page/1  note the recomendation is to turn of all the things people use queues for.

So what about the alternatives?

For a start we need to consider the alternative which is a plain WCF tcp service. We get security , scaling performance and transactions. We don't get persitence ( though long running services /WF does give you this ) and the concept of capturing as much critical data as possible even if there is a system fault. .

To handle this I prefer some simple code  (and hence reliable)  that stores critical information like orders immediately in a DB (after some basic validation eg not empty , has a customer etc ) and if successful returns this to the client , if not it will tell the client why and an appropriate error  ( which help the client use and like the system) .  Note this code is similar to the code that would put the information on the queue ( this is normally  pretty simple must work code also  ) .

You can then calls a process function via worker thread ( or in larger systems a seperate service can retrieve the record or workflow from the DB and process it this process can be sequential and  or parallel if needed , note DB maybe a persisted workflow) .   This remains simple , reliable and can scale . Best of all records that fail can be processed as normal eg an Order that fails to process is an order in the db and an order management app can probably bring it up allowing you to see it.   ( All apps including queues need to handle bad user data anyway)

Please note while a Workflow is almost ideal and can contain the data i would not do this and store the data seperately . The main reason for this  is Workflows (up to  3.5 , i dont 4.0 )  uses binary serialization which if you change the assembly version can give you a lot of grief in retrieving a workflow. Its better just to have the data seperate so you can resumbit it for processing if needed. If Workflow had an xml persitance representation id probably consider ( especially for smaller apps) just saving the workflow as you could manually retrieve the information in a meaningful format from the DB .