C# Development

C# High performance programming

Ben Kloosterman — Sun, 14 Feb 2010 02:10:25 GMT

I have noticed a number of comparisons with C++ and C# for high performance systems , and the same mistakes are frequently made. I will go over a C# queue which is frequently used in high performance systems.

1. Any linked list system in C# will preform badly unless -

You manage your memory yourself via unsafe pointers OR
You recycle nodes ( which will still not offer great or C++ performance)

The reason for this should be obvious since any new to create a new record is FAR more expensive than enqueing and most locks. Hence very high performance C# datastructures really need to be based on arrays or be unsafe .

2. Structs and Queues are very dangerous

structs can offer excellent performance especially when passed around via ref. however there are a number of issues..

You can pass in structs via ref and retrieve them via out. Note however if you use ,out a queue will reasign the space immediately and hence your data will get over written so this is prob a bad idea.

Also note when using ref with a struct it can have unintended consequences eg

AMethod( ref myStruct)

{

array[0] = mystruct;

}

is almost the same as the method without the ref , the ref will cause an additiona CIL loadobject and hence perform worse.

C# and F# are complementary

Ben Kloosterman — Sun, 11 Oct 2009 08:34:21 GMT

I just had a lot of repetative web work to do and im starting to look at F# , it looks like F# fits really well with C# replacing the batch files / scripts you use and great for proto typing.

Things where C# are better
Lib /UI Development
Writing large assemblies
When you need low level control or unsafe.
When you use TDD

When F# is better.
When you need quick and dirty codes ( utilities)
Screen / Web scraping
AI scripts
Mathematics / financial scripts
Workflow scripts
Frequently changing code.
Project control.

How often with C# did you have to do something really simple like put in a bunch of reg setting (from a reg file) , control nmake/make for daily builds , prototype code to get quick data to test something etc. F# seems to do all these things better. TDD test scripts should work very well with F# as well.

C# for low latency applications

Ben Kloosterman — Thu, 08 Oct 2009 04:31:38 GMT

There is no argument modern languages such as Java and C# allow quicker , safer and easier to maintain applications yest there is still a very large amount of C/C++ applications being written.

A lot of use of C++ is simply for 3 reasons
- A view of better performance
- Maintaining an existing application
- GC pauses
- Environment requires it.

Lets look at these issues

In practice with realistic budgets a Java/ C# applications will tend to have better performance due to better and easier to use collections and optomizations. The main performance difference relates to GC costs though the new Concurrent GC should remove most of these for multi core. The idea its not native is a fallacy cince you can use the Mono AOT compiler to generate x86 compiled code.

GC Pauses are quite valid and is a major obstacle for low latency applications. There are a number of things you can do
- There is nothing stopping you using C# and managing your own memory for key performance critical areas. While unsafe is discouraged C# with some unsafe code is better than C++.
- The new Concurrent GC should reduces pauses by 80%
- Use load balancing SOA/EDA services , this solves the issue for most cases such as Market data . Its worth noting there is a new event which can take the service out of a cluster while it does a collection.

Also Maintaining an existing application ( or staff skill set) is quite valid though the point comes when you are maintaing a Large C#/Java code base and have a few C++ bit left.

Lastly some environments do require it such as Windows device drivers , though i do question why more embedded systems dont use managed languages like Tiny OS ( which uses Java) . After all even operating systems are being built in C# ( Singularity , Midori , Cosmos , MOSA)

Silverlight 3- impressive stuff

Ben Kloosterman — Mon, 06 Apr 2009 09:33:16 GMT

Just saw this MIX Video

http://videos.visitmix.com/MIX09/T40F

Pretty impressive ( and they handle search engines) , if its as easy as it looks it will be great . Will check it out over the next few days. Silverlight has 3 major versions in 18 months mmm , could it be the focus for MS ( with Azure) , kind of tempted to go this way rather than WPF to support other platforms a multiplatform MS ??

Are projects getting more successful

Ben Kloosterman — Thu, 15 Jan 2009 16:11:40 GMT

Im curious at the increase rate of sucess in a number of surveys such as the later CHAOS surveys. Looking in the detail however it appears that projects after peaking in 2000 the amount of time overrun and the amount of features completed have dropped dramatically in the vast majority of projects. Not sure what it means but to me it DOESNT mean software development is getting better if features are getting dropped and there are increasing time blow outs.

Things that come to mind

Standards for success and projects have increasing amounts of over estimation to prevent failure and hence they come on time and on budget. (See should projects have risk contingencies)
Core elements which are critical to the business with good ROI are being prioritized and delivered while less critical features are not.
Less framework creation /coding is resulting in higher success. ( Projects with poor framework progress are/were often cancelled early as they had few deliverables)
How can the time blow out and the project still be considered a success
Projects are smaller. A lot less managers have the stomach for the $100M+ projects in the 90s

It also very interesting surveys with a loose success definition have a much higher rate

http://www.ambysoft.com/surveys/success2007.html

shows success rates as 63-72% vs 34% for Chaos . I think you MUST have a concrete definition to get anything from a survey in terms of project management and Development methodologies otherwise it is too subjective based on the team .

Chaos defines success as “on time, on budget, meeting the spec”,

With looser definitions of success you get some interesting things

"40.9% of respondents considered cancelling a troubled project to be a success. "
The offshore rate of 43% shows how subjective a non standard definition can be ( People will rate external teams harsher than teams they are part off) . I have yet to meet a company that allows offshore firms Agile methods. People often ask these firms fixed price - which no doubt affects quality and perception but the ultimate goal of ROI is likely to be better ( IMHO).

Yet project success and budgets in Agile environments are hard to relate back to the requirements and original plan.

My feeling is projects are getting a bit more succesfull however i also think the cost of new software has gone up massively. This is good in that business are willing to pay it but bad in terms of ultimate benefit and ROI . Part of this is due to higher demands from business while in the past internal applications had a simple grey form now people demand colours , fonts a web interface for access at home , animated indicators etc

Its worth stating the massive improvements in the tools ( where did the time go) . I'm definetly become a believer in Parkinsons law which has pretty big implications for software development

Work expands so as to fill the time available for its completion.

In its generalized form.

The demand upon a resource tends to expand to match the supply of the resource.

In software its more likely that the work expands to fill the time available +0-50% depending on how strong the management is .

Looking further i found this

http://www.focusedperformance.com/articles/ccpm.html pretty spot on in describing the problem.

An aggressive target duration schedule, along with elimination of task due-dates, minimizes impact of
"Parkinson's Law."

Anyway the thoughts are a bit loose in this post .

LINQ for SQL unit testing Part2

Ben Kloosterman — Wed, 14 Jan 2009 16:38:38 GMT

I'm not really sure whether the benefits are worth it unless you need very high quality code and for larger applications. I'm not sure why i feel this and whether this is due to compartmentalizing the LINQ code or whether its just TDD on LINQ code which can be tested via a test DB i suspect its the TDD.

Still storing all the sql in the various repositories is a plus.

}

---------------------------------------------------------------------------------------------------------------------------internal class ProvinceRepositry : RepositryBase<Province>, IProvinceRepositry{public ProvinceRepositry()base(new ProvinceUnitOfWork())public ProvinceRepositry(IUnitOfWork<Province> work)base(work)#region IProvinceRepositry Memberspublic ProvinceCollection GetWorld(string worldname)throw new System.NotImplementedException();public ProvinceCollection GetWorld()throw new System.NotImplementedException();public Province Get(int id)throw new System.NotImplementedException();public void Update(Province prov)Guard.ArgumentNotNull(prov, "PROVINCE"); public void Create(Province prov)Guard.ArgumentNotNull(prov, "PROVINCE"); throw new System.NotImplementedException();

{

}

{

}

{

}

{

}

{

}

{

}

{

}

#endregion

}

-------------------------------------------------------------------------------------------------------------------------

{

}

#region

{

context.SubmitChanges();

}

#endregion

#region

{

context.Dispose();

}

#endregion

}

internal class ProvinceUnitOfWork :UnitOfWorkBase<Province>, IUnitOfWork<Province>DataContext context;public ProvinceUnitOfWork()this.context = DataContextFactory<DataClassesDataContext>.GetThreadScopedDataContext(); ; IUnitofWork<World> Memberspublic override bool ReadOnly { get { return !context.ObjectTrackingEnabled; } set { context.ObjectTrackingEnabled = !value; } }public override void SubmitChanges() IDisposable Memberspublic override void Dispose()internal interface IProvinceRepositry{ProvinceCollection GetWorld(string worldname);ProvinceCollection GetWorld();Province Get(int id);void Update(Province prov);void Create(Province prov);

LINQ for SQL unit testing Part1

Ben Kloosterman — Wed, 14 Jan 2009 16:25:59 GMT

Ian Cooper had some nice articles on how to do unit testing , i have implemented his code and ideas ( shown below) .

See Part 2 for some Business code

{

impl = source;

queryableImpl = impl.AsQueryable();

}

{

}

{

impl.Add(item);

queryableImpl = impl.AsQueryable();

}

{

}

#region

{

}

{

impl.Remove(item);

queryableImpl = impl.AsQueryable();

}

{

impl.RemoveAll(MatchWithContainer);

queryableImpl = impl.AsQueryable();

}

{

}

#endregion

#region

System.Collections.

{

}

#endregion

#region

{

}

{

}

{

}

#endregion

}

public interface IDataSource<T> : IQueryable<T>, IEnumerable<T>,ITable, IQueryProvider

{

}

{

}

public

{

}

interface IUnitOfWork<T> :IDisposableILINQDataSource<T> DataSource{ get; set; } //table// bool ReadOnly { get; set; }void SubmitChanges(); // only call for inserts OR when all work is done

{

unitOfWork = work;

}

{

#region

{

}

{

dataSource =

}

#region

public abstract class UnitOfWorkBase<T> : IUnitOfWork<T>ILINQDataSource<T> dataSource; IUnitofWork<T> Memberspublic ILINQDataSource<T> DataSourcegetreturn dataSource;setvalue ;abstract public bool ReadOnly { get; set; }abstract public void SubmitChanges();#endregion IDisposable Membersabstract public void Dispose();#endregion}public class RepositryBase<T> where T : classprotected IUnitOfWork<T> unitOfWork;public RepositryBase(IUnitOfWork<T> work) Guard.ArgumentNotNull(work, "work");//public IQueryable<T> T//{// get { return gameSettings; }// set { gameSettings = value; }//} }public interface ILINQDataSource<T> : IQueryable<T>, IEnumerable<T> void Add(T item);void Attach(T item);void Remove(T item);void RemoveAll(IEnumerable<T> items);

publicclassMemoryDataSource<T> : ILINQDataSource<T> List<T> impl;IQueryable<T> queryableImpl;public MemoryDataSource(List<T> source)public MemoryDataSource(): this(newList<T>())publicvoid Add(T item)//public IQueryable<S> CreateQuery<S>(Expression expression)//{// return queryableImpl..CreateQuery<S>(expression);//}IEnumerator<T> IEnumerable<T>.GetEnumerator()return impl.GetEnumerator(); ITable<T> Memberspublicvoid Attach(T item)return; publicvoid Remove(T item)publicvoid RemoveAll(IEnumerable<T> items)internalbool MatchWithContainer(T item)return impl.Contains(item); IEnumerable MembersIEnumerator System.Collections.IEnumerable.GetEnumerator()return impl.GetEnumerator(); IQueryable Memberspublic System.Type ElementTypeget { return queryableImpl.ElementType; }publicExpression Expressionget { return queryableImpl.Expression; }publicIQueryProvider Providerget { return queryableImpl.Provider; }

LINQ for SQL and reattaching objects in nTier scenarios

Ben Kloosterman — Wed, 14 Jan 2009 15:00:23 GMT

One of the issues that keeps cropping up is LINQ for SQLs support for nTier scenarios and having to reattach the object to the DataContext.

However this problem is exhibited by all ORMs in nTier scenarios ( though not in web sites) and you basically have 2 choices

- refetch the object from cache/DB and replay all changes to see what has changed then you can craft a custom Update to just change the items changed.

- update all fields in the DB.

While the 2nd method is easier the first is more efficient especially when you consider that it is very likely that the object is still in a cache.

In both cases a row version/timestamp is normally required for concurrency .

ProcessOrder ( Order order)

{

var orderDB = ThreadDataContext.Context; //get dataContext
//Products is the table we draged and dropped in the dbml designer
var _oldOrder = (from _orders in order.Orders

select _order where _order.id = order.id).Single();

//note as you have the old order and new order you can apply some logic , this is much more efficient than doing it in a db

if ( _oldOrder.IsFinalized())

throw new InvalidOperationException("Finalized order cant be changed");

if ( _OldOrder.RowVer != _order.RowVer)

throw new ConcurrencyException("Order already Changed") ;

//update the fields

_oldOrder .Number = order.Number;
_oldOrder .CustomerId = order.CustomerId;

// etc ...

new BLL.OrderProcessor().Process(_oldOrder); //process

}

You could use reflection to automate the copying and then the new order would just overwrite the previous order but this has 2 issues

It would be slow
It would allow the client to trash the servers object by serializing on top of it .

It is also worth noting for medium and large services you are likely to have an internal object module and code to convert from messages to internal code anyway ( see OO with services article)

I think fetching the old order is good business practice anyway and stops issues like a client bug updatting a service and trashing a fulfilled order with a new one . Fetching the old order allows you to uphold one of the SOA tennats/rules that a service should never trust a client .

It is worth noting the same thing needs to be done for member objects or collections and hence its usefull to provide some simple converter classes for each type . The result of this is that services ( that write to a DB) using an ORM tend to have an external message structure different from the internal OO representation even when the OO model and service is quite small. Even though this is a burden it does not introduce a lot of time or any complexity and is a good practice to follow for all but the smallest services anyway. ( Again see OO with services article)

Are Datasets dead

Ben Kloosterman — Wed, 14 Jan 2009 14:01:45 GMT

Can anyone see a use for Datasets these days ? LINQ for SQL seems to have all the bases covered with a much easier and extensible model.

Performance

Performance varies dramatically for simpler queries and updates LINQ2SQL seems to have it , but for more complex queries datasets have an edge . Complex reports are better in datasets but the time for reports to complete is not that relevant as they don't execute so often.

Here are some benchmarks

http://www.codeproject.com/KB/database/linqVsADO.aspx provides a simple query result and shows simple retrieval.

http://visualstudiomagazine.com/features/article.aspx?editorialsid=2372 provides a more complex query which shows lazy loading an area where ORM all have issues but with a large cache these seem to be less in practice.

SELECT DISTINCT P.*
FROM [Products] AS P
   INNER JOIN [Order Details] AS D
      ON P.ProductID = D.ProductID
   INNER JOIN Orders AS O
      ON O.OrderID = D.OrderID
   INNER JOIN Customers AS C
      ON O.CustomerID = C.CustomerID
WHERE C.Country = @Country
ORDER BY P.ProductID

Its worth noting with services and caching LINQ2SQL can gain a significant edge as queries will be simpler.

Flexibility and Maintainability

The programming model is significantly easier for LINQ2SQL. Both have their issues with nTier systems though IMHO datasets have a significant issue with identity inserts and having to reconcile them ( much worse than LINQ where you have to re fetch/attach and re-apply the changes) .

Though both are set based systems the big gain LINQ has is the objects can have methods and additional fields via partial classes greatly improving code quality ,maintainability and flexibility.

Ease of use

Ease of use is similar for simple applications but as application get more complex LINQ2SQL gets a big edge while its clumsy to add a SUM field to a datatable adding something unusual like GMT date requires clumsy code which loops through and inserts the custom rows.

Here is an example

AdventureWorksDataContext AdventureDB = new AdventureWorksDataContext(
"Data Source=ISSY;Initial Catalog=AdventureWorks;Integrated Security=True");
//Products is the table we draged and dropped in the dbml designer
var _data = from _products in AdventureDB.Products
select _products;
dataGridViewLinq.DataSource = _data;
//Thats it!

SqlDataAdapter adapter = new SqlDataAdapter("select * from Production.Products",
"Data Source=ISSY;Initial Catalog=AdventureWorks;Integrated Security=True");
DataSet ds = new DataSet();
int x = adapter.Fill(ds);
dataGridViewADO.DataSource = ds.Tables[0];

Adding a query or altering it requires a new adapter for a Dataset for LINQ2SQL you just change the line

var _data = from _products in AdventureDB.Products
select _products;

Integration with other systems

IMHO ease of use with other packages is probably the most important thing and both systems are comparable but LINQ2SQL works better with the newer technologies.

	Datasets	LINQ2SQL
Windows Communication Foundation	Poor	Good
WPF	Excellent	Excellent
WF	Average	Good
ASP.NET	Excellent	Excellent
Forms	Excellent	Excellent
MS Report Engine	Excellent	Good
Crystal Reports	Excellent	Good

Longevity

Datasets and LINQ for SQL have similar development commitment from Microsoft ( the only one going forward seems the more bloated Entity Framework) however LINQ2SQL being based on POCO's will be compatible with most systems going forwards.

Overall

	Datasets	LINQ2SQL
Performance	Good	Good
Flexibility/Maintenance	Poor	Good
Ease of Use	Good	Excellent
Integration	Good	Good+
Longevity	Average	Good

I can see no real reason to continue using Datasets especially due to the difficult maintanence , ( its IMHO one of the main reasons RAD apps are hard to maintain) . The long term maintenance also makes the move worthwhile and all teams by now should have a sufficient skillset in working with POCO objects. If teams are working with Windows Communication Foundation they should move immediately.

Note im mainly referring to RAD style applications or services large OO type applications should consider EF(Entity Framework) , nHibernate or other auto generalting tools like LLBLGEN

Post by Frans Bouma

Ben Kloosterman — Tue, 13 Jan 2009 13:04:08 GMT

Loved this post so had to emphasize and post it.

"That's not the case. A framework user doesn't give a **** about how many different patterns the framework is using and how great and awesome it's internally and how it will ultimately evolve in the bringer of world peace. A framework user is interested in whether the framework takes care of things the framework user can't afford to spend time on. After all, the framework user can't afford to spend time paid by the customer on plumbing, on writing infrastructure code. The framework user is paid to write code directly related to the problem at hand: the application for the customer. No customer should accept that the team hired for writing the line-of-business application has spend time on writing a report-control or for example a grid control. So why should a customer accept to pay for time spend on other plumbing code? Just because the customer doesn't know any better? If so, isn't that erm... taking advantage of the inability of the customer to judge what the 'software guys' did was correct?

Yes, it's that extreme. The software industry should grow up and realize that it shouldn't be acceptable anymore to write frameworks and framework code, plumbing code etc. paid by the customer, just because the customer doesn't know any better. And we, the software developers, should grow up and accept that the job many of us have isn't to write plumbing code and infrastructure code, but to write code using the plumbing available. If the plumbing at the place where you work is broken, it's very likely that someone will call a plumber to fix it. So why accept that if the plumbing used for a project is broken, one has to fix it themselves? With all the bugs, loss of time, focus and ultimately: money, as a result? "

I dont think its extreme at all , the same attittude in the industry is pervasive ( See where has the productivity gone) . The industry routinely delivers software that is over engineered for the job and the additional code and complexity instead of reducing maintenance actually makes support and maintenance more costly. This also applies to implementing SOA and EDA architectures for organizations which are too small to see the benefit.