OOM Exceptions Max Addressable Space and Apache.NMS

by jmorris 15. June 2010 17:30

I kept running into OutOfMemoryExceptions in one my Unit Tests that tested some very basic pub/sub functionality with Apache.NMS. Basically, I could create aproximently 1835 consumers before the following exception was thrown:

I was a bit perplexed by this exception, first all because it happened at nearly the exact same number of consumers being created. Secondly, because that really wasn't that high of a number...I was expected failures to be in 100K or even 1M range...

Looking at my code, the problem became clear as day: I was creating a session per consumer.

Why is this bad? Little background on JMS (Java Messaging Service), which of course Apache.NMS is based (dotnet for JMS):

  • Each client should have a single connection
  • Each connection should manage less than 500 sessions
  • Each session is a thread
  • Each session can create n consumers/subscribers

The problem I was having is that I was running out of addressable space when I was creating 1:1 session to consumer. Each thread took up approximently 1MB thread stack * ~2000 session objects => 2GB max addressable memory space on a windows x86 process. Given that I was running a unit test with Resharper in VS2008, which undoubtedly consumes more threads, I reached the threshold at slightly lower than 2K threads.

With a simple refactor, I was able to run a subsquent unit test where 100K consumers were created on a single session!

Tags: ,

Apache.NMS | JMS

A Couple of Useful JSON Viewers

by jmorris 19. May 2010 09:40

I recently stumbled upon this viewer from a comment on another blog: http://jsonviewer.stack.hu/

A quick google search and I found a few more JSON viewers -

  1. A Firefox add-on: https://addons.mozilla.org/en-US/firefox/addon/10869
  2. Another online viewer 'service': http://chris.photobooks.com/json/default.htm
  3. A standalone viewer and a VS2005 plugin:  http://jsonviewer.codeplex.com/

To bad the plugin from codeplex only looks like it supports VS2005 and hasn't been updated in 3 years :(

Tags: , , ,

Using Datatables.net JQuery Plug-in with WCF Services

by jmorris 17. May 2010 13:42

There are several JQuery plug-ins that provide the basic functionality of an html grid: paging, sorting, etc. In essence these plug-ins provide a consolidated display of the data and typically means of which to interact with the data (update, insert, delete, etc). I recently had the opportunity to work with the DataTables.net grid and have been very impressed.

The following post describes the steps I used to make the DataTables grid work in an ASP.NET Webforms environment using a WCF Service as an endpoint. I’ll also go over some of the pitfalls I ran into which required a bit of ‘tweaking’ to get things working smoothly. One cool thing was that for every roadblock, a workaround was quickly discovered. This is huge when investing any time or resources on any third party plug-in or control.

First up, if you are not aware of what the DataTables is, it’s a JQuery Plug-in that written by Allan Jardine. According to the DataTables website:

“DataTables is a plug-in for the jQuery Javascript library. It is a highly flexible tool, based upon the foundations of progressive enhancement, which will add advanced interaction controls to any HTML table. “

It offers a plethora of features including pagination, filtering, sorting, auto loading of data via Ajax, Fully theme-able by CSS, etc. The documentation on the DataTables website is thorough and the quality excellent. There are also forums…good stuff. Additionally DataTables is licensed dually under the GPL v2 License and the BSD license for those who inquire…

Here is an example of Datatables with built in JQuery UI theme support:

The WCF Service
Datatables allows you to either have client-side paging, sorting and searching, which is the default, or you can do it on the server. Doing it on the client is the simplest and easiest way to get up and running with Datatables, however you will quickly find that it leads to larger page sizes and longer initial load times as all of the data must be piped from the server before Datatables is ready.  Performing the actions on the server makes for much a much snappier and scalable UI, but it adds complexity.

In order to use a WCF service with DataTables, you have to both honor the signature that DataTables expects and the data must be returned in a format that DataTables expects. Fortunately this is relatively easy to do; you simple need to a) map the HTTP variables from the GET request and b) generate a JSON string matches a specific format.

The HTTP variables that DataTables will post are as follows:

Note that the websiteId and categoryId are custom variables and that for each column specified, additional variables will be posted for the following fields with a consecutive numerical postfix: bSearchable_0, bSearchable_1, bSearchable_2, etc.

For my example the following WCF service method was used:

Note that I only specified a subset of the parameters in the WCF method that Datatables posts with its GET request:

Too be honest, I do not know why Datatables sends so many (looks like one variable per option per column for most), but from the WCF service method perspective they are optional. Also, note that Datatables uses some form of Hungarian notation to qualify the variables type. While a bit confusing at first, it did help a bit when debugging. The variables that we did specify in the WCF service method signature will be used for sorting, filtering, paging, etc.

Datatables expects to receive a JSON object with a two dimensional array called “aData” from this service. Here is an example of the server output that Datatables will receive:

Where “d” is the wrapper type appended by WCF and from the Datatables website:


Basically, note that the service must correctly compute and return the “iTotalRecords”, “iTotalDisplayRecords” and the data in “aaData”. Also, the length of each data returned must match the DataTables jQuery definition provided on the web page where the data will be displayed.

Here is the complete code listing for the internals of the WCF service method:


Note that depending your architecture, this code may look considerable different. However the steps are same:

  1. Query the datastore based upon the passed in parameters
    1. Calculated paging values
    2. Column to search by
    3. Column to sort by
  2. Create a JSON return result matching the JSON structure defined above.

Another thing to note here is that Datatables does not send column names; it sends the ordinal of the column with respect to the table. In order to sort by a column or filter by a column, you need to resolve the ordinal to a column name of your datastore:


The GetSortFieldFormOrder(int order) returns a string column name based upon the ordinal of the column so that the datastore (in this case via a sproc call) can sort the data by the correct column.

The Convert(object record) method makes an array from the values of properties of an object: each becomes an “aaData” record that is returned and used by Datatables on the client.

Just an architecture note here: in my opinion there is too much “busy” work going on here at the service level. Ideally this will be refactored into your entities so that the behavior is encapsulated outside of your data “delivery”.

The Client Code
The client JQuery code for Datatables can be very simple or rather complex depending upon the formatting and features your situation requires. Below you will see the code that I used and I will explain the major initialization required for using Datatables with a WCF service.


Lines 4-6 I get a reference to the DOM element I will load into using a typical JQuery selector. Let me explain the relevant settings:

A.    I am using the Datatables built in support for JQuery UI Themes
B.    I am setting the “bServerSide” flag to true so that Datatables will use AJAX to get the data source
C.    The URL of the AJAX method to call for loading data
D.    Since I want to set some custom variables to post to my service method and I want to override the default HTTP POST method that Datatables uses for invoking the AJAX method, I am overriding the “fnServerData” callback with a custom implementation.
E.    Datatables uses POST be default, I overridden the AJAX method to use a GET
F.    The results from the server are then passed into the fnCallBack callback and Datatables can now render the results.

The target for the selector is a div element on an ASP.NET UserControl:


   kick it on DotNetKicks.com

Tags: , , ,

Weird Exception Using Spring.NET NMS and ActiveMQ

by jmorris 28. April 2010 15:09

I am doing some testing with Spring.NET NMS and ActiveMQ (via Apache.NMS) and I came across this gem:

Note that the path in exception message does not exist on my local machine?!

Tags: , , ,

SerializationExceptions and DataContractJsonSerializer: XYZ Type is not expected

by jmorris 16. March 2010 12:47

I have been doing a bit of Web code where the state of an object must be persisted per Session, between HttpRequests within an HttpCookie. In the past I have stored data of this type in as an encrypted string of XML in the FormsAuthentication cookie and use XmlSerializer to serialize/de-serialize between the requests and responses, which ahas worked quite nicely. However, in this scenario I decided to persist as encrypted JSON string and use the DataContractJsonSerializer to do the serialization and de-serialization.

The DataContractJsonSerializer works similar to any other serializer, like XmlSerializer, and the code to do so is pretty simple:

However, I quickly ran into an issue while unit testing this code:

System.Runtime.Serialization.SerializationException: Type 'Genesis.Framework.Web.ADAM.Security.AdamPrincipal' with data contract name 'AdamPrincipal:http://schemas.datacontract.org/2004/07/Genesis.Framework.Web.ADAM.Security'is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.

The problem is that while it seems rather obvious that the serializer should be able to simple use the interface definition to perform the serialization/de-serialization, this is simple not the case. You have to supply a list of “known types”, which are simply the implementation of the said interface. The serializer uses the known type(s) to determine which data contract to serialize or de-serialize to: it is the target type.


Alternatively, instead of adding a list of known types to the DataContractJsonSerializer constructor like I did above, you can adorn your types with the KnownTypeAttribute, which has the same effect. In this case I chose _not_ to ‘dirty up’ my types with attributes ;)

Cookies are a Headache - A Cookie Pattern

by jmorris 11. March 2010 23:29
[No text]


Scott Guthrie and Leaky Abstractions

by jmorris 27. January 2010 22:02

Scott Guthrie is well known in the .NET world as being the main architect/designer of the ASP.NET framework. He has a very popular blog which is pretty much a press release blog for all Microsofts Web development technologies: ASP.NET, Silverlight, etc. On top of that he is extremely well respected amongst the .NET community and much of what he say's taken to be absolute "word" on best practices and usage of the Microsoft Web technologies.

One thing about his blog, and which I mentioned above, is that it is less of a blog (commentry, opinion, et al) and more of "press release" platform that the commmunity uses to see what's going on with the various Web technologies. Often times his posts are showcases for the various API's that will soon be released and are very tutorial based:

  1. Here is the feature
  2. This is the problem it solves
  3. This is what it does
  4. This is how you do it

Occasionally he'll post work arounds for various issues that developers are facing with technologies, integration or Visual Studio. However, one thing Guthrie rarely does is give an opinion or directly address the community. That is until this recent post in which he discusses the merits of technological debates and his opinion about them. To be honest I agree with him most wholeheartedly about most of his points, but then he begins to speak specifically about ASP.NET Web Forms and ASP.NET MVC debates and again while I follow him for the most part, at one point he really loses me. This assertion in particluar:

"d) I often find debates around programming model appropriateness and abstractions a little silly. Both Web Forms and MVC are programming web framework abstractions, built on top of a broader framework abstraction, programmed with higher level programming languages, running on top of a execution engine abstraction that itself is running on top of a giant abstraction called an OS.  What you are creating with each is HTML/CSS/JavaScript (all abstractions persisted as text, transmitted over HTTP – another higher level protocol abstraction)." 

What he is referring to here, is really the ongoing debate about whether or not ASP.NET WebForms is a "Leaky Abstraction"...here he is trivializing the argument via hasty generalization: "all things software are abstractions, thus all abstractions are good!" This is simply not true. While abstractions are all around us and for the most part are good, many times the abstraction causes more harm than good and MS has a history of creating these type of abstractions and many long-term developers are leery of such solutions.

To be fair, he does qualify his point with the following statement:

"The interesting question to debate is not whether abstractions are good or not – but rather which abstractions feels most natural to you, and which map best to the requirements/scenarios/developers of your project."

However, I feel that he does this not to clarify his point, but to qualify the agenda of the entire post:  WebForms is a good abstraction! However, after working with WebForms since 2002, classic ASP before that, and ASP.NET MVC now, I (and many others) disagree! I think many parts about WebForms are poor abstractions, in fact I am even leery about some of the newer features in ASP.NET MVC 2, but that's another story.

In fact, most of the poor abstractions that exist with WebForms were put there specifically to "shield" developers from the "complexity" of Web development. For example, in the late 90's there were many, many developers who had never developed anything other than desktop apps. To them, learning another technology such as HTML, CSS, or Javascript was a major blocker in moving from desktop to Web development. WebForms gave them what they thought they wanted: a nice abstraction over HTTP and HTML. Instead of hyperlinks, you got buttons. In place of POST and GET you got postbacks and events. This was all nice and good...for awhile.

However, pretty soon the abstraction became a hinderance. SEO became increasingly important and the ugly HTML the WebForms generated became a hinderance and developers had to clear huge hurdles to get the output clean enough to pass an SEO requirement. If it was possible at all. Additionally, the code behind concept created a great place for developers to combine BL, markup (oh, yes), data access, binding, html behavior (javascript),  hackery, etc... into a single co-dependent layer. Seperation of concerns was rarely (from my experiance) practiced as the framework itself lent itself this collusion of sorts.

In the end you get better and better at simply working around the limitations of the abstraction, instead of learning more about what the abstraction is trying simplify. This is bad in so many ways.

A better way to re-phrase Scott's qoute above would be: "The interesting question to debate is not whether abstractions are good or bad, but if the particular abstraction solves the problem is purports to and whether or not it causes more good than bad. If not, it is a poor abstraction." The debate is not about abstractions, it is about specific abstractions that cause developers more grief than good!


Tags: , ,

Resharper Hides Configuration File Error

by jmorris 21. January 2010 16:14

I just spent a frustrating two hours trying to get a unit test to run via Resharper for nada. The test would start and then exit as if the it was working but instead of showing a green or red icon as expected when the test was finished, it would remain 'grayed' out. I tried the usual: delete bin folders, run solution 'clean' via VS2008, delete the code directory and do a fresh SVN checkout. I even rebooted for nothing: no red, no green all gray:

I finally noticed that I had closed the output window of VS, so I couldn't see the build results. When I opened the output window I saw this exception at the very bottom of the list:

A first chance exception of type 'System.Configuration.ConfigurationErrorsException' occurred in System.Configuration.dll
A first chance exception of type 'System.Configuration.ConfigurationErrorsException' occurred in JetBrains.ReSharper.TaskRunnerFramework.dll
The program '[1012] JetBrains.ReSharper.TaskRunner.exe: Managed' has exited with code 0 (0x0).

What the f#*k? A configuration error was causing the Resharper taskRunner to die quietly? You have to be kdding me! Shouldn't the user be alerted if the software could not recover and had terminate? I mean, yeah it _did_ notify me, but not very explcitly - a pop up would have been nice!

Anyways, issue was found: I commented out the starting element of the ConnectionStrings section of the App.Config and when Resharper process tried to load the config it bombed. Crises averted, now back to work.

Tags: , , ,

So What is Microsoft Trying to Tell Me?

by jmorris 30. December 2009 23:10

This is the error message I woke up to this morning after installing VS 2010 Beta 2 last night:

One would think that an operation that had completed successfully would not be a problem, eh?

Tags: , , , ,

So What is Microsoft Trying to Tell Me?

by jmorris 30. December 2009 14:10

This is the error message I woke up to this morning after installing VS 2010 Beta 2 last night:

One would think that an operation that had completed successfully would not be a problem, eh?

Tags: , , , ,

Jeff Morris

Tag cloud

Month List

Page List