Jan 24, 2016

Roslyn Adventures: Metaprogramming with StackExchange.Precompilation

In this article we’ll wrap the StringBuilderInterpolationOptimizer from my previous article into an StackExchange.Precompiltion module, and use it to optimize an existing C# project.

First things first, we start off with an empty console project, to which we add some sample StringBuilder.Append calls, passing interpolated strings as parameters:

image


Now, let's add the StackExchange.Precompilation.Build package to our demo project, and make sure it still builds. This package replaces csc.exe with a custom and hookable compiler, which is basically a wrapper around the roslyn API. Unfortunately, roslyn doesn't expose any hooks for adding custom processing from the command line (yet). Luckily the API let's you do all kinds of crazy stuff, it even comes with a parser for the csc.exe command line arguments...

After this package is successfully installed, you should see the precompiler executable being called instead of csc.exe in the build output window:

image


Next, we need to build the compilation module that can be loaded into the compilation. So we create an empty class library, and install the StackExchange.Precompilation.Metaprogramming package (contains all the required base types).

After we’ve pulled in the SyntaxTree rewriter from the last article, we can implement the hook. The interface we need to implement is ICompileModule. We want to implement the BeforeCompilation method, which allows us to inspect and modify our compilation before any IL is emitted. Inside this method we have full access to all syntax trees in the compilation and their semantic model. Note that CSharpCompilation is immutable, so we have to replace the one in the compilation context with the modified one:

image


All that’s left to do now is to wire up our shiny new compile module, so it's picked up as by the compiler. This is done in the app.config (or web.config for ASP.NET project) file of the console application:

image


The modified build script forwards the project’s config file to the compiler (this is especially useful when it needs to actually precompile razor views, but this is out of scope of this post). Also, be sure to add the precompilation module as a project reference to the console application.

After everything is set up correctly, a quick build and decompilation can verify that the emitted assembly contains optimized string builder calls:

image


The sources for this demo, along with commits reflecting the described steps, can be found on GitHub.

P.S. a useful trick for debugging your compilation module is to add a Debugger.Launch() call to attach your current VS instance to the currently executing compilation.

Oct 19, 2015

Roslyn Adventures: Optimizing StringBuilder string interpolation

C# string interpolation is awesome. But we can make it even more awesome by making it less wasteful. Consider this line of code:
Currently Roslyn emits the following IL for this call:
You see what it does there? Let's translate that back to C#, to make it more obvious:
It allocates another string and possibly even another StringBuilder. The thing is, you wouldn't be using a StringBuilder if you weren't concerned about allocations. My initial idea of how to solve this was FormattableString. So basically something like this:
Unfortunately overload resolution doesn't work in favor of the method accepting the FormattableString whenever there is an overload for the string parameter. So the example above would write STRING: a test 42 to the console. If we could make the overload resolution smarter (e.g. make the compiler create FormattableString instances wherever there's an matching overload accepting FormattableString instead of a string argument) the solution would be as easy as creating Append/AppendLine(FormattableString) extension methods for StringBuilder.

Roslyn to the rescue

Luckily we can use Roslyn to do some metaprogramming magic and work around this. Basically we need to rewrite StringBuilder.Append/AppendLine calls to StringBuilder.AppendFormat:
... in IL speak:
Let's create a reusable Roslyn-based solution that knows how to do that optimization.
We can use the class above to rewrite each SyntaxTree in a CSharpCompilation:
Originally I wanted the optimization to do the same for TextWriter.Write/WriteLine and Console.Write/WriteLine calls, but it turns out that they actually call string.Format internally anyway. So, add that to the list of possible optimizations.

Conclusion

As you've seen by yourself, it's not particulairly difficult to optimize the back and forth between string interpolation and StringBuilder. I really think optimizations like that should be in Roslyn. As long as that's not implemented though, using string interpolation to build big string fragments (like for example HTML...) might be a bit more expensive than you might think.

Stay tuned for my next blog post, where I'll show you how to plug the optimization into the metaprogramming infrastructure of DNX and/or StackExchange.Precompilation, if you're not ready to migrate to vNext yet

Nov 14, 2014

Localization Adventures: Walk the #line

Intro

With Roslyn, C# developers now have a powerful tool which makes modifying the source code a breeze. At Stack Exchange we’ve invented our own little set of extension to C# for localization purposes, previously described on Matt Jibson’s blog. The project originally supported ASP.NET MVC views only, but we’ve expanded it to C# source files (.cs) because our projects have strings that simply don't fit into MVC views and rendering a view for each string is overkill. In this blog series I’ll try to highlight some of the fun things I've learned on this .cshtml ->.cs journey.

Localization Adventures: Walk the #line

With code rewriting being the biggest problem, one can quickly forget to support other things in the development workflow, like debugging. Take ASP.NET MVC’s .cshtml views for example. Although they are compiled to C# source code (you get CodeDOM from Razor, ASP.NET vNext will change that) you can still step through breakpoints in the .cshtml file. In C# we have the #line preprocessor directive that can enables us to do this. As trivial as it might seem, it’s not so easy to get right. The fun part is having more than one statement on a single line. Let’s say we have this simple program fragment that we need to rewrite:

… and the whole thing get’s rewritten into:

The question is, how to place the #line directives to hit a breakpoint on the baz = Baz() statement? A clue lies in the breakpoint itself, which Visual Studio shows as Program.Original.cs, line 7 character 67. As you can see though, there is no way to specify character 67 using the #line directive. To solve the problem we need to be aware of two simple facts the MSDN page forgets to mention:
  • the same line number can be used by multiple #line directives
  • whitespace is important when using the #line directive
Which brings us to the solution:

pics or it didn't happen

That's all for now. Next time in Localization Adventures, be ready to get you hands dirty with some Roslyn.

Sep 27, 2012

Aggregating Shared Google Reader Posts in the Google+ Era

Ever since Google overhauled the reader and removed the original share button I was searching for a way to get to my shared items in the RSS form. None of the online suggested solutions was acceptable for me so I figured, since I already have ASP.NET hosting, that creating a new site, that would tap into the Google+ API and parse the shared posts as RSS would be the way to go. Fortunately that was very easy to accomplish with a blank ASP.NET MVC solution, some NuGet packages for JSON and REST and a bit of API reference browsing. So, behold the code:

 

 

It should be however noted that you need to create a Google API key in their API console.

Now I can finally access my google reader shared items via RSS feed... Again... Hooray!

Apr 4, 2012

ZeroMQ XPUB sockets

Playing around with the new XPUB & XSUB ZeroMQ socket types that allow subscription forwarding, e.g. the producer (XPUB) can track what values he has to produce, which comes in handy when the value production is an expensive operation and the producer cannot afford to produce all values.


Apr 14, 2011

C# gotcha

Consider you have the following code:

SomeClass reference = null;
Console.WriteLine(reference.Echo("echo"));

Your first thought might be “NullReferenceException”. Ha, think again! What if you have an class defining an extension method like this:



public static class SomeClassExt
{
public static string Echo(this SomeClass x, string str)
{
return str;
}
}

You have been warned :)

Mar 9, 2011

Async Producer-Consumer with Linq, IEnumerable<T> and Threading.Tasks

Lately I’ve been dealing a lot with IEnumerable<T> classes and the yield return keyword. I found it really useful for stream like processing of otherwise really memory consuming operations. The workflow looks like this:
- Retrieve Item 1 (yield return)
- Process Item 1 (GetNext())
- ….
- Retrieve Item N
- Process Item N
The total time it takes to process the whole operation is:

N x AvgTime(Retrieve) + N x AvgTime(Process)

So I was thinking how to parallelize the whole operation, preferably in a Linq fashion, so it could fit into my existing code. Here is the result, using the (relatively) new System.Collections.Concurrent namespace and the Task Parallel Library:


using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

static class Extension
{
public static IEnumerable<TResult> SelectAsync<TProduct, TResult>(
this IEnumerable<TProduct> producer,
Func<TProduct, TResult> consumer,
int? productsBoundedCapacity = null,
int? resultsBoundedCapacity = null)
{
var productsQueue = productsBoundedCapacity.HasValue
? new BlockingCollection<TProduct>(productsBoundedCapacity.Value)
: new BlockingCollection<TProduct>();
var resultsQueue = resultsBoundedCapacity.HasValue
? new BlockingCollection<Tuple<int, TResult>>(resultsBoundedCapacity.Value)
: new BlockingCollection<Tuple<int, TResult>>();
Task.Factory.StartNew(() =>
{
foreach (var product in producer)
{
productsQueue.Add(product);
}
productsQueue.CompleteAdding();
});
Task.Factory.StartNew(() =>
{
var tasks = productsQueue
.GetConsumingEnumerable()
.Select((product, ind) => Task.Factory.StartNew(() =>
resultsQueue.Add(new Tuple<int, TResult>(ind, consumer(product)))))
.ToArray();
Task.WaitAll(tasks);
resultsQueue.CompleteAdding();
});
return resultsQueue
.GetConsumingEnumerable()
.OrderBy(x => x.Item1)
.Select(x => x.Item2);
}
}

Note however, that the producer part parallelizes correctly only with the usage of true enumerables that yield return the long running operation result (Arrays, List etc. already contain all the data). Here is a small proof of concept app:



class Program
{
private static IEnumerable<int> Producer()
{
for (int i = 0; i < 100; i++)
{
Thread.Sleep(50); // long running operation
yield return i;
}
}

private static int Consumer(int item)
{
Thread.Sleep(100); // long running operation
return item * item;
}

static void Main()
{
var sw = new Stopwatch();
sw.Start();
var resultSync = Producer().Select(Consumer).Sum();
sw.Stop();
Console.WriteLine("Result: {0}, {1}ms", resultSync, sw.ElapsedMilliseconds);
sw.Reset();

sw.Start();
var resultAsync = Producer().SelectAsync(Consumer).Sum();
sw.Stop();
Console.WriteLine("Result: {0}, {1}ms", resultAsync, sw.ElapsedMilliseconds);
Console.ReadLine();
}
}

See ya!

Feb 10, 2011

MVC – Unit Testing JsonResult Actions

Today I had to write a unit test for a Asp.net MVC controller that returned a JsonResult. The Controller’s code looked somewhat like this:


[HttpPost]
public JsonResult JsonData(Query query)
{
// ... process query, get total_row_cnt and rows
return Json(new { total_row_cnt = rowCount, rows });
}

I often return anonymous types as JsonResult, just because I don’t want to have additional return model classes for JavaScript that are used only by one controller action and nowhere else.


And what about unit testing such actions? Sure JsonResult exposes a property called Data (of type object), that contains the argument passed to the Controller’s Json() call. If the type of the Data is known you can easily cast it to the known type and do further processing. But what if you pass an anonymous type as data to the JsonResult? How can you verify the results then?


Well you have the following options:


  • convert the anonymous type to a class,

  • directly use reflection on the returned object (magic strings!),

  • or make use of C# 4 dynamics!


The first two options are trivial, and I’m going to discuss only the last option. Simply casting the JsonResult.Data won’t do, since the type object does not implement the necessary infrastructure. I decided to implement a wrapper class that implements DynamicObject. Since anonymous types can offer only readonly properties all I had to do was to override the TryGetMember method. So here is the implementation:



using System;
using System.Dynamic;
using System.Reflection;

public sealed class AnonymusDynamicGetWrapper : DynamicObject
{
private readonly Type _subjectType;
private readonly object _subject;
public static dynamic Create(object subject)
{
return new AnonymusDynamicGetWrapper(subject);
}
private AnonymusDynamicGetWrapper(object subject)
{
_subject = subject;
_subjectType = subject.GetType();
}

public override bool TryGetMember(GetMemberBinder binder, out object result)
{
result = null;
var propertyInfo = _subjectType.GetProperty(binder.Name);
if (propertyInfo == null) return false;
var getter = propertyInfo.GetGetMethod();
if (getter == null) return false;
result = getter.Invoke(_subject, null);
return true;
}
}

And finally the unit test method:




[TestMethod]
public void Simple_Data_Test()
{
// ... setup the test, controller and query
var data = controller.JsonData(query).Data;
var rowCnt = AnonymusDynamicGetWrapper.Create(data).total_row_cnt
Assert.AreEqual(25, rowCnt);
}

Feb 4, 2011

Obfuscated Hello World CodeGolf

 

Finally the codegolf Stackexchange site is in beta. So I decided to give it a try, as I found this nice codegolf:

golf

So…. Behold my solution with JavaScript:

javascript:(_=(_=[][(f=!!(_='')+_)[3]+(b=({}+_)[1])+($=(c=(d=!_+_)[1])+d[0])])())[f[1]+'l'+(a=d[3])+$]("H"+a+'ll'+b+' W'+b+c+'ld');

Feel free to try it out. Just paste it in your browser's address bar and let it do the magic.


Well if you would like to know how it works I suggest you read this blog post first (I've actually shared it in my Google Reader feed a month back, so you might have already seen it). The script uses the same principle to acquire the sort() function (and consequently the window object and alert() function) as described there. I also found this nice video that was very helpful.


Good fight, good night :)

Feb 3, 2011

jQuery.validation breaks jQuery 1.5 ajax API

Today I found out (the hard way) that the use of the jQuery.validation plugin breaks the jQuery.ajax API. Since the Asp.net MVC 3 uses jQuery.validation for its unobtrusive validation it is included in the default project template. You can imagine my surprise when I updated jQuery to v1.5, and my heavy ajaxified MVC page stopped working correctly. Every ajax request contained an parseerror with an error message stating that the callback jQuery[some random string] was not called. I tried to fiddle with the ajax settings for JSONP (namely jsonp and jsonpCallback), but no matter how I set them, the error was still present.

error_example
Error message

So I started digging around, trying to replicate the problem. When I ran simple html pages $.ajax worked as expected. Then I started to gradually test all the jQuery plugins and found out that the plugin causing the error was jQuery.validation. So here is my reproduction and a simple workaround (the whole project is available here):

The controller:

using System.Linq;
using System.Web.Mvc;

public partial class DefaultController : Controller
{
  private static readonly int[] numbers = Enumerable.Range(1, 10).ToArray();

  public virtual ActionResult Index()
  {
    return View();
  }

  [HttpPost]
  [ValidateAntiForgeryToken]
  public virtual JsonResult SimpleArray(int id)
  {
    return Json(numbers.Concat(new [] { id }));
  }
}

The view:

@{ Layout = null; }
<!DOCTYPE html>
<html>
<head>
<title>Index</title>
  <script src="/Scripts/jquery-1.5.js" type="text/javascript"></script>
  <script src="/Scripts/jquery.validate.js" type="text/javascript"></script>
</head>
  <body>
  <script type="text/javascript">
    $(function () {
      $('a').click(function (e) {
        var req = $.ajax({
            url: '@Url.Action(MVC.Default.SimpleArray())',
            type: 'POST',
            data: $('form').serializeArray(),
            dataType: 'json'
        });
        req.success(function (response, status, xhr) { alert('Success: ' + response); });
        req.error(function (xhr, error, msg) { alert('Error "' + error + '": ' + msg); });
      });
    });
  </script>
  <a href="javascript:void(0);">Do request</a>
  <form action="#">
    <input type="hidden" value="-1" name="id" />
    @Html.AntiForgeryToken()
  </form>
</body>
</html>

The workaround:

$(function () {
$.ajaxSettings.cache = false;
$.ajaxSettings.jsonp = undefined;
$.ajaxSettings.jsonpCallback = undefined;
})


The cause of the problem is this line of JavaScript in jQuery.validate.js, that overrides the settings you pass into the $.ajax call with all the default ones (and jQuery.ajaxSettings defaults to { jsonp: "callback", jsonpCallback: function() {...}}):


// create settings for compatibility with ajaxSetup 
settings = $.extend(settings, $.extend({}, $.ajaxSettings, settings));

Cheers!

Dec 14, 2010

inJsect – a JavaScript injection utility

Today I was fooling around on how to launch external JavaScript files in web pages. As a result a small utility called inJsect was created. Feel free to fork the code on jsfiddle (BTW great platform).

The final peace of the puzzle is a link that you can bookmark so you can run the tool from the toolbar of your browser (bookmarklet).

inJsect (drag&drop to your toolbar)

I host the code on m0sa.net so everyone can access it. You can find the expanded version of the bookmarklet code here.

inJsect can:

  • inject external JavaScript files into the remote page DOM
  • remove a script tag prior to the injection
  • execute addition JavaScript code after the external code is successfully injected

A pictures is worth a thousand words:

inJsect
Before

inJsect_after

After

Nov 24, 2010

KOTO - KO Tournament Organiser

I recently released my KnockOut Tournament Organiser (short. KOTO) application to the public. The application can be used to help you organise any kind of knockout elimination tournament events (karate, tennis, ...)




Features

  • Intuitive entry form, requiring minimal effort

  • Automatic draw (random, custom)

  • Simple billing system (global entry fee, customizable per category)

  • Draw printing and export to XSLX (customizable XLSX template)

  • Diploma printing (customizable XLSX template)

  • Various printable reports, fully customizable via CSS

  • Automatic version updates
  • The software has been actively developed, maintained and used for the organization of all official karate competitions hosted by the Karate Federation of Slovenia in the past 6 years.

    Requirements: Windows XP +, Microsoft Office 2007 +, .NET Framework 3.5

    If you want to evaluate the application feel free to contact me.

    Apr 15, 2010

    Čakanje na AppFabric RTM

    Kot že veste je 12.4.2010 bil dan-d, dolgo pričakovani datum, na katerega sta izšla .net ogrodje verzije 4 in Visual Studio 2010. Seveda smo pred tem vsi z velikim zagonom uporabljali beta in RC verzije, posebaj po ogledu videjev iz PDC. Tam sem tudi naletel na novo tehnologiji "velocity" in "dublin" združeni v novo platform za okenske strežnike imenovano Windows Server AppFabric.



    Seveda sem v času RC napisal tudi nekaj testnih storitev za AppFabric z uporabo Wf4 in jih uspel namestiti na AppFabric beta 2. Ko sem imel priložnost namestiti VS 2010 RTM, sem to seveda storil. A glej ga zlomka, pri razvojni ekipi za AppFabric so očitno zaspali, tako da zaenkrat lahko samo čakam in se igram z novo VS 2010 RC + AF B2 namestitvijo v virtualcu.


    Še nekaj povezav, za tiste ki neveste o čem govorim, da se vam malo pocedijo sline:



    Jun 2, 2009

    Semantic movie search


    Pri predmetu ekspertni sistemi smo s kolegi naredili zanimiv projekt za sikanje filmov, ki se trenutno predvajajo v slovenskihi kinematografih. S pomočjo xml sporedov Koloseja in Planeta Tuš ter imdb.com smo zgradili sistem, ki enkrat dnevno zgradi semantično mrežo vseh filmov, ki so trenutno na sporedu. Po teh podatkih je potem mogoče povpraševati z lastnimi SPARQL povpraševanji ali pa z že vgrajenimi iskanji.



    Vabljeni k ogledu!

    May 23, 2009

    Dec 19, 2008

    m0sa - mormon?

    Ta izredno zabaven članek me je opredelil kot mormona. Koliko je resnice v tem odločite sami. Naj pa povem, da ne mislim prestopiti v nobeno drugo "religijo".

    Med tem pa si še vedno prepevam odo snihu, katerega pa v Mariboru letos žal ne bom dočakal. Na srečo se nam je uspelo na faksu dogovoriti, da vse obveznosti naslednjega tedna "prestavimo", tako da se sedaj odpravljam domov.

    Dec 6, 2008

    Moje share-ane rss novice iz google readerja

    Za vse, ki vas zanima katere RSS novice so mi še posebaj zanimive, a vas nimam dodanih kot prijatelje na googlovem računu :)

    http://www.google.com/reader/shared/02236268932350252017

    Nov 28, 2008

    BBCode parser kontrola za WPF/Silverlight

    Na strani codeproject.com sem objavil majhno kontrolo, ki vam omogoči definiranj lastnih BBKod (ki jih vsi poznate iz phpBB forumov) in njihov prikaz v Silverlight WPF aplikacijah.

    Nov 20, 2008

    Asp.net Reverse Ajax Chat

    1. Motivacija
    Ker je bil danes tak turoben dan, sem se odločil napisati demonstracijsko aplikacijo, ki uporablja t.i. reverse ajax, s pomočjo tehnologije Asp.net in WCF. Pred kakim letom sem se udeležil poletne šole, kjer smo podoben primerček spisali v Javi. Danes sem se hotel prepričati ali je kaj podobnega mogoče tudi z asp.net. Gre pa zgolj za predstavitev koncepta, programska koda je zgolj provizorična.

    Do kode lahko dostopate preko tega naslova ali kar preko priljubljenega svn klienta:
    Uporabniško ime inu geslo se glasi "blog" (brez narekovajev).

    2. Kaj je Reverse Ajax?
    Predvidevam, da veste kaj je ajax in kako se uporablja. Problem pri klasičnem ajaxu je, da se podatki posredujejo na zahtevo uporabnika. Kot že ime pove pa pri reverse ajaxu gre za obraten koncept, želimo namreč, da se na klientu proži akcija na zahtevo strežnika.

    3. Implementacija
    Najvažnejša stvar pri implementaciji je zanka preko katere se klient prijavi na obvestila, ki jih potem pošilja strežnik. Strežnik za vsakega klienta hrani vrsto obvestil. Zahtevek za novo obvestilo preveri ali so za klienta na voljo kaka obvestila. Če jih ni zahtevek damo v spanje, dokler za klienta ne prispe novo obvestilo. Pri tem je med strežnikom in klientom ves čas odprt XmlHTTPRequest, preko katerega se (takoj ko je na voljo) pošlje novo obvestilo. Ob sprejetju in obdelavi obvestila klient nemudoma zahteva novo obvestilo.
    Spletna storitev - strežnik - izpostavi naslednje metode:

    Tukaj je najbolj pomembna metoda GetNextNotification, preko katere klienti zahtevajo nova obvestila. 

    Slabost tega pristopa je blokiranje niti spletnega strežnika. Ko je teh čakajočih niti več se delovanje slednjega znatno upočasni.

    V izvorni kodi je dodana tudi implementacija ki uporablja t.i. Async Pattern,  kar omogoči lastno asinhrono implementacijo izvajanja zahtevka. Dotična implementacija je kar zanimiva in malce lepše pokomentirana kot "navadna" verzija, tako da priporočam vsem, ki so željni malce bolj poglobljenega znanja, da si kodo tudi ogledajo.


    4. Testiranje
    Dodana je tudi ena majhna javaskripta za obremenjevanje vašega mlinčka. Zaganjal sem jo v po 3-5ih instancah v vsakem brskalniku (Chrome, IE, Firefox, Opera) in je zadeva delovala solidno, vendar ob 100% obremenjenosti CPU-ja. A upoštevati je potrebno, da vsak klient pošilja 1 sporočilo na sekundo, da se za vsako poslano sporočilo aktivira ~15 spečih niti (ker se je sporočilo pošiljalo vsem odprtim instancam), ter da je vse skupaj potrebno pomnožiti še z ~15! Sporočila so letela s tako hitrostjo, da jih IE ni uspel prikazovati in je nekajkrat popolnoma zamrznil.

    5. Zaključek
    Reverse ajax je mogoče implementirati tudi s pomočjo  asp.net tehnologije. Žal je java kar se tega tiče veliko pred asp.net saj obstajajo posebaj za te namene razviti strežniki (recimo putty) ter ogrodja (DWR). Obstajajo pa tudi implementacije s pomočjo Flasha in Java Appletov, kjer komunikacija poteka preko Socketov, ti vtičniki pa ob sprejemu obvestil kličejo javascript na klientovem spletnem brskalniku. Naslednjič se bom najverjetnje lotil iste naloge s pomočjo Silverlight-a in WCF-a z uporabo povratnih klicev. Stay tuned ;)