Archive

Archive for the ‘DLR’ Category

Calling IronPython from C#

September 16, 2011 2 comments

There are a lot of great Python libraries out there, and IronPython makes it really easy to call many of them from .NET. Over the years, IronPython has become easier to embed in your applications, and the DLR that was added in .NET 4 makes it dead simple.

Say you have a Python expression (could be an entire module) in a string variable called “expression” – the code to execute that is this simple:

var engine = IronPython.Hosting.Python.CreateEngine();
var script = engine.CreateScriptSourceFromString(expression);
var scope = engine.CreateScope();
dynamic result = script.Execute(scope);

When you execute that, your result will be whatever you returned from the Python expression. You could return a value or a function defined in Python. If your expression is a Python lambda taking three parameters, from your C# code you can write the following:

dynamic foo = result(a,b,c);

If you need to load additional .NET assemblies to expose them to the IronPython code, just call the following:

engine.Runtime.LoadAssembly(assembly);

What about passing parameters? The scope let’s you pass in a dictionary of parameters. The key to each dictionary entry is the name the parameter will have inside the IronPython scope, and the value is going to be the value of that parameter when script.Execute(scope) is called. To pass a dictionary of parameters, simply do this:

var parameters = new Dictionary<string,object>() {
   { "age", 30 }, { "name", "Vinny" }
}
scope = engine.CreateScope(parameters);
result = script.Execute(scope);

The parameters “age” and “name” will be passed into the scope of the IronPython script being executed.

Suppose you have additional Python modules that you want to call from your embedded IronPython. IronPython ships with quite a bit of the standard library, but your embedded code doesn’t necessarily know how to find it. A call to engine.SetSearchPaths(paths) adds a collection of strings with paths that IronPython should search when executing your code.

var paths = new List<string>();
paths.Add("c:\path\to\my\modules");
engine.SetSearchPaths(paths);

I encourage you explore the options for embedding IronPython in your own applications. The ScriptEngine is quite robust; you can execute string expressions or entire files, in the same AppDomain or in a new one.

Categories: C#, DLR, IronPython

JSON and the DLR

March 1, 2011 1 comment

JavaScript is a dynamic language, and with the DLR, C# can be as well. There have been more than a few times that I’ve wanted to pass JavaScript objects over to my C# code, but my object models didn’t match up, so I couldn’t easily deserialize them for server side processing. With the DLR, this is no longer a problem. We don’t need to modify a C# class to match the JavaScript object model. Instead, use the JavaScriptSerializer that ships with .NET 3.5 to get a Dictionary, then copy those elements into a DynamicObject.

First, our DynamicObject:

public class JsonDynamicObject : DynamicObject
{
	private Dictionary<string,object> properties = new Dictionary<string, object>();
	
	public override bool TryGetMember (GetMemberBinder binder, out object result)
	{
		object value;
		if(properties.TryGetValue(binder.Name, out value)) {
			result = value;
			return true;
		}
		else {
			result = null;
			return false;
		}
	}
	
	public override bool TrySetMember (SetMemberBinder binder, object value)
	{
		if(properties.ContainsKey(binder.Name)) {
			properties[binder.Name] = value;
		}
		else {
			properties.Add(binder.Name, value);
		}
		return true;
	}
	
	public override IEnumerable<string> GetDynamicMemberNames ()
	{
		return properties.Keys;
	}
}

Next, use the JavaScriptSerializer to parse a string of JSON into a Dictionary<string,object>, and build an instance of our JsonDynamicObject from that:

public static JsonDynamicObject Parse(string json) {
	var s = new System.Web.Script.Serialization.JavaScriptSerializer();
	return buildDynamicObject((Dictionary<string,object>)s.DeserializeObject(json));
}

private static JsonDynamicObject buildDynamicObject(Dictionary<string,object> props) {
	if(props != null) {
		JsonDynamicObject dynObj = new JsonDynamicObject();
		foreach(var kvp in props) {
			Dictionary<string, object> subProps = kvp.Value as Dictionary<string, object>;
			if(subProps != null) {
				dynObj.properties.Add(kvp.Key, buildDynamicObject(subProps));
			}
			else {
				dynObj.properties.Add(kvp.Key, kvp.Value);
			}
		}
		return dynObj;
	}
	else {
		return null;
	}

In the end, we have an object that is fully compatible with the DLR and can even be passed on to other dynamic languages, such as IronPython running in the server:

dynamic d = JsonDynamicObject.Parse("{'UserID':23, 'User':{'Name':'Billy', 'Age':28}}");
Console.WriteLine(d.Blah);
Console.WriteLine(d.Test.What);
var scriptEngine = Python.CreateEngine();
var scope = scriptEngine.CreateScope();
scope.SetVariable("jsObj", d);
var source = scriptEngine.CreateScriptSourceFromString("print jsObj.User.Age * 10");
source.Execute(scope);

An object defined in JavaScript passed to C# then on to IronPython all in just a few lines of code!

Categories: DLR, IronPython Tags: , ,

WCF Serialization of DLR dynamic types

January 1, 2011 24 comments

I’m a huge fan of the DLR, as it provides terrific interoperability between C# and dynamic languages like IronPython. To create a C# class that works with the DLR, the easiest thing to do is derive from DynamicObject. One limitation arises when trying to use a dynamic type in a WCF service. Trying to use a DynamicObject-derived type will result in a runtime exception when trying to serialize with WCF’s DataContractSerializer. This class (which fails serialization) was my first attempt:

[DataContract]
public class SerializableDynamicObject : DynamicObject
{
    [DataMember]
    private Dictionary<string, object> props = new Dictionary<string, object>();

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        return props.TryGetValue(binder.Name, out result);
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        if (props.ContainsKey(binder.Name))
            props[binder.Name] = value;
        else
            props.Add(binder.Name, value);
        return true;
    }
}

Trying to serialize an instance of SerializableDynamicObject results in the following exception:

Unhandled Exception: System.Runtime.Serialization.InvalidDataContractException:
Type ‘WCFDynamicObject.SerializableDynamicObject’ cannot inherit from a type that is not marked with DataContractAttribute or SerializableAttribute. Consider marking the base type ‘System.Dynamic.DynamicObject’ with DataContractAttribute or SerializableAttribute, or removing them from the derived type.

We can’t add attributes to DynamicObject, so we have to do this the *slightly* harder way by implementing IDynamicMetaObjectProvider rather than deriving from DynamicObject. The tricky part of this is creating the DynamicMetaObject which handles the evaluation of binding expressions. Luckily the DLR documentation on CodePlex has a great walkthrough for this.

The SerializableDynamicObject contains a dictionary of dynamic members and will serialize properly using WCF’s DataContractSerializer. Use it in place of DynamicObject when you want to be able to pass your dynamic types across WCF service boundaries.

[DataContract]
public class SerializableDynamicObject : IDynamicMetaObjectProvider
{
	[DataMember]
	private IDictionary<string,object> dynamicProperties = new Dictionary<string,object>();
	
	#region IDynamicMetaObjectProvider implementation
	public DynamicMetaObject GetMetaObject (Expression expression)
	{
		return new SerializableDynamicMetaObject(expression, 
			BindingRestrictions.GetInstanceRestriction(expression, this), this);
	}
	#endregion
	
	#region Helper methods for dynamic meta object support
	internal object setValue(string name, object value) 
	{
		dynamicProperties.Add(name, value);
		return value;
	}
	
	internal object getValue(string name) 
	{
		object value;
		if(!dynamicProperties.TryGetValue(name, out value)) {
			value = null;
		}
		return value;
	}
	
	internal IEnumerable<string> getDynamicMemberNames() 
	{
		return dynamicProperties.Keys;
	}
	#endregion
}


public class SerializableDynamicMetaObject : DynamicMetaObject
{
	Type objType;
	
	public SerializableDynamicMetaObject(Expression expression, BindingRestrictions restrictions, object value) 
		: base(expression, restrictions, value) 
	{
		objType = value.GetType();
	}
	
	public override DynamicMetaObject BindGetMember (GetMemberBinder binder)
	{
		var self = this.Expression;
		var dynObj = (SerializableDynamicObject)this.Value;
		var keyExpr = Expression.Constant(binder.Name);
		var getMethod = objType.GetMethod("getValue", BindingFlags.NonPublic | BindingFlags.Instance);
		var target = Expression.Call(Expression.Convert(self, objType),
		                             getMethod,
		                             keyExpr);
		return new DynamicMetaObject(target,
			BindingRestrictions.GetTypeRestriction(self, objType));
	}
	
	public override DynamicMetaObject BindSetMember (SetMemberBinder binder, DynamicMetaObject value)
	{
		var self = this.Expression;
		var keyExpr = Expression.Constant(binder.Name); 
		var valueExpr = Expression.Convert(value.Expression, typeof(object));
		var setMethod = objType.GetMethod("setValue", BindingFlags.NonPublic | BindingFlags.Instance);
		var target = Expression.Call(Expression.Convert(self, objType),
		setMethod, 
		keyExpr, 
		valueExpr);
		return new DynamicMetaObject(target,
			BindingRestrictions.GetTypeRestriction(self, objType));
	}
	
	public override IEnumerable<string> GetDynamicMemberNames ()
	{
		var dynObj = (SerializableDynamicObject)this.Value;
		return dynObj.getDynamicMemberNames();
	}
}

One warning, dynamic members can be anything, meaning at runtime someone could assign a method to one of these fields. If this is possible in your application, you’ll need to ensure any methods assigned to the dynamic type are not serialized. I’m leaving this as an exercise for the reader.

Categories: DLR, WCF Tags: , ,

Comparing method invocation using reflection and dynamic

March 23, 2010 Leave a comment

In C# 4.0, the ‘dynamic’ keyword was added to specify that you don’t want compile time checking – all operations on an instance declared as ‘dynamic’ will be resolved at runtime. This doesn’t add any functionality that wasn’t possible before, as you could use reflection or LCG to accomplish this, but the syntax is much more natural to use and feels like the duck typing you would have in a typical dynamic language.

So what’s the cost in terms of performance? With a little benchmarking, I’ve found that there is a cost to using dynamic, but it’s certainly much faster than invoking methods using reflection. Compiled code, as you would expect, is clearly the fastest way to invoke methods. In my quick little benchmark I was able to invoke the same method about 75 times more often in a two second period using compiled code than using reflection. Compiled is still faster than dynamic – I could invoke the method about 6 times more often in code using compiled invocation than using dynamic. More surprising is the method could be invoked using dynamic about 12 times more often than using reflection.

Here are some numbers from one of my samples – I’m counting the number of times I could invoke the same method in a two second period using all three types of invocation running simultaneously to avoid environmental factors skewing results. Bigger is better – that means the method could be invoked faster.

Compiled: 159,277,840.00 method calls.
Reflection: 2,102,121.00 method calls.
Dynamic: 24,702,097.00 method calls.

What this tells me is whenever I can choose between reflection and dynamic for invoking a method, dynamic is quite likely to perform better.

Here is the source, in case you want to run a similar comparison on your own.

class Program
{
    class MyTest
    {
        public ulong SomeField;

        public void Increment()
        {
            SomeField = checked((ulong)(SomeField + 1));
        }
    }

    static void Main(string[] args)
    {
        // Do any possible setup here so we're only invoking methods during the test.
        MyTest testCompiled = new MyTest();
        MyTest testReflection = new MyTest();
        dynamic testDynamic = new MyTest();
        Type t = typeof(MyTest);
        MethodInfo incrementMethod = t.GetMethod("Increment");

        // Define the ThreadStart for method invocation using compiled, reflection, and dynamic.
        ThreadStart threadStartCompiled = new ThreadStart(() =>
            {
                while (true)
                {
                    testCompiled.Increment();
                }
            });
        ThreadStart threadStartReflection = new ThreadStart(() =>
            {
                while (true)
                {
                    incrementMethod.Invoke(testReflection, null);
                }
            });
        ThreadStart threadStartDynamic = new ThreadStart(() =>
            {
                while (true)
                {
                    testDynamic.Increment();
                }
            });

        // Setup the threads and start them.
        Thread compiledTestThread = new Thread(threadStartCompiled);
        Thread reflectionTestThread = new Thread(threadStartReflection);
        Thread dynamicTestThread = new Thread(threadStartDynamic);
        compiledTestThread.Start();
        reflectionTestThread.Start();
        dynamicTestThread.Start();

        // Wait a couple of seconds to let each thread run then abort them all.
        Thread.Sleep(2000);
        compiledTestThread.Abort();
        reflectionTestThread.Abort();
        dynamicTestThread.Abort();

        // Get the output
        Console.WriteLine("Compiled: {0:n} method calls.", testCompiled.SomeField);
        Console.WriteLine("Reflection: {0:n} method calls.", testReflection.SomeField);
        Console.WriteLine("Dynamic: {0:n} method calls.", testDynamic.SomeField);
    }
}
Categories: DLR

DLR + IBatis – ORM mappings for dynamic objects

March 16, 2010 Leave a comment

I’m a big fan of using an ORM to hide the implementation details of the database from the people writing application logic. A lot of ORM’s depend on the structure of your object to determine what SQL they emit. That can be painful sometimes because you have to design your object model and database to fit the ORM. In the case of objects with dynamic members (like those supported by IronPython, IronRuby, and now the DLR in C# 4.0) the structure of your object isn’t necessarily determined at compile time. Instead, you have a backing collection like a Dictionary for your dynamic types that will get populated at runtime with any dynamic members. That dictionary can be full of more dynamic objects. As you can imagine, trying to persist a dynamic structure like this in a not-so-dynamic relational database can be tricky, but it also gives you some nice advantages. For one thing, you can change your database structure without having to deploy any code changes.

For an example of how to create a backing collection for dynamic types for .NET 4.0, follow the documentation on http://dlr.codeplex.com to derive from DynamicObject or implement IDynamicMetaObjectProvider. For IronPython in .NET 2.0, follow the steps regarding GetBoundMember and SetMemberAfter demonstrated in IronPython in Action for examples of how to do this.

Here’s an example to illustrate. I have a base class that all my entities derive from named DynamicBase. This class has the backing collection for any dynamic members, so if I’m in IronPython or dynamic C#, I can arbitrarily add properties to any class that derives from DynamicBase. In fact, if I didn’t want to derive any classes, I could probably stop with DynamicBase.

IronPython example:

simpleDynamicPerson = DynamicBase()
simpleDynamicPerson.Name = "Bill"
simpleDynamicPerson.Title = "Boss"
simpleDynamicPerson.Height = "6ft"

anotherDynamicPerson = DynamicBase()
anotherDynamicPerson.Name = "Mike"
anotherDynamicPerson.Title = "Assistant"
anotherDynamicPerson.YearsWorking = 4

complexDynamicTeam = DynamicBase()
complexDynamicTeam.Name = "MyTeam"
complexDynamicTeam.Leader = simpleDynamicPerson
complexDynamicTeam.Assistant = anotherDynamicPerson

After running this code, I have a DynamicBase instance, and in its DynamicProperties backing collection it has two entries: “Leader”, which contains the simpleDynamicPerson instance, and “Assistant”, which contains the anotherDynamicPerson instance. The simpleDynamicPerson instance’s DynamicProperties collection has entries for “Name”, “Title” and “Height” that are populated at runtime as well.

There aren’t really any classes to map here. There are just dictionaries of dictionaries that are holding all the data. using the IBatis DataMapper, I could run a statement like this:

Mapper.Instance().Insert("StoreDynamicTeam", complexDynamicTeam)

StoreDynamicTeam needs to be the name of an statement loaded in one of the SqlMap files loaded by SqlMap.config. The ORM mapping is the only thing that needs to know about the structure of the dynamic object, or at least the fields that need to be persisted.

<insert id="StoreDynamicTeam">
BEGIN TRAN
DECLARE @teamID INT
INSERT INTO TEAM (NAME) VALUES (#DynamicProperties.Name#)
SET @teamID = SCOPE_IDENTITY()
INSERT INTO PEOPLE (NAME, TITLE, HEIGHT, YEARSWORKING, TEAM_ID) 
VALUES (#Leader.DynamicProperties.Name#, 
#Leader.DynamicProperties.Title#, 
#Leader.DynamicProperties.Height#, 
#Leader.DynamicProperties.YearsWorking#,
@teamID)
INSERT INTO PEOPLE (NAME, TITLE, HEIGHT, YEARSWORKING, TEAM_ID) 
VALUES (#Assistant.DynamicProperties.Name#, 
#Assistant.DynamicProperties.Title#, 
#Assistant.DynamicProperties.Height#, 
#Assistant.DynamicProperties.YearsWorking#,
@teamID)
COMMIT
</insert>

Getting that data back out isn’t much more difficult.

dynamicPeople = Mapper.Instance().QueryForList("GetDynamicTeam", teamID)

My statement and the resultMap need to deal with the dynamic parameters:

<select id="GetDynamicTeam" parameterClass="int" resultMap="DynamicTeamResult">
SELECT p.NAME AS PERON_NAME, 
p.TITLE AS TITLE, 
p.HEIGHT AS HEIGHT, 
p.YEARSWORKING AS YEARSWORKING
FROM TEAM t
INNER JOIN PEOPLE p ON p.TEAM_ID = t.TEAM_ID
WHERE t.TEAM_ID = #value#
</select>

<resultMap id="DynamicPersonResult" class="MyClassLib.DynamicBase,MyClassLib">
   <result property="DynamicProperties" resultMapping="DynamicPropertiesResult" />
</resultMap>

<resultMap id="DynamicPropertiesResult" class="System.Collections.Generic.Dictionary`2[[System.String,mscorlib], [System.Object,mscorlib]], mscorlib">
   <result property="Name" column="PERSON_NAME" />
   <result property="Title" column="TITLE" />
   <result property="Height" column="HEIGHT" />
   <result property="YearsWorking" column="YEARSWORKING" />
</resultMap>

When you execute it, dynamicPeople will be a collection of DynamicBase objects with their dynamic properties populated. What’s very cool about these properties being dynamic is that you could change your query in your mapping file to add some new fields from some other table, update the resultMap to use those fields, and they’ll show up on your dynamic object.

Categories: DLR Tags: , , ,

Performance of IronPython vs. C#

October 21, 2009 25 comments

I’ve been using IronPython for a while, and every now and then I do a cursory check to see how it’s performs against the equivalent C# code. C# is generally a touch faster at most logic, but IronPython is faster at dynamic invocation of methods than reflection in C#. In general it’s a wash…over the course of an application you dabble in things that different languages are better optimized to handle. When performance is a wash, you get the freedom to choose entirely based on the features of each language, which is nice.

This was going well, until I had a need to do some recursion. I found that the recursive IronPython code was executing extremely slowly – code that was taking milliseconds in C# was taking minutes in IronPython. That’s not a wash anymore…it’s a few orders of magnitude. So to really figure out what’s going on with these recursive calls, I made two implementations of calculating Fibonacci numbers – one in C# and one in IronPython.

Updated 10/23 – I really need to rework this code a bit. The more I look at it, I don’t think I’m comparing apples to apples here, partly because Fibonacci isn’t all that representative of my actual problem where I’m recursing through complex types rather than primitives. After I do a little more research, maybe I can find out what performance metrics to look at when IPy (and dynamic code in general) executes appreciably slower than statically-typed C# code. The problem now is, I can get this code to run without a lot of GC, but it still does a lot of something else that doesn’t seem to show up in perfmon.

IronPython:

def fib(num):
   if num == 0:
      return 0
   elif num == 1:
      return 1
   else:
      return fib(num-1) + fib(num-2)

for i in range(0, 41): print "%i %i" % (i, fib(i))

C#:

class testfib
{
   private static int fib(int num)
   {
      if(num == 0)
         return 0;
      else if(num == 1)
         return 1;
      else
         return fib(num - 1) + fib(num - 2);
   }

   public static void Main(string[] args)
   {
      int i;
      for(i = 0; i < 41; i++)
         System.Console.WriteLine("{0} {1}", i, fib(i));
   }
}

I ran the C# version – 5.5 seconds on average. I ran the IronPython version, 12 2-1/2 minutes. What’s going on here?

Updated 10/23 – Perfmon turned out to tell me very little in compared to some profiling.

Function Time Consumed(units) Hit Count Time Consumed(%)
IronPython.Compiler.PythonScriptCode::Run 5876406555 14 27.7169028
IronPython.Runtime.PythonFunction.FunctionCaller::Call1 5870624528 37860134 27.68963105
IronPython.Runtime.Operations.PythonOps::GetLocal 2794453317 37860205 13.18043438
IronPython.Runtime.Operations.PythonOps::GetVariable 2532912650 37860205 11.94684082
IronPython.Runtime.Binding.PythonBinaryOperationBinder::IntSubtract 607137805 37860099 2.863651344
IronPython.Runtime.Operations.PythonOps::GetGlobalContext 325908336 37860134 1.53719277
IronPython.Runtime.Operations.Int32Ops::Subtract 320598896 37860099 1.512150045
IronPython.Runtime.Operations.PythonOps::GetParentContextFromFunction 315597320 37860134 1.488559404
IronPython.Runtime.Binding.PythonBinaryOperationBinder::IntAdd 282632451 18930035 1.333075937
MS.Win32.HwndSubclass::SubclassWndProc 260663364 4048 1.22945563
MS.Win32.HwndSubclass::DispatcherCallbackOperation 258788088 4048 1.220610625
MS.Win32.HwndWrapper::WndProc 258788088 4048 1.220610625

Compare this to my C# version, which has a lot less going on, since all it’s work is in the single fib method:

Function Time Consumed(units) Hit Count Time Consumed(%)
testfib::Main 1182320483 1 53.01288057
testfib::fib 1047930863 133691744 46.98711938
testfib::.cctor 0 1 0

The information from profiling turns out to be much more useful than the GC numbers I get from perfmon. What’s really going on? First off, IronPython is making millions of calls to PythonFunction.FunctionCaller.Call1. That casts the function to a PythonFunction, then does a second cast to a Func delegate, which then gets invoked.

After that, IronPython is making millions of calls to PythonOps.GetLocal, which calls PythonOps.GetVariable, which goes to the CodeContext and starts trying to lookup the variable in a few dictionaries, which appears to include the global dictionary by the number of calls to PythonOps.GetGlobalContext. The variables are local to the fib() function, but the fib() function itself is in the global context.

What else is in here? Int32Ops.Add and Int32Ops.Subtract are obviously called a lot to find Fibonacci numbers. These both have an overflow check in them in case the result is out of the range of an Int32, in which the result will be a boxed Int64 value. If the result happens to fit in a Int32, it will be performed again, with BigIntegerOps. All of this is to hide the work of handling integer overflow.

All this extra work – the casting, invoking, and dictionary lookups – are all by-products of working with a dynamic language. There isn’t a lot you can do to avoid it, but one thing that can cut down tremendously is to avoid recursion here. With an iterative solution, IronPython doesn’t have to continually lookup the functions to call, and you’re not going to be moving between function scopes so there is no need for looking up variables in dictionaries. The iterative solution in IronPython is much, much faster:

def fib(n):
   previous = -1
   result = 1
   for i in range(0,n+1):
      sum = result + previous
      previous = result
      result = sum
   return result

On with my post, and the GC numbers that are more relevant if recursing through objects, but less relevant to the code I presented with value types.

I fired up perfmon.exe and started adding various counters, but the one where you see the most visible difference is in the .NET CLR Memory, specifically the “# Gen 0 Collections”. In the IronPython version, the generation 0 collections numbered around 4 million 30,000! On my C# implementation, there actually were no collections. Everything was done on the stack. To be fair to IronPython which has to box everything in PythonType, I changed the C# code to box everything in objects, like the following:

public static object fib(object num)

The boxed C# implementation averaged 22 seconds to run, and also had about 12 6,000 generation 0 garbage collections. It’s still nowhere near the 4 million 30,000 collections taken by IronPython.

Two good things I learned:
1) Be careful with recursion in IronPython, as those PythonType objects carry some baggage that can keep the GC pretty busy.
2) Perfmon is a very important part of your toolbox if you’re fixing performance problems in IronPython code, or just .NET in general.

And, just for those Python purists out there, running it in CPython averaged 3 minutes, 18 seconds. Yes, much faster than IronPython, much slower than C#, and I have no idea how to figure out why (I have to get back to you on the CPython numbers). I also made a C version, and it averaged 3.8 6.5 seconds to execute. Obviously no garbage collection, but mysteriously slower than the C# version. Hopefully nobody cares about the C version, but here it is in case you want to try it for yourself:

int fib(int num)
{
   if(num == 0)
      return 0;
   else if(num == 1)
      return 1;
   else
      return fib(num - 1) + fib(num - 2);
}

int main(void)
{
   int i;
   for(i = 0; i < 41; i++)
      printf("%i %i\r\n", i, fib(i));
   return 0;
}