Quick Sample of Password Hashing in C#

August 27, 2012 11 comments

Since it seems like passwords are getting leaked way too often these days, I thought I’d throw a super quick sample out there for anyone with a need to store user passwords in their system.  Some important rules:

1. Don’t use reversible encryption, like 3DES or AES.
2. Always use a SALT for each set of credentials, otherwise you’re open to dictionary attacks should your hashes be compromised.  You’ll want to store this alongside your hash in whatever password store you’re using.
3. Don’t invent your own algorithms – use a proven one written by experts that write them professionally, like PBKDF2.

Some recommended reading on why you should follow these rules: http://crackstation.net/hashing-security.htm

And on to the code sample.  Six lines of C# that work on Windows and Linux / Mono will get you a solid method for generating secure hashes.  Enjoy.  I hope this helps people out and gets more developers to use effective password security measures.

var rng = new System.Security.Cryptography.RNGCryptoServiceProvider();
byte[] salt = new byte[8];
rng.GetBytes (salt); // Create an 8 byte salt
var iterations = 1000; // Choose a value that will perform well given your hardware.
var pbkdf2 = new System.Security.Cryptography.Rfc2898DeriveBytes(pwd, salt, iterations);
var hash = pbkdf2.GetBytes (16); // Get 16 bytes for the hash

Categories: Uncategorized

RabbitMQ Clustering on CentOS 6.2

August 6, 2012 1 comment

RabbitMQ takes a little extra effort to get it working on CentOS 6, particularly if using clustering.  After you’ve installed, you need to do the following before trying to set up the cluster.

Edit /etc/hosts to ensure all cluster nodes can resolve the hostnames of each other.  These hostnames need to match what is displayed after `rabbit@` in rabbitmqctl status.

Next, you need to take care of the ports needed for distributed Erlang processes to communicate. If you don’t do this, then no matter what you do, when you try to create the cluster it will give you ‘error_no_running_cluster_nodes’.

RabbitMQ runs on Erlang, and tor the nodes to talk, they need a few extra ports open beyond the standard 5672 that AMQP uses.  One is for epmd (Erlang Port Mapper Daemon) and that is a standard 4369.  Then you have a port range that Erlang nodes to communicate. You need to define this range so you can open these ports on your firewall.

To tell RabbitMQ to instruct Erlang to communicate on a certain range of ports, create a file at /etc/rabbitmq/rabbitmq.config with the following contents:

{kernel, [{inet_dist_listen_min, 9100},{inet_dist_listen_max, 9105}]}

This will force the nodes in the cluster to only communicate over these ports.  Restart RabbitMQ with `service rabbitmq-server restart` so the change takes effect.  Now for the iptables configuration.  Update /etc/sysconfig/iptables with the following rules:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 5672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4369 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9100:9105 -j ACCEPT

After those changes, reload iptables with `/etc/init.d/iptables restart` so the new rules take effect.  At this point, you should be good to follow the RabbitMQ clustering guide step by step.

Categories: RabbitMQ Tags: , , ,

Closing orphaned RabbitMQ connections

June 29, 2012 Leave a comment

I’ve been working with RabbitMQ a lot lately, and one problem I’ve run into is that occasionally during the course of development, something goes wrong and we leave a lot of connections open.  Enabling the heartbeat will take care of this for the most part, but if someone didn’t do that, you can end up with connections left open on the server.  Here is a quick shell script to close all the connections from a particular IP address:

for c in `rabbitmqctl list_connections peer_address pid | grep $peerToDisconnect`;
if [ $c != $peerToDisconnect ];
then rabbitmqctl close_connection $c "Orphaned - disconnecting";

Categories: RabbitMQ Tags: , , ,

Testing Code with the Mono Csharp REPL

March 21, 2012 2 comments

I’m porting a full enterprise application to Mono so it can be deployed on Linux as requested by customers. I’m using the term “porting” loosely, because really, I just need to verify the existing code will run under Mono and make some minor adjustments so everything runs cross-platform. The Mono csharp read-evaluate-print-loop (REPL) has been an invaluable tool in this process.

Take Active Directory authentication for example. This is one of the areas that the Mono framework is not compatible with the .NET framework. You can see this right away in the csharp REPL, and then work out a quick alternative with the lower level DirectoryServices libraries. Just type ‘csharp’ at the terminal on a Linux system with Mono installed:

csharp> LoadAssembly("System.DirectoryServices.AccountManagement");
error CS0006: Metadata file `System.DirectoryServices.AccountManagement' could not be found

No worries – the AccountManagement API’s aren’t there on Linux – it’s good to be able to find this out right away (you can also check the Mono GAC, but this is REPL school). Now we just need to verify that a lower level call will work:

csharp> LoadAssembly("System.DirectoryServices");
csharp> DirectoryEntry adsEntry = new DirectoryEntry("LDAP://,dc=local", "mydomain\\testUser", "testPa55w0rd");
csharp> var searcher = new DirectorySearcher(adsEntry);
csharp> var res = searcher.FindOne();
csharp> res.Properties;

It’s great to be able to do some quick API checks to understand and isolate compatibility issues right away. Go Mono REPL!

Categories: C#

Getting started with Twisted on Windows

March 10, 2012 Leave a comment

Twisted is an excellent networking framework for python applications, and like most of python, your code is easily portable from one platform to another. As an event-driven framework, you can handle very high loads without creating a large number of threads, so it scales quite nicely. Twisted comes preinstalled on recent versions of Mac OS X, and on Linux distros you can get it with your system’s package manager. Windows is a bit more work, so I’ll go through the steps.

The prerequisites:

Python – http://python.org/download
Setuptools – http://pypi.python.org/pypi/setuptools (needed to install the Zope Interface egg)
Zope Interface – http://pypi.python.org/pypi/zope.interface#download (used by twisted for its service interfaces)
Twisted – http://twistedmatrix.com

1. Install Python from http://python.org/download/. There are Windows installers aplenty.
2. Install setuptools. There is an installer for 32-bit windows. For 64-bit, just download the ez_setup.py locally, and then run “c:\Python27\python.exe c:\Users\me\Downloads\ez_setup.py”
3. Zope is a little tricky. When you installed setuptools, it added the easy_install.exe under your python\scripts directory. To install the python .egg file containing zope, you need to get the URL for the zope egg you want to download and pass that as a parameter to easy_install.exe, like this:

c:\Python27\Scripts\easy_install.exe http://pypi.python.org/packages/2.4/z/zope.interface/zope.interface-3.8.0-py2.4-win32.egg

If all goes well, you should be ready to install and run twisted.

4. Install twisted from http://twistedmatrix.com

At this point, you should be able to fire up python and run the sample from the Twisted home page:

from twisted.internet import protocol, reactor

class Echo(protocol.Protocol):
    def dataReceived(self, data):

class EchoFactory(protocol.Factory):
    def buildProtocol(self, addr):
        return Echo()

reactor.listenTCP(1234, EchoFactory())

I hope this helps you getting started with the excellent twisted network framework for your python apps.

Categories: python, twisted Tags: , ,

Loading XML into MongoDB

December 29, 2011 Leave a comment

I’m starting a new app today and building out the data layer with MongoDB as my database. The app uses a collection from the USDA, that I thought makes a good sample for getting started with the “Load” portion of ETL into MongoDB.

The data is available from the USDA here – the raw XML for MyPyramid: http://explore.data.gov/download/b978-7txq/XML

Step 1 – Define a Class for the data

Although not absolutely necessary as you could build a raw BSON document directly from XML, you kind of miss out on some of the C# driver’s niceties if you do. Looking at the raw data, I came up with this class, along with a constructor that takes an XElement to handle the parsing. Strict DTO people might move that parsing to a function within the ETL process…up to you. The only MongoDB specific code here is the BsonId attribute, which I’ll put on the FoodCode property – a unique ID from the source system.

public class Food {
	public int FoodCode {get;set;}
	public string DisplayName {get;set;}
	public float PortionDefault {get;set;}
	public float PortionAmount {get;set;}
	public string PortionDisplayName {get;set;}
	public float Factor {get;set;}
	public float Increment {get;set;}
	public float Multiplier {get;set;}
	public float Grains {get;set;}
	public float WholeGrains {get;set;}
	public float Vegetables {get;set;}
	public float OrangeVegetables {get;set;}
	public float DarkGreenVegetables {get;set;}
	public float StarchyVegetables {get;set;}
	public float OtherVegetables {get;set;}
	public float Fruits {get;set;}
	public float Milk {get;set;}
	public float Meats {get;set;}
	public float Soy {get;set;}
	public float DryBeansPeas {get;set;}
	public float Oils {get;set;}
	public float SolidFats {get;set;}
	public float AddedSugars {get;set;}
	public float Alcohol {get;set;}
	public float Calories {get;set;}
	public float SaturatedFats {get;set;}
	public Food(XElement elem) {
		this.FoodCode = Int32.Parse(elem.Element("Food_Code").Value);
		this.DisplayName = elem.Element("Display_Name").Value;
		this.PortionDefault = float.Parse (elem.Element("Portion_Default").Value);
		this.PortionAmount = float.Parse (elem.Element("Portion_Amount").Value);
		this.PortionDisplayName = elem.Element("Portion_Display_Name").Value;
		if(elem.Element ("Factor") != null)
			this.Factor = float.Parse (elem.Element("Factor").Value);
		this.Increment = float.Parse (elem.Element("Increment").Value);
		this.Multiplier = float.Parse (elem.Element("Multiplier").Value);
		this.Grains = float.Parse (elem.Element("Grains").Value);
		this.WholeGrains = float.Parse (elem.Element("Whole_Grains").Value);
		this.Vegetables = float.Parse (elem.Element("Vegetables").Value);
		this.OrangeVegetables = float.Parse (elem.Element("Orange_Vegetables").Value);
		this.DarkGreenVegetables = float.Parse (elem.Element("Drkgreen_Vegetables").Value);
		this.StarchyVegetables = float.Parse (elem.Element("Starchy_vegetables").Value);
		this.OtherVegetables = float.Parse (elem.Element("Other_Vegetables").Value);
		this.Fruits = float.Parse (elem.Element("Fruits").Value);
		this.Milk = float.Parse (elem.Element("Milk").Value);
		this.Meats = float.Parse (elem.Element("Meats").Value);
		this.Soy = float.Parse (elem.Element("Soy").Value);
		this.DryBeansPeas = float.Parse (elem.Element("Drybeans_Peas").Value);
		this.Oils = float.Parse (elem.Element("Oils").Value);
		this.SolidFats = float.Parse (elem.Element("Solid_Fats").Value);
		this.AddedSugars = float.Parse (elem.Element("Added_Sugars").Value);
		this.Alcohol = float.Parse (elem.Element("Alcohol").Value);
		this.Calories = float.Parse (elem.Element("Calories").Value);
		this.SaturatedFats = float.Parse (elem.Element("Saturated_Fats").Value);

You might notice I’m using float for my decimal values. That’s all the accuracy I need, but it does lose some precision. I’m rounding the data when I use it so it won’t really matter, but if your needs differ, choose a different numeric type.

Step 2 – Function for reading the XML file

This is a pretty small data file, only about 750 records, but loading it all into memory at once is a waste. I want to load the “Food_Display_Row” XML elements one at a time, convert to a Food object, store in MongoDB, and move on to the next. It’s a job for a streaming API and an iterator, powered by “yield return” to get one XElement at a time:

static IEnumerable<XElement> readElementStream(string fileName, string elementName) {
	using(var reader = XmlReader.Create(fileName)) {
		while(reader.Read()) {
			if(reader.NodeType == XmlNodeType.Element && reader.Name == elementName) {
				var e = XElement.ReadFrom (reader) as XElement;
				yield return e;
		reader.Close ();

Step 3 – Pull it all together and load the data

With the pieces in place, the load process is pretty simple. Connect to the server, get the database (MongoDB creates it on first use), get the collection (MongoDB also creates the collection), and use the iterator to read the XML file, load each element into a Food object and insert into the MongoDB collection. At the end, we have a MongoDB database with a collection of data from the food guide pyramid.

var server = MongoServer.Create ("mongodb://localhost");
var db = server.GetDatabase("gov");
var foods = db.GetCollection<Food>("food");

foreach(var elem in readElementStream("~/Downloads/MyFoodapediaData/Food_Display_Table.xml", "Food_Display_Row")) {
	var food = new Food(elem);
	foods.Insert (food);

My favorite thing about this is that I never had to leave C# to create the database, parse the source XML, or load the data. I don’t have to run a separate ETL process or use management tools to configure my database schema. It’s a simple, self-contained solution.

My second favorite part is that I ran all of this under Ubuntu and Mono. It should work just as well under Windows and .NET, but life is better running under an open source software stack.

I hope you find this helpful if you’re getting started with MongoDB and want a little data to play with.

Categories: C#, MongoDB Tags: ,

If C# is so awesome, why use anything else?

December 29, 2011 20 comments

Anyone who knows me professionally knows I work in C# most of the time. I think it’s a great language that’s been well designed and made very portable by way of being an open language specification. A lot of people look at C# and say, that’s just Java with some Microsoft-extensions. Sort of, since it’s framework (.NET) ships with quite a few libraries that interoperate well with Windows, although the C# language itself doesn’t have anything to do with Windows, and runs on Linux, OS X, Android, iOS, and so on. In my opinion, Java has stagnated over the years, while C# has been evolving with generics (which Java followed), lambdas (Java finally gets them years later), anonymous types, partial method and class declarations, language integrated query (LINQ), dynamic runtime integration, and soon a simplified asynchronous programming model with await and async that will allow the runtime to deal with the gory details of async programming rather than forcing the programmer to understand and properly implement callbacks and cleanup. Java isn’t catching up fast, so Scala is filling the gaps, but C# remains years ahead.

Every year, my family looks at me funny when I they give me new books. That’s right, I’m a geek that reads computer books. Most of these books are not on C#, but on JavaScript, Python, Haskell, and I even keep an old PERL book on my shelf. What is all this other stuff?

JavaScript – it’s pretty rare these days that other developers would say, “why would you ever want to write JavaScript?” It’s a ubiquitous language amongst web browsers, and it’s pretty rare that anyone can write much of a browser-based application at all without it. Besides, the latest trend is to write a “language X to JavaScript converter” and what good would that be if I didn’t know JavaScript and wasn’t willing to learn language X? There have always been some nice server-side implementations, like Spidermonkey and newer V8, powering trendy applications like MongoDB and Node.js. Until I started down the Python path, whenever I needed extensibility, I would embed Spidermonkey for some JavaScript fun.

Python – in the realm of C#, a lot of people are uncomfortable mixing in Python. They don’t like the idea of losing compile-time checks and worry about needing a myriad of Python frameworks to solve any sizable development tasks. However, Python is an excellent tool for large and small projects alike, and IronPython take the Python language and gives it access to the full .NET framework. In the last few years, I’ve felt constrained if I didn’t have a layer of extensibility that IronPython can add to CLR applications. Python scripts let you treat code as data, meaning you can store it, transmit it, and change it at runtime. Python gives you a new way to move the problem around, solve it at a different time in your overall solution. It’s a great piece of the toolbox.

I remember spending weeks building business rules engines so non-programmers could add some logic to enterprise applications. These engines would use reflection and Lightweight Code Generation (LCG) and a clumsy UI where end users would select data objects and operators and build expression statements. IronPython uses LCG, is highly optimized, and gives you a general purpose scripting language with access to CLR objects. Most end users prefer the ability to write an expression in script rather than fumble with the type of UI needed to build an expression tree. This is just scratching the surface of the Python language, but at the very least, it’s a great tool anywhere you want to offer runtime extensibility.

Haskell is pure functional programming – no state, just functions. I used F# a bit for professional work just to learn it, but it allows you to fall back into the OOP line of thinking. Haskell makes you take a fully functional approach. I recommend every developer that’s looking to expand their approach to problem solving to spend some time with it.

What about that PERL 5 book? Well, I don’t use that, to be honest. I did once upon a time, but I really do avoid PERL at all costs. Maybe one day, I’ll pick it back up.

Categories: C#, IronPython Tags: , ,