LINQ

I am looking at extending LINQ for a current project. I Googled for custom linq and found resources like Charlie Calvert’s Links to LINQ then Introducing Linq to Amazon and then LINQ in Action. Other resources, Bart De Smet’s many detailed posts on LINQ that included LINQ through PowerShell.

Reading LINQ in Action, the authors start with Extension Methods. Which got me thinking about extension methods in PowerShell. Experimenting a while back I created a static class, added a method, used the this syntactical sugar but loading it up PowerShell and trying it did not work.

Continuing research on customization of LINQ led me to Bart’s post Extension Methods in Windows PowerShell. He used PowerShell’s feature called the Extended Type System. With this approach you to create Xml and use the cmdlet Update-TypeData.

You can view the extend types shipped with PowerShell like this

notepad %windir%\system32\WindowsPowerShell\v1.0\types.ps1xml

Here is Bart’s illustration of what you can find in this file and its use.

<Type>
    <Name>System.Array</Name>
    <Members>
        <AliasProperty>
            <Name>Count</Name>
            <ReferencedMemberName>Length</ReferencedMemberName>
        </AliasProperty>
    </Members>
</Type>

PS > $a = "Jimmy", "John"
PS > $a.Count
2

Take a look at Jeffery Snover’s post Hate Add-Member? (PowerShell’s Adaptive Type System to the Rescue). He calls this feature the Adaptive Type System and shows how to extend every object so it can dynamically add Methods and Properties to itself.

Back to LINQ

Bart has a new post outlining one of the most important properties of LINQ: its flexibility.

The potential of LINQ is infinity², the reason for it being the infinite fan-in (LINQ though C#, VB, F#, PowerShell, various transport mechanisms, etc) multiplied by the infinite fan-out (LINQ to SQL, Entities, XML, Objects, DataSets, SharePoint, Active Directory, Amazon, etc).

Bart’s full post Who ever said LINQ predicates need to be Boolean-valued?

If you worked with SilverLight’s DataGrid and LINQ you likely noticed that you cannot select anonymous types in your queries. The application simply freezes. That sucks!

You can find some short term workarounds using LINQ to XML.

However, there is another approach. Microsoft has a small library called DynamicLINQ, which you can read about at http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx

You can download the solution and add Dynamic.cs to your SL project. It will not compile right away complaining about ReaderWriterLock not being available in SilverLight. I just commented those lines out for now. Here is a code snippet that creates a query on a generic list and returns a collection of dynamic data classes.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Animation;
using System.Windows.Shapes;
using System.Linq.Dynamic;

namespace SilverlightApplication1
{
    public partial class Page : UserControl
    {
        public Page()
        {
            InitializeComponent();
            List<Data> list = new List<Data>() { new Data() { Age = 5, Name = “Vasya”}};
            var query = list.AsQueryable().Select(”new(Name as FirstName, Age, \”Male\” as Sex)”);
            grid.ItemsSource = query;
        }
    }

    public class Data
    {
        public string Name { get; set; }
        public int Age { get; set; }
    }
}

So, with this, in addition to what anonymous types let you do, you can also make your projections dynamic.

In the book Programming Collective Intelligence, Toby Segaran uses the movie dataset from GroupLens Research to run distance algorithms against. From there he shows how to surface recommendations.

The dataset contains 100,000 lines with 4 fields separated by tabs. Here is the python code to read and split each line

for line in open(path+’/u.data’):
    (user,movieid,rating,ts)=line.split(’\t’)

Now the PowerShell version

ForEach($record in (Get-Content “u.data”)) {
  $user, $movie, $rating, $ts = $record.Split(‘`t’)
}

Not fast enough

When I run the python code in IronPython it is subsecond.

PowerShell takes 6.

I created a subsecond version for PowerShell by coding a C# LINQ version and compiling it inline to PowerShell.

While this approach requires leaving PowerShell syntax.

It highlights PowerShell’s deep integration possibilities when you hit a roadblock.

Invoke-Inline

Lee Holmes’ “Windows PowerShell Cookbook” has a script Invoke-Inline which provides library support for inline C#. Out of the box it works for .NET 2.0.

Mr. Holmes sent the changes for the compiler parameters to work with .NET 3.5.

The Result

Even with compiling the C# LINQ program inline, reading and parsing the dataset is now a subsecond operation.

Download it here.

Linq To SQL allows you to easily query data sources from a database to create in memory lists and enumerations. A typical query is:

IEnumerable<Person> people = from p in DBContext.Persons orderby p.LastName, p.FirstName select p;

You could then use "people" as your data context in WPF, but it’s very limiting since it’s only an IEnumerable<Person>. You can’t add new items, get an item by index, etc. The best option for WPF data binding would be to have an ObservableCollection<Person>, but you can’t get that from a Linq To SQL query and have the data context preserved. The next best option for WPF binding is BindingList<Person>, but how do you get one of those?

(BindingList<Person>)((IListSource)(from p in DBContext.Persons orderby p.LastName,p.FirstName select p)).GetList();

It takes a lot of casting, but this gives you what you need. This BindingList<Person> is still attached to the DBDataContext behind the scenes, so adding, removing, and updating items in the list works correctly. After manipulating the list, just call DBContext.SubmitChanges() and all changes will propagate to the database.

A colleague concurred this looks like O’Caml syntax for free/bound type variables. Interesting. var q = from i in File.ReadAllLines(@”test.txt”) let s = i.Split(‘ ‘) select new { parts = s }; foreach (var item in q) { Console.WriteLine(item.parts); }  

A colleague concurred this looks like O’Caml syntax for free/bound type variables.

Interesting.

var q = from i in File.ReadAllLines(@”test.txt”)
    let s = i.Split(‘ ‘)
    select new { parts = s };

foreach (var item in q) {
  Console.WriteLine(item.parts);
}

In scenarios where there is a need to retrieve data from a remote service, a programmer is usually faced with an additional task of manually defining a data structure (e.g. a class) that would hold the information returned. For example, data stored in a database table may be retrieved using either data reader or a data adapter and placed in a collection of custom objects, or even a DataSet.

With WCF, an object returned from a service must be serializable. In addition, it should use the new [DataContract] style serialization optimized for WCF, instead of the old fashioned [Serializable] one.

With Visual Studio 2008, the task of creating data structures ready for WCF serialization has been simplified greatly.

Let’s assume we have a User table in our database with a list of users and their information. We will need to add a Link To SQL data class file to our WCF Service project in our Visual Studio solution:

Once the design surface screen comes up, drag User table from Server Explorer onto it to create a class.

Hit “Save” button. Visual Studio automatically creates a corresponding User class ready to be used in your application. However by default the class is not marked with [DataContract] attribute and therefore cannot be used with WCF. To change this setting, first click your mouse on an empty space on the design surface and then adjust the Serialization Mode value in Properties grid from None to Unidirectional.

That’s it. Now you can create a WCF method to retrieve information and return it to a WCF client. Here is an example of such code:

public IList<User> GetUsers()
{
       MyDatabaseDataClassesDataContext db = new MyDatabaseDataClassesDataContext();
       var result = from u in db.Users
                       select u;
       return result.ToList();
}   

via Matt Davey and  Parallel Programming with .NET Consider an application where I’m rendering and writing out to a video file frames of a movie. […] access to my WriteToMovie method must be serialized Parallel Extensions using Future<T> var frames = (from i in Enumerable.Range(0, numberOfFrames) select Future.Create(() => GenerateFrame(i))).ToArray(); foreach (var frame in frames) […]

Microsoft released a new MVC extension for ASP.NET which allows users to build asp.net applications using the MVC framework.

A little about MVC:
The MVC framework enables you to easily implement the model-view-controller (MVC) pattern for Web applications. This pattern lets you separate applications into loosely coupled, pluggable components for application design, processing logic, and display.

To makes things easier, the release was accompany with a serious of nice videos given by Scott Hanselman. Even though this new framework is very new (and experimental I must add), there are some nice thing there that I’m sure will become a part of the next LINQ version.

via Tom Mertens Bart De Smet, founder of the LINQ-SQO project, steps through building LINQ without LINQ HERE. The CodePlex project LINQ-SQO is an implementation of the Standard Query Operators of  LINQ to Objects. This is a great reference video and project for understanding the System.LINQ namespace. PowerShell The video shows Process.GetProcesses() supplemented with extension methods for Where (predicates) […]

Seamless Linq is an extension of Microsoft “Linq to SQL”, which allows relational use of the “update”, “insert” and “delete” operations. Deferred update DataClassesDataContext db = new DataClassesDataContext(); SeamlessDataContext sdb = new SeamlessDataContext(db.Connection.ConnectionString); var q = from d in sdb.Details where d.DetailType.DetailTypeName == “Mouse” […]

LINQPad supports LINQ to objects LINQ to SQL LINQ to XML In fact, everything in C# 3.0 and .NET Framework 3.5. LINQPad is also a learning tool for experimenting with this new technology. MPI.Net .NET wrapper for  MS-MPI implementation to enable C# developers to create High Performance Computing Applications. Needs Microsoft Compute Cluster Pack SDK.

Billed as a lightweight event handling. LINQ to SQL employees this technique in their code generation. Allows you to bypass the targetEvent == null check. Partial methods must be declared within partial classes Partial methods are indicated by the partial modifier Partial methods do not always have a body (well look at this more below) Partial methods […]

Here are two ways to read an RSS feed with PowerShell. The first uses the Xml.Linq namespace and the second uses the Net.WebClient namespace.Both approaches retrieve the RSS feed differently. The Xml.Linq version returns an XDocument while the WebClient creates an XmlDocument.  Notice the final output is different between the scripts. The Xml.Linq returns Strings and […]

Here are two CSV files (comma separated values). The first file contains the host and port of the machines for different testing configurations. The second CSV file contains login information that can be used for each environment from above. I want to join the login information to each environment. There are several ways to do this in PowerShell. […]

Daniel Moth’s Webcast walks through some simple code taking a list of objects, filtering them and produces a different shape for output. He starts with C# 2.0, generics, loops and temporary types. Standard Classes (for the temp types) and the ForEach. Basic day to day plumbing. Next he brings in the System.Linq namespace and refactors. Performing the same tasks as above […]

Erik Meijer’s paper LINQ 2.0: Democratizing the Web. Being presented at the XTech 2007 Conference. LINQ 1.0 side-stepped the impedance mismatch “problem” with something better (monads and monad comprehensions), LINQ 2.0 will sidestep the mapping “problem” with something better (composable and programmable mapping). The other open problems we intend to attack in LINQ 2.0 are distribution and […]

An implementation of LINQ focused on streaming data. Examples include sensor networks, network monitors, financial services or any other push-based systems where the client needs to display or take action on “live” data. Things to look into, via OakLeaf Systems.

As Dinesh points out in this excellent post :

There is really nothing new about this problem - it is the problem of Object Relational Mapping (ORM). The two worlds are rules by languages and run-times that have different semantics. The problem is less acute when you are providing a window to the other through classic API and string query language based ORM components. It is more important when you are truly bridging the two with language integration - that is what LINQ does - with its LINQ to SQL (aka DLinq) component.

He goes on to explore one prominent example, that of null semantics (if a and b are null, in SQL, a ≠ b, whereas in .NET, a == b).

Worth a read especially for those planning their own LINQ implementations.