Here’s the first player, our code.

    % ls Repository/trunk/clementine/interfaces/gesture
    CLGestureCue.cxx           CLGestureCueAppearance.h   CLGestureListener.cxx      CLGestureParser.h
    CLGestureCue.h             CLGestureInterface.cxx     CLGestureListener.h        CLGestureTrainer.cxx
    CLGestureCueAppearance.cxx CLGestureInterface.h       CLGestureParser.cxx        CLGestureTrainer.h

    % cat CLGestureListener.h
    #ifndef __CLEMENTINE_INTERFACES_GESTURE_CLGESTURELISTENER_H__
    #define __CLEMENTINE_INTERFACES_GESTURE_CLGESTURELISTENER_H__

    #include "CLGestureParser.h"
    #include <string>

    /**
     * CLGestureListener is responsible for listening on the CLGestureInterface
     * for a single gesture type.
     */
    class CLGestureListener
    {
            public:
                    /**
                     * Instantiate the listener.
                     * @param gestureName the name of the gesture
                     * (and consequently, the filename to parse) that this object is created to listen for.
                     */
                    CLGestureListener(std::string gestureName);
    ... etc

It’s a whole bunch of header and implementation files for a C++ project I’m involved with. This project requires the use of thorough documentation, but our design documents are hundreds of revisions behind our code. While it’s important to have gone through initial designs, our code at this point is miles removed from those designs. It would be nice to be able to automagically update our design document to be kept current on what is occurring in the code. This will not make up our entire design document. It is meant to be used in conjunction with manual, hand-written and hand-proofed analysis of the larger architecture.

The second player is AsciiDoc, a wonderful formatting and markup system. You may know it from the Git User’s Manual. AsciiDoc affords us several advantages.

• It is in plain text, which means it can sit in source control, right alongside our code, and diffs are easily viewable.
• It is a templating language, which allows output into HTML, PDF, you name it. With enough hacking, you can produce really distinct output for webpages and PDF readers.
• It allows division of sections into different files, so authors can focus on the sections that concern them without being overwhelmed by the text they’re editing.

Act Two: The Tools

We want the comments in our code to be part of the documentation, but that’s not all. Since UML is the standard when it comes to software engineering description, it would be nice to have UML diagrams in our output as well. The ubiquitous free diagramming tool Dia provides, with its invocation, command line arguments to convert Dia diagrams into images. It goes like this, where you wish the output to be image_name.png:

    dia -t png -e image_name.png diagram_name.dia

So this part will be simple. Generating the Dia diagrams from code is already taken care of for us. Aaron Trevena has written an excellent script called AutoDia which provides this functionality. All we need to do is put these pieces together, along with writing the functionality to extract the comments we want to become documentation.

Act Three: The Program

I am calling it Amorfus, because I think client-side applications should start getting into the Web 2.0 naming crazes.

How do you use it?

1. Create a new CommentDocParser, with these parameters: a string pointing to the subdirectory of your trunk that you wish to document, a string containing the conceptual name of the code in that directory, and a string containing your trunk directory. You can leave this last one out, and it will default to '.'.
2. Run #parse on that parser.
3. Create a file handle, and output the return value of #to_asciidoc to that file.
4. That’s it!

Here’s an example:

	require 'amorfus'
	DIRECTORY_HEADER_LEVEL = 1 # This will become the section depth, in AsciiDoc, for each individual class we find
	r = CommentDocParser.new 'interfaces/gesture', 'Gesture Interface', 'Repository/trunk'
	r.parse
	t = File.new('output.txt', 'w')
	t.write r.to_asciidoc
	t.close

What does it do in the background?

1. It uses a dumb regex based heurestic to determine if a comment has value.
2. It classifies methods by what it can find of their name.
3. It generates Dia diagrams whenever it can find an object.
4. When generating images, it parses Dia diagrams using Nokogiri and removes irrelevant objects from that diagram, so that only the featured object shows up in the image.

Finally, we’ll need to make a small patch to AutoDia to make it recognize structs as equally valid objects. This is extremely hackish (see the Epilogue), but it manages to work for now.

--- Autodia-2.03/lib/Autodia/Handler/Cpp.pm     2009-04-15 01:10:46.000000000 -0400
+++ Autodia-2.03.orig/lib/Autodia/Handler/Cpp.pm        2005-04-15 08:02:49.000000000 -0400
@@ -72,7 +72,7 @@
          $i++;
 
          # check for class declaration
-         if ($line =~ m/^\s*(?:class|struct)\s+(\w+)/)
+         if ($line =~ m/^\s*class\s+(\w+)/)
            {
 
 #            print "found class : $line \n";

Act Four: The Results

The resultant HTML looks like this:

From our previous example, this can be generated on the command line like so:

    % asciidoc --unsafe -e data-uri output.txt

The --unsafe and -e data-uri allows AsciiDoc to embed the images you’ve created directly into the HTML, instead of referencing them externally. This makes the document self-contained, in a sense. You can ignore these flags if you wish. In that case, standard <img> tags will be generated.

Epilogue: Warnings

This functionality was hacked together in about two hours. It does horrible things to Dia’s XML documents, it guesses at what the current spec for AsciiDoc is, and requires a hastily applied patch to a third party tool, AutoDia, in order to accomplish its goals. Because of it’s nature, I can’t make any guarantees as to how effectively it will work, what circumstances it will work under, and the like. If you have any suggestions or improvements, I implore you to fork the code (currently hosted at this Gist) and see what you can come up with. For example:

• I don’t even think it recognizes variables correctly. You might want to fix this.
• It ignores public:, private:, and protected:. You might want to fix this, but it is meant for design documents, so all methods should be documented.
• If a class has more than one constructor, only one will be displayed in the documentation. You might want to fix this.

Okay, so here we go. This is going to be a bit of a rambling adventure, because that’s the only way it can follow what I actually managed to accomplish.

I’ve been obsessed with learning LINQ to SQL since I first heard of it, but never really had the time. Besides time constraints, I was also put off by the fact that there seemed to be nothing coming from the Oracle camp regarding the exciting new technology I saw unfolding. After a while I began hearing whispers of the Entity Framework, a complete object-relational mapping solution coming from the Microsoft camp, and was immediately interested. I had never really taken a look at the other available ORM solutions for .NET, NHibernate and the like, simply because I thought there would be a slim chance I would be able to use both Oracle and LINQ with them without too much trouble. As it stands, I had exactly that trouble with Entity Framework, but it feels justified in a weird way. I’m sure after this I’ll try taking a look around for other available solutions, but so far this looks extremely promising, if a little unpolished.

This all starts off with EFOracleProvider, an Entity Framework-friendly provider used to generate the necessary classes and mapping files for database access. Compiling EFOracleProvider gave me a DLL and some configuration settings I could toss at the already existing data model generator, EdmGen.exe, to generate the classes that map to the database tables and fields I specified.

I’m going to show you the files I worked with in all of this, but I’m not going to obfuscate them, or simplify them. I’m going to give you an idea of the exact situation I was in, in order to show you the problems I encountered, and how easy it is to encounter them. I’d also like to remind you that use of EFOracleProvider is not recommended in production environments, as it is extremely rough quality code. This was more of an adventure than a focused goal, but it raises a lot of questions.

Observing the Database

My first step, after compiling EFOracleProvider, was to create a storage model for the part of the database I wanted to present to .NET. I decided to go straight to our live database and take my queues from it. I took a small cross-section of our available schema — just a single table, with two columns, both VARCHAR2s.

    +--------------------------+
    | tblAsset                 |
    +--------------------------+
    | Asset_Number             |
    | Asset_Type               |
    +--------------------------+

Yes, Asset_Number is a string. It refers to alphanumeric serial numbers. But it’s also our primary key, so it’ll make do. In any event.

The storage model file is XML, and fairly easy to create, but has a lot of hidden gotchas that we’ll cover.

<?xml version="1.0" encoding="utf-8"?>    
<Schema Namespace="TestAppEFModel.Administration" Alias="Self" xmlns="http://schemas.microsoft.com/ado/2006/04/edm/ssdl" Provider="EFOracleProvider" ProviderManifestToken="10g">
  <EntityContainer Name="dbo">
    <EntitySet Name="Assets" EntityType="TestAppEFModel.Administration.Assets" Table="tblAsset" />
  </EntityContainer>	  
  <EntityType Name="Assets">
    <Key>
      <PropertyRef Name="Asset_Number" />
    </Key>
    <Property Name="Asset_Number" Type="varchar2" Nullable="false" StoreGeneratedPattern="Identity" />
    <Property Name="Asset_Type" Type="varchar2" MaxLength="50" />
  </EntityType>
</Schema>

Fairly straightforward. We’re reminded in the EFOracleProvider documentation to set the ProvierManifestToken to the relevant version number of our database. You’ll note that your namespace seems to require a secondary class. You cannot place your entities in TestAppEFModel, you need to create something like TestAppEFModel.Administration. Also note that I’ve manually specified the table name in the EntitySet tag, since it does not map cleanly to what .NET expects.

Fire up EdmGen.exe:

    C:\OracleEFTests>edmgen.exe /mode:FromSSDLGeneration \
        /c:"Data Source=TNSNAME;User ID=user_id;Password=password..." \
        /project:TestApp /prov:EFOracleProvider \
        /inssdl:TestAppEFModel.Oracle.ssdl /project:TestAppEFModel
    Microsoft (R) EdmGen version 3.5.0.0
    Copyright (C) Microsoft Corporation 2007. All rights reserved.

Creating conceptual layer from storage layer...
Writing msl file...
Writing csdl file...
Writing object layer file...
Writing views file...

Generation Complete -- 0 errors, 0 warnings

Preparing the Application

There are two basic steps to preparing your application: Providing a DLL of the object model, and configuring your application for use with EFOracleProvider and your database. Providing the DLL is easy enough — take the two .cs files created by EdmGen.exe and throw them into a project, and reference that project in your app.

I had initially put all of the test and generation files in one root directory on my disk, just to keep track of them. This turned out to be a good idea for a secondary reason. In your application configuration, you must provide access to the metadata files you’ve just created. The example code in the EFOracleProvider project used the following syntax in its App.config:

<add name="NorthwindEntities"
        connectionString="provider=EFOracleProvider;metadata=res://NorthwindEFModel/NorthwindEFModel.NorthwindEFModel.csdl|res://NorthwindEFModel/NorthwindEFModel.NorthwindEFModel.msl|res://NorthwindEFModel/NorthwindEFModel.NorthwindEFModel.ssdl;Provider Connection String='data source=XE;user id=edmuser;password=123456'" providerName="System.Data.EntityClient" />

Note those res:// addresses. I imagine those are used to refer to resource files, but I didn’t see any of the projects involved actually containing one. On top of that, there’s that double-namespace thing, i.e. NorthwindEFModel.NorthwindEFModel.csdl. I couldn’t get this syntax to even work a little bit with my configuration, so I just pointed my application at the root directory I created, that contained all of the necessary files. I imagine on a test or application server, leaving these files in a common place may be the most useful option.

<add name="TestServer"
    connectionString="provider=EFOracleProvider;metadata=C:\OracleEFTests\TestAppEFModel.csdl|C:\OracleEFTests\TestAppEFModel.msl|C:\OracleEFTests\TestAppEFModel.Oracle.ssdl;Provider Connection String='Data Source=TNSNAME;User ID=user_id;Password=password...'"  />

Don’t forget to add EFOracleProvider to the <DbProviderFactories> section of your configuration, as well. Also note that I didn’t use the providerName attribute, which in the test code was listed as System.Data.EntityClient, even though the provider in the connection string is our good old EFOracleProvider.

Finally, write a small piece of code to test the functionality.

using System;
using System.Collections.Generic;
using System.Data.EntityClient;
using System.Linq;
using System.Configuration;
using System.Text;
using TestAppEFModel;
 
namespace TestAppEFOracleTests
{
    class Program
    {
        public static void Main()
        {
            TestAppEFModelContext context = new TestAppEFModelContext( ConfigurationManager.ConnectionStrings["TestServer"].ConnectionString );
 
            var query = from asset in context.Assets
                        where asset.Asset_Type == "TEST ASSET TYPE"
                        select asset;
 
            foreach(var asset in query)
                Console.WriteLine("{0}, {1}", 
                    asset.Asset_Number, 
                    asset.Asset_Type);
        }
    }
}

The Troubles

And we’re good to go, right? Not quite.

ORA-00942: table or view does not exist

Whoops. What did we do wrong? Well, let’s look at the SQL that was generated.

SELECT
    "Extent1"."Asset_Number",
    "Extent1"."Asset_Type"
FROM
    "tblAsset" "Extent1"
WHERE
    (N'TEST ASSET TYPE' = "Extent1"."Asset_Type")

What’s wrong with that? Actually, something very subtle and dangerous that I’m sure is causing Oracle a lot of pain in trying to get a provider out for the Entity Framework. Observer the following session in SQL*Plus:

SQL> DESC tblAsset;

Name           Null?    Type          
-------------- -------- --------------

ASSET_NUMBER   NOT NULL VARCHAR2(50)  
ASSET_TYPE              VARCHAR2(50)  

SQL> DESC "tblAsset";
ERROR:
ORA-04043: object "tblAsset" does not exist

SQL> DESC "TBLASSET";
Name           Null?    Type          
-------------- -------- --------------

ASSET_NUMBER   NOT NULL VARCHAR2(50)  
ASSET_TYPE              VARCHAR2(50)  

This hearkens back to a long-standing conflict in dealing with Oracle and mixed-case or reserved identifier names, to wit: In order to ensure that table names that happen to be reserved keywords can be used, we need to enclose them in double-quotes. But we can create tables with or without quotes, with differing cases, and create two different tables. We may not be able to rely on the user telling us the exactly correct name of a table in our storage model file, but we can’t convert it to all caps without the risk of accidentally referring to a different table!

My solution at this point was to cave and refer to the table name in all caps in my storage model file. But now the problem is the column names! And we can’t refer to a column name in the storage model, but we can in the mapping file, the .msl, but we can’t simply generate the mapping file, change the column names to all caps, and have it compile. What I ended up doing was simply referring to all of the fields in all caps in the storage model, which of course breaks our code, making us refer to asset.ASSET_TYPE, as well as uglifying the constructor code for creating new objects:

/// <summary>
/// Create a new Assets object.
/// </summary>
/// <param name="aSSET_NUMBER">Initial value of ASSET_NUMBER.</param>
public static Assets CreateAssets(string aSSET_NUMBER)

One More Thing

So, those are the big things, the deal-breakers. The largest problem to me is the idea that we can specify a table name in our storage model file, but not column names. This seems entirely too half-baked to me, but maybe I just don’t understand the conceptual tactics behind a move like that.

And I just noticed this as I was signing off, a quick bizarre thing I wanted to try:

var query = context.Assets.Where( a => a.ASSET_TYPE.Contains("A") ).Take(10);

Take the top ten items that have an uppercase A in them. Simple enough, right? Nope. A rewriting of the generated SQL:

SELECT 
    asset_number, 
    asset_type 
FROM 
    tblAsset 
WHERE ((INSTR('A', asset_type)) >0) 
AND   ( ROWNUM <= (10) )

This is a mess. First off, that ROWNUM clause is not going to do what we expect, no matter what the rest of the query is. Secondly, INSTR takes its parameters in the reverse order. Little things like this are going to be the hell Oracle would have to go through to get a decent provider out there for Entity Framework, and I don’t even envy them a little bit.

One more thing I want to say about all of this. The work I did above took seven hours, a lot of Googling, a lot of searching through documentation, and a lot of guessing. The following is how everything above would be handled in my current favorite Ruby ORM.

class Asset
  include DataMapper::Resource
  property :asset_type, String
  property :asset_number, String
end
 
query = Asset.get :asset_type => 'TEST ASSET TYPE'

This is not to compare languages or ideologies, but methodologies. Simplicity always trumps complexity in the software engineering world. Always always always. Make Entity Framework easier for people to use, and it could be a godsend for shops that want a Microsoft-written ORM solution.

I’ll try and answer any questions anyone has, but this is literally the maximum scope of what I was able to accomplish. I’ll probably throw myself back at it later this weekend.

To the Apple-mobile, I cried, slamming developer.apple.com into my browser. A search for ‘ppc_thread_state_t’ was – less than helpful. A few references to it, but no actual useful documentation. Google was the next stop, where someone trying to get another older program working on Intel received a comment about the new ‘i386_thread_state_t’ — something I could have guessed but was useless without the new register names provided by the comment. Plugged that in, and the program built without error – but would not run.

Confused, I hit up developer.apple.com and searched for ‘i386_thread_state_t’ – and Apple returned no results. I googled around some more for ‘mach_inject intel’, finding that one of the developers of iCal had produced a port of mach_inject. Replacing this with my hacking around produced another clean build, and a failure with an explicit error – what GeekBind was trying to do involved being part of the procmon group.

It was at this point I figured I could do one of three things:

1. Do one of the arbitrary and unsafe methods recommended to make the running user part of procmon
2. Learn about the complexities of user elevation in Mac OS and see if that was applicable to GeekBind
3. Find another program that did what I wanted GeekBind to do.

So I did step 3. Window Dragon does what I want, for the most part, and only required the install of some hefty Application Extension Management System that deposited itself in my System Preferences and god knows where else on my system. It doesn’t work with applications started through X11, which is nearly a dealbreaker. But it’s what works for now.

Why do I write about this? Several things.

1. There’s always an alternate program. Someone on iusethis recommended paying 8 dolllars for a program with similar functionality – and the idea made me balk. Eight dollars for something that should be built-in? I’m not horrifically poor but I know when to shop around.
2. developer.apple.com’s search sucks something horrible. Maybe I don’t understand what it does, but looking up two object names returned absolutely nothing for me. I may be too used to MSDN, I guess, and should be checking the documentation that came installed with Xcode.
3. even knowing nothing about Xcode, Objective-C, Cocoa, or any part of the Mac development process, I was nearly able to port a PPC source code project to an Intel Mac binary. A little more work and I’d have what I want working on my system, and the source code to give to other people under GPL. This is a powerful incentive.

And that was my Apple Excursion.

And…

$ rake dm:db:automigrate
(in /home/.../Documents/Projects/Merb/Blog)
Connecting to database...
Fri, 04 Jan 2008 05:44:33 GMT: loading gem 'merb_datamapper' from config/dependencies.rb:13 ...
Started merb_init.rb ...
Loading Application...
Compiling routes..
Loaded DEVELOPMENT Environment...
rake aborted!
Your query failed.
Incorrect table name 'data_mapper/bases'
CREATE TABLE `data_mapper/bases` (`id` int(11) NOT NULL PRIMARY KEY AUTO_INCREMENT)

And based on the almost no documentation and out of date tutorials I’ve found, there’s nothing that could even remotely explain what is going on here.

EDIT: The second problem appears to be noticed here. Don’t know if the recommended course of action actually works yet.