• I got rid of the TableTool and replaced it with a real MSBuild task. I also added a task for generated opcode-related boilderplate and dispatch functions, to further decrease the amount of effort involved in implementin a new opcode in the interpreter.
• I reorganized the project files somewhat, in particular splitting the build process projects into a PspClr.Prerequisite.sln and leaving the actual loader, corelib implementation, et cetera in PspClr.Product.sln. The motivation behind this change was to work around the assembly unload limitation that makes developing MSBuild tasks in the same solution file you call them from a painful experience.
• I streamlined a lot of the metadata handling, especially as relates to runtime representations of the metadata — there is an actual object model that wraps the metadata row, making them much easier to work with in code.

After all of that, however, the basically functionality remains about the same as last time. With one small addition: newobj.

“4.21 newobj - create a new object”

newobj causes the CLR to create a new instance of either a reference or value type. Since nearly everything you’d ever want to work with is such an instance, it’s a fairly critical opcode to support. The only reason I was able to get away without it so far is that the assembly entry point method is static, as is System.Console.WriteLine.

newobj is very straightforward: it’s a single-byte instruction followed by a four-byte argument, which is a metadata token (essentially an encoded index) that refers to the constructor to invoke on the created object. The reference can be to the method definition metadata or the member reference metadata table — the former is for methods that exist in the same assembly, the latter for references outside the assembly. I’m only going to discuss the former, as the process for resolving the latter is essentially the same but with the extra layer of indirection of finding the assembly in your dependency list first.

When you encounter a newobj call, that constuctor method definition is all you have available. It looks like this:

• A four-byte address to the start of the method’s CIL opcode stream.
• Four bytes of flags.
• An index into the string heap, which provides the method’s name (”.ctor” for constructors).
• An index into the blob heap, which provides the method’s signature.
• An index into another metadata table which defines the parameters of the method.

Does it look like there’s anything missing? At first glance, there doesn’t appear to be any information about the type that owns the method — without knowing the type, you can’t know the size of an instance of that type, which means you can’t create that instance, which means you cannot call the constructor since it wouldn’t have anything to initialize.

You might try looking in the parameter list or in the signature in the blob heap to try and extract the information about the owning type, but that would not be a fruitful search. The existence of the this parameter is encoded with a flag, not a full parameter entry (in the interests of saving space). No, to find out who owns a method, you have to look through the type list until you find a type that has the desired method.

This isn’t as bad as it might seem at first glance — the metadata that defines a type contains an index to the first method definition for that type, and you are guaranteed that all method definitions are owned by at most a single type and that all definitions belong to a given type are contiguous in the method definition table. This makes it pretty easy to associate methods with a type — the fields of a type are stored in the same fashion, and you need to process them to create the in-memory layout for the type anyway (since you need to know how many bytes an instance of the type occupies). This is what spurred me to create a more robust object model for the runtime representations of all this metadata that the execution engine consumes.

At assembly load time, I iterate the type definition metadata, and produce a TypeDef object for each entry, which I store in the assembly. Creating a TypeDef object iterates the subsection of the method and field metadata tables that are owned by that type and creates MethodDef and FieldDef objects which are stored in the TypeDef. Methods and fields are imbued with a reference back to their owning TypeDef, which allows me to make the mapping between method and owning type O(1) at the expense of a small amount of storage.

With that object model in place, it becomes possible to implement newobj. Read the method reference argument, and find the appropriate MethodDef object (this is an O(1) lookup in an array stored in the assembly). Get the TypeDef for the MethodDef (another constant time lookup), and allocate enough space for that type on the heap. Then call the constructor.

The Horizon: ldfld and stfld

newobj and its supporting code changes are a pretty decent step forward for PSPCLR, but I’m still a long ways away from something useful. Next on my plate are the opcodes for loading and storing object fields — these are used when, for example, you read or write from members. Since constructors do this often, they make for a logical next step.

Cost and Benefit — Is There an Audience?

The gist of this point was that given that the majority of players are not versed in the specific technical domains that are involved in the process of developing a game, the size of the audience for “techy” updates is potentially rather small. Perhaps the audience is so small, in fact, that it’s not worth the productivity hit required to get the information out of the programmer’s brain and in to a tangible, presentable form.

The thing is, technical information does not have to be presented in its natural state. I’m not talking about posting your Perforce check-in logs on a website, here. I’m talking about communicating the progress of the technical aspect of the game, in consumer-friendly form. This isn’t impossible — it’s not neccessarily easy, though. It involves employing communications skills that are not neccessarily all that common among programmers. In particular, it calls for the ability to distill a technical factoid or event into bite-size chunks that can be digested by somebody with much less technical know-how. Usually this requires a fair command of metaphor and other literary or educational devices — which makes sense, because teaching is essentially what’s going on here. You will likely observe that team members who have had experience (successfully) teaching classes, speaking at conferences, and so on will likely be better people to solicit for input to a devblog.

The fact that what you’re essentially doing here is teaching your players a little about the technical magic behind the curtains is important. This is a good thing. The player base, as a whole, tends to get their impression of the inner workings of game development from the gaming media. But let us not kid ourselves here — the gaming press really doesn’t have a clue what’s going on, either. Is it really better for us to be letting them set the stage for what our industry is like? I think not.

Talking to the players about appropriately distilled technical progress helps improve their overall understanding of the industry. If you do it right, and maintain a spin on the presentation that focuses on the cool things the tech may enable players to see when the game ships, you can cultivate a much more trusting and pleasant relationship with your eventual playerbase — and you get a bit of a marketing push out of it, as well. Despite the fact that you’re releasing information to a wide audience, players will still get a kick out of the fact that you’re sharing “insider secrets” with them. However, this brings us to another issue.

Management of Expectations

If you recall, I described expectation management as the desire to limit discussion about things that might be in the game because of the possibility that those things might be cut and that players would thusly feel ‘cheated’ of features. Of the three excuses I discussed last time, I think that this one is the most prevalent in the real world. I only mention it second because I think that a significant root cause is the aforementioned lack of real understanding on the part of the players.

Features get cut for lots of reasons, but almost all of those reasons boil down to a cost/benefit ratio that is not in the developer’s favor, or because they sounded cool on paper but when implemented turned out bland or tedious. This is generally a good thing for the overall health of the product — but it isn’t always seen that way by the players. Doing more to educate and inform players about the state of the game as it develops would help alleviate the misconceptions that they commonly have about whether a feature was cut for compelling technical or fun-factor reasons.

I think the advantages weigh in more heavily here: talking about some aspects of gameplay can stimulate discussion among the fans and make them feel more like they’re a part of the process, even if things only end up turning out the way they want through pure coincidence.

I think that it can be harmful to keep information privatized. Certainly some information should be, but that sort of stuff tends to be the exception rather than the rule, I think. Tranparency and openness on a topic is an investment of the time and brainpower you devote to that topic, in this modern age. We’ve taken steps in the right direction with GDC and other trade shows (and to some degree, with the abolition of the farce that was the old E3), but we can do better.

If players become better educated about what making games is really about, maybe they won’t feel quite as put-out when the game shapes up to be different upon release from what it was during the early days. What’s the worst that could really happen anyway? We’d be in the same position we are now, which isn’t exactly a huge risk to be taking.

Publisher Mandate

Publishers are generally the controlling entity in the development process: they supply the money to the developer, persuant to the developer meeting certain milestones and following certain rules. So it’s entirely concievable that a publisher might want to curtail any unnecessary discussion of any ongoing projects.

On this point, I’m not sure what to say — “find a better publisher,” isn’t exactly a real-world option. Fortunately, I’m not entirely convinced this is an excuse that one hears all that often — I’ve heard far more stories about publishers signing off on the release of blatantly poor-quality information (screenshots with clear animation or art asset bugs, for example) than I have stories about publishers issuing gag orders on contractually obligated studios.

The End

In the end, it’s obviously going to be impossible to make everybody happy, and no matter how much, or how well, you communicate with your players there will always be those players who disagree with your choices no matter how rational or necessary those choices were from your perspective. To the player, if it sounded cool and you didn’t do it, that’s usually bad.

But being more outgoing and chatty has advantages for both parties involved; our historical tendency to keep our lips sealed has not been to our advantage. I’m glad to see a trend towards honest rapport between developers and players emerging, and I hope it continues.

Not surprisingly, there’s often a fairly strong correlation between candor and size. Small companies tend to be less formal in their operation, and the sorts of communication we’re talking about here is generally considered somewhat informal in nature. Of course there are exceptions. Spiderweb Software is a three-man shop that produces little in the way of development-related updates beyond “we’re working on X,” “almost done X,” and “X is out, we’re moving on to Y.”

Bungie is one of my favorite examples; back when they were working on Marathon they were active on AOL and Usenet groups. During the development of Halo, Matt Soell would produce weekly developer updates filled with all manner of interesting tidbits (they continued this tradition after Halo as well, but over time their community interaction has lost the style that attracted me to it originally).

Some other developer logs I recall fondly are those from Ambrosia projects, particularly Escape Velocity: Nova and its Windows port, and the port of Aquaria.

If you follow a few of those links and browse around, you’ll get a feeling for the specific type of communication I’m talking about here: discussion about the actual day-to-day process of crafting the game, venting about bugs and nerdy programmer anecdotes included. I’m not talking about more generalized community interaction concerning non-development issues — while that’s certainly an important aspect of overall relationship between development studio and players, it’s not my area expertise and I won’t comment on it too extensively, lest I look the fool.

With that context established, let’s consider some of the more popular reasons you don’t see more, larger, development studios actively producing developer blogs.

Cost and Benefit — Is There an Audience?

Producing content for public consumption involves human effort; real bodies, in real chairs, doing real writing and editing. Furthermore, although perhaps less true for smaller studios, larger companies have a certain level of professionalism they would presumably like to maintain. Part of that maintainence is making sure anything that is written as Official Company Word employs proper grammar and punctuation, doesn’t leak important trade secrets, doesn’t slander another developer — basically, make sure things look like they were written by somebody who knows what they’re doing, instead of some programmer who fancies himself a writer but who can’t even spell “maintenance.”

That editing, fact-checking, content review, and so on all take time out of somebody’s work day, and when you look at the bottom line you have to wonder if the returns justify the cost — that’s anybody’s guess when you’re talking about actual developer log-style (that is, containing some measure of technical nitty-gritty) content. Not everybody finds technical content interesting; if I had to guess I’d think that those people are the clear minority (compare the original EV Nova log, for example, to the Windows port version — the former is by and large a running checklist of ’stuff we checked in to the repository’ while the latter has actual narrative content with a wider appeal). Of course, you can blunt the edges of the technical mumbo-jumbo to some degree, and include commentary on other, more widely-palatable issues. Once again, Soell’s Halo updates did this really well. In fact, transforming some of the technical details about that sweet database voodoo you implemented last Thursday that makes your asset pre-build stage run twice as fast into layman’s terms can be an exciting challenge for some, and artists and designers usually enjoy taking time off from real work to stage a sexy screenshot to show off some new feature that just got added to the build.

But again, you ramp up the effort you put into wrangling content for your developer updates, you ramp up cost, and draw resources away from actually making the game — the real, important goal. It can escalate to the point where your team is in a mini “E3 crunch” every few weeks. The benefits in community engagement you might get back in return for that aren’t always tangible or measurable, which makes for risk — in some cases, enough risk that company’s simply adopt a blanket “we’ll talk when we’re done, thanks,” policy and forget about it.

Management of Expectations

People become attached easily, and often to surprisingly minor things. Everybody is different, everybody finds enjoyment in different things in a game. Most of us have an armchair game designer floating around in our soggy little brains, too, which always complicates things. My point here is that it’s often very difficult to get rid of some aspect of gameplay, even a minor one, once you’ve mentioned it.

Remember Guild Halls in Diablo 2? The fourth race in Rise of Legends? The deep, adaptive, immersive AI in Black and White or Oblivion? All of these were talked about, or even hyped in some cases, pre-release but vanished with barely a trace by the time the games hit shelves. The fan backlash didn’t kill the games, but a good deal of players had chips on their shoulders for some time, and blamed the developers for what they saw as an injustice. The damage wasn’t done to the games’ sell-through numbers, but to the atmosphere of those games’ communities, and community improvement is ostensibly what the whole developer-player communication thing is about, no?

It’s a fair assumption that the majority of players are non-technical (that is, they aren’t software developers themselves). It can be difficult to convince these people that cutting a feature was justified in terms of development cost, or because it wasn’t turning out to be fun. In the former case, the players tend to lack the technical acumen to grasp a completely literal explaination of the rationale, and/or such an explaination may require more background about the development history of the related aspects of the project than the studio may be willing to reveal. In the latter case, the players will have their own versions of the cut feature in their mind, almost certainly disconnected from the reality of how the feature feels in the context of the rest of the game, and can sometimes have a hard time looking past what’s in their own heads.

Because of this, it’s common for studios to err on the side of releasing less information because it doesn’t create expectations among the fan base that may be unrealistic. There’s also some measure of surprise when the game ships with a bunch of cool features nobody knew about, which can be a plus.

Publisher Mandate

There are a lot of studios that are owned by a publisher, and most of the rest are developing games on some other publisher’s dime via contract. Those sorts of relationships generally afford the publisher a certain measure of control over the developer, control which can manifest itself as an enforced “don’t talk about Project X,” policy.

Publishers, being beasts of an altogether different nature, are generally concerned more with the business ramifications (such as the effect on stock prices) that talking about projects in development have. In the interests of controlling the release of information they may deem related to some corporate strategy (investing in new intellectual property, entering into negotions to secure a license to some existing intellectual property, diversifying platforms, et cetera) or trade secret, publishers might choose to have the developer filter all related informational releases through them first, or simply ban them outright.

I’ve never personally been in a situation where I’ve observed this happen, as far as I know (I’m generally not involved in the details of the relationship between the studio where I work and its publisher), and I’m glad. It’s an extremely heavy-handed tactic, in my opinion, and it’s unfortunate that it happens at all.

On Balance

There are opposing points to be made here — I should note that I’m primarily playing devil’s advocate with this article; while I understand where studios are coming from with the above arguments, I don’t neccessarily agree with them. Next time I’ll talk about exactly how I disagree, and what I think can be done to improve things.

Well, that’s a waste of your time — and in many cases, a waste of quite a lot of your money, too.

Amateur game developers often have this sort of hero-worship attitude towards professional developers, their tools, and their products. But as I’ve touched on before, that’s not neccessarily a very productive stance to take. When it comes to tools in particular, reasoning behind the argument tends to be one of

• you’re considering a career in the industry and want to learn the “correct” tools, or
• you want to use “the tools” that professionals use because you believe it will give what you produce with those tools an edge over what you might produce with some other tool.

Both of these are bad reasons.

Argument The First

Consider the first argument. This is the stance that shady trade schools would like you to take, because it’s the based on the same core principle that they usually base their (usually rather weak) curriculum on. That is, the idea that game development is a tool-centric process that requires particular applications of specific tools and technology. The idea that there is, for any given problem domain for which a tool can be constructed (such as creating 3D models, or building game levels), a single tool that is the ‘best’ tool for that job. The One True Tool that all game developers are using for that task.

Now, if that assumption were true, there might be a valid argument there. But it’s not true. There is no such thing as the ‘best’ tool for a given problem (in general); while there are usually a handful of apps that are considered to be among the best, each of these will have strengths and weaknesses that need to be considered.

This is important: the criteria professionals use to evaluate tools are very different from the criteria you should use. The tools that do get used by professionals are a result of a selection process based on that criteria, so it should be clear that you, using a different set of criteria, may arrive at a different ideal tool. As for what your criteria should be? I can’t tell you — every problem is different, and you know yours the best. The only advice I can reasonably offer you is to try to get the biggest bang (in terms of increase in your productivity) for your buck.

Argument The Second

The second argument is built around an assumption that is similar to the first. Where the first assumption was that the craft requires the crafter to employ specific tools, the second assumption is that the tools imbue the crafter with the ability to perform the craft. This is equally false; the tools don’t make the games. Sure, Maya provides orders of magnitude more features, controls, and options than Milkshape 3D does — but those features don’t do anything without the artists skillful hands at the controls.

The skill that artist has is a core talent that comes from an understanding of the process of 3D modelling that is fundamental and application-agnostic. The understanding of how muscles interact with the skeletal structure of a bipedal humanoid to shape its movement, and how that interaction can be extroplated to create natural looking motion on a more bizzare, six-legged alien form — that doesn’t come from knowing where the Create Biped button is in the UI of a specific tool.

You can pick up a lot of the basic, fundamental important bits of working with a specific genre of tools by using free and/or inexpensive tools in that genre. Those skills will carry over as you migrate to other tools, even those big-name brands that you see advertised in your latest issue of Game Developer. In some cases you may even be able to produce more content at higher quality with simple tools, especially if you’re new to working with that specific type of content. Often the complexity of heavy-duty tools can be overwhelming.

Takeaway

What you should take away from this post is quite simple: you don’t need to emulate a professional game development studio, not in general, and not in terms of tool selection. Your goal is to make a killer game, right? Then focus on making a killer game. If you’re able to make your game without additional tools now, great, keep doing that. If you think you need to start using some external tools, grab some evaluation copies and give them a spin. If you can’t get an eval copy of something that you know (or think) professionals use, don’t sweat it — I assure you there is an alternative.

Above all, a tool is supposed to optimize the development process in some way, and this should be among your key requirements (and one of the only ones I can recommend in general, as I noted earlier). In order to be useful, a tool must make you more productive in some fashion. It should, ideally, do this as cheaply as possible. There are free tools out there for doing just about everything, most of which range from passable to excellent, so don’t be fooled into thinking that a $2,000 price tag is really getting you something.

Now, all of that said, you can usually get your hands on lightweight versions of some of the so-called ‘industry-standard’ tools for free. Microsoft publishes Express editions of Visual Studio, Softimage offers a ‘mod tool’ version of XSI, for example. As long as these tools meet your requirements, and you can work them competently, there’s no real reason not to use them. But if they don’t meet your requirements or you can’t work them well enough to be productive, don’t use them just because they’re what you heard [insert game studio here] used to make [insert popular title here].

You’re not in Rome, so who cares what the Romans are doing?

I already have a custom tool called transformadf.exe to do this. It reads .adf files (which describe the physical structure of in-game assets used by my code) and produces some C# source files containing the asset class and some supporting machinery. The .NET XPath query library is used to build an in-memory description of the structure from the .adf file, and then produces the code files based that structure and heavy use of StringBuilder. There are two things I don’t like about this approach:

• Modifying the resulting code is difficult. The boilerplate portions of the code, the “template” if you will, are all contained in string literals scattered around the file. Some characters need escaping, and a particular set of simple rules need to be obeyed with respect to line endings (not Unix / DOS / Mac, but rather whether or not you end the line via StringBuilder.AppendLine, or just use the StringBuilder.Append method, which will not include the newline). Making changes to the template is more cumbersome than it should be.
• transformadf.exe is part of the build process for the framework code itself (because the framework includes standard asset types), and the build process for the framework’s client games. The other tools in the framework are used only by the client games. This makes transformadf.exe unique within the framework’s build system, as it can’t use the standard set of prebuild/postbuild actions (since it’s part of those actions).

Back Into the Breach

I had not really done anything with XSLT since the last time I wrote about it, so I suffered through a few false starts before I ended up with something that wasn’t entirely embarrassing. The two things I took away from this study session with the language were the ancestor selection axis and the mode attribute on templates.

The ancestor Axis

XSLT uses XPath to query for (”select”) nodes in the XML hierarchy to operate on. Part of that selection process is the notion of the axis along which you are performing the selection — you can think of this axis as a direction, but I found it easier (although less intuitive, initially) to think of the axis as specifying the set from which you will select. The default selection axis, if you don’t specify one, is the child axis. To specify an axis other than the default, you use the syntax axis::expr, where axis is the axis specifier and expr is an XPath expression.

The ancestor axis is the set of nodes between the document root and the current context node, but not the context node itself (use ancestor-or-self for that). It’s important to note that the resulting set of nodes selected from this axis is organized in document order: matching nodes closer to the document root appear first. This is why I suggested thinking of the axis as a set rather than a direction, since “direction” and “ancestor” put me in the mind of traversing the node tree backwards, “towards” the ancestors, and thus returning the nodes closes to the context node first. Until I shifted gears, I kept making stupid mistakes in my xsl:for-each operations.

The mode Attribute

For every asset description, I generate three C# class: the asset class itself, a marshaller class that handles lifetime and serialization tasks, and a reference class that allows the asset to be pointed to by other assets (and allows the asset wrangling subsystem to ensure all dependent assets are automatically loaded, if desired). This requires three passes through the source XML. My original strategy was to copy the source tree into three variables, renaming the root of each tree. Then I applied templates to each tree in turn, and the templates themselves matched nodes that were descendants of specific named roots (using the XPath selection named-root//node-name).

This let me have a template for matching marshaller-class//field that was different from asset-class//field; desirable, since I needed to do entirely different things to process fields in each of the three classes. It worked, but it felt clunky and ugly — and it was. While researching some other aspect of XSLT that had slipped my mind, I stumbled across the mode attribute that can be applied to templates and the apply-template element that allows for exactly this kind of filtering without the bloated triple copying of the source tree.

It’s quite simple: an XSL template can be given a named mode, and will only match when templates are being applied in that mode. Now I can simply write the appropriate templates and mark them with an appropriate mode, and apply-templates three times — it is much cleaner to look at.

Results

In the end, deciding whether or not to use XSLT or my hand-rolled tool was very much a “six of one, half-dozen of the other,” kind of decision. The XSLT worked, and worked well. The time I spent struggling with new parts of the language (or those I’d forgotten) was about equal to the time I spent writing the generator initially. Maintaining the generator has been expensive, but I suspect maintaining the XSLT will be equally so. I didn’t gain a lot in terms of template code expressibility in XSL, either — that is to say, it’s nearly as difficult to write or maintain the non-generated boilerplate portions of the code in XSL as it was with transformadf.exe. The difficult is simply elsewhere (in particular, in coping with whitespace so that the generated code is at least passably clean).

While I haven’t made up my mind completely, I’m leaning towards continuing to use the XSLT. For the most part things stay roughly the same, but it does provide a solution to my second problem, that of transformadf.exe being a special-case project. We’ll see what happens.

Today we’ll discuss them.

SlimDX supports the ability to share — in a sense — the native objects of other APIs using System.IntPtr, an opaque managed representation of a pointer or handle. All SlimDX COM objects expose their native pointer as an IntPtr. They can also be constructed from IntPtrs.

With the old SlimDX design, we only ran into two serious issues with this process. The first was that since IntPtrs are entirely opaque, we’d more-or-less trust the user to construct the correct type and pass the correct pointer. Sure, we performed a QueryInterface(), but that may not have been the best idea in retrospect — there are known scenarios (for example, see here and here) where DirectX objects don’t behave the way you might expect a COM interface to behave — and so our code might have occasionally failed in situations it should not have.

The second issue was that we used these IntPtr constructors internally, occasionally in scenarios where it was impractical to know the proper type to construct. Remember the GetDevice example from last time? In Direct3D10, there’s a similar method for resource views (GetResource()) that is used to get a pointer to the resource that the view object is looking at. The method is a member of the ID3D10View base class, so naturally we placed it in the analagous SlimDX.Direct3D10.ResourceView class. The implementation would have looked something like:

1
2
3
4
5
6

Resource^ ResourceView::GetResource()
{
  ID3D10Resource* resource = 0;
  NativeComInterface->GetResource( &resource );
  return gcnew Resource( IntPtr( resource ) ); 
}

The problem with this approach is that it logically “slices” the object. This is similar to slicing in, say, C++ except that the underlying object is not physical destroyed, only the way you can interact with that object reference. In other words, even if the ID3D10Resource* returned from the native call pointed to a derived ID3D10Texture2D, it would be an error to downcast the managed Resource reference to a managed Texture2D reference, because we statically constructed a Resource, not a Texture2D.

A solution to this is a factory-type construction process — fortunately for us, most of the types involved here had base class methods that would indicate the actual runtime type of the object somehow — even though that would mean we’d have to push most implementation of the method lower in the class hierarchy. That was the solution we were considering, until the object table was brought up, which neatly solved the whole issue for us.

Almost.

You see, there’s one final caveat. The reason the object table eliminates the need for a factory is that it stores the native pointer as the key, and the actual runtime managed instance as the value. Thus, the aforementioned code snippet becomes a table lookup instead of a gcnew statement. The problem rears its ugly little head when you consider the possibility that the table lookup can fail to find the key. At first that might seem impossible (and thus not worth handling) because the view can’t be created with a resource, and given the existence of the object table, the only way the resource can cease to exist before the view is if the programmer intentionally disposed of it. This would be a programmer error, which is generally something we do not concern ourselves with handling gracefully (that is, we’d throw an exception indicating the programmer is a dim bulb, rather than try to bravely soldier on).

Unfortunately, since SlimDX can exchange native resources with other APIs via IntPtr, it’s possible to get into this sticky situation rather easily — simply construct a new ResourceView using an exposed IntPtr. SlimDX will correctly handle the ResourceView, incrementing the native reference count and placing an entry in the object table. But SlimDX won’t know about the original resource, so if you call GetResource() on the view, SlimDX will not find the Resource’s key in the table. What will happen, in that case, is that SlimDX will treat the object as a new one, increment the native reference count, and add an object table entry. This is unfortunate because the user now has to Dispose() of that resource, even though they didn’t explictly allocate it with new.

We’re currently considering this a pathological use case, and aren’t addressing the issue. Users doing that kind of advanced interop with SlimDX will simply need to be aware of the problem and the non-standard call to Dispose() they’ll need to make. I do believe it is a fixable issue, but the fix isn’t in the March release since it will not adversely affect the majority of our user base — you’ll have to wait for June, if we’re even able to remedy it all. In the interim, however, it’s important to be aware of the dangers that lurk in the shady corners of the API.

Getting the Tools

Homebrew console development is typically mired in obtuse (and frequently undocumented) complexity from line zero; it’s the grim reality of working with something that is essentially a giant, constantly-evolving hack. When I set out to get the PSP development toolchain on to my machine, I was prepared for all sorts of mucking about with Cygwin and various other Unix tools, and I was knee deep in getting everything sorted out and configured when I stumbled across this blog entry by Paulo Lopes and this related forum thread. Paulo’s SDK distribution is extremely pleasant to install, and with just one simple change (adding -lstdc++ to the LIBS in the makefile) I was able to build C++ applications from within Visual Studio without any of the mess I was expecting. I haven’t had any trouble with it yet, so I highly recommend it to anybody looking to get into PSP development instead of PSP screwing-around-with-makefiles-for-hours.

The Depths of the CLR

Like C#, the Common Language Runtime is described in an ECMA standard. Anybody looking to build themselves a CLR, or understand an existing CLR implementation, needs to have that document handy.

Code for the CLR is stored inside “assemblies,” which also includes resources and other modules. Assemblies are packaged up inside files in the Portable Executable format (the very same format used by Windows .exe and .dll files). The PE format is well-documented and discussed at length elsewhere, so I won’t go into it too much. It’s sufficient to say that the first task for my project was to implement a PE file loader, which was a relatively straightforward task.

Within the PE file is the CLR metadata, which pretty much contains all the information to load and run the application. There’s a header, with the kind of information you’d expect to find in a header, and there are “heaps” of string, GUID, and binary data. The meat of the metadata, however, are the tables. The tables consist of rows of data, each row having a specific number of columns. The data in each column of a particular table is generally a fixed constant value or an index into another table or the string/GUID/binary blob heaps. Each table is identified by a eight-bit ordinal (for example, the table containing method definitions is 0x06). Conceptually, there’s nothing too difficult here. In practice, however, there are few snags you can easily run afoul of.

Table Format Woes

For starters, the table columns that contain indices into other tables are not a fixed size. In the interests of saving space, all indices are two bytes in size if the table they’re indexing has less than two bytes worth of rows. Otherwise they’re four bytes. In the metadata table, you’re only given a bitfield indicating which tables exist and a series of integers specifying how many rows those tables each have. The tables are all contiguously stored in the metadata, but since the size of each table as a whole isn’t provided, you can’t just jump to any random table easily. You need to process every table definition, checking every column that’s an index against the reported size of the appropriate table, and build up a database of table row sizes at runtime.

The variable nature of the table sizes also means accessing them in a clean fashion is tricky, and best served by lightweight wrapper classes for each type of table row. The processing of the table definitions and implementation of the table row wrappers is rather mechanical, so it lends itself to automation by a build tool.

Data Alignment

On the PSP in particular, misaligned data access is a problem. The PSP’s main CPU is a MIPS chip, which has memory access alignment requirements: two-byte accesses must occur at an address divisible by two, four-byte accesses must occur at an address divisible by four, et cetera. The MIPS processor isn’t one I’m terribly familiar with; I know it has instructions supporting misaligned load/store operations, and I was able to see them generated by the compiler, but I’d still get hard crashes on the PSP when I tried to access misaligned data, so some further investigation is warranted.

For now, I simply use a set of wrapper routines to perform any memory IO. These routines ensure the address in question meets alignment requirements, and if it doesn’t they fetch data from the nearest aligned addresses and shift and mask the results to yeild the value I’m interested in. Misaligned access occurs quite frequently when implementing the CLR. For example, CIL opcode are a single byte, but they are occasionally followed by a four-byte argument token, which will usually be misaligned.

What About Mono?

The reason I embarked on this project, and in this particular fashion (rather than, say, porting Mono), was to learn more about the gritty details of the CLR. Mono’s very mature relative to my little upstart implementation, but by porting it I wouldn’t be learning as much about the CLR as I would be about Mono’s code structure and internals. I felt rather lost digging around in Mono’s code, as well, because not only did I not know their codebase, I didn’t know the domain within which their codebase was operating — and as I’ve said before, learning about that domain by reading their source code wouldn’t have taught me as much.

On the other hand, while I’ve learned a lot about the CLR’s guts so far, the actual amount of PSP-specific code I’ve had to write is quite small, and other than the memory alignment crashes I haven’t run in to many problems on the PSP that didn’t also manifest themselves in my Windows build. This will likely change as I start being able to implement more and more of the standard library.

Downloading the Project

I have a Google Code page for the PSP CLR, if you’d like to look around. I should warn you (if the low SVN revision numbers don’t tip you off) that this is an extremely early attempt. There are a lot of places I’m taking shortcuts, failing to check for errors, and making assumptions, and even if all of that works out, I only have support for a handful of opcodes and a single standard library function (System.Console.Write(string)). In other words, it doesn’t actually do much yet, so don’t expect too much. Nonetheless, it’s there if you want to take a look.

SlimDX is a managed wrapper for DirectX and other libraries. It’s written in C++/CLI, an offshoot of C++ whose niche is interoperability between managed and unmanaged code. Bridging that divide is never quite as simple as it looks, and SlimDX has certainly been no exception to that rule, even though it’s a relatively thin layer over the native APIs. One of the trickiest problems we’ve had to solve is how to to get SlimDX code to play nice on both sides of the fence: the managed side of the library needs to expose interfaces and semantics consistent with those of .NET, and the unmanaged side of the library needs to play by the COM rules that govern most of the native APIs we wrap.

In the COM world, the lifetime of objects are managed through reference counting. When you acquire a pointer to a COM interface, you call its AddRef() method, incrementing its reference count. When you’re done, you call Release(), which decrements the count and destroys the object if no other references exist. In languages like C++, where DirectX is most commonly used, you have rich support for the RAII pattern, which leads to smart pointer implementations such as CComPtr.

But C# doesn’t support automatic, scope-based, deterministic destruction. So, while incrementing the reference count correctly is a no-brainer, correctly decrementing it when the user is done with the object is a problem. C# has finalization, which is an entirely automatic, but nondeterministic, process for cleaning up managed resources. The framework also offers the IDisposable pattern and an idiomatic implementation thereof, which is intended for manual cleanup of unmanaged resources the garbage collector is not aware of: file handles, GDI handles, and in our case, COM objects.

Finalization and Disposal

Since the COM interface represents an unmanaged resource that is beyond the control of the runtime’s garbage collector, implementing IDisposable is an obvious step. The idiom for IDisposable in C# stipulates that a disposable object should also implement a finalizer (even though this is, strictly speaking, optional) to call Dispose() on the off chance the user fails to do so, ensuring that the unmanaged resources will at least get freed up eventually rather than leak.

This, however, causes trouble for SlimDX. Finalization, you see, is performed in its own thread, which means that the Direct3D runtime will emit warnings in debug mode if the underlying Direct3D device was not created with appropriate flag for enabling multithreading. Allowing those diagnostics to appear is against our design guidelines, and forcing device creation to be multithreaded is a heavy-handed and boorish solution, especially since the flag has performance implications.

This effectively makes finalization completely off-limits to SlimDX, since we can’t know at compile-time if we’re allowed to release resources from other threads. In fact, SlimDX’s finalizers do absolutely nothing.

The Non-Idiomatic Solution

Part of the semantics of IDisposable are that object creation (via new) and disposal should exist as a pair. Additionally, the client should only expect to have to call Dispose() on objects the client has explicitly created. SlimDX’s original design made that impossible; to clean up SlimDX resources correctly, you needed to call Dispose() on every single reference to an object you ever got from SlimDX.

Recall that in Direct3D, most resources are associated with the device object that created them, and you can at any time ask the resource to return a pointer to its device. If SlimDX implemented IDisposable in the traditional way, you would only need one call to Dispose() for the device object; ideally that call would be made from the same context that originally created the device in the first place. But, as I’ve said, SlimDX did things a bit differently — not only did you need to dispose of the original device, you also needed to dispose of the device reference given to you by the buffer’s GetDevice() method, even though they were both references to the same object. We initially chose this design because it is as similar to the native semantics as possible — each new reference to an object incremented the reference count, and thus required a Dispose() call to decrement the reference count.

Another annoying side-effect of this design was that SlimDX objects rarely compared equal, even if they represented the same native object. In other words, foo.GetBuffer() == foo.GetBuffer() was never true.

A Better Way

For a while, we simply lived with the strange disposal model. But as we started wrapping up open issues for the March release, it became painfully obvious we were going to have to take some drastic measures and refactor large portions of SlimDX’s internals, because the designs were making the library extremely difficult to use.

So, after a number of involved design discussions in #gamedev, we came up with a new design for the March release. This time around, we’d use an internal hash table to track the native objects and map them to an existing SlimDX object. When a new object is needed (for example, when GetDevice() is called), we first look in our hash table to see if we already know about the native object in question. If we do, we return the existing managed wrapping object, otherwise we create a new one and store the mapping in the table.

Additionally, we manually call Release() on native objects whenever the underlying DirectX API implicitly calls AddRef() — in effect, we ensure SlimDX’s only reference to the native object is the one stored in the hash table. You can call GetDevice() as many times as you like, and it will always return the same reference and the net COM reference count will remain the same after the calls as it was before. As a result, calling Dispose() on the object returned from GetDevice() is unneccessary — the only reference that needs to be disposed of is the first, exactly the way IDisposable is supposed to work.

It’s Not All Perfect

Unfortunately, while this new design is a huge step forward for SlimDX in terms of correctness and consistency on the managed side, there are still a handful of pitfalls. Thankfully, those pitfalls exist largely in edge cases that will affect a very small percentage of SlimDX users, as opposed to the original problem, which affected every SlimDX user. I’ll talk more about what those problems are next time.

An individual conversion in Shade is a simple thing, essentially a conversion is a highly-specialized shader feature. A conversion has semantic types for input and output, as well as a snippet of code code that will perform the conversion. Shade’s conversion search routine takes a source and target parameter, and returns a list of conversions and an integer representing the “difficulty” of the conversion. Difficulty is a heuristic. The only values with a concrete meaning are 0 (the types match exactly) and int.MaxValue (the conversion is impossible). Beyond that, difficulty is only useful as a relative measure — lower, of course, being better.

In order for a conversion to be possible at all, the source type must contain at least all of the trait names as the target type (but the values can, of course, be different). Conversions may not add new traits to a type, they can only change or remove them. So, for example, it is possible to convert from type:float4,basis:model to type:float4,basis:view. It is possible to convert from type:float3,use:normal,basis:clip to type:float3,basis:world (discarding the use trait in the process). However, it is not possible to convert from type:float3 to type:float2,basis:view (without information about the basis the vector originates in, there’s no sane way to bring it into a defined basis). This relationship is referred to as the compatibility of the two types, and it is not a transitive relationship. It’s also not a boolean relationship - types that have more traits in common are more compatible (type:float4,basis:world is more compatible with type:float4,basis:clip than it is with just type:float4. type:float4, however, is not compatible at all with type:float4,basis:world).

The traits that exist on both the source and the target, but whose values differ, are known as the “candidate traits.” The psuedo-type that is defined by the candidate traits is checked against every conversion available to Shade. Any conversion with an input that is compatible with the source type is a potential addition to the final result set. For every potential conversion found, a temporary type is created that represents the type of the original source parameter after application of the conversion, and the conversion search routine is repeated recursively using this new parameter as the source. The stopping case is when the source and target parameters are the same (success) or no acceptable conversions exist (failure). The child conversion with the lowest difficulty is selected as the result, and returned.

One thorny issue is that conversions discard any trait the source has that the target lacks. This is undesirable, because it means that the deeper the graph, the more likely you are to lose a lot of the useful semantic information that improves the stability and utility of the tool. One direction I plan on exploring in the future is modifying the conversion search so that such traits are simply propagated through unchanged, to prevent this problem from occuring.

Shade produces HLSL code when compiling a shader graph. Like most languages, HLSL’s type system is “simple” in the sense that it is only concerned with the fundamental type of a variable — this is perfectly acceptable for programming. Shade, however, requires more information to achieve its goals.

Take a look at the closeup of the example shader graph from the last article, below. The skinning node produces a “normal” output, containing the skinned normal. The diffuse lighting node expects a “normal” input so that it can perform the standard n-dot-l diffuse lighting calculation. Each shader feature is written in isolation, and generally has to make some assumptions about inputs. For example, this particular diffuse lighting feature assumes the input normal vector is in world space, because the shader global containing the light vector happens to be in world space. It would really clutter the UI to include all those assumptions on the display, and more importantly it would be exceedingly counterproductive for Shade to require the user to manually place little feature nodes to perform the appropriate conversions between coordinate spaces.

Fortunately, by adding additional information to the parameter types, we can have Shade inject those conversions automatically. This is the basis of Shade’s semantic type system. A semantic type is just a collection of name/value pairs called “traits” that describe contextual features of the type. The only trait that is required is that describing the fundamental type. Additional traits are optional, but more information is generally better (especially when finding conversions is concerned, as will be addressed in a later article). For example, the semantic type of the skinned normal is type:float3,use:normal,basis:model,range:unit. The input to the diffuse lighting feature is type:float3,use:normal,basis:view.

Imbuing the parameters with this extra information requires some additional effort on the part of the programmer crafting shader features, but the result is a more reliable interface for the artist building effects with those features since there is more information for the compiler to work with — it can perform more comprehensive sanity checks and automatic boilerplate. Without the extra semantic information, Shade could still generate HLSL that was correct and compiled. After all, the fundamental types of both parameters are the same — float3. But the extra information allows Shade to better understand the contexts in which the parameters exist, and makes it possible to ensure the values passed through those variables are put in the proper contexts before they are used.

The end result is a system that is both safer and easier to use, allowing more errors to be detected while at the same time keeping the graph uncluttered and not bogging the artist down with grunt work. Although the example shader I’m presenting here is quite simple and contains fairly low-level features, the ultimate goal of the tool is to have a system that involves fewer feature nodes (as a result, the feature nodes will be higher-level in nature). Part of that motivation is because this kind of visual pseudo-programming can become unmanageable quickly as the number of features increases; the semantic typing system allows for a great deal of clutter to be removed.

Next time, I’ll take a look at how Shade actually searches for and applies conversions between semantic types.