Delta Engine

Delta Engine Blog

All about multiplatform and game development

Writing your own programming language with the DLR (Dynamic Language Runtime)

Image stolen from this great article
In accordance to the very good blog with many useful links for IronPython at ironpython-urls.blogspot.com I will also try to post some useful links about my recent DLR (Dynamic Language Runtime for .NET) research.
I have been working a bit on the DLR before, mostly together with Silverlight, which was cool, but Silverlight was way too hard to work with and I still think it is not distributed enough. On that note if you are interested in Silverlight, check out the . And even before that quite a bit with the Visual Studio SDK, early IronPython versions and other language implementations in .NET and even with native c code (but usually I just modified existing samples). I have also modified Lua for my own needs recently and made it run on the Xbox 360 and PS3 and modified some behaviour for my projects, since we use it for our upcoming game and I use it quite a lot together with IronPython on the tools I write at work.`

But my overall goal remains: Writing my own language, preferbly running on .NET and the DLR seems to be a good fit. Today after we finished the latest demo of our game for the GDC next week, I had finally some time and motivation to get this thing started again :) I probably should do some more XNA projects since I'm a DirectX/XNA MVP and I have still something I work on from time to time, but it is still a long way to go until this project is completely done.

Okay, to get things started again I searched for some new DLR information and samples on the net and here are some helpful links:

The Tools:
Learning the basics and also some good blog posts:
Compiler Books:
Other useful DLR links:
Okay, back to improving my lexer and parser. I think I got the research aspect covered for now.

Making Lua run on the Xbox 360 and PS3 (native code)

I have been working quite a bit in Python and Lua last year and I also tried to use those languages in our current game project at exDream (will be announced shortly, so I'm finally allowed to talk about it ^^). While both Python and Lua are written in C (there are obviously other ports like the great IronPython project for .NET, but I'm not going to talk about that in this post), it was not easy at all to make Python run on the Xbox 360 or PS3. I was able in early testing to get some parts running on the Xbox 360, but too much was missed and commented out and the PS3 port looked like way too much work. So I decided to just use Lua, which is probably a better choice for our game anyway since I only use it to do some AI and physics code with it along with some settings. I'm not going to benefit from the Python libraries either way, they are not needed for our game and they won't run on those consoles without serious effort anyway.

Ok, so I was writing some Lua code and used mostly LuaInterface to make the Lua code run from .NET and of course to work with all my .NET code and even XNA. The second step was to make this run in our new game engine, which is written in native code (C++) since the whole engine is based on the Unreal 3 Engine. For that I used the normal Lua 5.1.4 release, which compiles just fine and works great. Since Lua is written completely in ANSI C I was able to quickly test some code on the Xbox 360 and PS3 too. This is important because our game will be a multi-platform title (PC, Xbox 360 and PS3), which is thanks to the Unreal 3 Engine a much simpler task than writing our own engine for all those platforms (it has obviously it own disadvantages, not only do we have to use native code, but we also have to work with a lot of code in the engine that we don't really need for our game).

So far so good. While it sounds like a simple task to make Lua run on all platforms, there is a lot of pain involved to make it run perfectly and behave the same way on all platforms. Since the Unreal 3 Engine is written in C++ and uses its own scripting Language UnrealScript, it does not help out a lot when you want to integrate your own code or libraries in there, so this was half of the work (just see my previous post about compile times and you see why this is a much lengthier process than writing some .NET code). Secondly there are many special rules on both the Xbox 360 platform and the PlayStation 3, some ANSI C methods are just missing or just don't do anything because of the platform design. For example on the Xbox 360 you can't use relative paths, you always have to specify a full path when opening a file, which is something that the Lua engine does not do out of the box. After figuring out where to store the Lua scripts on the consoles and how to load them, everything else just falls into place and works finally as expected.

To make things simpler for other people trying to run Lua code on the Xbox 360 or PS3 I wrote the following little solution:
As you can see on the right the solution consists of 4 projects:
  • LuaSupport, which is based on the current Lua 5.1.4 release from 22 Aug 2008 with my own additions to support the XBOX and PS3 platforms (uses XBOX and PS3 defines).

  • TestLuaPC is a simple test project for the PC to make sure everything still runs as expected on the PC. Please note that on the PC Lua assumes that all files can be found in the current directory. I just change to the Lua script directory and then execute the Lua files there normally.

  • TestLuaPS3 will probably only compile if you have the PS3 SDK installed, just unload this project if you don't have it installed yet. Also please test other SDK samples first to make sure everything is correctly setup (it uses a lot of extra include paths). In case you are not able to open the files, check for yourself if the /app_home/Lua/ directory contains the required files after building the project (use the ProDG Target Manager for that). Debugging can also be complicated, I guess it does not work on Windows 7 at all, but I was able to debug and run the app on a normal Windows Vista PC. In any case you can also just run the .elf file from the ProDG Target Manager and then see the output in the console.

  • TestLuaXbox360 is the project to test Lua on the Xbox 360. You will also need the XDK (Xbox Development Kit SDK) installed and have a Xbox 360 DevKit in your network for testing. A TestKit also works, but you won't be able to debug, you can just run the app and check the console output yourself.

Some of the Lua functionality is not available on the Xbox 360 or PS3 like lua_popen or lua_pclose (among others), and there is also some code that will not do anything like os_execute, os_getenv and some other os methods because they won't make sense on a console anyway or could not be implemented. I did not use these methods anyway, but if you need those and want to write your own code for them, go ahead.

Lets focus on the 2 most important code changes. First of all the Lua default directories had to be adjusted in luaconf.h:83 to support Xbox 360, PS3, PC and Linux platforms:

#if defined(XBOX)
/* On the Xbox we just use the D:\ drive for everything, all lua files should be
   located there */
#define LUA_LDIR    "D:\\Lua\\"
#define LUA_CDIR    "D:\\"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(PS3)
/* On the PS3 we just use the /app_home/Lua/ sub directory for everything,
   all Lua files should be located there */
#define LUA_LDIR    "/app_home/Lua/"
#define LUA_CDIR    "/app_home/"
#define LUA_PATH_DEFAULT  \
    LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua;" \
    LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#elif defined(_WIN32)
/*
** In Windows, any exclamation mark ('!') in the path is replaced by the
** path of the directory of the executable file of the current process.
** Note: We use the ..\Lua\ directory by default, but that is also the current
** directory, we change it in AIEngineInterface before running scripts!
*/
#define LUA_LDIR    "!\\..\\Lua\\"
#define LUA_CDIR    "!\\"
#define LUA_PATH_DEFAULT  \
        ".\\?.lua;"  LUA_CDIR"?.lua;"  LUA_CDIR"?\\init.lua;" \
        LUA_LDIR"?.lua;"  LUA_LDIR"?\\init.lua"
#define LUA_CPATH_DEFAULT \
    ".\\?.dll;"  LUA_CDIR"?.dll;" LUA_CDIR"loadall.dll"

#else
/* Default linux/mac directories */
#define LUA_ROOT    "/usr/local/"
#define LUA_LDIR    LUA_ROOT "share/lua/5.1/"
#define LUA_CDIR    LUA_ROOT "lib/lua/5.1/"
#define LUA_PATH_DEFAULT  \
        "./?.lua;"  LUA_LDIR"?.lua;"  LUA_LDIR"?/init.lua;" \
                    LUA_CDIR"?.lua;"  LUA_CDIR"?/init.lua"
#define LUA_CPATH_DEFAULT \
    "./?.so;"  LUA_CDIR"?.so;" LUA_CDIR"loadall.so"
#endif

And the most important code change is located in luaL_loadfile method, which is called by the lua_dofile macro. Please note that I did not change the code formating and tried to leave as much code as is. I have my own coding conventions, but it does not make sense to apply them when only changing <10% of the code of an existing library.
LUALIB_API int luaL_loadfile (lua_State *L, const char *filename) {
  LoadF lf;
  int status, readstatus;
  int c;
  char fullFilename[128];
  int fnameindex = lua_gettop(L) + 1;  /* index of filename on the stack */
  lf.extraline = 0;
  if (filename == NULL) {
    lua_pushliteral(L, "=stdin");
    lf.f = stdin;
  }
  else {
    // Note: Always open as the requested file, Lua should not care about
    // our crazy directory remapping.
    lua_pushfstring(L, "@%s", filename);

#if (PS3)
    // On the PS3, check always the /app_home/Lua/ directory!
    // This works both for dev-testing via remote HDD and on the game disc.
    // These work probably too just for developing, but app_home is fine!
    // /host_root/Lua/ or /dev_bdvd/PS3_GAME/USRDIR/Lua/
    CheckForValidFilenameWithPath(fullFilename, filename, "/app_home/Lua/");
#elif XBOX
    // For the Xbox we have to make sure always to load files from D:\ because
    // fopen ALWAYS expects a full paths, there are no current directories on the
    // Xbox 360 and therefore no relative paths! Check always "D:\Lua\<file>"
    CheckForValidFilenameWithPath(fullFilename, filename, "D:\\Lua\\");
#else
    // On the PC we just use the default search logic (see luaconf.h) and we
    // don't care about directories since we will already be in the correct one!
    // In earlier versions we had a lot of extra checks here.
    strcpy(fullFilename, filename);
#endif

    // Rest of this code is untouched, we just use fullFilename now!
    lf.f = fopen(fullFilename, "r");
    if (lf.f == NULL)
      return errfile(L, "open", fnameindex);
  }
  c = getc(lf.f);
  if (c == '#') {  /* Unix exec. file? */
    lf.extraline = 1;
    while ((c = getc(lf.f)) != EOF && c != '\n') ;  /* skip first line */
    if (c == '\n') c = getc(lf.f);
  }
  if (c == LUA_SIGNATURE[0] && fullFilename) {  /* binary file? */
    lf.f = freopen(fullFilename, "rb", lf.f);  /* reopen in binary mode */
    if (lf.f == NULL) return errfile(L, "reopen", fnameindex);
    /* skip eventual `#!...' */
   while ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;
    lf.extraline = 0;
  }
  ungetc(c, lf.f);
  status = lua_load(L, getF, &lf, lua_tostring(L, -1));
  readstatus = ferror(lf.f);
  if (filename) fclose(lf.f);  /* close file (even in case of errors) */
  if (readstatus) {
    lua_settop(L, fnameindex);  /* ignore results from `lua_load' */
    return errfile(L, "read", fnameindex);
  }
  lua_remove(L, fnameindex);
  return status;
}

The code in luaL_loadfile will call the new method CheckForValidFilenameWithPath, which helps us to build a full path filename for the Xbox 360 and PS3 platforms. It will just add D:\Lua\ or /app_home/Lua/ in front of the filename if it does not exist yet. Here we can see another pain point of C code, it takes so freaking long to do very simple tasks (in .NET this would be a 2-liner).

static void CheckForValidFilenameWithPath(char* fullFilename,
  const char* filename, const char* platformPath)
{
  int i = 0;
  int pathLen = (int)strlen(platformPath);
  // Already have a filename with the correct path?
  if ((int)strlen(filename) > pathLen)
    for (i = 0; i < pathLen; i++)
      if (filename[i] != platformPath[i])
        break;

  // Found path?
  if (i == pathLen)
  {
    // Then just use the existing file, happens when loading libraries from Lua
    strcpy(fullFilename, filename);
  }
  else
  {
    // Copy path
    for (i = 0; i < pathLen; i++)
      fullFilename[i] = platformPath[i];
    // Add the relative filename
    for (i = 0; i < (int)strlen(filename); i++)
      fullFilename[i+pathLen] = filename[i];
    // And finally close the string
    fullFilename[(int)strlen(filename)+pathLen] = 0;
  } // else
} // CheckForValidFilenameWithPath(fullFilename, filename, platformPath)

And that are the most important code changes to make everything run on the Xbox 360, PS3 and PC. Here is the main method for all the TestLua projects (always pretty much the same, only the Xbox 360 uses a slightly different way to print to the console):

int _tmain(int argc, _TCHAR* argv[])
{
    printf("Initialize Lua\n");

    Lua = lua_open();
    luaL_openlibs(Lua);
    lua_register(Lua, "print", lua_print);

    // Simple print test from Lua:
    luaL_dostring(Lua, "print(\"Hi from Lua\")");

    // Now load a Lua file, which also loads another Lua file (LuaUnit.lua)
    if (luaL_dofile(Lua, "SomeClass.lua") != 0)
    {
        printf("Error: Unable to execute Lua file %s: %s\n", "SomeClass.lua",
            lua_tostring(Lua, 1));
    } // if (luaL_dofile)

    // Stop the Lua engine and cleanup
    printf("Closing Lua\n");
    lua_close(Lua);
    Lua = NULL;

    // Keep command window for debug mode (ctrl+f5 does not have this issue)
    char input[100];
    gets_s(input);

    return 0;
} // _tmain


Summary:
  • Using Lua on other platforms is easier than Python or other dynamic languages I have checked because Lua is written completely in ANSI
  • Making code run on multiple platforms is still hard in C++ because you not only end up with ugly #define code, but you also have to test a lot
  • Building native applications is freaking time-intensive, it took me almost a full day to write this app and test it on every platform from serval PCs. In .NET it would take me less than an hour, but then again there is no .NET on the PS3 and our current game engine is using native code anyway.
  • On the Xbox 360 you always have to use full paths like D:\Lua\SomeClass.lua, relative paths won't work. The SDK says you should use Game:\, but since our game engine uses D:\ too, I left it that way in this sample too.
  • The PS3 is even more complicated with lots more possible mounted paths, but it seems /app_home did always work in both my samples and our game, so I'm sticking with it :)
If you are interested in Lua from .NET, check out the LuaInterface project. Someone also started NUA (Lua on the DLR) 2 years ago, but it seems this project (and many other attempts) have died a long time ago. There was also a port by the Lua guys (especially the great Fabio Mascarenhas, who has also started the LuaInterface project and is still involved) called LuaCLR, but it is still in alpha stage and pretty useless, LuaInterface on the other hand is a fully working library! The difference here is that LuaCLR emits CLR code like any other .NET language, while LuaInterface still relys on the Lua runtime. LuaInterface uses only 2 very simple C++ files, the rest is in C#, but because of that you currently cannot run it with XNA on the Xbox 360. IronPython, while fully written in C#, does not work on the Xbox 360 either because reflection is not supported by the Xbox 360 .NET framework.

And finally here is a link to Lua running on the iPhone: Lua v5.1.3 for iPhone/iPod, kinda cool, but since we can run C# code on the iPhone with help of the great Unity3D engine, I'm happy with that at the moment. The main problem for other languages on the iPhone is the missing libraries anyway (can't use Cocoa or any own UI) ..

Just in case you have not read about it yet: John Carmack has ported the good old Wolfenstein 3D game from 1992 to the iPhone as well.

Increasing performance on the iPhone

After finding our game not running as fast as we expected, I dedicated the last 3 days of work on finding ways to increasing the performance.

There are 2 issues which have a heavy impact on the framerate. Drawcalls and Physics.
Both have several roots and solutions.

For the graphics part you should decrease the drawcalls as much as you can. Since every object has its own drawcall it doesn't take too long to have several hundreds of themm, which is by far too much. What you want to do now is combining them. Basically telling Unity to draw them as one object. This only works for objects that share the same material, so you might consider using the same texture with some fancy UV layouts for several objects. You start by creating an empty gameobject and assigning the desired objects as its child. Next you need the CombineChildren script, which is found in the Standard Assets.unityPackage and assign it to the gameobject you created earlier. Tadaa, all those objects will be drawn in a single drawcall now.
If you're working on a 2d game you also might consider using the SpriteManager class, which is a convenient way to draw lots of sprites in a single call. It supports UV-Animation, Billboarding and other goodies you might enjoy.
You can optimise your lightning by creating lightmaps for your meshes in your 3d composing tool instead of using dynamic lights, since Unity has no baking function.
Another rather unlikely scenario is a too high polycount. But remember just in case: it's an iPhone, it has a tiny display, you won't even see that much of a difference between 500 and 2000 poly characters, and even if, it's still just a phone. :)
That said you should be fairly well if you stay within this stats:
Drawcalls < 30
Polycount < 10k

As for the physics, if you're using a lot of physical objects, you should consider using the simplest collider you can (e.g. enabling Convex mode on MeshColliders, using SphereCollider instead of BoxColliders)
If this doesn't help, you have the possibility to decrease the precision, or increase the collision checking interval. Head to the Menubar>ProjectSettings>Time and increase the Fixed Timestep. Remember that decreasing the physics precision can lead to penetrating objects, objects getting stuck in each other, and other graphical impacts, so you should thoroughly test which works best for you. If you still want to get more out of it you can consider slowing down your gameplay. While doing this you can increase the Fixed Timestep even further.

Fixed Timestep Defaultvalue: 0.02

That sums it up. You can find more detailed explanations and help in the Unity Forums and the irc channel #unity on freenode.org

Happy coding

Experiences with the Intel's X25-M 80GB SSD and compiling native code in Visual Studio

Last week I finally bought a Intel's X25-M 80GB SSD and I have been using it for about a week now at work. A few weeks back I also posted my impressions about the cheap OCZ 64 GB SSD I used before and how bad it worked out with Visual Studio compiling for me (also some helpful links in there if you want to optimize your SSD).

My first impressions of the Intel X25-M 80GB are as follows:
  • Initially very fast read, easily archived over 210 MB/s read performance with HDTune and other benchmarks.
  • Blocks above 4kb were faster, smaller blocks were slightly slower. 4096 bytes also seems to be a good NFTS block size for them.
  • Writing on it is slower with 70-80 MB/s, which is less than the OCZ SSD I had before in benchmarks. But real world usage tells another story as I was usually limited by the network, reading data from other disks and other factors. Other than the OCZ SSD the Intel one stayed very fast even with many files and programs open and doing crazy compiling in the background.
  • Using Windows Vista is fine, but I remembered how much better Windows 7 was for SSD when I was using it a month ago. You can tweak Vista quite a bit with the helpful tips from the OCZ Foraum, but it is not the same thing as having the operating system handle all the dirty stuff for you in a much better way.
  • So I'm back at Windows 7 again. Yes some apps and games do not work and I even have problems now doing some work related stuff (building content) for which I have to use another PC or use my secondary disk and boot Vista for that. But most tools I use everyday at work are just fine in Windows 7. Boot time is reduced, memory usage is way better, SSDs work great and the overall desktop experience is just so much better (I love the jiggle windows trick to show the desktop).
  • Compile times for our game have gone way down (more on that in a second).
  • When I'm going home and do some work there I clearly notice the difference. But even if I just open up FireFox or some other complex program, it seems like it takes forever on a normal hard disk. I was pretty happy with my system at home before, but I have not upgraded it for 2 years. It has never really bothered me before since I can still play all the games in Full-HD and everything is quick in Windows. I will probably wait until some cool new DirectX 11 cards will come out and of course some cool 8-core CPUs for some serious multicore action.
  • I can sense a slight slowdown to last week, probably due the fact that I use the SSD each day heavily and it is pretty full. Hopefully it will not get worse over time, but even then I could quickly just copy everything over to my secondary disk, reformat and copy everything back.
  • Do not believe most SSD reviews, it is really stupid that in the last 6 months almost no SSD review checked the usability of those things, even Linus Torvalds agrees with that (he is a Intel SSD user too ^^).
  • Also check out this very good new article on AnandTech: The SSD Anthology: Understanding SSDs
I could probably go on and on about this and there are certainly advantages (yeah, reading is fast) and disadvantages (wtf, this is costly, and you will run out of disk space sooner than you can say "copy this over there"). Lets take a look at some random numbers of compile-time from our current game project (which I'm still not allowed to talk about, so just pretend it is a huge C++ project with many many files). Please note that all those times are not in any way accurate, both because I just checked a few times and estimated others and it is also very different from PC to PC anyway plus there are many other factors like having intermediate files already present or not, etc. I'm also not a hardware tester like Tom's Hardware Guide, go there if you want more and real benchmarks, this is just what a noticed in the last weeks. Keep in mind that I could only overclock my current Intel i7 setup a few days ago since the default CPU cooler by Intel for i7 is really crap, loud and can't cool very good. So I bought a Zalman 9900 instead and it allowed me to go to 3.75 Ghz without any problems (up from 2.66 Ghz default).

Older PCs we have at work are obviously slower anyway, but the point here is that the SSD gives an additional bonus. I also only included the PCs I have worked on, some interns have slower PCs, but they usually do never have to do a full recompile anyway. Keep in mind that I do not compare .NET compile times here (they are all below 2 seconds on all those systems even in the larger .NET projects I have). This is all unmanaged C++ code, but since we also do a lot of our code in scripts we do not have to compile C++ most of the time - and even when - we usually only change certain parts of the code and then a incremental build is used.



PS: I was watching the current South Park S13E02 Episode while writing this and I think it was pretty crappy (superheroes, no real story). Last weeks episode with Kenny's girlfriend was way better and a really good season starter.

Fun: Apple Compatibility

Just in case you are bored like I am right now .. ups, getting back to work ..

Computer skills I have vs what friends and family think about

While I'm still preparing my big post about Lua and Consoles, here is something funny in the meantime ..

So true (found at GraphJam):

iPhone 3.0 will finally allow copy+paste


Lol, I had to laugh when I read about this:
iPhone 3.0: Push Notifications, Copy and Paste, MMS, and More

Our game is coming along quite nicely btw, we solved most performance issues and are testing on the iPod and iPhone heavily (prior to this we could only test on the simulator, which ran much faster .. strange).

iPhone/iPod testing

So we eventually got our distribution keys after ~1month of waiting. Apple really disappointed us in that matter. And after figuring out how to get our application run on the iPod took us half a day. The setup is not exactly intuitive. You have to go back and forth from the Developer Portal on apple.com, the XCode IDE, Keychain and whatever else.

So now we can actually Build & Run our game from within Unity3D on our iPod, and not a second to early. Lots of issues showed up, some of them being casual freezing, not switching to wide-view, frame dropping, and not running as fast as we expected. Hopefully they will all be sorted out by the end of the next week as the deadline draws nearer and we dont plan to push our release back.

Also, for some reason, the live preview feature doesn't work at all, i'll have to dig into that, because compiling and running the application everytime is pretty slow and bothersome, and doesn't easy the process of debugging in any sense.

So far so good,... i'm glad it works at all.

Wir sind drin! #offizieller iPhone Entwickler

Endlich ist es vollbracht - nach wochenlangem warten (insgesamt hat es nun fast einen Monat gedauert) und zahlreichen emails hin- und her konnte ich gerade endlich unseren Account aktivieren und wir sind nun stolzer offizieller iPhone Entwickler!

Dann wollen wir mal hoffen, dass die bisher entwickelten Sachen nicht nur virtuell sondern auch real auf dem iPhone laufen ...

Samsung SSD Awesomeness

One of my interns, Stefan, just send me this, this is hilarious:


The Samsung 256GB SSD MLC still seems to be expensive (and is not even available in online shops here in Germany), costs probably around 800 euro anyway. It does not seem to use the crappy JMicron controller, but I could not find any good benchmarks for it.

Today I bought a Intel X25-M 80G SSD today, which I will use instead of my current one from OCZ, which pretty much sucks for Visual Studio compiling as reported before.