Native app delivery, revisited

Web applications these days are taking increasingly more space on the market and attention from billions (if not more) users worldwide. They have been there for quite a long time and while the overall level of complexity is amazing, it is also preventing those applications from ever truly being the best, UX-wise, and thus are not capable of replacing native applications neither completely nor partially for most use cases.

Due to browsers becoming increasingly common, they have been made to work as application delivery platforms. Because of that, user experience suffers. Offline availability, performance, input responsiveness, graphical fidelity and uniformity across platforms – all of these things are compromised for the sake of getting something on the screen quicker.

It’s hard for me to watch this happen. UX is too important to let this quantity rush mess it up. This is why I’m trying to identify in what ways web applications have succeeded and outperformed native applications and finding ways to turn that around or at least minimize the difference.

Last week I was working intensively on a delivery method for native applications. This method is centered on an application that acts not only as a game archive and setup system but also as a platform for instant extract-launch-clean workflow support. For the user, it means that only two or three actions at most are necessary to run something – download the application, open it, and optionally click on a button to start it.

NAILER is the name. Native Application Installer / Launcher. v0.4 supports building a self-sufficient executable that can deliver an application and let people run it from anywhere, anytime, provided that they have some free space in their temporary file folder. Currently only Windows is supported since a lot of stuff has to be done in completely different ways for other operating systems. How fast the support will come depends on you. I’ve implemented the MVP, it’s time for you to decide how useful you think it is.

The repository can be found here.

Binaries / Sample images & script:

Script syntax can be currently read from the sample script. Dry-running nailer-builder displays command line syntax. I’ll write some docs if there’s interest.

First game package ever built (Windows version of my LD30 game: Connected Worlds – The Adventure) (Dropbox mirror)

In closing, all I want to say is – let’s test it and see how useful it is. I’ve found lots of unexplored space while building the system and I would like the chance to explore it – support the project by using it and I will make it even better.

Serialization of game data

I’ve taken some time to update my Ludum Dare game with some checkpoints (check the v1.1 version if you want to see them in action). Felt like I had to do this not only because people asked for it but also because it’s a very important thing to have for any bigger project. And because I’ve never attempted to do something like full state serialization for a SGScript game yet.

So I’ve wrote some code, and it didn’t go quickly enough, I felt like I hadn’t covered all use cases so I decided to write a plan first.

Most of the idea is rather well outlined there. Though there were some corrections made in the final version of the code that you can see in engine/system.sgs.

As for the user code – there is some work to be done – but it’s all quite simple. Here’s a few iconic bits of the entity serialization code:

  1. entity is fully parametrized so only recreation is necessary;
  2. entity should not be recreated if possible so a reuse-testing function is added, it checks if the entity has the same type and it loads the same mesh;
  3. entity has both parametrized and internal description data;
  4. a “singleton” entity (no parameters, created once, never rebuilt, no more than one instance) is serialized – reuse-testing function only checks the type.

Even though it was simple to implement, I’m glad I didn’t do it at the jam. It took some planning and well over 3 hours (split over two days) to plan the design, implement it, fully test it and learn to use it well.

After the LD release

Before I proceed with the review, I’d just like to mention – this is the game I made. You might want to have a look at it before reading further to have some context when I talk about the specifics.

What went right: development estimates, luck and controlling chaos & unfinished code.
What went wrong: lack of a fully developed engine / editor combo, deployment process.

Development estimates and luck – there’s not much to say about those. I always had spare time on my hands to fix whatever popped up, there was no stress to get something done in time, no major feature cutting. Might be partly related to the lack of a fully developed initial plan. I just went in with my tech and hoped I could get an idea of the gameplay halfway in the process. And that did happen.

Controlling chaos & unfinished code – what I had at the beginning was a completely unfinished engine. No working physics, not even fixed-key input handling or cursor locking/relative mouse movement, no pause screen, no 3D-related entities (sprites, lights, meshes), no character physics, no cutscene system. Things so many people take for granted to be working and polished when they begin working on a Unity or Unreal Engine based project.

It was very easy to write all that, provided that there was (as in my case) some previous experience with all of those things. It was also easy to add game action hooks to the engine (like non-entity tick/input events). I also dropped the original entity system in favor of a much simpler one (each entity is defined by a creation function), fixed some bugs in the engine too.

A note regarding code simplification: Every dynamic language takes a lot of weight off your hands with dynamic (re-)binding, runtime (re-)evaluation, first-class functions. My advice: trust those systems. It will save you a lot of code generally spent on the premise that any such system could fail to be robust enough. They almost never do.

Lack of a fully developed engine / editor combo - as much as I liked to write Bullet Physics wrapper code and fix bugs in my engine, I would prefer to spend that time somewhere else. Or maybe not, but others definitely would. People would also prefer (and here I join them) to use an editor to add things to scene, not copy & paste coordinates out of Blender. It’s what I did for the position data. Writing any other non-spatial/visual data is just fine, though. I would in fact prefer writing to fancy controls. But not things that need immediate visual feedback.

Here’s a few more things that should’ve been available to me:

  • Specular lighting – regardless of the type, it requires strong visual control. I have lights from far away leaking reflections through walls. I like unattenuated (by distance) specular because it’s the only kind that looks good and real. But it’s much harder to control in large scenes without shadows.
  • Occlusion culling – it was actually in development but not quite finished yet. It would probably help control those lights as well as improve frame rate and allow to increase levels of detail a lot.
  • Environment lighting generation – lightmaps and (blurred) environment maps would help a lot here.
  • Mesh data extraction from the exported file format – I actually had to write this one while working on the game, otherwise there would’ve been no physics, and no game.
  • Last but not least – support for other platforms. I really wanted to get more people to play this. Right now, it’s not happening.

Deployment process – there’s no errors like the ones you don’t get on your machine. The release started off with a “missing libgcc_s_dw2-1.dll” error. GCC was nice enough to screw with me at some point and didn’t let me know exactly when that happened. These things should be statically linked by default, though I should’ve checked all DLLs myself. It’s always too easy to forget that.

What followed was a silent failure. Most likely something simply wasn’t extracted from the archive but the engine should’ve notified the user. So that’s a bug, and it’s fixed now. Someone else couldn’t start the game without any explanation, that could’ve been the same bug. Or another one…

I received another report about a “variable of type ‘null’ cannot be called” error. Classic, means that the function doesn’t exist or the variable is not a function. Checked the file at the specified line – SS3D_CreateRenderer – SS3D addon silently wasn’t loaded. I’m going to upgrade all include messages to errors (SGS_ERROR) to fix that. Optional includes will be possible with either pcall or @include("...") (error suppression on function call). I should’ve handled such errors manually, though.

^ this is all that’s required to fix it. Yeah, a missing d3dx9_**.dll triggered that obscure error. Dependency errors vs. me – 2:0.

Edit: In fact, it would be best to not link optionally available DLLs statically. When I finally get to making multiple renderers for SS3D, it should be possible to run the OpenGL renderer if the Direct3D one isn’t available or at least notify the user about the action to take automatically.

So, what we can conclude from all that? Dependency issues are generally one-off (with some recurrence) so there isn’t really any need for automated tools. As for the rest – the value of a language strongly depends on the amount and value of bindings it has.

And that’s all I need to know to determine the direction of development.

As Fast As a Speeding Bullet

Before I state the successes that were already achieved, I just want to point out that the game is not finished yet, though well on it’s way to be delivered after 12 hours or so. Here are the blog posts about the game / my participation in LD30: 1, 2, 3 and 4.

As the preparation for LD30 went on, I became increasingly interested in making something really big. Something 3D.

The original idea was to make a third person platformer but as the theme was announced, I was not sure if I could pull it off. After all, the 3D addon to sgs-sdl does not yet support skinned meshes, which would certainly come in handy for the playable character.

What I did know and hold on to was the idea to quickly push out an incomplete 3D physics engine wrapper. In well under 24 hours of total investment (not an uninterrupted period of 24 hours, mind you, I’m not *that* crazy), a usable Bullet Physics wrapper was made, with a discrete dynamics world, rigid bodies and ghost objects, box/sphere/capsule/plane/triangle mesh shapes, raycasts, convex casts and possibly something else I’m forgetting now. Anyway, check the /bin and /src directories in the link above to see exactly what’s been implemented.

C++ binding compiler was slightly modified in the process to support inheritance. Without it, I couldn’t have done what I did and that’s not a part of the success. But it is a product of trial by fire, a process which I believe strongly in.

So, bottom line – one dare done successfully, one yet to be completed, a new library was created and SGScript was upgraded to support more use cases. I look forward to the moment when all of what I’ve used in these few days will be fully documented and available for others to use, with much less effort than what was required from me.

Connected Worlds - Intro Ending

Streamlining Production

As I’m preparing everything to get some quality stuff out (think native modules), not only I have to get documentation generation system in order for it (which I’ve almost done anyway), it is also important to improve other aspects of production, such as binary file generation and testing. Regarding those things, four distinct improvements have been implemented recently.

#1: build preparation script has support for partial behavior overrides

This basically means that it’s possible to change commands that will be executed for full rebuild / follow-up cleaning and a few other things. This enables the build script to be reused for projects that include SGScript, as shown here.

#2: makefile has been split for reuse of the core features

Reusing the platform-specific stuff handling code allows to reduce the amount of boilerplate code for additional modules, making things a lot clearer. Not a particularly big deal but removes a considerable amount of weight that would’ve been otherwise become a huge problem in the future.

#3: testing system is made to support running from specified folders

This has the benefit of being able to reuse the original SGScript test framework for other projects, making it easy to quickly see, at any moment, if the code is running as it should. Regression tests are a safe bet if you intend to deal with each bug only once, made easily available by this change.

#4: SGScript version checking macro

This thing should reduce the amount of version mismatch-related crashes to a very round number, zero. These things may happen if different versions of SGScript reside in current directory and the PATH environment variable. Generally the missing symbol errors should pop up but as the API becomes more stable over the time, it might be better to avoid relying on such luck.

So there you have it, four simple changes with substantial benefits. What can be inferred from this is that the first fully produced and implemented SGScript module should appear rather soon. Others are expected to follow that example. All goes well, by the end of the year lots of modules with tests, full documentation and binaries should be available.

But now, time to prepare for LD30.

Advanced property access

This post possibly should be called ‘Meta-objects, part 3′ but the focus has slightly shifted. This time we’ll look at the upcoming features of SGScript that will allow property access customization.

This code creates a class (metamethod-enabled object with a meta-object attached, if you care for the exact term), based on an array object, overriding its property retrieval to return indexed items based on specific names.

This reminds me… the storage could be more efficient on these things (arrays require 2 allocations – one for the object, one for the resizable list of variables), some day I might add a fixed array object if it proves to be a valuable investment.

Even though the concept is simple, it wasn’t possible to do this from scripts, only from the native environment. However, it is mostly a convenience feature, due to the vast amounts of user code being run here, it should not be expected that this is going to be fast enough for everything. At most, in terms of performance, it can only save memory by allowing to avoid copying data and put all kinds of links to that data instead.

But that’s not all. It is also possible to set properties in a similar way:

This piece of code enables setting the same nonexistent properties that could be retrieved. And this is where the really good part comes in: it is possible to intercept property assignments to allow custom code to be executed. This allows for various useful things to be implemented, such as:

  • property assignment debugging – know what has been assigned and when – by emitting a message or reading the call stack;
  • property modification callbacks – do something before or after the value of the property has been changed;
  • custom interfaces – property writes could be routed to some sort of messaging system;
  • a solid understanding of the system – this is really the same as point #2 but the goal is not as defensive – learning some of the dark parts of the system can be made easier;

Last point, illustrated:

…this code returns…

So there you have it, a way to get to know SGScript better. I deliberately left out a couple of things to give you a chance to find out what they are, when v0.9.6 is released.

But that’s not all just yet.

It’s possible to set per-property retrieval/setting functions too, just like in this code sample:

This is useful for localizing behavior of each property in the code (put related stuff together) rather than grouping code by system interfaces, which can be unintuitive at times. And the mm_(g|s)etindex_router functions guarantee two things:

  • you won’t need to write all that routing code over and over again (but you can if you really want to)
  • whenever a JIT virtual machine might be done, special optimizations could be attached to those functions to improve performance

So pretty much every angle has been covered at this point. As I’m writing the post, more changes and bugfixes are applied, all of the mentioned features are rather thoroughly tested.

And I guess that’s all for meta-objects. Or is it…?

The Graphics Framework

The thing I’ve been trying to heavily push forward for the last few days is the SGScript-SDL2 graphics framework (sgs-sdl).

I’m going for the same level of UX that SGScript has so it may take some time. A major push forward has been done recently - the documentation is written and external dependency list is optimized. The new makefile along with a binary building system are on the way as well, some changes in the API are also to be made (and some have already been implemented).

The idea is to create a fully portable system that can be picked up from any of the supported operating systems to any other such OS so that it works anywhere and can be quickly reconfigured for distribution. That, combined with the eloquent API and language, will soon be delivered to you for building your own games, presentations, visualizations or anything else of that kind.

If you want to follow the progress of building the library, you can find all about it in GitHub, specifically the development branch.

As for web support, which I’m sure has been quite a hot topic around the world, all I can say right now is that I’m not prepared to give up anything to get there. If it can be done while maintaining the goals of flexibility and portability, it will be done. But not without either of those things. Excellency in experience must be put high above catering to the lazy and clueless. That is why I do what I do.

Eating native modules without chopsticks

Up until now (one of the latest commits in development branch, as of 2014-08-07), it was not exactly convenient to cleanly construct the directory trees for some work involving SGScript, particularly with binary module loading. It was not possible to easily load native modules from the executable directory if it was not the current one.

All that and a bit more is considered to be fixed at this very moment, since the introduction of process directory into SGS_PATH (the universal loader search path variable) and a function that helps to generate additional entries for that same SGS_PATH.

Not a very huge change but one that I myself have for some reason been delaying all the time, leading to a huge disappointment every time I was working with the “fun pack” that was supposed to be released quite some time ago. I would say that it clearly underlines the necessity for this very feature.

As a nice side effect, the documentation generator is one step closer to being easily run on SGScript-based projects, as shown here, in the new, v0.9.6-compatible dev branch of sgs-sdl.

The update is yet to be tested on other platforms but I’m quite sure that there should be no new problems related to binary loading since an absolute path is used to access them, thus bypassing the seeking systems.

In other news, there’s that documentation for sgs-sdl I’ve been working on. Should be fully covered rather soon and I’ll also fix all the things that don’t make any sense to me anymore. I also look forward to removing the old C++ binding method from v0.9.6, it seems too obscure (even though it’s simple and fast) to be ever truly understood by anyone.

That’s all for now. I still hope I’ll have something more interesting to talk about this week.

SGScript in a single file

I’ve written a single file generator for SGScript to simplify its inclusion into projects.

There are two versions already generated – the full implementation and the no-standard-library version:

Both files have the SGScript version 0.9.6 [API v13], a preview of the upcoming release, mostly compatible with v0.9.5 (and its documentation). The updated documentation, should you choose to accept it, can be found here, though it will probably be easier to download the whole package and extract it from there.

Standard libraries are the formatting (fmt), I/O (io), basic/trig. math (math), O/S (os), regular expression (re), string handling (string) libraries. The no-stdlib version will not have the sgs_LoadLib_* functions as well as any useful output from the include_library SGScript function and things like include "io" will not yield the expected results either. This is good for those that have their own functions/objects to bind for I/O (or perhaps they don’t want any custom I/O at all).

Hopefully this will simplify the introduction to the library. It goes without saying that in programming, nothing more should ever be required to be done than what is actually necessary. Please report any problems you find in the comments section or via mail/Twitter, I’ll try to fix them ASAP.

In the name of serialization

The second mode/algorithm for the serialization system is finally finished.

The whole thing started as a vague ambition to finish the system in the previous day. That, of course, didn’t happen but nonetheless the system is finally done and available for testing in the development branch.

What went right was the idea to keep a written plan like I usually have, it helped me when I needed to refer to the outline of the algorithm. Another good thing is that now serialization won’t interfere with output interface. Or, rather, the other way around.

What went wrong was working late. I guess I’d spent about an hour chasing bugs I created some time ago due to not being careful enough. Can’t really avoid these things but there’s always hope for that.

A very positive side effect was the development of a new debug API. It helped me diagnose and fix the last bugs rather quickly. It’s very simple, along the lines of “you never debug the debug API”, meaning that there is only one thing the function accepts for input and only one thing it gives back as output and it should always be clear what exactly happened. Usage code is as follows:

This works due to the reason that it always pushes a valid string on return. Doesn’t matter if there’s an error or a successful dump. Behavioral minimalism is a good foundation for a simple one-liner.

As for the serialization system, the expected goal is tested – no variable is ever serialized twice. There’s only the limitation of being able to only serialize the caller object, for now. Won’t interfere with existing code but will require an additional API function eventually.

Anyway, that’s all for now. I’d like to move towards other sub-projects now to provide compatibility for the upcoming 0.9.6 version of SGScript. Hopefully there will be some rather interesting things to write about next week.