Jump to content

Elite Force MDR Format


  • You cannot reply to this topic

64 replies to this topic

#1 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 13 March 2009 - 02:53 PM

I just thought I should put this here, just as an update to a crazy thing I'm trying to do.

As we currently know it, here is the MDR format that EF uses for all of it's character models.

/*
==============================================================================

mdr.h

The necessary data formats to define and utilize the EF MDR format.
Credit goes to James Monroe of Ravensoft for providing information on the MDR
format.

==============================================================================
*/

#ifndef __MDRFORMAT_H__
#define __MDRFORMAT_H__

// surface geometry should not exceed these limits
#define    SHADER_MAX_VERTEXES    1000
#define    SHADER_MAX_INDEXES    (6*SHADER_MAX_VERTEXES)


// the maximum size of game reletive pathnames
#define    MAX_QPATH        64
/*
==============================================================================

    MDR file format

    v1 - The original MDR format, provided generously as reference from
         members of Ravensoft. This code should match as-is, the MDR code present
         in EF.

         It appears to be a slightly modified version of an older revision of
         the MD4 format.
         Notable changes include:

         - LOD models are also contained within this file
         - Bones do not carry information on their hierarchy or name (only tag bones do)
         - Bone data is compressed into an array of 24 bytes (as opposed to 12 floats)

==============================================================================
*/

#define MD4_IDENT            (('5'<<24)+('M'<<16)+('D'<<8)+'R')
#define MD4_VERSION            2
#define    MD4_MAX_BONES        128

#define    MD4_MAX_TRIANGLES    8192    // per surface
#define MD4_MAX_VERTS        4096    // per surface
#define MD4_MAX_FRAMES        2048    // per model
#define    MD4_MAX_SURFACES    32        // per model
#define MD4_MAX_TAGS        16        // per model

#define MD4_MAX_LODS        3

typedef struct {
    int            boneIndex;        // these are indexes into the boneReferences,
    float        boneWeight;        // not the global per-frame bone list
    vec3_t        offset;
} md4Weight_t;

typedef struct {
    vec3_t        normal;
    vec2_t        texCoords;
    int            numWeights;
    md4Weight_t    weights[1];        // variable sized
} md4Vertex_t;

typedef struct {
    int            indexes[3];
} md4Triangle_t;

typedef struct {
    int            ident;

    char        name[MAX_QPATH];    // polyset name
    char        shader[MAX_QPATH];
    int            shaderIndex;        // for in-game use

    int            ofsHeader;            // this will be a negative number

    int            numVerts;
    int            ofsVerts;

    int            numTriangles;
    int            ofsTriangles;

    // Bone references are a set of ints representing all the bones
    // present in any vertex weights for this surface.  This is
    // needed because a model may have surfaces that need to be
    // drawn at different sort times, and we don't want to have
    // to re-interpolate all the bones for each surface.
    int            numBoneReferences;
    int            ofsBoneReferences;

    int            ofsEnd;                // next surface follows
} md4Surface_t;

typedef struct {
    float        matrix[3][4];        //bone axis matrix + origin
} md4Bone_t;

typedef struct {
    vec3_t        bounds[2];            // bounds of all surfaces of all LOD's for this frame
    vec3_t        localOrigin;        // midpoint of bounds, used for sphere cull
    float        radius;                // dist from localOrigin to corner
    char        name[16];
    md4Bone_t    bones[1];            // [numBones]
} md4Frame_t;

typedef struct {
    unsigned char Comp[24];            // MC_COMP_BYTES is in MatComp.h, but don't want to couple
} md4CompBone_t;

typedef struct {
    vec3_t        bounds[2];            // bounds of all surfaces of all LOD's for this frame
    vec3_t        localOrigin;        // midpoint of bounds, used for sphere cull
    float        radius;                // dist from localOrigin to corner
    md4CompBone_t    bones[1];            // [numBones]
} md4CompFrame_t;

typedef struct {
    int            numSurfaces;
    int            ofsSurfaces;        // first surface, others follow
    int            ofsEnd;                // next lod follows
} md4LOD_t;

typedef struct {
    int            boneIndex;
    char        name[32];
} md4Tag_t;

typedef struct {
    int            ident;
    int            version;

    char        name[MAX_QPATH];    // model name

    // frames and bones are shared by all levels of detail
    int            numFrames;
    int            numBones;
    int            ofsFrames;            // md4Frame_t[numFrames]        // this will be -ve for compressed models

    // each level of detail has completely separate sets of surfaces
    int            numLODs;
    int            ofsLODs;

    int            numTags;
    int            ofsTags;

    int            ofsEnd;                // end of file
} md4Header_t;


#endif
The ultimate goal is to finally let us make our own models with this format, which would then ultimately work in both EF and ioEF.

I've been looking at the source code that Gongo has provided on the Quake 3 MD4 format here: http://gongo.quakedev.com/

Fundamentally, the formats are the same, with only a few small changes here and there (Enough to make the formats incompatible at least).

At the moment, I've made my own modded version of Gongo's SKL2MD4 converter program, redubbing it SKL2MDR. :)

Unfortunately... I've now found out that the 3DSMAX SKL export plugin I'm using seems to be writing the SKL files incorrectly (it's treating geometry like they're bones), so it's at this point I'm stuck.

I'm hoping with a little more research and effort, it's possible to perfect a method of generating totally brand new MDR files from 3DSMAX. :)

#2 khamseen

khamseen

    Member

  • Members
  • 387 posts

Posted 13 March 2009 - 03:33 PM

Very awesome project idea I wish you the best of luck :)

#3 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 13 March 2009 - 04:33 PM

Thanks... I'm going to need it.

Matrix math still confuses the hell out of me. XD

#4 khamseen

khamseen

    Member

  • Members
  • 387 posts

Posted 13 March 2009 - 06:04 PM

I wouldn't even know where to begin so I really do not envy you the task lol.

#5 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 13 March 2009 - 06:58 PM

The beginning is easy. The file header chunk (ie that md4Header_t struct up there). XD

The hard bit is after that when you gotta write in all of the animation and vertex data and stuff, which of course has to be translated from whatever co-ordinate system the 3d modeler program uses to Quake 3 co-ordinates... XD

The hardest bit right now is the SKL exporter I'm looking at is actually a MAXscript file... and that requires I gotta learn MAXscript before I can edit it... :$

#6 Ryan

Ryan

    Member

  • Members
  • 368 posts

Posted 13 March 2009 - 07:57 PM

We all have utmost faith in you TiM :P

#7 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 16 March 2009 - 03:36 PM

:o

YES!!!!!!! I MADE SOME PROGRESS!!! :rocks:

I as screwing around with the exporter script again tonight.... I think I've preeeetty much figured out how MaxScript works (ie, not like C++ :P ) and although it requires tweaking some settings for it to come out right... it's (more or less) working. :)

So here is a screenshot. Hopefully this is the very first ever non Raven official MDR model made from scratch!!!!!! ^_^
It's a really basic model... it's curved to allow for lighting tests and segmented in the middle to test bone deformation (ie, a real programmer model).

Some stuff that still isn't working right:
-If you choose to compress the model, all of the polys fly out of whack (My bad. I know how to fix that)
-As evident in this pic, the vertex normals are completely messed up (no idea why.... ;_; )
-Not sure why, but surface names are getting destroyed as well. :skeptical:
-Haven't tested textures at all yet. Assuming the worst. :P
-There's next to no error checking yet... so if you don't do it right, the script will crash (and possibly take MAX with it :P )

But either way... here we go. With a little bit more work, we'll finally have a way to properly generate custom MDR models. ^_^

Enjoy!

Oh yeah... jsut an extra interesting note. The tools I've modified to be able to do this were both originally designed for the Ritual UberTools engine, and specifically, the MaxScript was designed for Elite Force 2.

So... suffice it to say, albeit indirectly, it's thanks to EF2 that we can have this tool for EF har har. XD

Attached Thumbnails

  • ef_skl2mdr.jpg


#8 khamseen

khamseen

    Member

  • Members
  • 387 posts

Posted 16 March 2009 - 05:23 PM

Fantastic progress TiM, congrats! :cheers:

So... suffice it to say, albeit indirectly, it's thanks to EF2 that we can have this tool for EF har har.


I'm sure all the EF2 haters will love that lol.

#9 Lee

Lee

    Safety not guaranteed.

  • Members
  • 1,418 posts

Posted 16 March 2009 - 07:47 PM

oh i do...
3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117067982
14808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196442881
09756659334461fnord28475648233786783165271201909145648566923460348610454326648213393607260249141273724587
006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609

#10 Scooter

Scooter

    The Canon Man

  • UberGames Admin
  • 2,545 posts

Posted 18 March 2009 - 12:58 AM

The irony of it is hilarious.

#11 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 18 March 2009 - 10:37 AM

Hahaha I love the irony. XD
I don't really consider myself an EF2 hater though.... but I definitely prefer EF much much much much more. :)
It's probably what I think is the greatest example of how greater graphics and improved engine technology don't make a better game. XD

w00t! I got vertex normals to work!! (ie this means that it will be shaded properly in the ingame lighting system) :rock:

Turns out although the bones (and subsequently the vertices attached to them) were getting translated to the Quake 3 co-ordinates system, the vertex normals weren't (so although they were technically correct, they were pointing in the wrong direction).

That's making me curious though. This might mean this is a problem in Gongo's original utility as well...? :confused:

Attached Thumbnails

  • ef_skl2mdr_normals.jpg


#12 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 18 March 2009 - 10:53 AM

Bargh! :@

If I enable frame compression (ie it basically compacts all of the bone data into half the size, which ultimately makes the model file a good portion smaller), this is what happens. :eek:

Not sure wtf is causing it though. :ffs:

Attached Thumbnails

  • ef_skl2mdr_screwed.jpg


#13 sharpkiller

sharpkiller

    Queen Fanatic

  • Members
  • 491 posts

Posted 18 March 2009 - 01:00 PM

Bargh! :@

If I enable frame compression (ie it basically compacts all of the bone data into half the size, which ultimately makes the model file a good portion smaller), this is what happens. :eek:

Not sure wtf is causing it though. :ffs:


It's dead, Jim - I mean TiM.

:(

Sharpkiller.png


#14 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 18 March 2009 - 01:37 PM

ROFL!!!!! :lol:

I have a friend here who likes to say that everytime there is a possibility of something dying. So geeky... yet so good haha XD

Oookay.... I think I got it working! ^_^

So basically what this frame compression Ravensoft concocted up is, is it's a quick and dirty way of cutting down each bone in the model from taking up 48 bytes of space to 24 bytes each (ie 50% reduction booyeah!). It does this by scaling the floating point values up to really high integer values and then saving them as short ints in 16 bit chunks.

Basically this means, you can have half the file size at the cost of a teeny tiny bit of precision loss. XD
From what I gather, every MDR in EF was compressed in this manner.

In any case, I found out that the reason it was breaking was because some Raven guy must have been screwing with the code, and a few bytes were getting truncated off the end... something pretty simple... but enough to keep my stumped for a good few hours. :$

Anyway, here's a pic of the model in both compressed and uncompressed formats. See if you can find any changes. ;)
From the looks of it, any precision loss from it at all is so small it's un-noticable by humans (ie, JPEG, MP3, etc), which means it's probably best to always use it. :)

Whew... okay.... stuff I still haven't checked:
-Bug involving surface names
-UV Texture co-ordinates
-LoD loading
-Tags (*gulp* ;_;)

Either way... we're getting there... slowly but surely. XD

Attached Thumbnails

  • ef_skl2mdr_comp_compare.jpg


#15 khamseen

khamseen

    Member

  • Members
  • 387 posts

Posted 18 March 2009 - 07:00 PM

I wouldn't say it was all too slowly, I think you're making brilliant progress in a very short period of time, Congrats :)

#16 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 19 March 2009 - 03:21 PM

Haha, well thanks for the kind words of encouragement.

I reeeally hate it in programming though when you spend a tonne of time trying to track down a single error, and it always turns out to be something really simple you missed at the beginning. Gah! :$

Either way, not much more progress. I spent most of the time deciphering this code on how tags are generated (in MDR files they get converted into special bones)... I think it's okay, but hopefully if not, it should be pretty easy to trace.

Also, something I totally forgot... I gotta find a way so users in MAX can write file routes to the textures which can then be embedded in the file (ie, so you don't HAVE to use .skin files like MD3 files can).

Hmmm...

#17 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 20 March 2009 - 05:47 PM

I've hit a pretty epic snag and I'm totally stuck. :

If you try and export more than one surface object (which I would expect to be VERY common), on the second loop around, the script terminates with this error.

I have no clue what it means.... I'm guessing something required isn't being properly selected.

Either way... I guess it's time for lengthly googling and documentation deconstruction. >_<

Attached Thumbnails

  • skl2mdr_fail.jpg


#18 Thilo

Thilo

    Member

  • Members
  • 145 posts

Posted 21 March 2009 - 03:48 AM

Awesome work Tim! Don't get demotivated on those roadblocks, you have made much progress already.

I never tried to grasp the maths behind all the 3d model stuff... so it's up to bright guys like you, to figure this out :D

I also often spent several hours to hunt down bugs in my ioEF engine port before I finally got it to work right. This was one of the most gratifying feelings, you know :)
As to that compression stuff: Just forget about it. In my opinion, this 50% decrease is not worth all that trouble, considering the model animation data size already has decreased considerably compared to its md3 counterpart.
As you know, the format fully supports storing uncompressed bones, so if I were you, I wouldn't bother.

Anyways, I am not completely dead yet, though I was very busy over the course of the last year. I promised to release a new build of ioEF... well, let's see whether I will keep it, shan't we? :D

#19 TiM

TiM

    Administrator

  • UberGames Admin
  • 3,425 posts

Posted 21 March 2009 - 06:03 AM

Hey Thilo!! Great to see you again! :D

Ahaha, thank you very much! I think you just re-invigorated my resolve to try harder here. ^_^

ROFL! Really? Wait... you managed to get the MDR code into the Q3 code base without understanding how it worked properly....?
Hahaha I dunno about bright. Just a lot of time and a lot of googling of terms like 'unproject matrix', 'transpose matrix' and 'euler angles matrix conversion'. XD

Yeah? Haha given how huge the Q3 codebase is and how much potential there is for stuff to completely fail, I can easily understand that hehe. Haha yeah I know that feeling. And I'm amazed you stuck through and managed to get EF classic servers working on ioQ3. Definitely very gratifying. ^_^

Haha yeah. I was actually thinking about that too. I did a comparison of a compressend and uncompressed model, and the size difference wasn't overly big.
But then again, on an RPG-X model, that easily starts to add up to potentially saving a megabyte or so of data.
Either way, thankfully I got it working and I'm not going to spend any more time on compression hehe.

Haha I know the feeling, but it's good to see you come back. Haha yes!! Please do!! :D

#20 Thilo

Thilo

    Member

  • Members
  • 145 posts

Posted 21 March 2009 - 11:50 AM

ROFL! Really? Wait... you managed to get the MDR code into the Q3 code base without understanding how it worked properly....?
Hahaha I dunno about bright. Just a lot of time and a lot of googling of terms like 'unproject matrix', 'transpose matrix' and 'euler angles matrix conversion'. XD


I know I could grasp the math behind it if I were to get behind the format itself, but that was never a necessity as there was lots of sample code. Firstly, the quake3 engine itself always came with the basic vertex generation code, and secondly there was a GPLed model viewer explicitly for EliteForce's models.

By the way, I got classic servers working thanks to the information you got me from the Raven guy. I would not have been able to do it without.



0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users