59 replies to this topic

[dscoshpe]

As Brad says, this is the victory posting

We have successfully built a program which will easily convert .exe programs to rxe!

The program is CALiPSO dRXE, and I will have it released very soon, I just need to clean it up a bit as it has been in a purely functional "experimental tool" stage for a while. For your amusement and as a demonstration I have uploaded a rxe version of Duobab's Boxworld (the original one).

There are only minor features of the rxe format that have not been addressed in the program, but they are all infrequent. We have been having issues with all/most Borland compiled programs as they make some writes to the code block which is physically flash and in rxe terms is a big critical issue. Boxworld contains a hack to workaround this problem but we have yet to make it a universal fix for all Borland programs, but it is a start. If you have information that may be useful in this regard do mention it.

Otherwise, I have successfully converted assembly programs and those from other compilers such as Digitial Mars.

Expect the first release of dRXE soon! Pick up Boxworld from file sharing!

- dscoshpe -

[2072]

Thank you so much That's a great breakthrough!

So how much usable ram is left for an RXE program?

[BradN]

Thank you so much :) That's a great breakthrough!

So how much usable ram is left for an RXE program?

<{POST_SNAPBACK}>

I believe 260 bytes of RAM are used for the RXE code, so the rest of the open RAM is available for data. To get the most of RXE conversion, it might be beneficial to either store static data as code within assembly objects, or separate files loaded on demand - string constants and such will become part of the data segment and will be copied to RAM otherwise.

Also, we haven't tested with all the different borland C memory models yet... they may require different hacks than the ones boxworld needed, but again, digital mars doesn't seem to need any, and assembly would depend on the actual program (as long as no writes are done to the code segment, it should be fine).

[CrimsonCasio]

dose this mean the Casio Elite are back in action?

[BradN]

dose this mean the Casio Elite are back in action?

<{POST_SNAPBACK}>

I know dscoshpe's been working on this project, but I haven't done any casio stuff lately except helping debug the borland issues with dRXE. One of these days, I'll get around to fixing my link cable...

[dscoshpe]

Short answer "yes", long answer "no".

I work on stuff off and on, like in streaks, and I happen to be in one now. The last time was probably in March. I suffer from Real-Life Syndrome and some of those days I just cant sit in front of a computer long enough to concentrate. But dRXE is a program I have wanted to finish for a while (I started it like 2-3 years ago) because it hasn't been done yet and so I have chipped away at it gradually. I plan on releasing one more version of AIP which will probably have RXE conversion built in because of the difficulties associated with converting programs seperately and then combining them on a drive, but I dont think that case will be in demand too much. Commander is, of course, the other never ending story (and a tragedy to me) but I don't know if it will ever see the light of day, but I am fond of the unique way it handles transfers, so who knows.

So I'm saying that you should check up with us from time to time. I know everytime someone asks I always open up the source and see where I was which then sometimes leads to getting something done!

PS - About Casio Elite the website, I think I'll try to dump it somewhere safe. It always seems to have bad luck with hosts. It will be a closed site though. I think the site I had with the assembly archive closed . (for those of you that remember it)

- dscoshpe -

[huhn_m]

great!! This is the best thing I heared for the last 2 years!!!

I hope we'll be able to use the conversions soon

Maybe crimson can host your page ...

[4nic8]

It's really useful project
ex. I could use ~40 kb of RAM more in MLCafx since MLCafx (un-compressed) takes ~40 kb
I would be ~ 2 times RAM memory more then I can use in MLCafx now

I hope you'll finish it as soon as possible.

btw. I use tcc to compile the MLCafx but I could port it to DM if it's really needed

way to go dscoshpe!!!

[dscoshpe]

I tried to upload the first release of the program yesterday but I was getting error 500s from 2072's server, I'll try again in a few minutes.

4nic8: As of right now it would be a good idea to port to Digital Mars because the hack hasn't been worked out as a universal solution. Boxworld works because we spent sometime analyzing it to determine what to do but we can't do that for every program. Besides, I think DM gives you a natural speed increase over Borland (Borland has lots of messy init code - which is also part of the problem for us).

But! Neither Brad or I have much expertise with the Borland compilers so maybe there a memory model or option that can be set to avoid the problem completely? I think maybe I will add a function to detect if a hack would be needed.

- dscoshpe -

[dscoshpe]

dRXE v1.50 has been added to file sharing.

- dscoshpe -

[dscoshpe]

I wanted to make something I overlooked more obvious:

The RXE base address corresponds directly to the location in the flash (ie - drive letter) so to target an RXE for another drive, add 2000h per letter on top of the default L.. I always tested on drive L, and I tried something on O and it didnt run (after running on L) so I may have forgotten somehting in the conversion in this regard (or made something static )

A future version will make this more clear, and I will fix or resolve the issue with different drives.

- dscoshpe -

[4nic8]

it worked with my game in C (DM). But it only on drive L... I'm sure that dscoshpe will fix it

[2072]

The 500 server error is because a PHP script cannot run more than 30 seconds... So it depends how fast you can send the file.

[SoftCalc]

The 500 server error is because a PHP script cannot run more than 30 seconds... So it depends how fast you can send the file.

<{POST_SNAPBACK}>

Is it possible to change this timeout? I know you can change the timeout in the PHP config file, but can it be changed for a specific scripts?

I noticed that classpad.net uses PHP and doesn't have this problem with some really big downloads. Ricky somehow worked around this problem on classpad.org and cpsdk.com. I'll find out how he did it.

[2072]

Changing PHP setting won't do anything, the server kills any CGI that exceed that limit...

[dscoshpe]

2072: That would do it. I'm a big torrent seeder and I reserve about 5kB upstream for general use.

- dscoshpe -

[dscoshpe]

OK so I did some investigating...

Brad assured me that it is not neccessary to change the base because of how the flash is handled on the calc, so that simplifies that. The base never has to change except to specify a new place in the rom-disk image (if there are more than 1 files). The problem must be something limited to 4nic8's program. Boxworld and another program compiled with DM worked from other drives.

All is well now except for the Borland hack which I will be working on.

- dscoshpe -

[4nic8]

The problem must be something limited to 4nic8's program.

My program works very well! In fact I used only Orwell's gxlib and hooked animation engine to 0x1C. I haven't made anything special there, so I guess that problem will come back with some other game...

[huhn_m]

how about turbo pascal programs? I know they are borland but has
anyone tested them?!?!? maybe I'll do

[4nic8]

figured out

Now it works with all drives

So what was wrong then?
HUH surely it wasn't my fault . I changed dmc options from

-2 -Aa -Ab -msd -gg -o+space

to

-2 -Aa -Ab -cpp -w7 -msd -gg -o+space

and now my program is forced to be C++ (even it's written in C - ).

You can try it on your own (there is no warranty):
dRXE version (this must be at the top of a drive - but now it works with L:-Q: drives):
http://max.iem.pw.ed...xe/attt.exe.rxe
UPX-ED normal exe version:
http://max.iem.pw.ed...wj/drxe/ttt.exe

btw. strange thing:
C command coreleft (returns a measure of unused memory) doesn't seem to work fine with dRXE:
- in the emulator it gives 0 - but everything works fine
- on calc it gives the same value as with normal exe file

any cl00s why's that? / any way to test *real* left ram space?

[huhn_m]

so it seems not to work with TP.

It gave me a Runtime Error 9 (Out of valid Range).

I can give you the data if this helps you.

[4nic8]

@huhn_m: btw. does it work with your asm stuff?

[dscoshpe]

For Pascal, I would need to add support for it because it looks like dRXE is scanning outside of the file. Give me a link to the program or e-mail it and I will see about it.

For assembly, you *have to* manually declare the start of the code block in the file image because there is nothing to search for in the file that will hint to where it is (short of a full dissassembly). You can also do this for Pascal programs to get them to work. I've been thinking about adding .map file support to make this easier.

4nic8: Good job

- dscoshpe -

[huhn_m]

just a tip:

make a function where the program can search a term like "RXE_START" that
the coder (in asm) would have to include at the beginning of the code block.
This could be usefull and used for easy codeblock identification ...

I'll e-mail you the test program.

[dscoshpe]

huhn: Pascal seems like it should work fine but I am not yet able to add support because it wasnt clear whether the sample you sent me was representative of most Pascal programs. For the sample you sent, the data block starts at 6e0h. The good news is that you can enter this value manually under the "Conversion" tab, the bad news is that dRXE 1.50 has a bug which prevents it from accepting a disasm line that starts at the same address so your program will be missing a fix and will therefore not convert properly. I have corrected this problem already and I will have an updated dRXE uploaded by the end of the day (my day ).

I forgot to mention that the sample you sent converted fine and ran on the calc.

- dscoshpe -

[huhn_m]

great! So I can continue my <{GNULINUX}> development on Pascal Thank you a lot
this will make stuff a HUGE deal easier. You could try some of the programs
made by marco since he is coding is Pascal

[4nic8]

Hey guys! So when I program in nasm my code should look something like it, right??:

SEGMENT _TEXT class=CODE

%include "LIBS\macros.asm"

;==============================
RXE_START:
start:
	mov     ax,_TEXT
        mov     ds,ax
        mov     es,ax
	mov     ax,stack
        mov     ss,ax
        mov     sp,stack_end

	call	init_graphics_mode
............
............

[huhn_m]

no. not really. This was just a suggestion. If dscoshpe implements it
it would look like the folowing

...
db 'RXE_START'
start:
    ;CODE
      ....
end start

the stack could also be used like alway with the .STACK statement.
the same is valid for the .DATA & .CODE statements

[4nic8]

Ok then!

You can download a NASM game with the beginning:

db 'RXE_START'
start:

CLICK HERE

[dscoshpe]

So I like the RXE_START idea, and in 4nic8's example it works great. But I didn't notice earlier that you were talking about the code block... Unfortunately its not the code block thats the problem, its finding the data block which is tricky.

As a reference: In RXE, only the data block is moved to ram when run by dos, the code block stays resident on the flash. Ideally you want a program with a code block which is as large as possible. The <{GNULINUX}>.exe example is very good in that regard, only a few hundred bytes makeup the data block! So virtually all of the program executes from flash!

Anyways, perhaps the RXE_START method can be used for that just as easily? (Sorry but my assembly knowledge doesn't answer that question for me)

- dscoshpe -

[huhn_m]

yeah. then you'd have to write

.DATA
  db 'RXE_START'
 ;insert data here (include files with data also here and files with
 ;procedures in code segment)
.CODE
start:
 ;Insert code here
end start

[dscoshpe]

dRXE 1.51 has been uploaded to the file sharing

It now has bug fixes, Pascal support, "DATA_START" support, and an experimental hack for Borland 2 (3 is coming soon....)

Pascal support: I couldn't figure out a way for having 100% success in figuring out where the data block should be short of a full disassembly analysis which Im not willing to do. So I compromised on assuming that every Pascal program's data block starts with 2 bytes 0000h which seems to be true, and that I have better odds of landing on the data block (since there are usually multiple instances of 0000h) if I figure the bytes to be positioned at x * 10h and to have 2 preceding bytes of 0h, in addition to this I take the first instance I find. I checked all of Marco's progs that I could and only one of them (sound.exe) did not succeed. I have dRXE list the other possibilities for your own reference, and for use in specifying manually.

"DATA_START": To implement this idea easier I changed it to "DATA_START" so it would be 10 bytes long. To use, just make sure it appears at the start of the data block and dRXE will mark its position as just that.

Borland 2 hack: Very experimental and relatively untested. Boxworld should work 100% with it but other than that I'm not sure because I just finished it before uploading for you. I was going to disable it again but I figured it might be fun. Remember, its only for Borland 2, if you forget, dRXE will remind you

You will also find you can now add and remove fix handler strings (the things in the "Fixes" tab) as well as edit them. Its really useful to know what their use is for forcing a program to convert properly without waiting for an update by me.

Also, I have made many more bug fixes which will help de-VB this VB program. (things like unsigning certain variables etc)

- dscoshpe -

[dscoshpe]

Just wanted to announce the progress that has been made lately:

dRXE now supports fixing of stack pushs (which it hasn't so far) which should make the Borland hacks more reliable as we dont have to guess at which registers are safe to use.

dRXE will also now append a small chunk of information on the end of generated files for use in other programs. This will enable RXE's to be relocated after the conversion process is done making them able to run from within a drive rather than at the begining of it. It will require programs to be converted with the new version to take advantage of it.

On a similar note, Add-In Packager will get a new release as well...

New features include a more streamlined and more assuming interface. It also just so happens to support packaging of RXE's anywhere in a drive. If you are curious, there is no user intervention needed to work with RXE's in AIP as the work has been taken care of, meaning that the programs may remain in the form program.exe.rxe or whatever and be packaged as program.exe. The only thing that lets you know an RXE was packaged is a list of programs that were identified and adjusted as RXE's. Quite simple

There are many other less exciting changes to both programs too....

Expect these releases soon.

- dscoshpe -

[dscoshpe]

dRXE 1.53 has been uploaded to file sharing, here's whats worth knowing:

Borland 3 hack has been added with mixed results. Some programs convert wonderfully while others do not, I'm not sure what I'm missing but I will continue to investigate it.

A thing I call a 'dRXE Tag' is now placed on the end of each file converted and is meant to let programs that create rom-disk drives adjust the rxe files properly within the rom-disk. I have included a text file with the dRXE distribution called 'Relocation.txt' which explains how to use the dRXE tag. It's as easy to handle as possible I think.

A new Add-In Packager is being readied to support dRXE tags.

Only 1 more core conversion feature is left for me to include support for before dRXE is 100% compatible with the entire format!! In 1.53 I added support for 2 of these features. The last would be added by now but I have yet to encounter a situation that would need it.

- dscoshpe -

[huhn_m]

Now thats nice ... I have holidays now so I'll continue my <{GNULINUX}> development and thus will make extensive use of you RXE program I think.

I'll drop of notes if sth. doesn't work or goes wrong.

Btw. does RXE need the DOS interrupt to be working or can I disable it?

And could someone (glimpses at 2072) make sure that this infos are always posted on the french forum. The crteators of Flash100 linger there so it might me interresting for them to add this relocation thing to their program

Cu huhn

[AlephMobius]

Does this compression give your program more RAM to work with? Could someone explain how that works or point me somewhere that does? Just curious.

[dscoshpe]

huhn: Yeah the interrupt is required and there is actually another interrupt used which is INT1 + 1 = INT2 ... INT1 being F0h, usually. The reason for the interrupts is that they are the method that the code block, which when run is executing from flash, can find the data block, which is loaded in to ram. INT1 handles code->data fixes that involve registers or memory addresses and INT2 handles code->data fixes that involve pushs.

Aleph: This is not a form of compression. Essentially what an RXE tries to achieve is program that doesn't have to be completely loaded in to RAM from flash in order to run thus saving space that would have been used "needlessly". Dos compatible exes usually have 3 parts, the header, the code block (for the program instructions), and the data block (for game maps, graphics, etc). As an RXE the exe will loaded specially such that only the data block takes up RAM space and the code from the code block is executed directly from the flash, this can potentially free the entire amount of calc usable ram for other purposes, and if you need MORE space than that, you can start using special tricks so that your data that does NOT change during program execution becomes recognized as a part of the code block. If you want the nitty gritty, and my resource for understanding the idea see the RXE Theory of Operation straight from Datalight!

I'm going to put a file together that explains what really happens in more understandable terms and pictures and post it in the near future.

I enjoy these questions!

- dscoshpe -

[X-thunder28]

great job ^^

I want to know if the functions pointers like that:
void *(paf)(void);

would work as well?

(yes I'm in love with functions pointers )

[dscoshpe]

x-thunder: It is hard to say because I believe you are showing me C syntax and I really can't guess how it will compile. Go ahead and try all the usual things you would do and when you have compiled, run your program through dRXE and check the fixes section for any entries with "E!" in them, these are the fixes which cannot be made. If you see none then everything you did should be OK!!! Good luck!

- dscoshpe -

[X-thunder28]

no problem it works perfectly!!

even perfectly that my program doesn't crash anymore in RXE
(instead of the EXE format it crashed a lot of time (memory alloc are strange... huhu in fact I forgot to free when the program ended!! yay thanks RXE, I can find my errors now )