Looking for a python script in the now defunct archives

0 users browsing Discussion. | 17 bots

Main » Discussion » Looking for a python script in the now defunct archives

Pages: 1 2 Next Last

MysticLord

Posted on 20-08-01, 04:33

Post: #1 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

Hello,

I am looking for a python script which a user "xibalba" requested and which was graciously provided by the users of I think the first iteration of bboard. The script is referenced in post #416 in this thread (posts per page set at 50, your page-view will vary):
https://gamefaqs.gamespot.com/boards/198537-saga-frontier/77194324?page=8#416

Use the python script that opens a single file and splits that file into it's 10 subfiles.

It's the last one on the first page of this link, in a post by "funkyass":
http://helmet.kafuka.org/byuubackup/viewtopic.php@f=10&t=2421.html

I think it's used as:
python arc_split.py M*.arc

There could be some issues if the version of Python isn't correct, post them and I'll help you troubleshoot them. Asking on a Python forum would help too, though I'm not sure what exactly to ask off the top of my head.

The script was used to split M???.ARC files for Saga Frontier into their subfiles.

If it can't be recovered and no one has it saved anywhere, I can pick up python in a week or two (I am very lazy) and recreate it, so no worries if it's lost forever. Just thought I would try the simple, obvious solution before I rouse myself.

Thanks,
MysticLord

Kawaoneechan

Posted on 20-08-01, 09:12

Brony

Post: #504 of 599
Since: 10-29-18

Last post: 195 days
Last view: 4 hours

Recovered from my personal copy of the archived board:

from struct import unpack
from io import FileIO
import sys,os,glob


def split_file(fd):
    datafile = FileIO(fd,"r")
    num_pointers = unpack("<I",datafile.read(4))
    pointer_fmt = "<%dI"%num_pointers[0]
    pointers = unpack(pointer_fmt,datafile.read(num_pointers[0]*4))
    pointers+=(os.stat(datafile.name).st_size,)
    i = 0
    datafile.seek(pointers[0],0)
    while i < (num_pointers[0]):
   
        splitname = "%s_file%u.bin"%(datafile.name,i)
        with FileIO(splitname,"w") as splitfile:
            splitfile.write(datafile.read(pointers[i+1]-pointers[i]))
        i+=1

for arg in sys.argv[1:]:
    for files in glob.glob(arg):
        split_file(files)

MysticLord	Posted on 20-08-01, 23:29 (revision 1) Hi.
Post: #2 of 27 Since: 08-01-20 Last post: 1327 days Last view: 1227 days	Thank you. edit Link for posterity https://pastebin.com/qqe5Cvdx

MysticLord	Posted on 20-08-21, 01:24 Hi.
Post: #3 of 27 Since: 08-01-20 Last post: 1327 days Last view: 1227 days	Sorry for the double-post, feel free to merge it once someone notices me. Could you find another script that xibalba e-begged off you all, which IIRC was something that would write concatenated text in a CSV to a file? I think there were multiple versions posted, for C and Python (not sure which version of Python either). I'll take all of them, if possible. Thank you once again.

wertigon

Posted on 20-08-26, 21:36

Post: #137 of 205
Since: 11-24-18

Last post: 155 days
Last view: 27 days

Posted by MysticLord
Sorry for the double-post, feel free to merge it once someone notices me.

Could you find another script that xibalba e-begged off you all, which IIRC was something that would write concatenated text in a CSV to a file?

I think there were multiple versions posted, for C and Python (not sure which version of Python either). I'll take all of them, if possible.

Thank you once again.

Would probably solve this with a one-liner bash script to be honest:


cat test.csv | cut -f3,4 -d, | sed s/,//g > output.txt

Works like this:

* move csv file to stdin (cat)
* Select the fields that are desired, and use comma as delimeter (cut)
* Remove any commas and other unnecessary characters (sed s///g)
* Optional: use > to save output to a text file

MysticLord

Posted on 20-08-27, 08:20

Post: #4 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

Posted by wertigon
Would probably solve this with a one-liner bash script to be honest:
cat test.csv | cut -f3,4 -d, | sed s/,//g > output.txt
Works like this:

* move csv file to stdin (cat)
* Select the fields that are desired, and use comma as delimeter (cut)
* Remove any commas and other unnecessary characters (sed s///g)
* Optional: use > to save output to a text file

Alas, it's for skrub windoze lusers. He used spreadsheets to edit data, concatenated all the hex strings together, and put them in a file intended to be exported as a CSV and written to whatever game files are needed with the Python script. I think there's a command-line tool that lets you import and export entire directories and wildcarded files into a disc image, so it should be possible to edit monster data in SaGa Frontier this way.

Format of the CSV was IIRC:
[code]
filename,address,text-string-representing-hex-bytes
file.bin,0,BEEFCAFEB00B1E55
[code]

Doesn't matter if you can't find the python script, I can whip it up in a day or two.

funkyass	Posted on 20-08-27, 17:02 Hi.
Post: #156 of 202 Since: 11-01-18 Last post: 660 days Last view: 15 days	https://stackoverflow.com/questions/4441827/windows-command-for-cutting-columns-from-a-text

MysticLord	Posted on 20-08-28, 08:11 Hi.
Post: #5 of 27 Since: 08-01-20 Last post: 1327 days Last view: 1227 days	While I appreciate your thoughtfulness and effort, my time is better spent picking up a little Python than ~~beating my testicles with a hammer~~ learning esoteric Windows command line tools.

wertigon

Posted on 20-08-29, 14:08

Post: #138 of 205
Since: 11-24-18

Last post: 155 days
Last view: 27 days

Posted by MysticLord
While I appreciate your thoughtfulness and effort, my time is better spent picking up a little Python than ~~beating my testicles with a hammer~~ learning esoteric Windows command line tools.

In that case:


import csv

with open('file.csv', newline='') as f:
    reader = csv.reader(f)
    data = list(reader)

Gives you the csv file as a Python list, then it's just a matter of looping over the list:


for line in data:
    output = ""
    for index,field in enumerate(line, start=0):
       output += field
       if index < 2:                  # Keep the first two fields but remove , from the rest
           output += ','

    print(output)

This is the inefficient but simple way to do it. Otherwise, do a text parser for it. :)

MysticLord	Posted on 20-09-02, 04:09 Hi.
Post: #9 of 27 Since: 08-01-20 Last post: 1327 days Last view: 1227 days	https://python-forum.io/Thread-How-do-I-write-a-single-8-bit-byte-to-a-file Python 3 is apparently far less useful than Python 2 for writing a single byte to a file. Looks like I'm going back to Java.

funkyass

Posted on 20-09-02, 05:58 (revision 4)

Post: #157 of 202
Since: 11-01-18

Last post: 660 days
Last view: 15 days

nmbr is a string that is 2 characters long, so yeah its not going to write a single byte, especially if you tell it to treat it as an array of utf-8 characters. Cast to an int, and stuff that int into an array:

writeFile.write(bytes([int(nmbr,base=16)]))

however, make use of the csv module. you be writing less code

code flex:


import sys,io, struct
 
csvName = sys.argv[1]
with open(csvName, "r") as f:
    line = f.readline()
    while line != '':
        lineList = line.split(',')
        with open(lineList[0],"r+b") as writeFile:
            writeFile.seek(int(lineList[1]))
            #cntr = 0
            
            nmbr = [int(lineList[2][i:i+2],base=16) for i in range(0, len(lineList[2])-1,2)]

            
            writeFile.write(bytes(nmbr))
            #print(bytes(nmbr,'utf-8'))
            #print(nmbr)
            #print(bytes(nmbr))
            #need to step through above for string slicing
            #while cntr < lineList
            #next do a loop where you grab lineList[3][n:n+1],
            #convert it from a hex string to an int,
            #and write it.
            #maybe int(lineList[3][n:n+1],16)
            #cntr += 1
        line = f.readline()

MysticLord

Posted on 20-09-03, 12:37 (revision 2)

Post: #10 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

It turns out that I'm a moron.

Posted by 'bowlofred'

Those aren't ints. You describe the bit at the end as "concatenated 8-bit bytes", but I'm guessing they're actually hexadecimal strings, so each character represents a 4-bit value.

You can create a byte object directly from a hex string like this:
>>> nmbr = bytes.fromhex("0102030405")
>>> nmbr
b'\x01\x02\x03\x04\x05'
>>> len(nmbr)
5
Since you've opened the file as binary and you have a bytes object, you can write it directly. There's no encoding necessary here.

Final code. Tested, works great. Tested on files padded with 0x00 and 0xFF to ensure that nothing is getting zero padded.

import sys,io
 
csvName = sys.argv[1]
with open(csvName, "r") as f:
    line = f.readline()
    while line != '':
        lineList = line.split(',')
        with open(lineList[0],"r+b") as writeFile:
            writeFile.seek(int(lineList[1]))
            writeFile.write(bytes.fromhex(lineList[2]))
        line = f.readline()

I looked at the CSV package but it seems like overkill given that I'm only using readline() twice and split() once. It's not like it's a complicated or script CSV after all.

This is my second screw up with bytes objects, I really need to do my research before I herp a derp in the future.

funkyass	Posted on 20-09-03, 16:56 Hi.
Post: #158 of 202 Since: 11-01-18 Last post: 660 days Last view: 15 days	reading the documentation is the best first step.

MysticLord

Posted on 20-09-27, 14:06 (revision 2)

Post: #11 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

If this should be moved to a new thread in a different board, I encourage the mods to do so and to give it a witty title.

Python 3 is a truly awful programming language. It's a struggle to do things that I could do in minutes in Java, but unfortunately my users are living caricatures of technological illiteracy when it comes to reading the fucking documentation and following clearly written instructions - and generally dysfunctional in a fractal manner - so I'm stuck with Python.

I'm trying to dump the spark talents data for Saga Frontier in a format that can be easily copied and pasted into a spreadsheet. It needs to be done because my users to too dumb/lazy/entitled/insane to simply provide the spreadsheets and production documents (assuming they made any to start with lol) they used to make their mods, and I need to make scripts that allow one to extract their changes from their mods so one can understand what changes they made and make informed decisions such as:
1. Do I want to play this?
2. How do I play this?
3. Was he just mashing the keyboard at random when he made these changes?

The table is currently organized in a 16x16 table of bytes, where each row is the talent listing (0 - 15) and each column is a set of bit that represents 8 sparkable (or not) skills.

FB 2C D2 93 40 00 00 00 00 00 00 00 EA C8 61 00 //talent listing 1
F9 AA F0 D7 C0 00 00 00 00 00 00 00 FD 28 CA 00 //talent listing 2
E2 49 B4 8B 50 00 00 00 00 00 00 00 F2 4B CE 00 //...
E8 99 A4 4B B0 00 00 00 00 00 00 00 E1 37 D7 00 
82 0E 00 C3 40 00 00 00 00 00 00 00 DE 89 19 00 
F5 F1 FB 1F F0 00 00 00 00 00 00 00 E3 FE E6 00 
FA E7 D2 DF D0 00 00 00 00 00 00 00 F5 EC F5 00 
FF F9 FF 1F F0 00 00 00 00 00 00 00 F4 BD 79 00 
FB C6 92 D3 20 00 00 00 00 00 00 00 FF FA FE 00 
FB C9 54 03 D0 00 00 00 00 00 00 00 FF 13 0A 00 
C9 96 92 CB B0 00 00 00 00 00 00 00 C5 1E BD 00 
FB EE D2 C3 77 00 00 00 00 00 07 FC EA E9 F9 00 
FD F9 FD 57 E0 00 00 00 00 00 00 00 F7 7D 7D 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 //talent listing 16

I want to turn it into a 128x16 list, where each column is the talent listing (0 - 15) and each row is a set of sixteen 1 or 0 representing talented or not talented, for all 128 skills.


1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0 //Slash
0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0 //StunSlash
1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0 //DoubleSlash

Note that the above isn't actually correct or exactly how I want it formatted (I don't want commas), but it's close enough for our work and I can easily change the formatting later once it works.

Here's the Python 3 code I've written so far:

import io,sys,pdb,copy

#with open("SCUS_942.30", 'rb') as scus:
    #scus.seek(58364)
with open("sparks.bin", 'rb') as scus:
    scus.seek(0)
    tblList1 = []
    cntr = 0
    while cntr < 16:
        tblList1.insert(cntr, bytearray(scus.read(16)))
        cntr += 1
    tblList2 = copy.deepcopy(tblList1)
    
    for i in range(len(tblList1)):
        for j in range(len(tblList1[0])):
            tblList2[j][i] = tblList1[i][j]
                
    tblList3 = []
    for row in range(128):
        tmpRow = []
        for column in range(16):
            tmpRow.append((row * 16) + column)
        tblList3.append(tmpRow)

    ecntr = 0
    fcntr = 0
    
    while ecntr < 16:
        while fcntr < 16:
            #something's wrong in your column/row calculations
            #need to multiply something somewhere, it seems.
            #still doesn't work. double check earlier stuff prints correctly
            #then walk through this code once and make sure
            if tblList2[ecntr][fcntr] & 128 == 128:
                tblList3[(ecntr * 8) + 0][fcntr] = 128
            else:
                tblList3[(ecntr * 8) + 0][fcntr] = 0
            if tblList2[ecntr][fcntr] & 64 == 64:
                tblList3[(ecntr * 8) + 1][fcntr] = 64
            else:
                tblList3[(ecntr * 8) + 1][fcntr] = 0
            if tblList2[ecntr][fcntr] & 32 == 32:
                tblList3[(ecntr * 8) + 2][fcntr] = 32
            else:
                tblList3[(ecntr * 8) + 2][fcntr] = 0
            if tblList2[ecntr][fcntr] & 16 == 16:
                tblList3[(ecntr * 8) + 3][fcntr] = 16
            else:
                tblList3[(ecntr * 8) + 3][fcntr] = 0
            if tblList2[ecntr][fcntr] & 8 == 8:
                tblList3[(ecntr * 8) + 4][fcntr] = 8
            else:
                tblList3[(ecntr * 8) + 4][fcntr] = 0
            if tblList2[ecntr][fcntr] & 4 == 4:
                tblList3[(ecntr * 8) + 5][fcntr] = 4
            else:
                tblList3[(ecntr * 8) + 5][fcntr] = 0
            if tblList2[ecntr][fcntr] & 2 == 2:
                tblList3[(ecntr * 8) + 6][fcntr] = 2
            else:
                tblList3[(ecntr * 8) + 6][fcntr] = 0
            if tblList2[ecntr][fcntr] & 1 == 1:
                tblList3[(ecntr * 8) + 7][fcntr] = 1
            else:
                tblList3[(ecntr * 8) + 7][fcntr] = 0
            fcntr = fcntr + 1
        ecntr = ecntr + 1
    
    for g in tblList3:
        print(g)

It reads from SCUS_942.30, which is the Saga Frontier main compiled executable file, but I also ripped the data and commented out the SCUS-reader code at the first few lines of code. It now uses a dummy file that is just the spark talents table, which is here:
https://ufile.io/tizyn3fx

If you don't have or don't want to download Saga Frontier, use the dummy file above; though any file that is at least 256 bytes long, named sparks.bin (or whatever you change that line of code to), and contains easily identified patterns will do.

Here's the output:
https://pastebin.com/zATnhKbw

Note that I'm using bit flags on each line to ensure I'm not mixing things up that shouldn't be mixed. Those will be replaced with 1s and 0s once I'm sure everything else is working correctly.

The first 8 lines print correctly, but every line after that just prints the value I stored when I initialized the array. I wasn't sure if the array initialization was working correctly (it wasn't earlier until I used copy.deepcopy(...)), so i initialized the 2D array I'm storing it in with the effective/human-understandable index of each element within said 2D array.

I have no idea why it's only working on one line instead of all lines, and the incredibly helpful people over at the Python.org forums tell me things I've already figured out and which are totally irrelevant to the (previous) issues I have. I could have written this a hundred times over in Java, but Java isn't an option.

Note that 8 lines of printed stuff in the output is equivalent to 1 byte of talent data for all 16 spark talent listings.

I have no idea why the loop-in-a-loop starting with "while ecntr < 16:" isn't working, and frankly I'm ready to sharpen a pointy stick so I can hunt my users and consume their flesh for sustenance. Assistance will be greatly appreciated and I'm sure the peaceful emanations from my aura will be felt within a 5000 mile radius, calming the nation and preventing the Second American Civil War and the resulting World Wars III and IV. Though if you'd prefer a two-digit amount of US dollars, that works too.

funkyass

Posted on 20-09-27, 18:59 (revision 6)

Post: #169 of 202
Since: 11-01-18

Last post: 660 days
Last view: 15 days

Its because you are not resetting fcntr at the top of the outer loop.

its the reason why for x in range() is perfered for loops of a fixed number


for fcntr in range(0,15):

to fix your loops:


    while ecntr < 16:
        fcntr = 0

MysticLord

Posted on 20-09-27, 22:42

Post: #12 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

Posted by funkyass
Its because you are not resetting fcntr at the top of the outer loop.

its the reason why for x in range() is perfered for loops of a fixed number
for fcntr in range(0,15):
to fix your loops:
    while ecntr < 16:
        fcntr = 0

Ahhhh, I feel my sanity returning. Do you have a paypal, gofundme, patreon, or onlyfans (lol) I can donate money to? Or would you prefer I donate $10-$99 to some cause you support?

funkyass

Posted on 20-09-27, 23:07

Post: #170 of 202
Since: 11-01-18

Last post: 660 days
Last view: 15 days

Posted by MysticLord
Posted by funkyass
Its because you are not resetting fcntr at the top of the outer loop.

its the reason why for x in range() is perfered for loops of a fixed number
for fcntr in range(0,15):
to fix your loops:
    while ecntr < 16:
        fcntr = 0
Ahhhh, I feel my sanity returning. Do you have a paypal, gofundme, patreon, or onlyfans (lol) I can donate money to? Or would you prefer I donate $10-$99 to some cause you support?

its a typical brainfart that happens to everyone, i just fixed a similar bug in my own code. Save the reward for a bigger knot.

wertigon

Posted on 20-09-27, 23:48 (revision 2)

Post: #149 of 205
Since: 11-24-18

Last post: 155 days
Last view: 27 days

Nice attempt, but knowing a thing or two about python, this should take you where you want to go:


infile = open("input.txt", "r")
outfile = open("output.txt", "w")

skills = ["one",      "two",      "three",   "four",
          "five",     "six",      "seven",   "eight",
          "nine",     "ten",      "eleven",  "twelve",
          "thirteen", "fourteen", "fifteen", "sixteen"]

for line in infile.readlines():
    writeline = ""
    for index, word in enumerate(line.split()):
        try:
            num = int(word[:2],16)
            writeline += "1 " if (num & 128) else "0 "
            writeline += "1 " if (num &  64) else "0 "
            writeline += "1 " if (num &  32) else "0 "
            writeline += "1 " if (num &  16) else "0 "
            writeline += "1 " if (num &   8) else "0 "
            writeline += "1 " if (num &   4) else "0 "
            writeline += "1 " if (num &   2) else "0 "
            writeline += "1 " if (num &   1) else "0 "
            writeline += "//" + skills[index] + "\n"
        except ValueError:
            pass
    outfile.write(writeline)

There is (almost) always a shorter solution. Anyway, you're welcome, a fun little 10 minute problem for me. :)

MysticLord

Posted on 20-11-12, 23:01

Post: #15 of 27
Since: 08-01-20

Last post: 1327 days
Last view: 1227 days

Posted by wertigon
Nice attempt, but knowing a thing or two about python, this should take you where you want to go:


infile = open("input.txt", "r")
outfile = open("output.txt", "w")

skills = ["one",      "two",      "three",   "four",
          "five",     "six",      "seven",   "eight",
          "nine",     "ten",      "eleven",  "twelve",
          "thirteen", "fourteen", "fifteen", "sixteen"]

for line in infile.readlines():
    writeline = ""
    for index, word in enumerate(line.split()):
        try:
            num = int(word[:2],16)
            writeline += "1 " if (num & 128) else "0 "
            writeline += "1 " if (num &  64) else "0 "
            writeline += "1 " if (num &  32) else "0 "
            writeline += "1 " if (num &  16) else "0 "
            writeline += "1 " if (num &   8) else "0 "
            writeline += "1 " if (num &   4) else "0 "
            writeline += "1 " if (num &   2) else "0 "
            writeline += "1 " if (num &   1) else "0 "
            writeline += "//" + skills[index] + "\n"
        except ValueError:
            pass
    outfile.write(writeline)

There is (almost) always a shorter solution. Anyway, you're welcome, a fun little 10 minute problem for me. :)

Recommend me a book?

wertigon	Posted on 20-11-15, 10:11 Hi.
Post: #155 of 205 Since: 11-24-18 Last post: 155 days Last view: 27 days	Posted by MysticLord Recommend me a book? What I did was pretty much text processing combined with knowledge about low-level programming, so nothing is covered 100%, but as for text processing this book seems to do the trick; https://www.nltk.org/book/ The dead tree version is not updated to Python 3, but otherwise it seems legit.

Pages: 1 2 Next Last

Main » Discussion » Looking for a python script in the now defunct archives