0 users browsing Discussion. | 3 bots  
    Main » Discussion » Looking for a python script in the now defunct archives
    Pages: 1 2 Next Last
    Posted on 20-08-01, 04:33
    Post: #1 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Hello,

    I am looking for a python script which a user "xibalba" requested and which was graciously provided by the users of I think the first iteration of bboard. The script is referenced in post #416 in this thread (posts per page set at 50, your page-view will vary):
    https://gamefaqs.gamespot.com/boards/198537-saga-frontier/77194324?page=8#416

    Use the python script that opens a single file and splits that file into it's 10 subfiles.

    It's the last one on the first page of this link, in a post by "funkyass":
    http://helmet.kafuka.org/byuubackup/viewtopic.php@f=10&t=2421.html

    I think it's used as:
    python arc_split.py M*.arc

    There could be some issues if the version of Python isn't correct, post them and I'll help you troubleshoot them. Asking on a Python forum would help too, though I'm not sure what exactly to ask off the top of my head.


    The script was used to split M???.ARC files for Saga Frontier into their subfiles.

    If it can't be recovered and no one has it saved anywhere, I can pick up python in a week or two (I am very lazy) and recreate it, so no worries if it's lost forever. Just thought I would try the simple, obvious solution before I rouse myself.

    Thanks,
    MysticLord
    Posted on 20-08-01, 09:12
    I said, put the bunny back in the box!

    Post: #504 of 598
    Since: 10-29-18

    Last post: 86 days
    Last view: 9 hours
    Recovered from my personal copy of the archived board:

    from struct import unpack
    from io import FileIO
    import sys,os,glob


    def split_file(fd):
    datafile = FileIO(fd,"r")
    num_pointers = unpack("<I",datafile.read(4))
    pointer_fmt = "<%dI"%num_pointers[0]
    pointers = unpack(pointer_fmt,datafile.read(num_pointers[0]*4))
    pointers+=(os.stat(datafile.name).st_size,)
    i = 0
    datafile.seek(pointers[0],0)
    while i < (num_pointers[0]):

    splitname = "%s_file%u.bin"%(datafile.name,i)
    with FileIO(splitname,"w") as splitfile:
    splitfile.write(datafile.read(pointers[i+1]-pointers[i]))
    i+=1

    for arg in sys.argv[1:]:
    for files in glob.glob(arg):
    split_file(files)
    Posted on 20-08-01, 23:29 (revision 1)
    Post: #2 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Thank you.

    edit

    Link for posterity
    https://pastebin.com/qqe5Cvdx
    Posted on 20-08-21, 01:24
    Post: #3 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Sorry for the double-post, feel free to merge it once someone notices me.

    Could you find another script that xibalba e-begged off you all, which IIRC was something that would write concatenated text in a CSV to a file?

    I think there were multiple versions posted, for C and Python (not sure which version of Python either). I'll take all of them, if possible.

    Thank you once again.
    Posted on 20-08-26, 21:36
    Post: #137 of 203
    Since: 11-24-18

    Last post: 9 days
    Last view: 5 days
    Posted by MysticLord
    Sorry for the double-post, feel free to merge it once someone notices me.

    Could you find another script that xibalba e-begged off you all, which IIRC was something that would write concatenated text in a CSV to a file?

    I think there were multiple versions posted, for C and Python (not sure which version of Python either). I'll take all of them, if possible.

    Thank you once again.


    Would probably solve this with a one-liner bash script to be honest:


    cat test.csv | cut -f3,4 -d, | sed s/,//g > output.txt


    Works like this:

    * move csv file to stdin (cat)
    * Select the fields that are desired, and use comma as delimeter (cut)
    * Remove any commas and other unnecessary characters (sed s///g)
    * Optional: use > to save output to a text file
    Posted on 20-08-27, 08:20
    Post: #4 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Posted by wertigon
    Would probably solve this with a one-liner bash script to be honest:


    cat test.csv | cut -f3,4 -d, | sed s/,//g > output.txt


    Works like this:

    * move csv file to stdin (cat)
    * Select the fields that are desired, and use comma as delimeter (cut)
    * Remove any commas and other unnecessary characters (sed s///g)
    * Optional: use > to save output to a text file

    Alas, it's for skrub windoze lusers. He used spreadsheets to edit data, concatenated all the hex strings together, and put them in a file intended to be exported as a CSV and written to whatever game files are needed with the Python script. I think there's a command-line tool that lets you import and export entire directories and wildcarded files into a disc image, so it should be possible to edit monster data in SaGa Frontier this way.

    Format of the CSV was IIRC:
    [code]
    filename,address,text-string-representing-hex-bytes
    file.bin,0,BEEFCAFEB00B1E55
    [code]

    Doesn't matter if you can't find the python script, I can whip it up in a day or two.

    Posted on 20-08-27, 17:02
    Post: #156 of 202
    Since: 11-01-18

    Last post: 421 days
    Last view: 54 days
    https://stackoverflow.com/questions/4441827/windows-command-for-cutting-columns-from-a-text
    Posted on 20-08-28, 08:11
    Post: #5 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    While I appreciate your thoughtfulness and effort, my time is better spent picking up a little Python than beating my testicles with a hammer learning esoteric Windows command line tools.
    Posted on 20-08-29, 14:08
    Post: #138 of 203
    Since: 11-24-18

    Last post: 9 days
    Last view: 5 days
    Posted by MysticLord
    While I appreciate your thoughtfulness and effort, my time is better spent picking up a little Python than beating my testicles with a hammer learning esoteric Windows command line tools.


    In that case:


    import csv

    with open('file.csv', newline='') as f:
    reader = csv.reader(f)
    data = list(reader)


    Gives you the csv file as a Python list, then it's just a matter of looping over the list:


    for line in data:
    output = ""
    for index,field in enumerate(line, start=0):
    output += field
    if index < 2: # Keep the first two fields but remove , from the rest
    output += ','

    print(output)


    This is the inefficient but simple way to do it. Otherwise, do a text parser for it. :)
    Posted on 20-09-02, 04:09
    Post: #9 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    https://python-forum.io/Thread-How-do-I-write-a-single-8-bit-byte-to-a-file

    Python 3 is apparently far less useful than Python 2 for writing a single byte to a file. Looks like I'm going back to Java.
    Posted on 20-09-02, 05:58 (revision 4)
    Post: #157 of 202
    Since: 11-01-18

    Last post: 421 days
    Last view: 54 days
    nmbr is a string that is 2 characters long, so yeah its not going to write a single byte, especially if you tell it to treat it as an array of utf-8 characters. Cast to an int, and stuff that int into an array:

    writeFile.write(bytes([int(nmbr,base=16)]))


    however, make use of the csv module. you be writing less code

    code flex:

    import sys,io, struct

    csvName = sys.argv[1]
    with open(csvName, "r") as f:
    line = f.readline()
    while line != '':
    lineList = line.split(',')
    with open(lineList[0],"r+b") as writeFile:
    writeFile.seek(int(lineList[1]))
    #cntr = 0

    nmbr = [int(lineList[2][i:i+2],base=16) for i in range(0, len(lineList[2])-1,2)]


    writeFile.write(bytes(nmbr))
    #print(bytes(nmbr,'utf-8'))
    #print(nmbr)
    #print(bytes(nmbr))
    #need to step through above for string slicing
    #while cntr < lineList
    #next do a loop where you grab lineList[3][n:n+1],
    #convert it from a hex string to an int,
    #and write it.
    #maybe int(lineList[3][n:n+1],16)
    #cntr += 1
    line = f.readline()
    Posted on 20-09-03, 12:37 (revision 2)
    Post: #10 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    It turns out that I'm a moron.

    Posted by 'bowlofred'

    Those aren't ints. You describe the bit at the end as "concatenated 8-bit bytes", but I'm guessing they're actually hexadecimal strings, so each character represents a 4-bit value.

    You can create a byte object directly from a hex string like this:


    >>> nmbr = bytes.fromhex("0102030405")
    >>> nmbr
    b'\x01\x02\x03\x04\x05'
    >>> len(nmbr)
    5


    Since you've opened the file as binary and you have a bytes object, you can write it directly. There's no encoding necessary here.


    Final code. Tested, works great. Tested on files padded with 0x00 and 0xFF to ensure that nothing is getting zero padded.
    import sys,io

    csvName = sys.argv[1]
    with open(csvName, "r") as f:
    line = f.readline()
    while line != '':
    lineList = line.split(',')
    with open(lineList[0],"r+b") as writeFile:
    writeFile.seek(int(lineList[1]))
    writeFile.write(bytes.fromhex(lineList[2]))
    line = f.readline()


    I looked at the CSV package but it seems like overkill given that I'm only using readline() twice and split() once. It's not like it's a complicated or script CSV after all.

    This is my second screw up with bytes objects, I really need to do my research before I herp a derp in the future.
    Posted on 20-09-03, 16:56
    Post: #158 of 202
    Since: 11-01-18

    Last post: 421 days
    Last view: 54 days
    reading the documentation is the best first step.
    Posted on 20-09-27, 14:06 (revision 2)
    Post: #11 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    If this should be moved to a new thread in a different board, I encourage the mods to do so and to give it a witty title.

    Python 3 is a truly awful programming language. It's a struggle to do things that I could do in minutes in Java, but unfortunately my users are living caricatures of technological illiteracy when it comes to reading the fucking documentation and following clearly written instructions - and generally dysfunctional in a fractal manner - so I'm stuck with Python.

    I'm trying to dump the spark talents data for Saga Frontier in a format that can be easily copied and pasted into a spreadsheet. It needs to be done because my users to too dumb/lazy/entitled/insane to simply provide the spreadsheets and production documents (assuming they made any to start with lol) they used to make their mods, and I need to make scripts that allow one to extract their changes from their mods so one can understand what changes they made and make informed decisions such as:
    1. Do I want to play this?
    2. How do I play this?
    3. Was he just mashing the keyboard at random when he made these changes?

    The table is currently organized in a 16x16 table of bytes, where each row is the talent listing (0 - 15) and each column is a set of bit that represents 8 sparkable (or not) skills.

    FB 2C D2 93 40 00 00 00 00 00 00 00 EA C8 61 00 //talent listing 1
    F9 AA F0 D7 C0 00 00 00 00 00 00 00 FD 28 CA 00 //talent listing 2
    E2 49 B4 8B 50 00 00 00 00 00 00 00 F2 4B CE 00 //...
    E8 99 A4 4B B0 00 00 00 00 00 00 00 E1 37 D7 00
    82 0E 00 C3 40 00 00 00 00 00 00 00 DE 89 19 00
    F5 F1 FB 1F F0 00 00 00 00 00 00 00 E3 FE E6 00
    FA E7 D2 DF D0 00 00 00 00 00 00 00 F5 EC F5 00
    FF F9 FF 1F F0 00 00 00 00 00 00 00 F4 BD 79 00
    FB C6 92 D3 20 00 00 00 00 00 00 00 FF FA FE 00
    FB C9 54 03 D0 00 00 00 00 00 00 00 FF 13 0A 00
    C9 96 92 CB B0 00 00 00 00 00 00 00 C5 1E BD 00
    FB EE D2 C3 77 00 00 00 00 00 07 FC EA E9 F9 00
    FD F9 FD 57 E0 00 00 00 00 00 00 00 F7 7D 7D 00
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 //talent listing 16


    I want to turn it into a 128x16 list, where each column is the talent listing (0 - 15) and each row is a set of sixteen 1 or 0 representing talented or not talented, for all 128 skills.


    1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0 //Slash
    0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0 //StunSlash
    1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0 //DoubleSlash

    Note that the above isn't actually correct or exactly how I want it formatted (I don't want commas), but it's close enough for our work and I can easily change the formatting later once it works.

    Here's the Python 3 code I've written so far:
    import io,sys,pdb,copy

    #with open("SCUS_942.30", 'rb') as scus:
    #scus.seek(58364)
    with open("sparks.bin", 'rb') as scus:
    scus.seek(0)
    tblList1 = []
    cntr = 0
    while cntr < 16:
    tblList1.insert(cntr, bytearray(scus.read(16)))
    cntr += 1
    tblList2 = copy.deepcopy(tblList1)

    for i in range(len(tblList1)):
    for j in range(len(tblList1[0])):
    tblList2[j][i] = tblList1[i][j]

    tblList3 = []
    for row in range(128):
    tmpRow = []
    for column in range(16):
    tmpRow.append((row * 16) + column)
    tblList3.append(tmpRow)

    ecntr = 0
    fcntr = 0

    while ecntr < 16:
    while fcntr < 16:
    #something's wrong in your column/row calculations
    #need to multiply something somewhere, it seems.
    #still doesn't work. double check earlier stuff prints correctly
    #then walk through this code once and make sure
    if tblList2[ecntr][fcntr] & 128 == 128:
    tblList3[(ecntr * 8) + 0][fcntr] = 128
    else:
    tblList3[(ecntr * 8) + 0][fcntr] = 0
    if tblList2[ecntr][fcntr] & 64 == 64:
    tblList3[(ecntr * 8) + 1][fcntr] = 64
    else:
    tblList3[(ecntr * 8) + 1][fcntr] = 0
    if tblList2[ecntr][fcntr] & 32 == 32:
    tblList3[(ecntr * 8) + 2][fcntr] = 32
    else:
    tblList3[(ecntr * 8) + 2][fcntr] = 0
    if tblList2[ecntr][fcntr] & 16 == 16:
    tblList3[(ecntr * 8) + 3][fcntr] = 16
    else:
    tblList3[(ecntr * 8) + 3][fcntr] = 0
    if tblList2[ecntr][fcntr] & 8 == 8:
    tblList3[(ecntr * 8) + 4][fcntr] = 8
    else:
    tblList3[(ecntr * 8) + 4][fcntr] = 0
    if tblList2[ecntr][fcntr] & 4 == 4:
    tblList3[(ecntr * 8) + 5][fcntr] = 4
    else:
    tblList3[(ecntr * 8) + 5][fcntr] = 0
    if tblList2[ecntr][fcntr] & 2 == 2:
    tblList3[(ecntr * 8) + 6][fcntr] = 2
    else:
    tblList3[(ecntr * 8) + 6][fcntr] = 0
    if tblList2[ecntr][fcntr] & 1 == 1:
    tblList3[(ecntr * 8) + 7][fcntr] = 1
    else:
    tblList3[(ecntr * 8) + 7][fcntr] = 0
    fcntr = fcntr + 1
    ecntr = ecntr + 1

    for g in tblList3:
    print(g)


    It reads from SCUS_942.30, which is the Saga Frontier main compiled executable file, but I also ripped the data and commented out the SCUS-reader code at the first few lines of code. It now uses a dummy file that is just the spark talents table, which is here:
    https://ufile.io/tizyn3fx

    If you don't have or don't want to download Saga Frontier, use the dummy file above; though any file that is at least 256 bytes long, named sparks.bin (or whatever you change that line of code to), and contains easily identified patterns will do.

    Here's the output:
    https://pastebin.com/zATnhKbw

    Note that I'm using bit flags on each line to ensure I'm not mixing things up that shouldn't be mixed. Those will be replaced with 1s and 0s once I'm sure everything else is working correctly.

    The first 8 lines print correctly, but every line after that just prints the value I stored when I initialized the array. I wasn't sure if the array initialization was working correctly (it wasn't earlier until I used copy.deepcopy(...)), so i initialized the 2D array I'm storing it in with the effective/human-understandable index of each element within said 2D array.

    I have no idea why it's only working on one line instead of all lines, and the incredibly helpful people over at the Python.org forums tell me things I've already figured out and which are totally irrelevant to the (previous) issues I have. I could have written this a hundred times over in Java, but Java isn't an option.

    Note that 8 lines of printed stuff in the output is equivalent to 1 byte of talent data for all 16 spark talent listings.

    I have no idea why the loop-in-a-loop starting with "while ecntr < 16:" isn't working, and frankly I'm ready to sharpen a pointy stick so I can hunt my users and consume their flesh for sustenance. Assistance will be greatly appreciated and I'm sure the peaceful emanations from my aura will be felt within a 5000 mile radius, calming the nation and preventing the Second American Civil War and the resulting World Wars III and IV. Though if you'd prefer a two-digit amount of US dollars, that works too.
    Posted on 20-09-27, 18:59 (revision 6)
    Post: #169 of 202
    Since: 11-01-18

    Last post: 421 days
    Last view: 54 days
    Its because you are not resetting fcntr at the top of the outer loop.

    its the reason why for x in range() is perfered for loops of a fixed number


    for fcntr in range(0,15):


    to fix your loops:

    while ecntr < 16:
    fcntr = 0
    Posted on 20-09-27, 22:42
    Post: #12 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Posted by funkyass
    Its because you are not resetting fcntr at the top of the outer loop.

    its the reason why for x in range() is perfered for loops of a fixed number


    for fcntr in range(0,15):


    to fix your loops:

    while ecntr < 16:
    fcntr = 0

    Ahhhh, I feel my sanity returning. Do you have a paypal, gofundme, patreon, or onlyfans (lol) I can donate money to? Or would you prefer I donate $10-$99 to some cause you support?
    Posted on 20-09-27, 23:07
    Post: #170 of 202
    Since: 11-01-18

    Last post: 421 days
    Last view: 54 days
    Posted by MysticLord
    Posted by funkyass
    Its because you are not resetting fcntr at the top of the outer loop.

    its the reason why for x in range() is perfered for loops of a fixed number


    for fcntr in range(0,15):


    to fix your loops:

    while ecntr < 16:
    fcntr = 0

    Ahhhh, I feel my sanity returning. Do you have a paypal, gofundme, patreon, or onlyfans (lol) I can donate money to? Or would you prefer I donate $10-$99 to some cause you support?


    its a typical brainfart that happens to everyone, i just fixed a similar bug in my own code. Save the reward for a bigger knot.
    Posted on 20-09-27, 23:48 (revision 2)
    Post: #149 of 203
    Since: 11-24-18

    Last post: 9 days
    Last view: 5 days
    Nice attempt, but knowing a thing or two about python, this should take you where you want to go:


    infile = open("input.txt", "r")
    outfile = open("output.txt", "w")

    skills = ["one", "two", "three", "four",
    "five", "six", "seven", "eight",
    "nine", "ten", "eleven", "twelve",
    "thirteen", "fourteen", "fifteen", "sixteen"]

    for line in infile.readlines():
    writeline = ""
    for index, word in enumerate(line.split()):
    try:
    num = int(word[:2],16)
    writeline += "1 " if (num & 128) else "0 "
    writeline += "1 " if (num & 64) else "0 "
    writeline += "1 " if (num & 32) else "0 "
    writeline += "1 " if (num & 16) else "0 "
    writeline += "1 " if (num & 8) else "0 "
    writeline += "1 " if (num & 4) else "0 "
    writeline += "1 " if (num & 2) else "0 "
    writeline += "1 " if (num & 1) else "0 "
    writeline += "//" + skills[index] + "\n"
    except ValueError:
    pass
    outfile.write(writeline)


    There is (almost) always a shorter solution. Anyway, you're welcome, a fun little 10 minute problem for me. :)
    Posted on 20-11-12, 23:01
    Post: #15 of 27
    Since: 08-01-20

    Last post: 1089 days
    Last view: 988 days
    Posted by wertigon
    Nice attempt, but knowing a thing or two about python, this should take you where you want to go:


    infile = open("input.txt", "r")
    outfile = open("output.txt", "w")

    skills = ["one", "two", "three", "four",
    "five", "six", "seven", "eight",
    "nine", "ten", "eleven", "twelve",
    "thirteen", "fourteen", "fifteen", "sixteen"]

    for line in infile.readlines():
    writeline = ""
    for index, word in enumerate(line.split()):
    try:
    num = int(word[:2],16)
    writeline += "1 " if (num & 128) else "0 "
    writeline += "1 " if (num & 64) else "0 "
    writeline += "1 " if (num & 32) else "0 "
    writeline += "1 " if (num & 16) else "0 "
    writeline += "1 " if (num & 8) else "0 "
    writeline += "1 " if (num & 4) else "0 "
    writeline += "1 " if (num & 2) else "0 "
    writeline += "1 " if (num & 1) else "0 "
    writeline += "//" + skills[index] + "\n"
    except ValueError:
    pass
    outfile.write(writeline)


    There is (almost) always a shorter solution. Anyway, you're welcome, a fun little 10 minute problem for me. :)

    Recommend me a book?
    Posted on 20-11-15, 10:11
    Post: #155 of 203
    Since: 11-24-18

    Last post: 9 days
    Last view: 5 days
    Posted by MysticLord

    Recommend me a book?


    What I did was pretty much text processing combined with knowledge about low-level programming, so nothing is covered 100%, but as for text processing this book seems to do the trick;

    https://www.nltk.org/book/

    The dead tree version is not updated to Python 3, but otherwise it seems legit.
    Pages: 1 2 Next Last
      Main » Discussion » Looking for a python script in the now defunct archives
      Yes, it's an ad.