tehtmi

Opmap is slightly off. Collisions (two codes mapping to the same op) are suspicious. Math ops like addi must be followed by the matching metatable operation like mmbini. Not sure if you were trying to decode by hand or using a reference (not sure why it would be wrong). move and loadk are easy to mix up in a way since they both assign a register and don't use a second parameter, but constant usage is usually pretty easy to track since constants are (first) used mostly in order. close is always easy...

Opmap is slightly off. Collisions (two codes mapping to the same op) are suspicious. Math ops like addi must be followed by the matching metatable operation like mmbini. Not sure if you were trying to decode by hand or using a reference (not sure why it would be wrong). move and loadk are easy to mix up in a way since they both assign a register and don't use a second parameter, but constant usage is usually pretty easy to track since constants are (first) used mostly in order. close is always easy...

Opmap is slightly off. Collisions (two codes mapping to the same op) are suspicious. Math ops like addi must be followed by the matching metatable operation like mmbini. Not sure if you were trying to decode by hand or using a reference (not sure why it would be wrong). move and loadk are easy to mix up in a way since they both assign a register and don't use a second parameter, but constant usage is usually pretty easy to track since constants are (first) used mostly in order. close is always easy...

Yes, I think it is reasonable for you to modify unluac in this way. (The other option is to do a separate preprocessing step where you undo any encryption and standardize the file, which is also a fine approach -- you have to write a binary Lua parser, but it's pretty easy.) The encryption scheme you described sounds plausible given what I was seeing. The kind of decryption you want could easily be applied in LFunctionType.parse_code (LFunctionType.java). While you're at it you could also make unluac...

Opmap won't be sufficient for decoding the bytecode. There could be some kind of encryption or obfuscation going on. It's weird because the A bitfield seems like it's encoded as-is, but B isn't and probably C isn't. And it seems not consistent between functions, but maybe consistent within a function. Pretty confident op112 is closure in the main function, but then it would need exactly one usage in main/f7 which is not the case. op111 is probably call in main/f0, but then doesn't appear anywhere...

You can use unluac's disassembler with --disassemble. This works if you change the first byte like you said. It doesn't rely on the opmap being correct, and it also doesn't care about string contents (only lengths are needed to parse correctly). So yes, it looks like both these things may be the case. Opmap should be a text file where each line is .op <opcode> <opname>, for example the standard Lua 5.4 opmap would start like: .op 0 move .op 1 loadi .op 2 loadf .op 3 loadk .op 4 loadkx Op names match...

Custom instructions look like bitwise operators. Unluac supports this natively using Lua 5.3+ semantics. (Your code is Lua 5.1 which usually doesn't support bitwise operators.) To do this, you need to provide an unluac "opmap" file via --opmap opmap.txt that tells unluac how to decode the custom bytecode. I think this opmap should work (based on contextual usage, although shl, bxor, and bnot do not appear in my code sample): .op 0 move .op 1 loadk .op 2 loadbool .op 3 loadnil .op 4 getupval .op 5...

Custom instruction look like bitwise operators. Unluac supports this natively using Lua 5.3+ semantics. (Your code is Lua 5.1 which usually doesn't support bitwise operators.) To do this, you need to provide an unluac "opmap" file via --opmap opmap.txt that tells unluac how to decode the custom bytecode. I think this opmap should work (based on contextual usage, although shl, bxor, and bnot do not appear in my code sample): .op 0 move .op 1 loadk .op 2 loadbool .op 3 loadnil .op 4 getupval .op 5...