Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing AMF parsing #6753

Open
6600024d opened this issue Mar 24, 2024 · 5 comments
Open

Fixing AMF parsing #6753

6600024d opened this issue Mar 24, 2024 · 5 comments
Labels
kind/feature New features / enhancements

Comments

@6600024d
Copy link

Problem Description

AMF ( https://en.wikipedia.org/wiki/Action_Message_Format ) parsing support vanished at some point in the early mitmproxy releases, and didn't come back. I'm planning to re-plug pyamf or some AMF lib in mitmproxy to decode AMF.

CONTRIBUTING says :

If you intend to work on a larger contribution to the project, please come talk to us first.

There we go :)

Proposal

I figure out how to plug some AMF lib (the python ones are pretty old, but might work and one of them has a nice C implementation, there's surely some lib for that in ruffle.rs but I'm no cross-language packaging expert, and the spec itself doesn't seem that overly complicated, so as a last resort I'll try to write one myself).

I've got some X-AMF traffic there, not sure I need an extensive dataset as long as I can plug an existing lib.

Alternatives

Not parsing AMF with mitmproxy.

Questions

Is there anything specific I should be aware of ? I'll follow the doc otherwise. Cheers !

@6600024d 6600024d added the kind/feature New features / enhancements label Mar 24, 2024
@mhils
Copy link
Member

mhils commented Mar 24, 2024

Thank you for raising an issue before starting your work! 🍰

Generally happy to add AMF parsing, my only concern here is that I'd like to avoid adding a dependency on an additional library. In particular, I'm very strongly -1 on anything that involves file format parsing in C. Both from a memory safety perspective as well as a "needs a compiler to install" perspective. Is there maybe already an existing Kaitai Struct definition for AMF somewhere? Not sure how complex AMF is, but if this can be done with Kaitai this would be by far my preferred option. :)

@6600024d
Copy link
Author

Hey, status update: I managed to get some amf0 parsing and it works well against the https://github.com/sile/amf/tree/master/src/testdata dataset. Here's the Kaitai Struct YAML file:
amf0.v1.ksy.txt, (I couldn't find amf ksy files online, maybe I'm bad at searching) and here's what parsing amf0-ref-test.bin gives:
image
It's pretty verbose, maybe there are KS tricks to reduce the amount of layers.

I didn't have a chance yet to plug that in mitmproxy to observe my own intercepted data. I'm specifically wondering if it's raw amf0/amf3 objects or if they're packed in the custom amf-packet format. Because both could be used, and there's no way to tell amf-packet data from amf0/3 data without attempting to parse it. I guess I'll have to offer both, even if having amf-packet and amf as types feels superfluous, I don't foresee a viable alternative.

Are there examples of Kaitai Struct -parsed formats in mitmproxy I could borrow code from ?

@mhils
Copy link
Member

mhils commented Mar 27, 2024 via email

@6600024d
Copy link
Author

6600024d commented Apr 2, 2024

Hey, status update:

  • amf0 : more or less OK. could be enhanced: boolean, undefined, null, references, times
  • amf-packet : OK (!) (or is it ? :D)
  • amf3 : just started, I'm struggling with the U29 integer format they invented, pretty much UTF-8-for-uint32. There will be the same possible enhancements as for amf0 on data types, and I'll also have to sort out some bitfield issues. (Wouldn't be fun if these bitfield flags were not applied to U29 integers, yay.)

I got some pointers from the peeps developing ruffle.rs, the flash runtime. My own dataset is amf-packet with headers packing amf0 objects which sometimes are amf3 objects. I've been pointed at some random flash games that do generate amf-packet data as well. Unless I see raw amf0 objects on the wire then the parsing root will be amf-packet.

@6600024d
Copy link
Author

Status update

I managed to implement the U29 format, but am stuck at amf3_object_type, and can't reasonably invest more energy & time into this initiative « at this time ».

Blockers

  • I couldn't get the meaning of the specification for amf3_object_type or the few implementations I could find. I tried reading sile/amf, ruffle-rs/rust-flash-lso and hydralabs/pyamf and still could not make sense of the amf3 object type, notably when compared to real-world data.
  • The resources I can invest in this are too scarce. Real life. This, or I'm too stupid to read a spec. I guess I'll never know 🤷 :D.

Remaining work

  • Ideally, you just have to sort the amf3_object_type structure, possibly array and dictionary will require similar handling.
  • Then, plugging the spec in plugging it in mitmproxy might not be that hard. I don't know hehe.

Contribution so far

  • amf-packet : implemented
  • amf0 : implemented, and tested to some extent against the sile/amf dataset and some custom flash app.
  • Limitation: "null", "undefined" and other JavaScript-ish objects are not well handled by Kaitai Struct so we just skip them
  • Limitation: so far, there's no support for references, I'm not sure if it's possible (or worth it) to perform bookkeeping like that in Kaitai Struct.
  • I managed to sort out that u29 format, which isn't the vlq_base128 found in the Kaitai Struct standard set.

Enclosed .KSY file

Anyways, here's my amf.ksy file so far, I'm releasing it as public domain for whomever wants to finish tackling this down. I'd be very honored if anyone managed to reuse my work to some extent.

amf.ksy.txt

Thanks for mitmproxy, it really is a great tool I keep recommending anytime I get a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New features / enhancements
Projects
None yet
Development

No branches or pull requests

2 participants