Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(library): Serial Transmitter Interface #103

Draft
wants to merge 27 commits into
base: Main-Trunk
Choose a base branch
from
Draft

Conversation

ZZ-Cat
Copy link
Owner

@ZZ-Cat ZZ-Cat commented Mar 19, 2024

Overview

This Pull Request will bring in an exciting new feature: The Serial Transmitter Interface.

Note

This Pull Request focuses on bi-directional full-duplex non-inverted communication with the internal transmitter module in your handset.
The external module bay of your handset manages communication in a way that is outside the scope of this Pull Request, and it MAY warrant a Pull Request entirely of its own.

Details

If you recall the Serial Receiver Interface is the middleware layer between your Sketches (via the Sketch Layer) and your connected ExpressLRS or TBS Crossfire reciever (via the Hardware Abstraction Layer, which is managed by the Arduino API for back-end functionality).
The Serial Receiver Interface is so-called this because it implies CRSF for Arduino communicated with your connected ExpressLRS receiver. EG A RadioMaster RP3.

The Serial Transmitter Interface will add bi-directional communication to-and-from your handset's internal transmitter module.

Up until now, I have been focusing on the receiver-to-"flight controller" end of The Crossfire Protocol, and I had not even thought about the handset's side of the protocol... until it was brought to my attention by a group called Atopile and their Swoop project.
Atopile have asked me to put the Serial Transmitter Interface together, so that it MAY serve as the communications between the handset itself and its internal ExpressLRS transmitter module.


How this has impacted my quality-of-life and what can be done about it

This is the first time a company has approached me to implement a feature they need

In fact, Atopile are the first company I have ever worked with in this context, period.

At first, I thought this was pretty exciting and wanted to get it done ASAP for them.
However, as time marched on, the novelty of that wore off and reality set in... and that reality is "Oh f*ck. A bona fide company has asked me to essentially 'work for them' for free." Thus, putting me into the same bucket as most other FOSS developers out there.

Since that realisation, I have felt an unnecessary amount of pressure to get this done... even more than what I normally get when someone asks me to implement something. To be clear here, I can easily handle feature requests and what-not with no problems at all. Also, the company themselves have not done anything to make me feel this way.
It is simply me getting in my own head and over-thinking into Oblivion about it.

With that being said, it has still burned me out all the same.
Because of that, I have been stuck on how to actually go about this, and I had overwhelmed myself with "option paralysis" when I learned that on the handset side of things the internal and external transmitter modules handle things in completely different ways as far as their physical layers are concerned.
So, I figured it would be a good idea at the time to implement both in this Pull Request... only to burn myself out on option paralysis even harder.

So now, I need to remember to....

Focus on one feature at a time

In other words: Don't sacrifice myself in favour of my own project!

Instead of me being ambitious at the expense of my own quality-of-life, I have decided to drop implementing the external module bay version of the Serial Transmitter Interface for the time being.
I had a long time to think this one through, and the more I thought about it, the more I felt tackling both versions at once in one Pull Request was too overwhelming for me.

Atopile have only requested I implement communication to-and-from an internal transmitter module, and me doing both the internal transmitter module and external transmitter module bay communications would be me going above-and-beyond.... and what has that done to me as a developer? It burned me out to the point of getting dangerously close to shutting this project down.... which is something that I had a history of doing with other projects in my past. CRSF for Arduino is the only project of mine that has seen any success at all (for real: 100+ stars and 4,000 users is nothing to sneeze at), and that is what's currently keeping me from hitting that "delete repository" button. That success is all from you guys, and I am very grateful for that.

Tackling the much wider issues - Developer burnout, social engineering, and supply chain attacks

Currently, there is a lot on my plate as far as the upcoming Version 1.1.0 is concerned... probably a lot more than what I am used to. Most of which is all security-related stuff in light of the XZ Utils Backdoor incident.
I know, I know. I keep banging on about XZ Utils, and I probably should shut up about it. However, it's not just the back door itself. It's how it happened. It was a supply chain attack. A burned out sole maintainer was about to chuck in the towel on their project, and some clever cybercriminal named Jia Tan came along and socially engineered the maintainer into making them (Jia Tan) the primary maintainer before they (the original maintainer) left the project.
Over time, Jia Tan got to work on writing (and implementing) dodgy code... and a few years later... hey presto, we have a back door on our hands what's slated to be released into the wild that could have spelled devastating results to an ecosystem what prides itself on its security.

I keep bringing up XZ Utils, 'cause I be like "Who's to say the same thing won't happen to me?" 'Cause I am in the exact same position - A burned out and overwhelmed solo maintainer of (an increasingly) widely used project.
Okay, sure, I write firmware, and CRSF for Arduino is a firmware library that targets microcontrollers. How could malware possibly be deployed to that? Quite easily. In fact, it's easier than you may think. All a threat actor would need to do is convince me that I can trust them (good luck with that, 'cause I got severe trust issues to begin with, and because I'm neurodiverse, social engineering doesn't work on me anyway), they gain access to my project, do their thing, and the next thing you know... malware has been covertly deployed under the guise of my legitimate project... and I could be powerless to do anything about it, because either I have been removed from the project (because the cybercriminal was granted admin priveleges) or I decided "enough is enough" with my project and door-slammed and completely washed my hands of it. That is a situation I don't want to happen to my project, and I am taking all steps I deem necessary to negate the chances of that happening, at all.

This begs the next question: Who would want to inject malware into my project in particular?
Well, ask yourself this: Who would want to come up to you while you're on your daily commute to/from work, and mug you for your money and your ID, and impersonate you while they get up to the most heinous of crimes? If you answered "that will never happen to me", then that's all the more reason why I am doing this. Because "it will never happen to me" is the epitome of complacency. It is that complacency that I view as being equal parts as dangerous as that identity thief in my analogy. I feel as a responsible maintainer, I have an obligation to protect my project from supply chain attacks and any other potential vector a cybercriminal could use to hijack CRSF for Arduino.

Coming full circle - Reducing burnout

I have decided to return to my "old schedule" where I work only between two and four hours per working day, and that is only in the mornings, during the week and on days where I'm not in physical pain (I also have rheumatoid arthritis, which can severely impact my ability to do literally anything, let alone code).

At this point, I MUST prioritise taking care of myself before I can take care of my project(s).
Because when I am constantly burned out, how can anyone expect me to write any code at all, let alone anything that's of good quality?
Up until a few weeks ago, my philosophy with CRSF for Arduino has been "Quality over quantity" and "less is more".
Why did I change this? Because I let my own intrusive thoughts win out, and I fell back into old habits what led me to the current state I am in. So, yes, burning myself out is my own fault, and this is what I am doing to rectify that.

What this means to you

This means that it will take longer for things to be implemented, yes. So, please be patient.
I would rather provide you all with a code-base that is of good quality that does what it says on the tin, as opposed to a code-base that is focused on sacrificing quality in favour of quantity.

...and after seeing several great projects become abandonware, I prefer to not abandon CRSF for Arduino. Especially when I am well aware of how popular it is becoming. You guys are what's holding me accountable here, and I love it. =^/.~=

This is to ensure the relevant pull request for this branch gets created as early as possible, because this branch will introduce an exciting new feature...
@ZZ-Cat ZZ-Cat self-assigned this Mar 19, 2024
@ZZ-Cat ZZ-Cat added ✨️ Enhancement ✨️ New feature or request ...in progress 🚧 Development on this is in progress labels Mar 19, 2024
@ZZ-Cat ZZ-Cat added this to the Version 1.1.0 milestone Mar 19, 2024
@ZZ-Cat ZZ-Cat linked an issue Mar 19, 2024 that may be closed by this pull request
1 task
…er Interface

This sets the precedence for the Serial Transmitter's API

This is not yet functional.
@ZZ-Cat ZZ-Cat removed the ⤴️ Rebase Needed ⤴️ This needs to be rebased. label Apr 19, 2024
…to commonly used targets

This temporarily speeds up development of the Serial Transmitter Interface. Once development on it is complete, running Quality Control on all supported targets will be reinstated.
@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Apr 20, 2024

It appears I may have hit a wall with this.
According to the documentation, external CRSF transmitter modules are using inverted half-duplex UART.
This could severely limit my options here, because a lot of supported targets don't have the ability to invert their GPIO pins what service UART. The SAMD51J19A being one of those.

I'm not entirely sure on where to go from here, because of that actually. I really feel like I have hit a wall here.

Probably the best thing for this Pull Request for the time being is to have it so that the Serial Transmitter Interface only works with internal transmitter modules, because those use full-duplex non-inverted UART. That is more straightforward.

Writing full-duplex is incredibly straightforward, because it means I don't need to get into the bare metal side of things.
Whereas with half-duplex, that's a little more involved, and not all hardware natively supports it (let alone the ability to assign custom UART pins). Even with targets that support it, I still need to write the parts where one pin and pad combination is swapped over (EG from Tx to Rx and vice versa), and that almost invariably involves bare metal coding - IE Dicking around under the hood with direct hardware registers.

IIRC, I did enable the ability to assign custom UART pins on supported hardware... so, it's likely that the Serial Transmitter Interface may only work with these targets if someone wants to use CRSF for Arduino with an external module (such as the RadioMaster Ranger).

@ABLomas
Copy link

ABLomas commented Apr 21, 2024

One transistor and resistor can solve inversion problem, so maybe this is not big deal?

Also, would like to clarify - i would be interested if it can just "output uninverted signal", just for TX, something like:
image
(in other words - read channel values, modify them and output to FC)
You may ask "is there anything that FC cannot do?" and i would answer "yes". For example - many GPIO's (on pi side) and autopilot mode trigger based on inputs, stepper motor control (using the same control channel, for example ch1-8 for FC, 9-16 for Pi with other stuff) and so on, so on.
This would not require inverted output, just two separate UART's, right?

@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Apr 21, 2024

One transistor and resistor can solve inversion problem, so maybe this is not big deal?

Also, would like to clarify - i would be interested if it can just "output uninverted signal", just for TX, something like:
image
(in other words - read channel values, modify them and output to FC)
You may ask "is there anything that FC cannot do?" and i would answer "yes". For example - many GPIO's (on pi side) and autopilot mode trigger based on inputs, stepper motor control (using the same control channel, for example ch1-8 for FC, 9-16 for Pi with other stuff) and so on, so on.
This would not require inverted output, just two separate UART's, right?

GET OUTTA MY HEAD!!! 🤣

No, seriously. This is why I spit ball my ideas in the comments section of my Pull Requests like this. Because I know someone can chime in and give me a hand on something that I could be having a hard time with on my own; and that's what this project is all about.

...and what you have suggested here has given me some extra food for thought.

I was thinking of starting the Serial Transmitter Interface off with standard non-inverted full duplex (IE separate Tx and Rx lines), because this is more straightforward to implement. Then possibly adding half-duplex on targets that are able to support it (through customisable UART pin assignments), and having the option to have it inverted on (currently few and far between, according to my information) targets that support GPIO inversion.

As far as I am aware, internal transmitter modules (such as the one in the RadioMaster TX16S Mk2) use standard full-duplex non-inverted UART, and the Rx and Tx lines are separate (like what we're all familiar with, on our receivers).


in other words - read channel values, modify them and output to FC

This would not require inverted output, just two separate UART's, right?

That's in the context of receiver-to-flight-controller.
Therefore your physical layer is two separate Tx and Rx lines, full-duplex non-inverted UART.
So, no. Hardware inversion isn't needed there.

I would be interested if it can just "output uninverted signal"

Yup. Can be done. As I said, that's straightforward to implement.

…ace prototype

This is initially populated with the typical Arduino "boiler plate" code what prints "Hello, World!" to the terminal.
…d enable it

This development environment will remain enabled until development of the Serial Transmitter Interface is complete.
@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Apr 25, 2024

Right! Now, I know where to go with this.

I am focusing on getting a "working prototype" of the Serial Transmitter Interface, where I am sending out the CRSF RC Channels packet at an arbitrary packet rate over non-inverted full-duplex UART.

I have created a file where development on this is taking place. It currently prints the phrase "Hello, World!" to the terminal, simply because I wanted to get the development environment set-up and ready-to-go.

The environment itself is slightly different from how I used to do it, in the fact that I have turned on several build flags related to compile-hardening. I have done this, so I can practically shoot two stones with one bird, because I want to bring in compile hardening to CRSF for Arduino's build process... especially in the context of my Quality Control.
So, now is a good time to trial-run that at the same time as I am working on the Serial Transmitter Interface.

Additionally, this development environment only builds the target that I am currently using to develop the Serial Transmitter Interface. Currently, this target is my tried-and-true Adafruit Metro M4 Express.
Eventually, I will port over to one of the RP2040 targets, because I have a fair amount of folks with RP2040-based targets watching this development with great anticipation.

Once development on this is complete, I will disable the development environment (in the same way as I have always done) and re-enable the entire build process across all compatible targets.

…croseconds) to the terminal at the selected packet rate

Pakcet rates are based on ExpressLRS' packet rates.
…rial1` at the selected packet rate

This sets the stage for CRSF packets.
This drops a frame when the time jitter is more than two microseconds.
…ironment

CppCheck is used to check for any defects during development.
Using the latest version of C++ keeps its security up-to-date.
Instead of spamming the time delta in the Terminal, a simple notification is sent; and the prototyp test now halts when the `time_us_max_allowed_error` is exceeded.
@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Apr 25, 2024

Progress update 2024-4-26

I now have a serial_transmitter_prototype.cpp in the $PROJ_DIR/examples/platformio directory, which was introduced in 86e9337.
As of 0515624, a blank full CRSF packet is sent out on Serial1 at a packet rate of 50 Hz. This test lasts for about three seconds and terminates.

The prototype itself watches the micros() counter during execution, and derives a time->time_us_error from the selected packet rate (in this case, the selected packet rate is 50 Hz, but all of ExpressLRS' standard packet rates (as of ExpressLRS v3.3.0) are available). This error is compared against a maximum allowed error time->time_us_max_allowed_error that's set at 2 µS.
When time->time_us_max_allowed_error is exceeded, the test terminates with an error showing how far out the time->time_us_error is.

I am still "fleshing out" the CRSF RC Channels packet itself.
Once I have that done, the serial_transmitter_prototype.cpp will send out centred (except for ch5, which will be locked to 1000 µS) RC channels, along with the correct CRC.

Effectively creating a "proof-of-concept" that I can refine as time marches on.

…ions

These will be used to calculate the CRC to be appended to the CRSF RC Channels data packet.
All RC Channels are 'centred' except for `ch5` which is initialised to `1000 µS`.
…acked data...

...and send it at the selected packet rate.
`CRC.cpp` and `CRC.hpp` will be shared across the Serial Receiver Interface and the Serial Transmitter Interface.
@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented May 19, 2024

Right-oh. Changing tack with this slightly.... well... more accurately, keeping it in-line with the rest of the library where I am not providing any drivers at all. You write/bring your own.
By the time I have encapsulated this into the code-base, you will be REQUIRED to create your own means of scheduling of sending out and receiving CRSF packets.

Doing it this way offloads the need for me to maintain my own drivers on top of the rest of CRSF for Arduino. This is precisely the reason why I made it so from the very beginning that this library does not provide its own drivers for anything. You essentially provide your own, and adapt CRSF for Arduino to that instead.

From what I understand of the implementations of the handset's side of the protocol is... internal transmitter modules use the same hardware layer as what we are using on our receiver-to-flight-controller side of things: Separate Rx and Tx lines that are full duplex and are non-inverted.
The external transmitter module bay is more complicated in the sense that it relies on a combined Rx/Tx line that is half-duplex and the data signal is inverted. This complicates things to the extent where it is outside the scope of this Pull Request, as this Pull Request dominantly focuses on communicating with an internal transmitter module.

I will update the OP shortly, to reflect this.

@dbloemhard
Copy link

Found this project while looking for ELRS libraries for a small transmitter project I am doing.

The Inverted UART should be a non-issue. Users can select whether CRSF is inverted or not on the TX module side, at the time of flashing. So this just needs to be a caveat in your firmware - be sure to uncheck the inverted UART checkbox when flashing your TX module. Its the same requirement for DeviationTX transmitters, and i see the same note in the SimpleTX project.

@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Jun 12, 2024

Found this project while looking for ELRS libraries for a small transmitter project I am doing.

The Inverted UART should be a non-issue. Users can select whether CRSF is inverted or not on the TX module side, at the time of flashing. So this just needs to be a caveat in your firmware - be sure to uncheck the inverted UART checkbox when flashing your TX module. Its the same requirement for DeviationTX transmitters, and i see the same note in the SimpleTX project.

Thank you for the tip. =^/.^=

I have had MAJOR Writer's Block over the last few weeks because of this.

@dbloemhard
Copy link

Definitely wouldn't stress over it :)
All ELRS modules are expected to be flashed at some point, so you just add a point that CRSF for Arduino library needs the transmitter module to be flashed with the Inverted UART option unchecked and thats all. I am sure there are more difficult parts to code without worrying about signal inverters!

@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Jun 13, 2024

Definitely wouldn't stress over it :)
All ELRS modules are expected to be flashed at some point, so you just add a point that CRSF for Arduino library needs the transmitter module to be flashed with the Inverted UART option unchecked and thats all. I am sure there are more difficult parts to code without worrying about signal inverters!

Yeha.
The folks what originally spurred this into action simply wanted me to make it so my library communicates with the internal module of their controller.
That is easy for me to implement, because the protocol for internal modules is basically the same as what we have between our receivers and flight controllers: Separate Tx and Rx lines.
So, it's trivial for me to write the code. Therefore, this is what I am deciding on implementing first.

The communication between your controller and your external module is half-duplex and it relies on a single wire line. Some of the targets my library supports may require additional hardware in order to make that single wire line possible, if it is not possible to configure a chipset's internal UART and respective pins to service that.

At this stage, I'll tackle the internal transmitter module side of things first, and get this Pull Request squared away.
Then, at a later date, I may look into the external module side of things and open a separate Pull Request for that, just to cut down on my perception of this being "HOLY SHIT, THIS IS SO OVERWHELMING!!!"

@dbloemhard
Copy link

Interesting. So I am using an RX flashed as a TX, with one of the receivers that has a 100mw PA onboard (100mw has been enough for me for all of the flying i do)... With the TX firmware flashed by default it becomes the same full duplex transmitter module with the RX and TX pads both being used. So this PR could be very useful to me as well.

Right now i am using the SimpleTX project as it is just a simple transmitter, no screen, no options to be set. But if your PR works, i may switch to using this library. Eventually the goal is to make a motion controller though, basically exactly what Swoop is, but no custom PCBs and as small as possible. I'd like to use a Betaflight FC flashed with the special firmware that Limon FPV described in his recent video (so it outputs gyro data via SBUS), an Arduino to translate that into CRSF, and then the RX flashed as a TX to then push that out to a tiny little whoop or micro drone.

@ZZ-Cat
Copy link
Owner Author

ZZ-Cat commented Jun 13, 2024

Interesting. So I am using an RX flashed as a TX, with one of the receivers that has a 100mw PA onboard (100mw has been enough for me for all of the flying i do)... With the TX firmware flashed by default it becomes the same full duplex transmitter module with the RX and TX pads both being used. So this PR could be very useful to me as well.

Right now i am using the SimpleTX project as it is just a simple transmitter, no screen, no options to be set. But if your PR works, i may switch to using this library. Eventually the goal is to make a motion controller though, basically exactly what Swoop is, but no custom PCBs and as small as possible. I'd like to use a Betaflight FC flashed with the special firmware that Limon FPV described in his recent video (so it outputs gyro data via SBUS), an Arduino to translate that into CRSF, and then the RX flashed as a TX to then push that out to a tiny little whoop or micro drone.

What's funny is... the guys that asked me to do this are the same folks that are working on Swoop. They sent me one of their prototypes to test this Pull Request on.
So, I can f*ck around and find out here. =^/.~=

Setting the version date to tomorrow's date, because that's when I'm picking development up on CRSF for Arduino again... after a nealry two-month long quiet hiatus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
✨️ Enhancement ✨️ New feature or request ...in progress 🚧 Development on this is in progress
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

writeRcChannel - Control CRSF Transmitters with CRSFforArduino
3 participants