Perry (M0PYL)'s advice for learning Digital Signal Processing. This is extended from an email I sent to someone asking for advice.
I'd suggest starting with an RTL SDR, they're super cheap, RX only enough to start playing around. They're relatively cheap, and so less concerning if you damage or mod it or whatever. TX capable SDRs are expensive (relative to RX SDRs) and the cheaper TX capable SDRs often introduce a lot of noise. None of the transmit capable SDRs output significant power, you'll need additional filters and a power amplifier if you want to transmit for a reasonable distance.
If you want to transmit, I'd strongly suggest getting your Amateur License if you don't have it already. While you can transmit at extremely low power (enough to go across a room) without a license, you're going to want to transmit on the quieter bands and higher power levels that Amateur radio is for. In the UK the foundation level is 26 multiple questions and you need 19/26 correct. You can sit the exams online with a windows or macos PC. If you've done physics at school, you probably already vaguely remember the complicated parts (Ohms law) and how SI units work, the most of the rest of the work is learning about the rules and conventions of amateur radio. https://rsgb.org/main/clubs-training/for-students/foundation/ has more information.
Prototype what you're doing in GNU Radio Companion to start with until you get the hang of the various algorithms and what they do, how they work and how to properly tune them to do what you need. Start by building the an FM Broadcast radio decoder (The “Hello World” of DSP) and tinker with it to find out how all the settings impact things and tune it until it works satisfactorly.
Then start with experimenting with decoding something pretty simple, FSK is a good place to start. It's extremely easy to decode, even by eye - you can probably decode 2FSK by eye just by looking at the carrier. There's lots of information about how to decode AX.25 (2-AFSK at 1200 baud). Once you have FSK working, you can start on more complex modulation system. There's lots of lessons to learn with prototyping in GNU Radio campanion.
Once you have a handle on the high level concepts, I'd start on writing some software. I'd suggest starting with jupyter so that you can easily plot things and have visualisations of what's going on. Visualising what's going on is critical - I spent a month trying to figure out why my code wasn't working until I eventually hooked up something to visualise it and could see that my initial filter wasn't working properly….
Once you have a version working in Jupyter, then you can write it in whatever code you want.
Another trick is to have your software dump out a file that can be read into gnuradio so you can use the gnuradio visualisation tools and check your working for individual parts of your decode pipeline.