I recently got the much-hyped Arturia Minibrute - terrific small analog synth that would be fun for a starter. If you don’t have the money for classics like the Roland SH-101 or Korg MS-10, this might be a good alternative. The pre-orders are quite backed up now though, so availability will be limited for most of the year.
If I was starting out today and needed a budget setup, I’d probably just buy a used MIDI keyboard and a few of the many affordable monophonic modules out there now: Meeblip, DSI Mopho / Evolver, Moog Slim Phatty or Minitaur, Vermona Mono Lancet, Doepfer Dark Energy, Oberheim SEM reissue, Analog Solutions Semblance, MFB Zwerg series, etc.
For polyphonic sounds: Roland JX-3P/8P or Alpha Juno, Korg Poly 800/61 or DW-6000/8000, or maybe one of the Casio CZ’s. There’s also some great and cheap “toy” Yamaha’s that do nice lo-fi digital FM sounds (PSS-470/480/570, etc.)
As for a drum machine, I’d just get a cheap sampler like a used Korg Electribe ES or Akai MPC 500/1000, and fill them with self-made sounds or WAV samples of vintage gear available online.
Also, effects: Electro-Harmonix are quite good (Electric Mistress, Memory Toy/Boy/Man, Holy Grail, Small Stone, etc) and even some Behringer pedals are impressive for the price if you’re desperate.