Understanding relativity is not a prerequisite for understanding quantum physics. But it helps. And understanding quantum physics helps in understanding relativity. I’ve found that it’s helpful to go back and forth between the two fields. But even more important for an understanding of quantum physics is grounding in classical physics, the physics of Isaac Newton.
So, do you need to understand the math of quantum physics and classical physics? That would mean being able to work with complex numbers, trigonometry, linear algebra (matrices & vectors), and calculus. If you’re just going for the concepts and if you pick the right books, you can get away without knowing the math. Of course, knowing the math really helps get an in-depth understanding of both classical physics and quantum physics.
I started with trig only and little understanding of linear algebra and calculus. Out of a desire to understand quantum physics better, I eventually studied both linear algebra and calculus. I was able to find free on-line courses, where I could study at my own pace (which was very, very slow!). These courses were offered by Ohio State University and Khan Academy.
Wishing you the best in your pursuit of understanding!