I have a question about learning Japanese.?

This Kanji thing is agitating.

For all Japanese speakers, did you find it better do learn the Kanji as you learned the words, or did you learnt to speak Japanese first and then worry about the Kanji later?
I can read and write hiragana and katakana. I