I walked back to Kiyosumi-Shirakawa station and got back on the Toei Oedo line to go Northwest 7 stops to Kasuga station. There I changed to the Metro Namboku line at Korakuen station to go 3 stops north to Komagome station. Although this was a slightly longer route, it involved less walking outdoors and given the rain that seemed like my best option.
Once at Komagome it was a short walk to the garden, and of course I had my Touch instructions.