In general Security Director relies on the ports needed for Junos Space. The required ports are listed in https://kb.juniper.net/InfoCenter/index?page=content&id=kb18148
In summary only ssh from Space/SD towards the SRX gateways is needed. SD does netconf via ssh.
Ping and snmp-read (udp/161) are optional but nice to have available.
In general your assumption is correct... but it depends on your setup.
It could also be that you only allow ssh as host-inbound-service system-services on the relevant zone/interface and then have a RE protection firewall filter to handle which IPs can access via ssh on this zone.
Alternative could also be a global policy which allows management across all zones to avoid doing multiple src-zoneX/Y/Z to junos-host policies (if ssh access is needed from multiple different zones)
Junos provides you many ways to accomplish the same goal 🙂