There are couple of things you can do to limit access to the SSH service on your system.
If you have a static IP address that you want to use to accesss the
server, setting the ROUTER variable in the firewall
scripts is a good idea because the SSH port for everybody not coming from
that address will be closed by the firewall.
Other things you should consider:
Disabling root login, installing sudo and adding your username to the sudoers list.
Changing SSH server to listen on a non standard port (default is 22). Just remember to change your firewall rules.
Disabling login via the use of passwords and use keys to login to your server.
Use port knocking.