What the error is telling you is that a connection attempt was made, but the session authentication was not provided before SQLNET.INBOUND_CONNECT_TIMEOUT seconds.

As far as adverse effects in the long run, you have a user or process that is unable to connect to the database. So someone is unhappy about the database/application.
————————————————————————————————
In 10gR2, SQLNET.INBOUND_CONNECT_TIMEOUT the parameters were set to have a default of 60 (seconds).

Set the parameters SQLNET.INBOUND_CONNECT_TIMEOUT and INBOUND_CONNECT_TIMEOUT_listenername to 0 (indefinite).
Before setting SQLNET.INBOUND_CONNECT_TIMEOUT, verify that there is not a firewall or Network Address Translation (NAT) between the client and server. Those are common cause for ORA-3136.
————————————————————————————————
The parameter is to be added in sqlnet.ora on the server, so wont require a shutdown.
$ORACLE_HOME/network/admin/sqlnet.ora
and edit it at server side.