Manage a separate collection of users that should be affected by time with a pre-calculated BaseNetworkable.SaveInfo that contains the packet data for time syncing, removing serialization of the entity from the hook completely.
Update this collection when ever the desired time for a user changes, reusing the previous save info.
Subscribe/unsubscribe to the hook based on whether there are any players affected by time.