May I ask how you are automating the ELB/TLS configuration and how that ties into the Ingress controller? Do you somehow specify which ELB it should use? We're in a similar situation.
You can annotate any Service of type LoadBalancer in order to configure various aspects[1] of the associated ELB, including which ACM-managed certificate you want to attach to each listener port.