Internal Communication Between Microservices on Heroku Common Runtime: Why Eureka and Load Balancing Won't Work?
This blog discusses the problem I faced while working with microservices deployed in the Heroku Common Runtime. Before diving into the problem, let me first give you a short introduction to microservices and Heroku.
What is Microservices?
Microservices is an architecture that decomposes a large application into small, independent units based on their functionality. Each module is independently developed and deployed separately.
In a distributed environment, how does one service know about another? This is where service discovery comes into play. All services register themselves with a service discovery system. When one service needs to communicate with another, it discovers the target service through this service discovery. In my setup, I use Netflix's Eureka Server for service discovery.
To isolate services from the outside world, an API Gateway is introduced. Clients cannot access microservices directly, They must go through the API Gateway, which provides security and other cross-cutting concerns.There are also many architecture patterns which simplify the implementation of microservices.
I just created a simple Messenger application which has 4 microservices. Main objective of this application is to broadcast messages to all users. It has a Notification Service that needs to call User Service to retrieve the email addresses of all users and then broadcast a message to everyone. They communicate with each other through API service calls.
4 Microservices are
Eureka Server - Service discovery for all the services
API Gateway - Route all the request to the respective microservices
User Service - Handles user management.
Notify Service - Handles sending messages to the users.
All the microservices are initially deployed locally and working well. Finally I have to move these microservice to Heroku.
What is Heroku?
Heroku is a cloud platform where you can easily run your apps without worrying about servers or setup. In Heroku’s Common Runtime, all services are deployed in dynos which share the same network, and each service gets its own public URL. The applications don’t have the fixed IP addresses. Once the service is deployed on Heroku, you will get a public URL like https://your-app-name.herokuapp.com. So you can access via postman or swagger.
What are Dynos in Heroku Common Runtime?
Dynos are Heroku managed lightweight linux containers. All applications run on dynos. The Common Runtime has a single dyno manager per region which is responsible for all dynos and all customers running in a region.
Dynos are classified based on the Use Case
Web Dynos are used to run your web applications, API services, microservices, and so on. They handle incoming HTTP requests and provide responses. I used web dynos for all my microservices.
Worker dynos are used to run background jobs, queueing systems, and timed jobs. You can have multiple kinds of worker dynos in your application.
One-of-Dynos are created for a specific task that are not part of the application. You can use one-off dynos to initialize a database, run a command-line tool, or interact with your application's database console.
Dynos in the Common Runtime can only receive requests from the routing layer (This was the point I missed). Heroku's router receives the request from the client and forwards it to the respective dyno. Only web dynos can receive the request in this way. Each dynos are secured with a strong firewall even though they run in a single network, they are isolated from each other.
I have deployed all my microservices in Heroku. The Eureka Server is running, and all other services are accessible through the API Gateway. The API Gateway successfully discovers all services from Eureka, and everything is working fine through the Gateway.
However, there is one problem:
The Notify Service is unable to call the User Service directly using WebClient. All other endpoints are working fine.
Notify Microservice
WebClientConfig.java
@Configuration public class WebClientConfig { @Configuration @LoadBalanced @Bean public WebClient.Builder builder() { return WebClient.builder() .baseUrl("http://USER-MS"); //Eureka service name } } |
NotifyService.java
public class NotifyService { private final WebClient.Builder webClientBuilder; public List<String> getUserLoginList() { List<String> emailList= webClientBuilder.build() .get() .uri("/users/fetch-emails") .retrieve() .bodyToMono(ArrayList.class) .block(); return emailList; } } |
UserEmailFetchController.java
@RequestMapping("/fetch-emails") public class UserEmailFetchController { private final NotifyService notifyService; @GetMapping("/webClient") @ResponseStatus(HttpStatus.OK) public List<String> fetchLMSUserEmails () { return notifyService.getUserLoginList(); } } |
When I access the notify service to fetch the emails from the user service.
It throws the below error.
Error
error code=H12 desc="Request timeout" method=GET path="/notify-service/fetch-emails/webClient" host=notify-service-ca24c535831c.herokuapp.com |
Why is the Notify Service not able to call the User Service directly?
In Heroku’s Common Runtime, services cannot communicate internally. Each service is given a public URL like https://user-service.herokuapp.com. Eureka usually tells services to connect using their internal names (like http://USER-MS), but in Heroku, that won’t work because
Apps don’t have private IP addresses
They can be accessed only over the public internet
Internal names like http://USER-MS are not valid. You must use the public Heroku URL
That's why WebClient inside my Notify Service fails to connect when it tries to use the service name from Eureka.
Why can API Gateway access services using Eureka service names in Heroku Common Runtime, but Notify Service cannot?
When API Gateway tries to access another service using the service name (like http://USER-MS), it does not actually call the service directly over the network.
Instead, it uses Eureka service discovery, gets the registered instance info, and builds a new HTTP request to the public URL (like https://user-service.herokuapp.com)
API Gateway knows the public URLs of the services through Eureka because the services are registered with their Heroku public URLs in Eureka.
Notify Service, when using WebClient, tries to connect directly to http://USER-MS, but in Heroku Common Runtime, there is no DNS, so the connection fails. Eureka and client-side load balancing don't work as expected.
Solutions
Use Heroku Private Spaces
If private communication between dynos is a must , go for Private Spaces In which dynos are attached to a private network. Private service discovery and load balancing work. Eureka-based architectures become possible. Private Spaces are very costly.
Communicate via Public URLs
Simple and cost effective approach is Communicate via public URLs. Use Spring Profiles to separate local development from cloud deployment.
In my application I used the second method. Incorporated some changes in WebClientBuilderConfiguration.
WebClientConfig.java
//Load Balanced WebClient Builder for local deployment @Bean @Profile("local") //webclient is created for local profile @LoadBalanced public WebClient.Builder loadBalancedWebClientBuilder() { return WebClient.builder() .baseUrl("http://USER-MS"); } // Plain WebClient - for communication between services using public URLs @Bean @Profile("dev") // WebClient is created for Heroku development public WebClient.Builder plainWebClientBuilder() { return WebClient.builder() .baseUrl("https://user-service.herokuapp.com"); } |
If you know any other ways to handle this scenario, please share them in the comments section.
Happy Coding !!!
