Sometimes the “easy” path turns out to be the hard one. I recently integrated a legacy Java/Spring Boot system with a managed Kafka cluster (Aiven) in an enterprise environment. The twist: we had to route through a corporate proxy to reach external services.

The recommended approach was Kafka REST Proxy. It seemed reasonable - HTTP traffic through existing proxy infrastructure, no new firewall rules needed. Six months later, I can tell you: go native if you possibly can.

Here’s what I learned.

Why We Chose REST Proxy

The constraints were typical enterprise:

  • Legacy Java 17 / Spring Boot application
  • All external traffic must route through corporate HTTP proxy
  • Opening new firewall ports requires security review and documentation
  • Managed Kafka cluster (Aiven) outside the corporate network

REST Proxy seemed like the path of least resistance. HTTP traffic, existing proxy infrastructure, no firewall changes. The platform team had used it before. Ship it.

The Problems Started Immediately

1. Stateful Consumers Are Awkward

Native Kafka consumers maintain state - partition assignments, offsets, consumer group coordination. The broker tracks all of this.

With REST Proxy, you’re making stateless HTTP calls to a stateful system. The proxy maintains consumer instances that you must explicitly create, poll, and destroy:

// Create consumer instance
POST /consumers/my-group
{
  "name": "my-consumer",
  "format": "json",
  "auto.offset.reset": "earliest"
}

// Poll for messages (must use same instance)
GET /consumers/my-group/instances/my-consumer/records

// Commit offsets
POST /consumers/my-group/instances/my-consumer/offsets

// Delete when done
DELETE /consumers/my-group/instances/my-consumer

If your consumer crashes without deleting the instance, you have orphaned state on the proxy. If you poll too slowly, the instance times out. If you create too many instances, you exhaust resources.

Native Kafka handles all of this automatically.

2. Partition Ordering Gets Complicated

We needed ordered processing within partitions - events for the same entity must be processed in sequence. With native Kafka, you just use a partition key:

producer.send(new ProducerRecord<>(topic, entityId, event));

All events with the same entityId go to the same partition. Done.

With REST Proxy, you specify the partition key in the request body, but you’re now depending on the proxy to hash consistently. We had 3 partitions and needed to verify the proxy’s partitioning matched what we expected. Extra testing, extra documentation, extra doubt.

3. Performance Overhead

Every Kafka operation becomes an HTTP request:

  • TCP connection overhead (or connection pooling complexity)
  • JSON serialization/deserialization
  • Proxy hop latency
  • No batching optimizations

For high-throughput scenarios, this adds up. Our message rates were modest (hundreds per minute), but latency percentiles were noticeably worse than native benchmarks.

4. Error Handling Is Different

Native Kafka clients have sophisticated retry and error handling built in. Transient broker issues, rebalances, network blips - the client handles most of it.

With REST Proxy, you get HTTP error codes. A 5xx might mean the proxy is down, the broker is unreachable, or your consumer instance expired. You build retry logic around HTTP semantics, not Kafka semantics.

// Native: built-in retries, backoff, etc.
producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        // Kafka-specific exception handling
    }
});

// REST Proxy: HTTP error handling
try {
    response = httpClient.post(url, body);
} catch (HttpException e) {
    // Is this retryable? Depends on status code.
    // Is the consumer instance still valid? Who knows.
}

5. Observability Gaps

Kafka’s native metrics (consumer lag, producer throughput, partition distribution) are well-supported by monitoring tools. Prometheus exporters, Grafana dashboards, alerting - it all just works.

With REST Proxy, you’re monitoring HTTP endpoints. You can see request rates and latencies, but the Kafka-specific insights require extra work. How far behind is your consumer? The proxy knows, but exposing that cleanly takes effort.

When REST Proxy Makes Sense

To be fair, REST Proxy isn’t always wrong:

  • Polyglot environments where not every service can run a JVM
  • Serverless functions that can’t maintain long-lived connections
  • Quick prototypes before committing to native integration
  • Firewall constraints that genuinely can’t be changed (rare, but real)

For our use case - a persistent Java service running 24/7 - none of these applied.

The Native Path

Six months in, we started the work we should have done initially: native Kafka integration.

The requirements:

  1. Document the traffic flow for IT security
  2. Request firewall rules for Kafka ports (typically 9092-9094)
  3. Configure SASL/SSL authentication
  4. Update the application to use native Kafka clients

The firewall process took weeks of back-and-forth with security teams. But once approved, the integration was cleaner:

@Configuration
public class KafkaConfig {

    @Bean
    public ProducerFactory<String, Event> producerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootstrapServers);
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        config.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
        config.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
        config.put(SaslConfigs.SASL_JAAS_CONFIG, saslJaasConfig);
        return new DefaultKafkaProducerFactory<>(config);
    }
}

Spring Kafka handles consumer groups, partition assignment, offset commits, and retries. The code is simpler, the behavior is predictable, and monitoring works out of the box.

Lessons Learned

1. “Easy” infrastructure choices often aren’t. REST Proxy avoided firewall paperwork but created months of application complexity.

2. Fight for native integrations. The security review process is painful, but the operational simplicity is worth it. Document thoroughly, explain the benefits, and push back on “just use HTTP” shortcuts.

3. Prototype with production constraints. We tested REST Proxy in isolation before hitting the edge cases in production traffic patterns. Earlier load testing with realistic scenarios would have surfaced issues sooner.

4. Talk to teams who’ve done it. We eventually connected with another team that had completed native Kafka integration. Their documentation and lessons learned saved us weeks. Should have reached out earlier.

Conclusion

Kafka REST Proxy works. It’s a legitimate tool for specific use cases. But for persistent services that need reliable, high-throughput Kafka integration, native clients are almost always the right choice.

The overhead isn’t just performance - it’s operational complexity, debugging difficulty, and fighting semantics that don’t quite fit. If you’re choosing REST Proxy to avoid firewall paperwork, reconsider. The paperwork is finite; the technical debt compounds.

Building Kafka integrations in enterprise environments? I’ve been there: [email protected]