Optional peer dependencies should not be required

[StefanTheWiz]

I've tried running the code at https://github.com/floydspace/effect-kafka/blob/main/examples/consumer.ts with kafkajs

It appears that in order to run that, it also requires @confluentinc/kafka-javascript which is marked as an optional peer dependency here

My set up:

yarn add effect-kaka kafkajs
cp examples/consumer.ts example.mjs
node example.mjs

node:internal/modules/cjs/loader:1225
  const err = new Error(message);
              ^

Error: Cannot find module '@confluentinc/kafka-javascript'
Require stack:
- /path/to/project/node_modules/effect-kafka/lib/ConfluentKafkaJSInstance.js
- /path/to/project/node_modules/effect-kafka/lib/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1225:15)
    at Module._load (node:internal/modules/cjs/loader:1051:27)
    at Module.require (node:internal/modules/cjs/loader:1311:19)
    at require (node:internal/modules/helpers:179:18)
    at Object.<anonymous> (/path/to/project/node_modules/effect-kafka/lib/ConfluentKafkaJSInstance.js:30:28)
    at Module._compile (node:internal/modules/cjs/loader:1469:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)
    at Module.load (node:internal/modules/cjs/loader:1288:32)
    at Module._load (node:internal/modules/cjs/loader:1104:12)
    at Module.require (node:internal/modules/cjs/loader:1311:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/path/to/project/node_modules/effect-kafka/lib/ConfluentKafkaJSInstance.js',
    '/path/to/project/node_modules/effect-kafka/lib/index.js'
  ]
}

I can make it work by changing to more granular imports but I wonder if there are better ways to handle this

import * as KafkaJSInstance from "effect-kafka/lib/KafkaJSInstance.js";
import * as Consumer from "effect-kafka/lib/Consumer.js";
import * as ConsumerRecord from "effect-kafka/lib/ConsumerRecord.js";
import * as MessageRouter from "effect-kafka/lib/MessageRouter.js";

[floydspace]

hi @StefanTheWiz

It appears that in order to run that, it also requires @confluentinc/kafka-javascript which is marked as an optional peer dependency here

it is indeed should not be the case, thank you for raising it. This happens due to all the implementations are reexported through index file, so good point.

Idealy it should be like this:

import { Consumer, ConsumerRecord, MessageRouter, InMemoryKafkaInstance } from "effect-kafka";
import * as KafkaJS from "@effect-kafka/kafkajs";
import * as RdKafka from "@effect-kafka/rdkafka";

But for now I can fix it to use like this:

import { Consumer, ConsumerRecord, MessageRouter, InMemoryKafkaInstance } from "effect-kafka";
import * as KafkaJS from "effect-kafka/KafkaJSInstance"
import * as RdKafka from "effect-kafka/ConfluentRdKafkaInstance"

[StefanTheWiz]

Thinking it could potentially be simplified / reduced to a default export or a named export (or both)?

import { Consumer, ConsumerRecord, MessageRouter, InMemoryKafkaInstance } from "effect-kafka";
import KafkaJS from "@effect-kafka/kafkajs"; // OR import { KafkaJS }  from "@effect-kafka/kafkajs";

Also I'm not sure if there would be a case when someone might want to use both KafkaJS and RdKafka in the same file / app?

import { Consumer, ConsumerRecord, MessageRouter, InMemoryKafkaInstance } from "effect-kafka";
import { Kafka } from "@effect-kafka/kafkajs"; // OR import { Kafka }  from "@effect-kafka/rdkafka";

If they do, they would likely be migrating from one lib to another and they could probably do

import { Consumer, ConsumerRecord, MessageRouter, InMemoryKafkaInstance } from "effect-kafka";
import KafkaJS from "@effect-kafka/kafkajs";
import RdKafka from "@effect-kafka/rdkafka";

[floydspace]

Thinking it could potentially be simplified / reduced to a default export or a named export (or both)?

The effect itself does not use default exports, not sure what is the main reason, maybe it is becasue default exports behaviour is not predictable from user to user, but @tim-smart could answer better
so I prefer to be consistent with effect here.

Also I'm not sure if there would be a case when someone might want to use both KafkaJS and RdKafka in the same file / app?

It is indeed possible and there is already such use case from one user, so I will keep unique naming

[floydspace]

@StefanTheWiz fix is in the v0.5.1

[StefanTheWiz]

mm seems to break for ESM?

Error: Cannot find module '/path/to/project/node_modules/effect-kafka/dist/esm/internal/consumerRecord' imported from /path/to/project/node_modules/effect-kafka/dist/esm/ConsumerRecord.js
    at finalizeResolution (node:internal/modules/esm/resolve:265:11)
    at moduleResolve (node:internal/modules/esm/resolve:933:10)
    at defaultResolve (node:internal/modules/esm/resolve:1169:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:540:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:509:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:239:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:96:40)
    at link (node:internal/modules/esm/module_job:95:36)

[floydspace]

@StefanTheWiz could you share your setup? how do you run and if you are bundling?

I tried all examples with esbuild bundler, and tried running with tsx and ts-node, an they all work perfect

[StefanTheWiz]

I've put together a demo with tsc, available here: https://github.com/StefanTheWiz/debug-effect-kafka

My actual project uses NX + esbuild, which was already set to bundle the app as a single esm file. It ran with v0.4.x but not with v0.5.1

[floydspace]

hey @StefanTheWiz it is fixed in v0.5.2

thanks for the repro 👍