Simplify your stack and build anything. Or everything.
Build tomorrow’s web with a modern solution you truly own.
Code-based nature means you can build on top of it to power anything.
It’s time to take back your content infrastructure.

several webpack polyfill errors with node:fs, jsdom and fast-csv

default discord avatar
djkatolast year

Hello, I'm trying to import a csv and convert it to collection data, but both "fast-csv" module and "node:fs" throw polyfill issues.


import { readFileSync } from "node:fs";
import { parse } from "fast-csv"

I'm trying to read a file, and then parse it.



Also, while doing lexicals headless html parsing to parse my old blog texts from html to lexical, webpack complains about it too.



ERROR in ./node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js 4:22-46
Module not found: Error: Can't resolve 'child_process' in '/home/node/app/node_modules/jsdom/lib/jsdom/living/xhr'

ERROR in ./node_modules/jsdom/lib/jsdom/living/websockets/WebSocket-impl.js 3:16-30
Module not found: Error: Can't resolve 'url' in '/home/node/app/node_modules/jsdom/lib/jsdom/living/websockets'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
    - add a fallback 'resolve.fallback: { "url": require.resolve("url/") }'
    - install 'url'
If you don't want to include a polyfill, you can use an empty module like this:
    resolve.fallback: { "url": false }


ERROR in ./node_modules/@fast-csv/parse/build/src/CsvParserStream.js 5:17-34
Module not found: Error: Can't resolve 'stream' in '/home/node/app/node_modules/@fast-csv/parse/build/src'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
    - add a fallback 'resolve.fallback: { "stream": require.resolve("stream-browserify") }'
    - install 'stream-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
    resolve.fallback: { "stream": false }


includes:

https://payloadcms.com/docs/rich-text/lexical#lexical-html
import { $generateNodesFromDOM } from "@lexical/html";
import { $getRoot, $getSelection, EditorConfig } from "lexical";
import { JSDOM } from "jsdom";
import { createHeadlessEditor } from "@lexical/headless"; // <= make sure this package is installed
import {
  getEnabledNodes,
  sanitizeEditorConfig,
  defaultEditorConfig,
} from "@payloadcms/richtext-lexical";
import { readFileSync } from "node:fs";
import { Payload } from "payload";
import { parse } from "fast-csv"

async function parse_csv(csv: string): Promise<CsvType> {
  return new Promise((resolve, reject) => {
    let cumulator = []
    const parser = parse({ headers: true })
    parser.on("data", (data) => { cumulator.push(data) })
    parser.write(csv)
    parser.end()
    parser.on("end", () => resolve(cumulator))
    parser.on("error", (reason) => reject(reason))
  })
}

export const migration = async (payload: Payload): Promise<void> => {
  const csv_text = readFileSync("./old-blogs.csv").toString();
  const csv_data = await parse_csv(csv_text)

  const headlessEditor = createHeadlessEditor({
    nodes: getEnabledNodes({
      editorConfig: sanitizeEditorConfig(defaultEditorConfig),
    }),
  });

  for (const row of csv_data) {
    headlessEditor.update(
      () => {
        const dom = new JSDOM(row.HTML);
        const nodes = $generateNodesFromDOM(headlessEditor, dom.window.document);
        $getRoot().select();
        const selection = $getSelection();
        selection.insertNodes(nodes);
      },
      { discrete: true }
    );
    const obsah = headlessEditor.getEditorState().toJSON();
    payload.create({
      collection: "clanok",
      data: {
        autor: {
          relationTo: "autor",
          value: editor.id.toString()
        },
        nadpis: row.title,
        createdAt: row.created_at,
        updatedAt: row.updated_at,
        obsah: { root: obsah.root }
      },
    });
  }

};


on wait is everything inside the paylad.config considered browser side? Do I have to somehow do this part outside the app, in the express part and then pass only the data down to payload?



Yup, I moved that part to the server.ts and it works.

    Star on GitHub

    Star

    Chat on Discord

    Discord

    online

    Can't find what you're looking for?

    Get dedicated engineering support directly from the Payload team.