Importing / migrating data?

burgerlove25803 years ago

3

• 

  zoul08133 years ago

  I'm currently doing a similar WordPress to Payload migration, and what I've done is create some WP-CLI commands in WordPress to export the Posts to JSON.

  
  
  

  I then wrote a script for Payload that uses the Local API, reads all of these JSON files, performs any necessary transformations, and then calls create/update as needed.

  
  
  

  I created a

  seed.ts

  file next to my

  server.ts

  , at the core of this is the

  start

  function, which initializes Payload ...

  
  
  

  const start = async () =>
    await Payload.init({
      secret: process.env.PAYLOAD_SECRET,
      mongoURL: process.env.MONGODB_URI,
      mongoOptions: {
        dbName: process.env.MONGODB_DB_NAME ?? 'payload-konghq',
      },
      local: true,
      onInit: (_payload) => {
        payload = _payload;
        payload.logger.info(`Payload initialized...`);
        seed().then(
          () => {
            process.exit(0);
          },
          (err) => {
            console.error(err);
            process.exit(-1);
          },
        );
      },
    });
  start();

  
  
  

  The

  seed

  function just acts as a wrapper around various other functions, which are purpose built for seeding data in the correct order ... for example, I create/update my Payload Users first, then I create my uploads, then I create my globals, and then I create my Posts ... this ensures that related data is in place before I try to associate things.

  
  
  

  Ignoring my transformations, and some other bits ... the main

  seedCollection

  function looks like this (it uses

  createDocument

  ,

  updateDocument

  and

  setDocumentParent

  )

  
  
  

  async function createDocument(
    collection: string,
    data: any,
  ): Promise<TypeWithID> {
    return payload
      .create({
        collection: collection,
        disableVerificationEmail: true,
        data: {
          ...data,
          parent: null,
        },
      })
      .then((doc) => {
        if (data.parent) {
          return setDocumentParent(collection, data.id, data.parent);
        }
        return doc;
      });
  }
  
  async function updateDocument(
    collection: string,
    id: string,
    data: any,
  ): Promise<TypeWithID> {
    return payload
      .update({
        collection: collection,
        data: data,
        id: id,
      })
      .then((doc) => {
        if (data.parent) {
          return setDocumentParent(collection, data.id, data.parent);
        }
        return doc;
      });
  }
  
  // nested docs parent
  async function setDocumentParent(
    collection,
    childId,
    parentId,
  ): Promise<TypeWithID> {
    return payload.update({
      collection: collection,
      id: childId,
      data: {
        id: childId,
        parent: parentId,
      },
    });
  }
  
  async function seedCollection(collection: string) {
    const dirPath = path.resolve(__dirname, '../seeds', collection);
    let filenames = [];
  
    try {
      filenames = fs.readdirSync(dirPath);
    } catch (err) {
      console.warn('[WARNING]', err);
    }
  
    for (let filename of filenames) {
      const filepath = path.resolve(dirPath, filename);
      if (fs.lstatSync(filepath).isDirectory()) continue;
  
      const json = fs.readFileSync(filepath);
      const data = JSON.parse(json.toString());
      try {
        const doc = await payload.findByID({
          collection: collection,
          id: data.id,
        });
        const updated = await updateDocument(collection, doc.id, data);
      } catch (err) {
        console.log('[WARNING]: ', err);
        const created = await createDocument(collection, data);
      }
    }
  }

  
  
  

  It's important to note that to keep things in sync across multiple calls, I've overridden the collection's ID field and use a custom ID. This ensure if I change data in WordPress, run the export commands, then run the import commands, the import will update the appropriate content ... rather than deleting/recreating ...

• 

  altanbln3 years ago

  I'm using the WordPress API (wpapi package) to query posts and import them via payloads rest API in a simple node script.

  
  
  

  Also importing media files like that and check against the slug to avoid duplicate imports. It's def not super performant, but works well enough for my usecase

• 

  joephuz.3 years ago

  Working something similar porting from keystonejs, I agree with

  @190191205907169280

  , I’m querying my current db and just hitting the payload rest api. It’s worked out well… the trickiest part is media files, I just wrote some notes on my post about if you would like my experience

  https://discord.com/channels/967097582721572934/1070191210834182194