User CSV Import for a Collection

jj22wham2 years ago

7 1

• Selected Answer
  

  DanRibbens

  2 years ago

  I can think of two good ways to get this done. I typically write an import script that starts up Payload with local: true, reads the CSV file and loops over the rows, mapping data to a data object and calls save on your collection(s). The other option is more work, but you can create a custom endpoint that handles the import from an uploaded file along with adding a custom component in Payload to the UI for users to submit data to.

  Here is a limited example of what I've used for an import script:

  import dotenv from 'dotenv';
  import path from 'path';
  import csv from 'csvtojson';
  import payload from 'payload';
  import { User } from '../payload-types';
  import { UserSource } from './types'; // I like to make types as all strings of my CSV data so that I know I have the field names correct, especially from legacy systems with poor naming conventions.
  
  dotenv.config();
  payload.init({
    secret: process.env.PAYLOAD_SECRET,
    mongoURL: process.env.MONGO_URL,
    local: true,
  });
  
  (async () => {
    const usersSource: { [key: string]: UserSource } = {};
    const csvPath = path.resolve(__dirname, 'source/users.csv');
    csv()
      .fromFile(csvPath)
      .then(async (fromCSV) => {
        fromCSV.forEach((source: UserSource) => {
          usersSource[source.id] = source;
        });
        // eslint-disable-next-line no-restricted-syntax
        for (const key of Object.keys(usersSource)) {
          const source = usersSource[key];
  
          const data: Omit<User, 'id' | 'createdAt' | 'updatedAt'> = {
            id: source.ID,
            email: source.EMAIL,
            name: source.NAME,
            // other fields you might have
          };
          try {
            // eslint-disable-next-line no-await-in-loop
            await payload.create({
              collection: 'users',
              overrideAccess: true,
              data,
            });
            console.log('created: ', data.email);
          } catch (e) {
            console.log(e);
            // eslint-disable-next-line no-continue
            console.log('skip: ', data.name);
            // eslint-disable-next-line no-continue
            continue;
          }
          console.log('DONE');
        }
      })
      .then(() => process.exit(0));
  })();

  This is a ts file, which you can run with npx ts-node -T imports/users.ts modifying the path for the directory you want.

  These tend to get longer and more complex if you need to also save relationships, but it all be handled this way as well.

  For users specfically, if you're using the local authentication (default) auth, you'll want to have a workflow for users to reset their passwords with a token for first time on the new system.

  Hopefully this gives you what you need. If you need any help we do offer enterprise support agreements that cover this kind of work.

  1 reply
• 

  opalepatricklast year

  I have adapted this to upload a collection with related collections, but I fall at the first fence in that I cannot initialise the collections for some reason. There is a lot of rubbish in it, but this is it. If anyone can help this newbie it would be appreciated.

  `
  import dotenv from 'dotenv';
  dotenv.config();
  import path from 'path';
  import csv from 'csvtojson';
  import payload from 'payload';

  import Organisations from '../collections/Organisations';
  import SocialMediaDetails from '../collections/SocialMediaDetails';
  import PhoneNumbers from '../collections/PhoneNumbers';
  import AdditionalEmails from '../collections/AdditionalEmails';
  import Tags from '../collections/Tags';
  import Urls from '../collections/Urls';

  import { Organisation } from '../scripts/types';

  payload.init({
  secret: process.env.PAYLOAD_SECRET,
  mongoURL: process.env.MONGODB_URI,
  local: true,
  onInit: async () => {
  console.log("Payload initialized. Collections:", payload.collections);

  const csvPath = path.resolve(__dirname, '../resources/networks_import.csv');
  const csvData = await csv().fromFile(csvPath);
  
  for (const source of csvData) {
    let socialMediaDetailId: string | null = null;
    let phoneNumberId: string | null = null;
    let orgEmailId: string | null = null;
    let tagId: string | null = null;
    let addressId: string | null = null;
  
    // Create Social Media Detail
    if (source.SOCIALMEDIADETAILS && source.URL) {
      const socialMediaResult = await payload.create({
        collection: 'social-media-details', // Corrected line
        overrideAccess: true,
        data: {
          socialMediaPlatformName: source.SOCIALMEDIADETAILS,
          url: source.URL,
        },
      });
      socialMediaDetailId = socialMediaResult.id;
    }
  
    // Create Phone Number
    if (source.PHONENUMBER) {
      const phoneResult = await payload.create({
        collection: 'phone-numbers',
        overrideAccess: true,
        data: {
          phoneNumber: source.PHONENUMBER,
        },
      });
      phoneNumberId = phoneResult.id;
    }
  
    // Create Organisation Emails
    if (source.ORG_EMAIL) {
      const orgEmailResult = await payload.create({
        collection: 'additional-emails',
        overrideAccess: true,
        data: {
          additionalEmailAddresses: source.ORG_EMAIL,
        },
      });
      orgEmailId = orgEmailResult.id;
    }
  
    // Create Tags
    if (source.TAGS) {
      const tagResult = await payload.create({
        collection: 'tags',
        overrideAccess: true,
        data: {
          name: source.TAG,
        },
      });
      tagId = tagResult.id;
    }
  
    // Create Address
    if (source.ADDRESS_LINE1 && source.TOWN && source.COUNTRY && source.POST_CODE) {
      const addressResult = await payload.create({
        collection: 'addresses',
        overrideAccess: true,
        data: {
          addressLine1: source.ADDRESS_LINE1,
          addressLine2: source.ADDRESS_LINE2 || null,
          town: source.TOWN,
          province: source.PROVINCE || null,
          country: source.COUNTRY,
          postCode: source.POST_CODE,
        },
      });
      addressId = addressResult.id;
    }
  
    // Create Organisation with linked data
    const data: Partial<Organisation> = {
      title: source.NAME!,
      PhoneNumbers: phoneNumberId ? [phoneNumberId] : [],
      Tags: tagId ? [tagId] : [],
      SocialMediaDetails: socialMediaDetailId ? [socialMediaDetailId] : [],
      Addresses: addressId ? [addressId] : [],
      AdditionalEmails: orgEmailId ? [orgEmailId] : [],
      // ... other fields
    };
  
    try {
      await payload.create({
        collection: 'organisations',
        overrideAccess: true,
        data: data as any,
      });
      console.log('created: ', data.title);
    } catch (e) {
      console.error(`Error creating organisation ${data.title}:`);
      console.error('Error Details:', e);
      console.error('Stack Trace:', e.stack);
      console.log('Skipped: ', data.title);
    }
  }
  
  process.exit(0);
  

  }
  });
  `

  Sorry formatting not working properly.

  I get this in the terminal: "Payload initialized. Collections: {}"