User CSV Import for a Collection

default discord avatar
jj22whamlast year
7 1

What is the best way to go about adding a CSV import to a Collection for Users?

  • Selected Answer
    discord user avatar
    DanRibbens
    last year

    I can think of two good ways to get this done. I typically write an import script that starts up Payload with local: true, reads the CSV file and loops over the rows, mapping data to a data object and calls save on your collection(s). The other option is more work, but you can create a custom endpoint that handles the import from an uploaded file along with adding a custom component in Payload to the UI for users to submit data to.

    Here is a limited example of what I've used for an import script:

    import dotenv from 'dotenv';
    import path from 'path';
    import csv from 'csvtojson';
    import payload from 'payload';
    import { User } from '../payload-types';
    import { UserSource } from './types'; // I like to make types as all strings of my CSV data so that I know I have the field names correct, especially from legacy systems with poor naming conventions.
    
    dotenv.config();
    payload.init({
      secret: process.env.PAYLOAD_SECRET,
      mongoURL: process.env.MONGO_URL,
      local: true,
    });
    
    (async () => {
      const usersSource: { [key: string]: UserSource } = {};
      const csvPath = path.resolve(__dirname, 'source/users.csv');
      csv()
        .fromFile(csvPath)
        .then(async (fromCSV) => {
          fromCSV.forEach((source: UserSource) => {
            usersSource[source.id] = source;
          });
          // eslint-disable-next-line no-restricted-syntax
          for (const key of Object.keys(usersSource)) {
            const source = usersSource[key];
    
            const data: Omit<User, 'id' | 'createdAt' | 'updatedAt'> = {
              id: source.ID,
              email: source.EMAIL,
              name: source.NAME,
              // other fields you might have
            };
            try {
              // eslint-disable-next-line no-await-in-loop
              await payload.create({
                collection: 'users',
                overrideAccess: true,
                data,
              });
              console.log('created: ', data.email);
            } catch (e) {
              console.log(e);
              // eslint-disable-next-line no-continue
              console.log('skip: ', data.name);
              // eslint-disable-next-line no-continue
              continue;
            }
            console.log('DONE');
          }
        })
        .then(() => process.exit(0));
    })();

    This is a ts file, which you can run with npx ts-node -T imports/users.ts modifying the path for the directory you want.

    These tend to get longer and more complex if you need to also save relationships, but it all be handled this way as well.

    For users specfically, if you're using the local authentication (default) auth, you'll want to have a workflow for users to reset their passwords with a token for first time on the new system.

    Hopefully this gives you what you need. If you need any help we do offer enterprise support agreements that cover this kind of work.

    1 reply
  • default discord avatar
    opalepatrick10 months ago

    I have adapted this to upload a collection with related collections, but I fall at the first fence in that I cannot initialise the collections for some reason. There is a lot of rubbish in it, but this is it. If anyone can help this newbie it would be appreciated.

    `
    import dotenv from 'dotenv';
    dotenv.config();
    import path from 'path';
    import csv from 'csvtojson';
    import payload from 'payload';

    import Organisations from '../collections/Organisations';
    import SocialMediaDetails from '../collections/SocialMediaDetails';
    import PhoneNumbers from '../collections/PhoneNumbers';
    import AdditionalEmails from '../collections/AdditionalEmails';
    import Tags from '../collections/Tags';
    import Urls from '../collections/Urls';

    import { Organisation } from '../scripts/types';

    payload.init({
    secret: process.env.PAYLOAD_SECRET,
    mongoURL: process.env.MONGODB_URI,
    local: true,
    onInit: async () => {
    console.log("Payload initialized. Collections:", payload.collections);

    const csvPath = path.resolve(__dirname, '../resources/networks_import.csv');
    const csvData = await csv().fromFile(csvPath);
    
    for (const source of csvData) {
      let socialMediaDetailId: string | null = null;
      let phoneNumberId: string | null = null;
      let orgEmailId: string | null = null;
      let tagId: string | null = null;
      let addressId: string | null = null;
    
      // Create Social Media Detail
      if (source.SOCIALMEDIADETAILS && source.URL) {
        const socialMediaResult = await payload.create({
          collection: 'social-media-details', // Corrected line
          overrideAccess: true,
          data: {
            socialMediaPlatformName: source.SOCIALMEDIADETAILS,
            url: source.URL,
          },
        });
        socialMediaDetailId = socialMediaResult.id;
      }
    
      // Create Phone Number
      if (source.PHONENUMBER) {
        const phoneResult = await payload.create({
          collection: 'phone-numbers',
          overrideAccess: true,
          data: {
            phoneNumber: source.PHONENUMBER,
          },
        });
        phoneNumberId = phoneResult.id;
      }
    
      // Create Organisation Emails
      if (source.ORG_EMAIL) {
        const orgEmailResult = await payload.create({
          collection: 'additional-emails',
          overrideAccess: true,
          data: {
            additionalEmailAddresses: source.ORG_EMAIL,
          },
        });
        orgEmailId = orgEmailResult.id;
      }
    
      // Create Tags
      if (source.TAGS) {
        const tagResult = await payload.create({
          collection: 'tags',
          overrideAccess: true,
          data: {
            name: source.TAG,
          },
        });
        tagId = tagResult.id;
      }
    
      // Create Address
      if (source.ADDRESS_LINE1 && source.TOWN && source.COUNTRY && source.POST_CODE) {
        const addressResult = await payload.create({
          collection: 'addresses',
          overrideAccess: true,
          data: {
            addressLine1: source.ADDRESS_LINE1,
            addressLine2: source.ADDRESS_LINE2 || null,
            town: source.TOWN,
            province: source.PROVINCE || null,
            country: source.COUNTRY,
            postCode: source.POST_CODE,
          },
        });
        addressId = addressResult.id;
      }
    
      // Create Organisation with linked data
      const data: Partial<Organisation> = {
        title: source.NAME!,
        PhoneNumbers: phoneNumberId ? [phoneNumberId] : [],
        Tags: tagId ? [tagId] : [],
        SocialMediaDetails: socialMediaDetailId ? [socialMediaDetailId] : [],
        Addresses: addressId ? [addressId] : [],
        AdditionalEmails: orgEmailId ? [orgEmailId] : [],
        // ... other fields
      };
    
      try {
        await payload.create({
          collection: 'organisations',
          overrideAccess: true,
          data: data as any,
        });
        console.log('created: ', data.title);
      } catch (e) {
        console.error(`Error creating organisation ${data.title}:`);
        console.error('Error Details:', e);
        console.error('Stack Trace:', e.stack);
        console.log('Skipped: ', data.title);
      }
    }
    
    process.exit(0);
    

    }
    });
    `

    Sorry formatting not working properly.

    I get this in the terminal: "Payload initialized. Collections: {}"

Star on GitHub

Star

Chat on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.