Migration of CSV Data into Paragraphs

Posted by Charlotte León on December 9, 2016
Development
Migration & Upgrades

With the daily work of the office it is natural that there are challenges and potential research topics. With the addition of our developer Lucas as a Drupal core Migrate system maintainer, we decided to delve into the little documented area of migrating data into paragraphs.

Paragraphs is the new way of creating content. It allows site builders to make things cleaner so they can give more editing power to their end users.

In this blog I will explain in a simple way how to migrate data from CSV into paragraphs using the scenario of a Landing Page content type that has a banner field (single value paragraph) and a tiles field (multi-valued paragraphs field). There is a difference in how you have to structure the migration into single or multi-valued paragraphs fields, so this covers both nicely.
Let's first review our two CSV files. The first line in the files is the header row. Followed by rows of the values of the CSV. Very typical stuff so far.

# files.csv
banner
banner.jpg
img1.jpg
img2.jpg
img3.jpg

 

# landing_pages.csv
id,title,banner_image,banner_link,banner_link_text,desc1,img1,desc2,img2,desc3,img3
1,Services,banner.jpg,http://www.mtech-llc.com,"Go to site",Desc1,img1.jpg,Desc2,img2.jpg,Desc3,img3.jpg

Next we should build our paragraph types. The banner type has two fields. This should seem very familiar if you have ever dealt with Paragraphs or building a Drupal site.

Paragraphs Type: Banner

Field Name Field Type
Call to Action Link
Image Image

Paragraphs Type: Tile

This tile paragraph type has two fields too and allows someone to print some text with an associated image field. The author of the landing page is going to build a well structured set of tiles for the main content region of the landing page. Of course, you could swap out these fields and make your paragraphs however you want. But this just gives you some very typical paragraph types and migration approaches.

Field Name Field Type
Description Text (formatted, long)
Image Image

Finally, we build our landing page node content type.

Content Type: Landing Page

Field Name Field Type
Banner Entity reference revisions (paragraph field)
Tile Entity reference revisions (multi-value paragraph field)

At this point, we are ready to migrate data. You've already downloaded and installed Entity Reference Revisions and Paragraphs, but you'll also need to install Migrate Plus, Migrate Source CSV and Migrate Tools. And for migrating into Paragraphs, you'll also need to install this patch.

Now we'll build a custom module using Drupal Console. I've named mine paragraph_migration and added dependencies on migrate_plus, migrate_source_csv & migrate_tools. For convenience of this example, inside of this module at paragraph_migration/assets/csv I've placed my CSV files and atparagraph_migration/assets/img I've put my source images. Lastly, I store my migration yaml at paragraph_migration/config/install.

First up, a simple migration of files.csv: For more info on files migrations, head to this earlier blog post: How to Migrate Images into Drupal 8 Using CSV Source

When we are done with the migration, the files will show up in  /admin/content/files.

dependencies:
  enforced:
    module:
      - paragraph_migration
id: files
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/files.csv
  header_row_count: 1
  keys:
    - name
  constants:
    source_base_path: modules/custom/paragraph_migration/assets/img
    destination_base_path: 'public:/'
process:
  filename: name
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/source_base_path
        - name
    -
      plugin: urlencode
  destination_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/destination_base_path
        - name
    -
      plugin: urlencode
  uri:
    plugin: file_copy
    source:
      - '@source_full_path'
      - '@destination_full_path'
destination:
  plugin: 'entity:file'
migration_dependencies:
  required: { }
  optional: { }
  • Migration of multi-valued paragraph fields from banner, tile_1, tile_2 and tile_3: We will use the destination plugin provided in the previously mentioned patch for Entity Reference Revisions. To use that plugin, enter 'entity_reference_revisions:paragraph' in the destination. **It is very important to use a singular paragraph, not paragraphs for the second part of the destination**
dependencies:
  enforced:
    module:
      - paragraph_migration
id: banner
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/landing_pages.csv
  header_row_count: 1
  keys:
    - id
process:
  field_image:
    plugin: migration
    migration: files
    source: banner_image
    no_stub: true
  field_call_to_action/title: banner_link_text
  field_call_to_action/uri: banner_link
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: banner
migration_dependencies:
  required:
    - files
  optional: { }
dependencies:
  enforced:
    module:
      - paragraph_migration
id: tile_1
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/landing_pages.csv
  header_row_count: 1
  keys:
    - id
    - img1
process:
  field_image:
    plugin: migration
    migration: files
    source: img1
    no_stub: true
  field_description: desc1
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: tile
migration_dependencies:
  required:
    - files
  optional: { }
dependencies:
  enforced:
    module:
      - paragraph_migration
id: tile_2
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/landing_pages.csv
  header_row_count: 1
  keys:
    - id
    - img2
process:
  field_image:
    plugin: migration
    migration: files
    source: img2
    no_stub: true
  field_description: desc2
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: tile
migration_dependencies:
  required:
    - files
  optional: { }
dependencies:
  enforced:
    module:
      - paragraph_migration
id: tile_3
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/landing_pages.csv
  header_row_count: 1
  keys:
    - id
    - img3
process:
  field_image:
    plugin: migration
    migration: files
    source: img3
    no_stub: true
  field_description: desc3
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: tile
migration_dependencies:
  required:
    - files
  optional: { }

For each tile YML migration file it is necessary to add the id and the name of the image as the id or key for the migration. This will get added to the mapping table and used by the follow-up node migration. These keys need to be unique.

 

  • Node migration (final destination of our content): Lastly, let's review the YML file that should pull everything together from all our previous migrations into Paragaphs. There is a different approach used for multi-valued paragraphs as compared to single valued paragraphs, so pay careful attention to the details.
dependencies:
  enforced:
    module:
      - paragraph_migration
id: landing_pages
source:
  plugin: csv
  path: modules/custom/paragraph_migration/assets/csv/landing_pages.csv
  header_row_count: 1
  keys:
    - id
process:
  title: title
  field_banner/target_id:
    -
      plugin: migration
      migration: banner
      no_stub: true
      source: id
    -
      plugin: extract
      index:
        - '0'
  field_banner/target_revision_id:
    -
      plugin: migration
      migration: banner
      no_stub: true
      source: id
    -
      plugin: extract
      index:
        - 1
  combination_1:
    plugin: get
    source:
      - id
      - img1
  combination_2:
    plugin: get
    source:
      - id
      - img2
  combination_3:
    plugin: get
    source:
      - id
      - img3
  combination:
    plugin: get
    source:
      - @combination_1
      - @combination_2
      - @combination_3
  field_tile:
    -
      plugin: migration
      migration:
        - tile_1
        - tile_2
        - tile_3
      no_stub: true
      source: @combination
    -
      plugin: iterator
      process:
        target_id: '0'
        target_revision_id: '1'
destination:
  plugin: 'entity:node'
  default_bundle: landing_page
migration_dependencies:
  required:
    - banner
    - tile_1
    - tile_2
    - tile_3
  optional: { }

**field_banner/target_id y field_banner/target_id : The extract value of '0' (with single quotes) is important and isn't a mistake. If you don't do it this way, the extract process plugin will treat zero as empty and skip extracting the value. A value of 1 (without quotes) is fine for the extract plugin. It doesn't skip a numeric value of one.

**combination: This is a temporary or psuedo field that is used to collect all the source values together into the right format for later inserting into the actual destination. You can tell it is a temporary field because it has a somewhat random name that doesn't begin with the 'field_' prefix and later on it is used with a prefix of at sign (@). We use the combination field as an intermediary for the final destination of 'field_tile'.

The second part of the multi-valued paragraph migration, the iterator only accepts string values. It doesn't like numerics. The single quotes around both the '0' and the '1' are not a mistake. This is different than the earlier mention for field_banner/target_id y field_banner/target_id.

With our fully-backed migration yamls and our created content types, it is now time to see how it works.

$ drush en paragraph_migration -y

Now, run the migration:

$ drush mi --all

Yeah!!! Now we are able to migration source data into Paragraphs.

You can obtain the whole code base for the examples used in this post on GitHub.

Are you looking for help with a Drupal migration or upgrade? Regardless of the site or data complexity, MTech can help you move from a proprietary CMS or upgrade to the latest version–Drupal 8.

Write us about your project, and we’ll get back to you within 48 hours.


Migration of CSV Data into Paragraphs