Running a large import (over 50,000 rows) can be stressful and time consuming. We've had many customers do imports from 50,000 to a few million. These imports can take a long time to run and because of the large number of orders, there are bound to be some data issues. Based on what we've seen in the wild, we've put together a few guide lines for large imports.

Why Break Import Into Multiple Files?

EZ Importer was built for both large and small imports. We could allow a CSV file of any size to be imported but we've found this causes more trouble for the uploader than not. So we've put a few restrictions on single imports. The maximum file size you can import is 20MB and the maximum number of rows per import is 50,000. You may be asking yourself, why these limitations? It just makes it hard to do my import.

At first glance this makes sense but we've seen enough of these large imports to recognize this is not the case. We recommend you import files larger than 20MB and/or with more than 50,000 rows (in your CSV file. This can be line items or orders) in multiple files because of the following:

1. Large files take a long time to import. The Shopify API typically allows us to import a single order every 2 seconds, however we've seen them respond in up to 7 seconds per order. This means your import will run for hours. This is a long time for a process to run uninterrupted and it means more can go wrong. We have precautions in place to help ensure the imports run successfully but the longer orders take to run the more likely it can fail. Breaking large imports into multiple files helps mitigate this risk
2. If any data issues arise during your import, it will be easier to manage them when you break your import into multiple files. With an order of this size, you are bound to run into data issues no matter how much you've tested. Our customers have found managing data errors with large imports is easier when broken into multiple files.
3. There is a chance you may need to cancel or delete your import. If you have an import that took hours to run, it will take hours to delete. This is extremely time consuming and breaking your import into multiple files will help you manage this.

Breaking Import Into Multiple Files

In order to break your large CSV file into multiple imports, you'll need to manually break up the file. Each file will need to have the same header row. You can do this by copying/pasting orders into multiple files by hand or if you're more technical you can write a small program to do this. How you break the import into multiple files is up to you.

Remember you must put all line items for your order into the same file. If you put an orders line items in multiple files, EZ Importer will not be able to merge them together and it will duplicate the order.

It's extremely import that you run tests before doing your full import, this will save you a ton of time as it will prevent you from finding data errors later in the process.

We hope this helps explain why we recommend you import orders into multiple files. As always if you have any other questions about this process please don't hesitate to contact us!