This article explains how Metapack Intelligence (MI) Feed, a new add-on product for the Advanced tier, provides shippers with a data feed in CSV files made specifically for your bespoke reporting needs. With MI Feed, you can integrate modeled shipping, despatch, tracking, and delivery data to empower your in-house Business Intelligence tools.
-
Types of Feeds: The 5 separate feeds include pre-shipping, post-shipping, tracking of inflight shipments, warehouse activity (Pre-ship Fraud), and a formatted replica of the Custom Report from the Delivery Manager. Included are time stamps for the raw events as well as modeled data to reflect delivery performance, pick-up times, exception states, and more. For more information, see the Overview of Data Feed Tables.
-
Hosting: Metapack will host these feeds, and each feed will be a snapshot of the underlying data generated that hour or that day. Depending on the feed, the CSV file will contain 30-day, 90-day, or 13-month spans. The Overview of Data Feed Tables gives time details for each feed. Subsequent feeds will overlap, but there may be changes to the underlying records.
-
Availability: Customer feeds will be available in MI Feed for 7 days before they are purged from the storage location.
-
Credentials and Storage: Metapack will manage the storage locations within AWS S3 and will provide credentials to the subscriber(s).
Connection Requirements:
Metapack customers will qualify to use Metapack Intelligence Feed if they have or meet the following requirements:
-
Already shipping on Delivery Manager
Eligibility Restrictions on Metapack Intelligence Feed
Merchants are ineligible to use Metapack Intelligence Feed if they are:
-
A third-party shipper via TAPI, or
-
A Metapack Parcel Manager (MPM) shipper.
This table gives an overview of the name, purpose, and update intervals for each of the data feeds available with MI Feed. The 4 initial files are Parquet formatted files, as that better suits larger-sized files. The DM Custom report will be in CSV format for migration purposes.
|
Name |
Purpose |
Update Interval |
Contains |
|---|---|---|---|
|
Pre-ship feed |
Shows parcel-level data for shipments in the warehouse from creation, allocation, printing, and manifesting. Does not include personal information such as Contact details and addresses, or parcel contents. Designed to merge with Warehouse Management System and Order Management System data to see throughput in the warehouse operation. |
Every 15 mins |
Parquet formatted file. Shipments created in the past 30 days |
|
Post-ship feed |
Shows manifested and despatched parcels from the shipper at parcel-level detail. Shows order number, tracking number, as well as planned and actual despatch dates. Designed to show what was despatched over time. Allows access to trend data and information in the event of a Customs regulation audit. |
Once Daily |
Parquet formatted file. Shipments created in the past 13 months |
|
Inflight & Delivery Feed |
For shipments currently in-flight or delivered, this shows parcel-level data that includes the parcel identifier, originating order number, details about exceptions (if applicable), delivery date, and more. Designed to work with external systems to generate bespoke performance reporting and customer-exceptions systems. |
Hourly |
Parquet formatted file. Shipments created in the past 30 days |
|
Warehouse Audit |
Shows which users are creating and modifying parcel details such as address, content, value, etc., in the warehouse before despatch. Designed for profit protection teams to review access to the shipping platform and where in the warehouse changes are being introduced to shipments. |
Hourly |
Parquet formatted file. Shipments created in the past 30 days |
|
DM Custom Report |
Replicates the format of the existing, daily Custom Report file structure in Delivery Manager (DM) with all the fields selected. Designed for retailers who use the DM Custom Reports. The MI Feeds will also generate those reports from the operational database, and they will replicate the DM Custom Reports capability. |
Once Daily |
CSV formatted file (GZ zipped in S3) Shipments created in the past 90 days |
The following tables each represent the structure of the Parquet file or CSV file that will be placed into S3. The column names here are the field headers of the CSV.
Click any of the following FAQs to learn more about MI Feed.
No. At launch, the MI Feed set will not allow for the extraction of all data held in DM. Noticeable missing fields will include:
-
Personal information fields such as Addresses, Contact names, emails, and telephone numbers.
-
Parcel contents information (i.e. Products, quantities, and compositions)
No. At launch, MI Feed will not have parity with the Daily Manager (DM) custom reports as it will have data missing. Significant fields include:
-
Address line 1: The field will show an empty value. We will add the value for Address line1 in Q3 of 2024.
-
Parcel contents information (i.e. Products, quantities, and compositions): Available in the DM Custom Reports feed.
Yes. Information about carrier tracking data quality will not appear in the feed and is only available in the MI Advanced tier. If you require this data, you can obtain it via the PTS feed (a separate post-ship product). The following 3 reports are only available as visualisations in MI.
As described in the Data Feed Tables Overview, the file format will be Parquet. The only exception to this format is the DM Custom report, which will be in CSV format (GZ zipped in S3). To save transmission bandwidth, it will be a compressed CSV file, encoded as UTF-8.
Each feed will contain a complete set of all translations and their respective statuses. Depending on the type of feed, it will be generated hourly or daily.
The data feeds will be posted to a Metapack-managed S3 location.
Each shipper will get unique, secure credentials to be created via Amazon’s IAM console and sent via email. To reset your password or unlock your account, you will need to contact Metapack Support through the Customer Support Portal.
No, not currently. Metapack must manage storage and access to the MI Feed data on your behalf. This solution avoids many technical, connection-based issues, and it allows us to offer additional services for you like data-storage retention and clean-up options.
Right now we have rolling storage for a week's worth of extracts with each feed.
With this in mind, consider these file estimates on the feed we produce for a shipper shipping 2.5 million parcels a month may be about XGB per day:
-
Inflight & Delivery Feed: XGB per feed x 24 (hourly) XGB
-
Pre-ship feed: X MB per feed x 24 (hourly) X GB
-
Live user report: X per feed x 24 (hourly) X GB
-
Post-ship feed: X per feed x 1 (daily)
-
DM Custom Report: X per feed x 1 (daily)
On average, retailers we currently produce feeds for see something like a Total Storage of 3400 GB, before compression. The following is an average for uncompressed sizes of extracts, and these will be further compressed before we launch.
-
Inflight & Delivery Feed - 785 GB
-
Pre-ship feed - 1431 GB
-
Post-ship feed - 730 GB
-
Warehouse Audit - 454 GB
We anticipate that customers will collect from S3 at least daily. The feed files will be retained in S3 for a rolling 7 days post generation and that
Any files older than 7 days old will be purged from storage and there is no capability offered to re-generate historical feed files.
-
The MI Feed product is not a direct replacement for DM Custom Reports, as Product data currently does not appear in the feed files.
-
These feeds do not directly replace the MI Dashboards. Some Data Quality reports are not included in the feeds because that data originates from PTS Feeds.
Feed Files Include the Full Data Set
The CSV files in the feed are not incrementally updated. Instead, they include the full data set each time. This makes for bigger files, but they include the state of all parcels in the last month or year.
Information in the full data set is helpful because parcels can sit for a long time before they are despatched or updated. This can help you to better handle pre-orders and delayed tracking.