Moving annotations between CoralNet and CPCe

Coral Point Count (CPCe) has been one of the most popular software programs for point-count surveys over the years. CoralNet provides ways to import points/annotations from CPCe to CoralNet, as well as export points/annotations from CoralNet to CPCe.

Basics

When you work with an image in CPCe, the software saves point positions and annotations to a .cpc file format - one .cpc file per image. These .cpc files are the files that you'll be uploading to CoralNet, or exporting from CoralNet.

Note that Windows Explorer may not show file extensions such as .cpc by default, so an IMG_3402.cpc file may show up as IMG_3402 for example. You can also look at the "Type" column to determine that it's a .cpc file:

Whether you're importing or exporting, the point/annotation data doesn't get mixed and matched. The destination ends up with the exact same points/annotations as the source, with any other points/annotations getting overwritten.

For example, suppose you have an image called 0001.JPG which has 20 points in CPCe, and 10 points in CoralNet. If you import from CPCe to CoralNet, 0001.JPG doesn't get a total of 30 points. Instead, the 10 existing points in CoralNet get overwritten, and 0001.JPG just ends up with the 20 points from CPCe.

In cases like this, the upload preview should give a warning about data getting overwritten, such as the following:

Setting up the labelset

For the porting process to work, all of the label (category) short codes used must be the same between CPCe and CoralNet.

When annotating points in CPCe, each point gets a label short code saved to it.

These short codes must match the short codes used in your CoralNet source. You can go to your CoralNet source's Labelset page to check the short codes.

As a reminder, every label on CoralNet is 'global', available for all sources to use. Each label on CoralNet has a default short code. However, once you add a label to your source's labelset, you don't have to use the default short code. Each source can specify custom short codes for each label.

To specify/change your source's custom short codes, click "Edit (customize) label codes" and use the form there. After changing the short codes, don't forget to click "Save Changes".

  • To import points/annotations from CPCe to CoralNet, every short code used in the CPCe annotations must be present in your CoralNet source's labelset.

  • To export from CoralNet to CPCe, every short code used in the CoralNet annotations must be present in your CPCe environment. (This is determined by a "code file" in CPCe; consult CPCe's help document for more detail.)

Importing points/annotations from CPCe to CoralNet

First you must upload the image files to CoralNet, if you haven't already. Go to your source on CoralNet and click "Upload" at the top. You must have Edit or Admin permissions in your source to see this button.

Click "Upload Images".

Use the form to upload images. Click the "?" button for more details about image uploads.

The "Name prefix" field lets you add a prefix to the names of the images being uploaded. For example, uploading IMG_0001.JPG with a name prefix of 20m_ gives a CoralNet image name of 20m_IMG_0001.JPG. Be careful, though - in most cases, the .cpc import step won't work if you add a name prefix, because the CoralNet image name must match the image filename recorded in the corresponding .cpc file. There is one notable exception - see the "Folder tree structures in image names (advanced usage)" section below.

Next, you must upload the .cpc files. Click "Upload" at the top, then this time click "Upload Points/Annotations (CPCe files)".

Use the form field to choose one or more .cpc files. Click the "?" button for more details.

Note that, like images, you can only choose .cpc files within a single folder. If your .cpc files span multiple folders, you'll need to do one upload per folder.

Once you've selected .cpc files, CoralNet will examine the .cpc files and their contents to see if the data can be uploaded to the source. If there are no errors, you'll see a preview showing the data detected by CoralNet. As mentioned previously, there may be a warning about overwriting existing data for this image. If it all looks fine to you, proceed and save the points/annotations.

If something is wrong, it should show an error. For example, here's what it shows if the .cpc files include a label short code that's not in the source's labelset:

Exporting points/annotations from CoralNet to CPCe

Go to your source on CoralNet and click "Images" at the top.

If you want to export .cpc data for a subset of images in your source, rather than all images, use the search fields to specify the subset of images you want, then click Search.

Scroll down to the "Image Actions" box. You must be signed in to see this box, even if the source is public.

Select "Export" in the first dropdown, and then "Annotations, CPCe" in the second dropdown. You must have Edit or Admin permissions in your source to see this option, because this export can reveal slightly sensitive information, namely file paths on the computer where CPCe is being used.

In the third dropdown, select whether you want to export .cpc data for all images in the search, or just for the images on this page of results (can be useful as a test run).

For more details on this .cpc export form, click the "?" button and read the "Annotations, CPCe" section of the help text.

When you are ready to export, click "Go" and wait for your .zip export file to be generated. This may take a long time, and you must stay on this page until the export file is generated. Once it has finished generating, your browser will start downloading the file (or will prompt you to choose a download location).

Note: CoralNet makes an effort to accept .cpc files from older CPCe versions such as CPCe 3.5. However, there are no guarantees that CoralNet's exported .cpc files can be used in older CPCe versions. The only fully compatible CPCe version is the latest one, CPCe 4.1.

Caveat on point positions

CoralNet stores point positions to pixel granularity. So for an image which is 4000 by 3000 pixels, there are 4000 x 3000 possible positions where CoralNet can place a point. When CoralNet decides a pixel for a point position, it places the point at the center of the pixel.

As a contrived example, here is what it looks like when we have a 6 x 6 pixel image, with one point at every single pixel. (Note: the annotation tool isn't designed to display extremely low-resolution images like this, so there is a lot of blurring here.)

CPCe, on the other hand, actually stores point positions to sub-pixel granularity. When points are generated in CPCe, the point positions typically have a granularity of 1/15th of a pixel. So a 4000 x 3000 pixel image would have about 4000 x 15 x 3000 x 15 possible point positions in CPCe. (In older versions of CPCe, the granularity can be something else like 1/12th depending on the computer's DPI setting. Note again, though, that CoralNet may not be fully compatible with older CPCe versions.)

Here's what it means in terms of point precision:

  • If point positions are generated in CPCe and imported into CoralNet, the positions are rounded to the nearest pixel for usage in CoralNet. This means that the point positions seen on CoralNet may be at most a half-pixel off from the positions seen in CPCe.

  • If point positions are generated in CoralNet and exported to CPCe, the positions are rounded to the 1/15th nearest to the center of the pixel. This means that the point positions seen in CPCe may be at most 1/30th of a pixel off from the positions seen in CoralNet.

  • If you import .cpc files into CoralNet, change some annotations, and then export .cpc files back to CPCe, the export will contain the original point positions with 1/15th granularity. The reason is that CoralNet retains previously uploaded .cpc files for each image, and uses these previous .cpc files as templates for future exports. These .cpc files are retained as long as the corresponding images do not get their points re-generated.

Note that it's best to use images with a fairly high pixel resolution, so that sub-pixel precision isn't much of a concern in the first place.

Side note: both CoralNet and CPCe have a chance of generating two or more points on the exact same position. Both CoralNet and CPCe are fine with this situation, but it may be visually confusing to you when you're trying to annotate points which are perfectly overlapping.

Folder structures in image names (advanced usage)

If you have a lot of images, there is a good chance that you've organized them into multiple folders on your computer, perhaps with multiple levels of folders.

CoralNet sources do not have folder structure, but generally there is no issue with going through your computer's folders one by one and uploading the images to CoralNet.

However, if you have images with the same filenames in different folders (such as Site 2/1m/0001.JPG and Site 2/5m/0001.JPG), CoralNet will detect name conflicts when you try to upload those images. In this case, you can use the "Name prefix" field on the Upload Images page to avoid name conflicts.

In general, you can use any name prefixes you want. But there's an additional consideration if you want to later import/export data between CoralNet and CPCe. Each .cpc file stores the full file path of its associated image file. You can see this if you open a .cpc file in a text editor program like Notepad:

When you upload a .cpc file to CoralNet, CoralNet reads this full file path from the .cpc file to determine which CoralNet image the points/annotations should be applied to.

The trick here is that CoralNet accepts image names with slashes or backslashes, such as Site 2/1m/0001.JPG. This fact can be used to achieve a longer match between the .cpc full file path and the CoralNet image name, thus disambiguating between images of the same filename.

So if you have images named Site 2/5m/0001.JPG and Site 3/5m/0001.JPG in your CoralNet source, and you upload a .cpc with the image file path C:\Surveys\Bocas Del Toro\Site 2\5m\0001.JPG, CoralNet will match that .cpc with the image Site 2/5m/0001.JPG. (Note that CoralNet considers slashes / and backslashes \ to be interchangeable.)

To achieve this naming scheme, you can use name prefixes such as Site 2/5m/ when uploading images:

And here is what it might look like when uploading .cpc files:

This scheme has another advantage when exporting back from CoralNet to CPCe. CoralNet recognizes slashes and backslashes as folder delimiters in the image names, and uses this information to export a .zip containing a folder tree of .cpc files:

It's then possible to extract the .zip at the root folder of your project (in the example above, C:\Surveys\Bocas Del Toro), thus updating all of the .cpc files at once. This can save a lot of time compared to manually copying the .cpc files one sub-folder at a time. However, since this will overwrite all the old .cpc files, you should make a backup of your old files first if you're not completely sure what you're doing.

Note: While a single export operation can take care of the entire folder tree, the uploads of the images and .cpc files still must be done one folder at a time.