Creating aerial imagery with a bike helmet camera (GoPro) and OpenDroneMap

See comments on Hacker News (24 comments, 220 points)

This technical guide details how you can create your own orthorectified (aka satellite view/bird mode) imagery, point clouds and 3D models of streets with nothing but a 360 degree camera mounted on bicycle helmet, and the open source photogrammetry software OpenDroneMap.

Why might you want to do this? With your own up-to-date and highly detailed orthorectified imagery you could:

  • quantify and communicate inefficient road space allocation
  • record necessary infrastructure repairs
  • take measurements such as lane and cycleway widths
  • measure footpath obstructions in 3D and rate pedestrian amenity
  • map kerb features on OpenStreetMap
  • survey street parking using the new OSM spec: wiki.openstreetmap.org/wiki/Street_parking
  • 3D print a model of your home street!

Subscribe to Jake’s blog

Or follow on Mastodon – @[email protected]

* indicates required


Continue reading below


Viewing orthoimagery generated with a GoPro in the OpenStreetMap ID editor
Making measurements in WebODM. Generating 2d elevation profiles is also possible.

Drones are great for surveying and mapping things on the kerb (parking/public space/road widths/building shadows) but there are a lot of places you can’t fly a drone.

OpenDroneMap is usually used to combine a set of geotagged drone images into one coherent 3d model, but it can also be used with any geotagged images.

Roughly the process is as follows:

  • Take spherical images on a 360 degree camera
  • Import the files from the camera
  • Optional: Convert the photos into rectangular, spherical images (for GoPro: convert from the proprietary GoPro format to standard spherical images using GoPro Fusion v1.2)
  • Start/login to WebODM, upload the images with fisheye lens setting for 180 degree images or spherical for rectangular 360 degree images and generate imagery & model

Please comment any projects you make after reading this guide, or if you have any questions! There are certainly issues with this process with I’ve added under Limitations.

Table of Contents

Buying a 360 degree camera

360 degree cameras can be very expensive (the ones mounted on Google Street View cars are possibly $45,000!), but GoPro makes relatively affordable models. See https://help.mapillary.com/hc/en-us/articles/115001465989-About-360-cameras for more suitable cameras. One common second hand (and now unsupported) model is the GoPro Fusion, which I’ll base this guide on.

Note: Seth Deegan has written a PowerShell script for using Insta360 cameras with ODM. I haven’t tried it but it seems worth a look: https://github.com/lectrician1/Insta360-2-ODM

To generate a detailed model you will need to capture enough images close together. You could mount it to the top of your car, but unless you bought an expensive elevated mount much of the image would just be your car roof!

Some advantages of taking imagery from a bicycle:

  • The unobstructed field of view is larger so ODM gets more data for a better model
  • The average speed is lower permitting more photos (the maximum self timer frequency of this camera is 2 shots/second)
  • You can capture images from different places in the street for more data, such as the road, footpath and bike lanes
  • If you’re putting in this much effort to study public space and urban planning you probably like bikes 🙂

You’ll need to buy a helmet mount for the GoPro or your chosen camera. GoPro makes a bicycle helmet mount: https://gopro.com/en/us/shop/mounts-accessories /vented-helmet-strap-mount/GVHS30.html (though many non-branded mounts exist on eBay/Amazon too). Be aware a camera mounted on a bicycle helmet is possibly a safety risk to yourself if you crash – be careful.

If you have the ability to mount the camera on a long pole above your head, this will increase the unobstructed field of view and improve the perspective of the street which will improve the results. I imagine this would attract even more attention! If you have tips for building rigs like this please leave a comment. Andrew Harvey wrote an OpenStreetMap diary entry on his setup here: https://www.openstreetmap.org/user/aharvey/diary/42139

GoPro Fusion specific tips

Set the camera to:

  • Timelapse photo mode (icon of camera & timer circle)
  • Image frequency to 0.5 seconds
  • Enable GPS geotagging

Mount the camera to the helmet, wait until the GPS location icon turns solid, and start capturing! Sometimes the first few shots don’t have any location in the EXIF data but this doesn’t seem to confuse WebODM, you can check this with identify -verbose image or your metadata viewer of choice.

Preprocessing the images

The GoPro Fusion will output two 180 degree “fisheye” images to two separate SD cards. Copy all these images to a folder onto your computer.

For an alternate method, see instructions under the generating equirectangular image heading.

Unfortunately, only the “front” images have geotagged information, so you’ll need to copy the exif data from each “back” image to each similarly named “front” image.

A quick and dirty way of doing this is generating a list of terminal commands, that copies all exif tags from each “front” to each “back” photo:

ls -l | grep "GF" | sed -E "s/^.* GF(.*).JPG*/exiftool −overwrite_original_in_place -gps:all -tagsFromFile GF\1.JPG GB\1.JPG/g" > script.sh

I’m sure there are better and easier ways of doing this, please let me know if you come up with one.

Generating the model and orthorectified imagery with Web OpenDroneMap

WebODM (https://opendronemap.org) is an absolute marvel of open source engineering. Unfortunately, generating 3D models takes some serious computing horsepower. You can either use the paid cloud version (WebODM Lightening) at https://webodm.net/ (fast) or run the software on your own computer (a few hours/overnight/days depending on the number of images).

You can directly upload images to WebODM Lightening to process, however you don’t get some friendly/useful features like a browsable map and 3d model viewer. I recommend setting up WebODM locally and adding WebODM Lightening as a “processing node”, so you get the power of the cloud and the extra features of WebODM.

Setting up WebODM locally

Running the software using Docker is a breeze. Install Docker from https://www.docker.com/ (or your preferred method), allocate as much memory & CPUs as you can, and then:

git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
cd WebODM
./webodm.sh start 

See https://github.com/OpenDroneMap/WebODM#getting-started more more details including GPU acceleration.

You’ll now be able to open WebODM on http://localhost:8000

Optional: Adding WebODM lightening as a processing node

Click lightening network in your WebODM sidebar and login.

Generating the model & selecting the correct options

Add a new project, click “Select images & GCP”, select your images, then you will see options for processing your imagery. If you’re using WebODM Lightening, make sure to set the Processing Node appropriately.

Some options I’ve found are required:

If you are using the raw 180 degree “fisheye” images straight out of the GoPro Fusion, set camera-lens to fisheye (this is critical). If you are using equirectangular images set to spherical. WebODM seems to be unable to auto-identify the lens type in both cases.

  • enable sky removal – this uses AI to mask out the sky in each frame so that there are less artifacts in the final model
  • enable auto-boundary
  • enable bg-removal

This is just from my trial and error, there may be better option configurations.

You will likely need to enable “resize images” to 2000px or it is very easy to run out of memory.

Though it is out of scope for this article, you can also set up a VPS instance to speed up the process if you don’t want to use the hosted cloud processing tool. Ive tried this, but it’s probably more effort than it’s worth unless you’re making a business out of this.

64 cores and 124GB of RAM!

Even with an EC2 instance with 124GB of RAM and 324 images at 5760 × 2180 pixels I ran out of memory – remember to enable some swap space if you want to run at full size: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04

Output accuracy: taking measurements

OpenDroneMap can make measurements of the 3D model. I’ve found these are usually quite accurate; they may have small “measurement” errors due to artifacts or distortion but will tend towards the correct value – GPS coordinates ensure the scale is correct.

For example, measuring the width of the green paint of the cycle path:

Large path:

  • OpenDroneMap: 4.54 metres
  • Tape measure: 4.56 metres

Small path:

  • OpenDroneMap: 1.89 metres (though this varies slightly due to distortions)
  • Tape measure: 1.89 metres

Alternative: Generating and processing equirectangular spherical images

I’m currently unclear whether this method is faster or produces better results than the raw 180 degree images. I’m very interested to hear if you have experience (and I’ll update this if I get more evidence). I believe the GroPro Max generates these images in camera, so this won’t be required.

If you use the GoPro Fusion, unfortunately because this camera is no longer supported GoPro discontinued the software: https://gopro.com/en/au/news/fusion-end-of-life

You’ll (unfortunately) need to use the proprietary GoPro Fusion software to generate spherical images. I’ve found on an M1 Mac (Monterey) the only version that still runs is 1.2, 1.4 just crashes. The least dodgy download I can find is https://macdownload.informer.com/fusion-studio/1.2/.

If you make an open source solution (maybe building from https://stackoverflow.com/questions/37796911/is-there-a-fisheye-or-dual-fisheye-to-equirectangular-filter-for-ffmpeg) please let me know! Edit: Looks like this repo will do it. https://github.com/trek-view/fusion2sphere

This camera has two SD card slots – one for the front facing camera, and one for the back facing camera. Sometimes only one camera works for part of the shoot and the proprietary software refuses to stitch any of the photos at all! The only way I’ve found to combat this is to format the card in camera before each shoot. Spending more on a GoPro Max may solve a lot of headaches!

Limitations

Little helmets appear! I’ve tried some techniques but haven’t found a proper solution that doesn’t degrade the model/orthophoto structure.

Appendix: Things that didn’t work

Removing helmet artifacts by cropping

I tried using the convert and mogrify tools (part of the amazing open source ImageMagick suite) can crop the spherical photos and retain the geographical information! The cropping works great, but OpenDroneMap doesn’t seem to be able to understand a cropped equirectangular image. If you’d like to try to get this working, here are the steps I took.

First install ImageMagick (sudo apt-get install imagemagick on Linux/brew install imagemagick on macOS).

The general syntax is:
convert -crop {x_size}x{y_size}+{x_offset}+{y_offset} inputfile outputfile

See https://deparkes.co.uk/2015/04/30/batch-crop-images-with-imagemagick/ for more detail and helpful diagrams.

Helpfully, the helmet artifact is always at the bottom of frame, so the x offset and y offset will be zero.

To crop 700 pixels off the bottom of a spherical image, you can use

convert -crop 5760x"$((2880-700))"+0+0 inputfile outputfile

To batch convert a number of files, where the input files are in ./input/, and you have made an output directory ./output/, you can use:

mogrify -monitor -crop 5760x"$((2880-700))"+0+0 -path ./output ./input/*

Removing helmet artifacts by adding a mask

Adding custom drawn masks, either covering just the helmet or also the surrounding black “void” either:

  • removed the helments but caused extra “holes”/missing segments in the ground
  • didn’t remove the helmets

I think this may be due to clashing behaviour of the bg-removal and sky-removal flags with manual masks.

ODM requires a mask image for every image with _mask.{EXT} as a suffix (where EXT replaces the original extension).

The two mask types I tried:

If you know what may be happening please let me know!

Further research/experimentation

  • How many “capture lines” per street are necessary to make a decent model? I didn’t have any luck with one but it may have been my settings. I used ~4 for the above model.
  • How much better are models when the camera is on a stick?
  • Improved WebODM settings for generation

Prior art

16 responses to “Creating aerial imagery with a bike helmet camera (GoPro) and OpenDroneMap”

  1. bikesaint Avatar
    bikesaint

    Wow!

  2. Don Avatar
    Don

    This is some amazing stuff. You certainly have put in a lot of work so far. It is all beyond my capabilities but I really appreciate your effort on this project. Good luck and Merry Christmas.

  3. zidane Avatar

    yeah thanks mate so good

  4. StephaneP Avatar
    StephaneP

    Hi!

    What a coincidence, I’m exactly trying the same process right now. Thank you for your guide, it will save me some time.

  5. Peter Elderson Avatar
    Peter Elderson

    I have tested a helmet mount setup for capturing imagery while walking. I had good imagery using the vented helmet strap and a flexible gooseneck extension. Very little helmet artifacts. https://www.openstreetmap.org/user/Peter%20Elderson/diary

    1. Jake C Avatar

      Thanks for that post!

      > I ended up using a bicycle helmet with a vented helmet strap, and on top of that a 20 cm gooseneck, then the camera on top of that

      I think I’ll try that and see if I get less artifacts.

  6. Tobias Avatar
    Tobias

    I was just reminded by https://twitter.com/lars_schwarz/status/1611011065776578560 that the Polycam App https://poly.cam/ can also create a 3D Model. That would only work for small areas, but might be feasible to get detailed data on Crossings etc. to map in OSM.
    One would need a process to transform the 3D model to a OpenAerialMap-“Aerial image”, though. And this will probably need manual steps to “orient” the image on the globe.

    1. Jake C Avatar

      That’s a good point – looks like a great use case for small crossings.

      I haven’t yet come across any tools for converting a 3D model to an orthophoto – are you aware of any? I know OpenDroneMap does it behind the scenes but doesn’t provide an API for it yet.

      I’ve wanted to generated oblique orthophotos for some time and there’s some talk of tinkering with the model to image code: https://github.com/OpenDroneMap/ODM/issues/1549

      This comment appears to show how OpenDroneMap generates the orthophoto from the model: https://community.opendronemap.org/t/creating-2-5d-oblique-orthophoto/13579/4

      [INFO] running "/code/SuperBuild/install/bin/odm_orthophoto" -inputFiles /var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_texturing/odm_textured_model_geo.obj -logFile "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/odm_orthophoto_log.txt" -outputFile "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/odm_orthophoto_render.tif" -resolution 20.0 -outputCornerFile "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/odm_orthophoto_corners.txt"
      [INFO] Creating GeoTIFF
      [INFO] running gdal_translate -a_ullr 427920.073538 4579970.179102 427951.115709 4579948.016147 -co TILED=YES -co COMPRESS=DEFLATE -co PREDICTOR=2 -co BIGTIFF=IF_SAFER -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co NUM_THREADS=64 -a_srs "+proj=utm +zone=17 +datum=WGS84 +units=m +no_defs +type=crs" --config GDAL_CACHEMAX 46.8% --config GDAL_TIFF_INTERNAL_MASK YES "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/odm_orthophoto_render.tif" "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/odm_orthophoto.tif" > "/var/www/data/526f1a12-91b7-43b8-af96-0f0231360dde/odm_orthophoto/gdal_translate_log.txt"

  7. Seth Deegan Avatar
    Seth Deegan

    I have created a tool to do this for Insta360 cameras: https://github.com/lectrician1/Insta360-2-ODM

    1. Jake C Avatar

      That’s brilliant! I’ll add a link to that repo into the article.

  8. Tobias Avatar
    Tobias

    FYI I wrote https://www.openstreetmap.org/user/tordans/diary/401804 to further talk about the possibilities to scale this pipeline.

  9. Matt Avatar
    Matt

    I’m about to generate a 3d map of an entire apartment complex but I need lots of ground level photos to go WITH my aerial photos. I was hoping something like this was doable with a 360 camera, but my question is, has anyone figured out a way to import / combine the images from a drone with the equirectangular images from the GoPro on import, or a way to add images after import to increase the quality of the model?

  10. aisun Avatar
    aisun

    Hi!
    Can you please give access to your 360 degree images?
    I am working on an university project and I need some images as you have used but I am unable to find any from the internet.

    1. Jake C Avatar

      I unfortunately don’t think I have these images any more – they weren’t the highest quality with the helmet taking up much of the image leading to artefacts. It might be possible to reproduce with multiple takes on a phone camera using the wide angle lens.

  11. Andrew Harvey Avatar
    Andrew Harvey

    I’ve used a process documented at https://github.com/andrewharvey/lg360-mapillary-helpers to inpaint the helmet area, which shouldn’t affect any of the image metadata since it’s just replacing pixels.

Leave a Reply

Your email address will not be published. Required fields are marked *