Talks

2025-11-14T00:00:00+00:00

A Rusty Geospatial Adventure (OmniOpenCon 2025)</a></li> </ul>

Recovering information from a PDF map

2023-02-23T00:00:00+00:00

This post is going to be about a question that recently came up in the GDAL Matrix room</a>. A user had a discrete heat map of noise levels in their town as a PDF, and wanted to extract some usable information out of it. Let's see if we can help them.

Looking at the data</h2>

If you want to play along, you can download the file from here</a>. Pick the first one</a>, with Lzsn</code> in its name. It's a 42 MB PDF, looking like this (click on the image for a better version):

</a>

If that seems unreadable, it's partly because my PDF viewer</a> can't zoom into large documents. A better choice is probably QGIS</a>, so you can install it.

</a>

This would probably make a real cartographer unhappy, but fortunately I'm not one. And yes, some labels are doubled, don't ask me why. It happens in Evince, Okular and QGIS, but I don't have a "real" PDF reader to test with.

If you're following along, you might have noticed that QGIS took about 20 minutes to open the file and probably just as long to close it. That's not great and puts our software in a bad light, so let's convert it to a better format.

Using GDAL</a>, you can convert it to a GeoTIFF like this:

$</span> gdal_translate drumuri_Lzsn.pdf drumuri_Lzsn_temp.tif
</span>$</span> gdal_translate</span> -f</span> COG</span> -co</span> NUM_THREADS=ALL_CPUS drumuri_Lzsn_temp.tif drumuri_Lzsn.tif
</span></code></pre>
A COG</a> is just a GeoTIFF laid out so it can load faster while zooming in.
If you're wondering why I'm doing the conversion in two steps, it just happens to be very slow otherwise, while the commands above should only take a couple of seconds.
COGs are tiled (stored in blocks), and going from a striped (stored by lines) to a tiled layout can be slow, but I don't think that's the case for our PDF.
Anyway, once you have a TIFF, QGIS should be quite snappy.</p>
Extracting the image</h2>
The user would like to have a nice vector dataset, and overlay it on top of OpenStreetMap</a>.
This means that we're not really interested in the roads, road labels and building footprints, which are already available there.</p>
I initially missed this, but the user pointed out that the roads in the PDF seem to be stored in a vector format.
Maybe there's a chance we can get rid of them?</p>
After some searching, I found the pdfimages</code> tool, part of Poppler.</p>
$</span> pdfimages</span> -list</span> drumuri_Lzsn.pdf
</span>page</span>   num  type   width height color comp bpc  enc interp  object ID x-ppi y-ppi size ratio
</span>--------------------------------------------------------------------------------------------
</span>   </span>1</span>     0 image   16377 12649  rgb     3   8  jpeg   no         7  0   200   200 39.7M 6.7%
</span></code></pre>
This sounds promising, as we have a large image which we can extract.
It's a JPEG, which might cause us some problems later, but we'll cross that bridge when we get there.</p>
$</span> pdfimages</span> -j</span> drumuri_Lzsn.pdf drumuri_Lzsn
</span>$</span> ls</span> -lh</span> drumuri_Lzsn*.jpg
</span>-rw-r--r--</span> 1 user group 40M Feb 21 19:04 drumuri-Lzsn.jpg
</span>$</span> gdal_translate</span> -f</span> COG</span> -co</span> NUM_THREADS=ALL_CPUS drumuri-Lzsn.jpg drumuri_Lzsn-data.tif
</span></code></pre>
Yes, GDAL can read JPEG, that's pretty nifty.
Let's see what we got:</p>
</a></p>
Yup, we still have the legend and other stuff on the right, and we can reasonably suspect that our map was scanned.
But is it at least cleaner?</p>
</a></p>
Success — it looks like we're on the right track!
We still have the building footprints, but the labels are gone.
We won't miss them.</p>
Georeferencing</h2>
Unfortunately, if we open both images in QGIS, they won't show up in the same place.
This is because the one we extracted is missing the georeferencing and projection information.
To put it differently, QGIS doesn't know where the image is located, its orientation and how large a pixel is on the ground.</p>
The original PDF had this information, and it was propagated to the TIFF we made from it:</p>
$</span> gdalinfo drumuri_Lzsn.tif
</span>Driver:</span> GTiff/GeoTIFF
</span>Files:</span> drumuri_Lzsn.tif
</span>Size</span> is 12992, 9448
</span>Coordinate</span> System is:
</span>PROJCRS[</span>"</span>WGS 84 / Pseudo-Mercator</span>"</span>,
</span>    </span>BASEGEOGCRS[</span>"</span>WGS 84</span>"</span>,
</span>        </span>ENSEMBLE[</span>"</span>World Geodetic System 1984 ensemble</span>"</span>,
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (Transit)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G730)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G873)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G1150)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G1674)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G1762)</span>"</span>],
</span>            </span>MEMBER[</span>"</span>World Geodetic System 1984 (G2139)</span>"</span>],
</span>            </span>ELLIPSOID[</span>"</span>WGS 84</span>"</span>,6378137,298.257223563,
</span>                </span>LENGTHUNIT[</span>"</span>metre</span>"</span>,1]],
</span>            </span>ENSEMBLEACCURACY[2.0]],
</span>        </span>PRIMEM[</span>"</span>Greenwich</span>"</span>,0,
</span>            </span>ANGLEUNIT[</span>"</span>degree</span>"</span>,0.0174532925199433]],
</span>        </span>ID[</span>"</span>EPSG</span>"</span>,4326]],
</span>    </span>CONVERSION[</span>"</span>Popular Visualisation Pseudo-Mercator</span>"</span>,
</span>        </span>METHOD[</span>"</span>Popular Visualisation Pseudo Mercator</span>"</span>,
</span>            </span>ID[</span>"</span>EPSG</span>"</span>,1024]],
</span>        </span>PARAMETER[</span>"</span>Latitude of natural origin</span>"</span>,0,
</span>            </span>ANGLEUNIT[</span>"</span>degree</span>"</span>,0.0174532925199433],
</span>            </span>ID[</span>"</span>EPSG</span>"</span>,8801]],
</span>        </span>PARAMETER[</span>"</span>Longitude of natural origin</span>"</span>,0,
</span>            </span>ANGLEUNIT[</span>"</span>degree</span>"</span>,0.0174532925199433],
</span>            </span>ID[</span>"</span>EPSG</span>"</span>,8802]],
</span>        </span>PARAMETER[</span>"</span>False easting</span>"</span>,0,
</span>            </span>LENGTHUNIT[</span>"</span>metre</span>"</span>,1],
</span>            </span>ID[</span>"</span>EPSG</span>"</span>,8806]],
</span>        </span>PARAMETER[</span>"</span>False northing</span>"</span>,0,
</span>            </span>LENGTHUNIT[</span>"</span>metre</span>"</span>,1],
</span>            </span>ID[</span>"</span>EPSG</span>"</span>,8807]]],
</span>    </span>CS[Cartesian,2],
</span>        </span>AXIS[</span>"</span>easting (X)</span>"</span>,east,
</span>            </span>ORDER[1],
</span>            </span>LENGTHUNIT[</span>"</span>metre</span>"</span>,1]],
</span>        </span>AXIS[</span>"</span>northing (Y)</span>"</span>,north,
</span>            </span>ORDER[2],
</span>            </span>LENGTHUNIT[</span>"</span>metre</span>"</span>,1]],
</span>    </span>USAGE[
</span>        </span>SCOPE[</span>"</span>Web mapping and visualisation.</span>"</span>],
</span>        </span>AREA[</span>"</span>World between 85.06°S and 85.06°N.</span>"</span>],
</span>        </span>BBOX[-85.06,-180,85.06,180]],
</span>    </span>ID[</span>"</span>EPSG</span>"</span>,3857]]
</span>Data</span> axis to CRS axis mapping: 1,2
</span>Origin</span> = (2886267.104982524644583,5550384.975221752189100)
</span>Pixel</span> Size = (3.554565159517144,-3.554573706431164)
</span>Metadata:
</span>  </span>AREA_OR_POINT</span>=</span>Area
</span>  </span>CREATION_DATE</span>=</span>D:20180606221414
</span>  </span>CREATOR</span>=</span>þÿ
</span>  </span>NEATLINE</span>=</span>POLYGON </span>((</span>2886267</span>.</span>10498252 5550384</span>.</span>38278758</span>,</span>2886267</span>.</span>10498252 5516801</span>.</span>36284339</span>,</span>2932447</span>.</span>42309224 5516801</span>.</span>36284339</span>,</span>2932447</span>.</span>42309224 5550384</span>.</span>38278758</span>,</span>2886267</span>.</span>10498252 5550384</span>.</span>38278758</span>))
</span>  </span>PRODUCER</span>=</span>Qt </span>4.8.5</span> (C) </span>2011</span> Nokia Corporation and/or its subsidiary(-ies)
</span>  </span>TITLE</span>=</span>þÿ
</span>Image</span> Structure Metadata:
</span>  </span>COMPRESSION</span>=</span>LZW
</span>  </span>INTERLEAVE</span>=</span>PIXEL
</span>  </span>LAYOUT</span>=</span>COG
</span>Corner</span> Coordinates:
</span>Upper</span> Left  ( 2886267.105, 5550384.975) ( </span>25d55</span>'</span>40.00"E, 44d32</span>'</span>46.88</span>"</span>N)
</span>Lower Left  ( 2886267.105, 5516801.363) ( 25d55'40.00</span>"</span>E,</span> 44d19'</span>51.43"N)
</span>Upper Right ( 2932448.016, 5550384.975) ( 26d20</span>'33.46"</span>E, 44d32'46.88</span>"N)
</span>Lower</span> Right ( 2932448.016, 5516801.363) ( </span>26d20</span>'</span>33.46"E, 44d19</span>'</span>51.43</span>"</span>N)
</span>Center      ( 2909357.560, 5533593.169) ( 26d 8' 6.73</span>"</span>E,</span> 44d26'</span>19.51"N)
</span>Band 1 Block=512x512 Type=Byte, ColorInterp=Red
</span>  Min=0.000 Max=255.000 
</span>  Minimum=0.000, Maximum=255.000, Mean=222.565, StdDev=77.753
</span>  Overviews: 6496x4724, 3248x2362, 1624x1181, 812x590, 406x295
</span>  Metadata:
</span>    STATISTICS_APPROXIMATE=YES
</span>    STATISTICS_MAXIMUM=255
</span>    STATISTICS_MEAN=222.56487158937
</span>    STATISTICS_MINIMUM=0
</span>    STATISTICS_STDDEV=77.753237328653
</span>    STATISTICS_VALID_PERCENT=100
</span>Band 2 Block=512x512 Type=Byte, ColorInterp=Green
</span>  Min=0.000 Max=255.000 
</span>  Minimum=0.000, Maximum=255.000, Mean=214.219, StdDev=83.659
</span>  Overviews: 6496x4724, 3248x2362, 1624x1181, 812x590, 406x295
</span>  Metadata:
</span>    STATISTICS_APPROXIMATE=YES
</span>    STATISTICS_MAXIMUM=255
</span>    STATISTICS_MEAN=214.21891418763
</span>    STATISTICS_MINIMUM=0
</span>    STATISTICS_STDDEV=83.65912385424
</span>    STATISTICS_VALID_PERCENT=100
</span>Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
</span>  Min=0.000 Max=255.000 
</span>  Minimum=0.000, Maximum=255.000, Mean=211.690, StdDev=87.032
</span>  Overviews: 6496x4724, 3248x2362, 1624x1181, 812x590, 406x295
</span>  Metadata:
</span>    STATISTICS_APPROXIMATE=YES
</span>    STATISTICS_MAXIMUM=255
</span>    STATISTICS_MEAN=211.69040207532
</span>    STATISTICS_MINIMUM=0
</span>    STATISTICS_STDDEV=87.03195681233
</span>    STATISTICS_VALID_PERCENT=100
</span></code></pre>
You can see it's using EPSG:3857 "Web Mercator" as the coordinate reference system.
This tells QGIS and other software how to convert the coordinates in the image to latitude and longitude on the globe.
Just below we also have the corner coordinates and pixel spacing (size), which is about 3.55 meters or 3.89 yards.</p>
But the image we extracted using pdfimages</code> doesn't have this information:</p>
$</span> gdalinfo drumuri_Lzsn-data.tif 
</span>Driver:</span> GTiff/GeoTIFF
</span>Files:</span> drumuri_Lzsn-data.tif
</span>Size</span> is 16377, 12649
</span>Image</span> Structure Metadata:
</span>  </span>COMPRESSION</span>=</span>LZW
</span>  </span>INTERLEAVE</span>=</span>PIXEL
</span>  </span>LAYOUT</span>=</span>COG
</span>Corner</span> Coordinates:
</span>Upper</span> Left  (    0.0,    0.0)
</span>Lower</span> Left  (    0.0,12649.0)
</span>Upper</span> Right (16377.0,    0.0)
</span>Lower</span> Right (16377.0,12649.0)
</span>Center</span>      ( 8188.5, 6324.5)
</span>Band</span> 1 Block=512x512 Type=Byte, ColorInterp=Red
</span>  </span>Overviews:</span> 8189x6325, 4095x3163, 2048x1582
</span>Band</span> 2 Block=512x512 Type=Byte, ColorInterp=Green
</span>  </span>Overviews:</span> 8189x6325, 4095x3163, 2048x1582
</span>Band</span> 3 Block=512x512 Type=Byte, ColorInterp=Blue
</span>  </span>Overviews:</span> 8189x6325, 4095x3163, 2048x1582
</span></code></pre>
So what do we do?
Can we copy it from the original one?
Sure, and it's quite easy!</p>
$</span> python
</span>Python</span> 3.10.9 (main, Dec 24 2022, 19:48:26) </span>[GCC</span> 12.2.0] on linux
</span>Type </span>"</span>help</span>", "</span>copyright</span>", "</span>credits</span>" or "</span>license</span>" for more information.
</span>>>> from </span>osgeo</span> import gdal
</span>>>> image = </span>gdal.Open</span>("</span>drumuri_Lzsn-data.tif</span>")
</span>>>> reference = </span>gdal.Open</span>("</span>drumuri_Lzsn.tif</span>")
</span>>>> reference.GetProjection() </span># just to check
</span>'</span>PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]</span>'
</span>>>> reference.GetGeoTransform() </span># just to check
</span>(</span>2886267.1049825246,</span> 3.5545651595171437, 0.0, 5550384.975221752, 0.0,</span> -3</span>.5545737064311638)
</span>>>> image.SetProjection(</span>reference.GetProjection</span>())
</span>0
</span>>>> image.SetGeoTransform(</span>reference.GetGeoTransform</span>())
</span>0
</span>>>> image.FlushCache() </span># write our changes to disk
</span>
</span>$ gdalinfo drumuri_Lzsn-data.tif
</span>Driver: GTiff/GeoTIFF
</span>Files: drumuri_Lzsn-data.tif
</span>       drumuri_Lzsn-data.tif.aux.xml
</span>Size is 16377, 12649
</span>Coordinate System is:
</span>PROJCRS["WGS 84 / Pseudo-Mercator",
</span>    BASEGEOGCRS["WGS 84",
</span>        DATUM["World Geodetic System 1984",
</span>            ELLIPSOID["WGS 84",6378137,298.257223563,
</span>                LENGTHUNIT["metre",1]]],
</span>        PRIMEM["Greenwich",0,
</span>            ANGLEUNIT["degree",0.0174532925199433]],
</span>        ID["EPSG",4326]],
</span>    CONVERSION["unnamed",
</span>        METHOD["Popular Visualisation Pseudo Mercator",
</span>            ID["EPSG",1024]],
</span>        PARAMETER["Latitude of natural origin",0,
</span>            ANGLEUNIT["degree",0.0174532925199433],
</span>            ID["EPSG",8801]],
</span>        PARAMETER["Longitude of natural origin",0,
</span>            ANGLEUNIT["degree",0.0174532925199433],
</span>            ID["EPSG",8802]],
</span>        PARAMETER["False easting",0,
</span>            LENGTHUNIT["metre",1],
</span>            ID["EPSG",8806]],
</span>        PARAMETER["False northing",0,
</span>            LENGTHUNIT["metre",1],
</span>            ID["EPSG",8807]]],
</span>    CS[Cartesian,2],
</span>        AXIS["easting",east,
</span>            ORDER[1],
</span>            LENGTHUNIT["metre",1]],
</span>        AXIS["northing",north,
</span>            ORDER[2],
</span>            LENGTHUNIT["metre",1]],
</span>    ID["EPSG",3857]]
</span>Data axis to CRS axis mapping: 1,2
</span>Origin = (</span>2886267.104982524644583,5550384.975221752189100</span>)
</span>Pixel</span> Size = (3.554565159517144,-3.554573706431164)
</span>Image</span> Structure Metadata:
</span>  </span>COMPRESSION</span>=</span>LZW
</span>  </span>INTERLEAVE</span>=</span>PIXEL
</span>  </span>LAYOUT</span>=</span>COG
</span>Corner</span> Coordinates:
</span>Upper</span> Left  ( 2886267.105, 5550384.975) ( 25d55'</span>40.00"E, 44d32</span>'46.88"</span>N)
</span>Lower Left  ( 2886267.105, 5505423.172) ( 25d55'40.00</span>"E, 44d15'</span>28.05"N)
</span>Upper Right ( 2944480.219, 5550384.975) ( 26d27</span>' 2.58"</span>E, 44d32'46.88</span>"N)
</span>Lower</span> Right ( 2944480.219, 5505423.172) ( </span>26d27</span>'</span> 2.58"E, 44d15</span>'</span>28.05</span>"</span>N)
</span>Center      ( 2915373.662, 5527904.074) ( 26d11'21.29</span>"</span>E,</span> 44d24'</span> 8.11"N)
</span>Band 1 Block=512x512 Type=Byte, ColorInterp=Red
</span>  Overviews: 8189x6325, 4095x3163, 2048x1582
</span>Band 2 Block=512x512 Type=Byte, ColorInterp=Green
</span>  Overviews: 8189x6325, 4095x3163, 2048x1582
</span>Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
</span>  Overviews: 8189x6325, 4095x3163, 2048x1582
</span></code></pre>
If you're wondering, the "Geotransform"</a> is a set of parameters for an affine transformation</a>.
It can describe a combination of translation (where the image is located), scaling (how large a pixel is), rotation (the orientation of the image) and shear, which I suspect nobody uses.
These are sufficient for any "well-behaved" image like those you'll find on my blog, but that's not always the case.</p>
We can load them both in QGIS, but something seems a little off.
In the screenshot below, I reduced the transparency of drumuri_Lzsn-data.tif</code>:</p>
</a></p>
That's pretty awful, and it seems that our approach was not quite right.
In retrospect, this makes sense.
The PDF had a correct georeference, but it doesn't necessarily transfer to the one we extracted, because we didn't take into account the position and scale of the image on the page.</p>
Georeferencing, again</h2>
Fortunately, not all is lost, as this must be a problem a lot of QGIS users must running into.
Digitizing a map doesn't seem like such a rare use case, and I knew that QGIS has a feature that helps with this.
What we'd like to do is pick some points on our image, find their pair in the original one, and let QGIS figure it out.
In GIS-speak, these are called GCPs, for Ground Control Points.</p>
We can do this using the Georeferencer tool.
If you're looking it up, a lot of documentation online is outdated, and it's now found Layer menu.
The documentation makes this seem pretty complex, but it's not that bad.</p>
Note: if you're following along, the file you load must not have a geotransform, otherwise QGIS will get very confused</a>.
So we have to undo the work from the previous section.
The easiest way is probably to rebuild the file:</p>
$</span> gdal_translate</span> -f</span> COG</span> -co</span> NUM_THREADS=ALL_CPUS drumuri-Lzsn.jpg drumuri_Lzsn-data.tif
</span></code></pre>
I opened the tool, loaded the raster (drumuri_Lzsn-data.tif</code>), then zoomed in a lot and clicked on a pixel that seemed easy to spot.
A new window showed up asking for some coordinates, I clicked "From Map Canvas" and selected the corresponding pixel from the original (drumuri_Lzsn-data.tif</code>):</p>
</a></p>
The points you click on don't get snapped to pixel centers, so I tried to be somewhat precise when clicking.</p>
After hitting "OK", the GCP showed up at the bottom of the window.</p>
You need at least two pairs of points, but I picked three because it was pretty fun.
It's probably a good idea to choose points that are far apart from each other.</p>
Then I went to "Transformation Settings", checked "Set target resolution", but left the horizontal and vertical values set to 0 (you get a weird error if you don't do this).
Linear and Nearest Neighbor should be fine.
Once you hit start, you should see a drumuri_Lzsn-data_modified.tif</code> layer in QGIS.</p>
Here I opened the layer settings and added 255</code> as an additional no data value in the Transparency tab, then added the OpenStreetMap XYZ tileset:</p>
</a></p>
</a></p>
Unfortunately, I couldn't get a pixel-perfect match over the original, but it should be good enough for practical purposes.</p>
Next steps</h2>
Our result looks encouraging, but we still need to do something about the compression artefacts (remember, the image we extracted is a JPEG) and, ideally, fill in the building footprints.
If you'd like to read more, stay tuned for the next part!</p>



About a fun little NFS bug
2022-06-29T00:00:00+00:00
One cloud provider</a> I sometimes work with has a large repository of images, accessible using the S3 protocol or over an NFS gateway. Today I spent a bit of time debugging an issue related to the NFS gateway, so read on for more — and for a twist at the end.</p>
Something is broken</h2>
I have a workload which involves relatively many of those images at once, accessed through the s3fs</code></a> FUSE helper (a way to make an S3 bucket look like a file system).
For some reason, sometimes requests start failing after a while, and using NFS is a way to work around that.</p>
But something strange was happening today when I was trying to use the NFS share.
One version of GDAL</a> was able to read the files, while another, older one, complained they were invalid and wasn't able to open them.
Using s3fs</code>, both GDAL versions worked fine.</p>
This made me think that NFS is returning corrupted files (maybe with some junk at the end), and the newer GDAL was able to ignore that.
But on a closer look, both s3fs</code> and NFS returned identical (up to a checksum, at least) files.
This was the cause of some head scratching, since reading a the same file from different places should yield the same results —</p>
My favourite tool</h2>
— unless maybe it shouldn't?
This story seemed fishy and I broke out what's probably my favourite debugging tool, strace</code>.
strace</code> sets breakpoints on every interaction of a program with the operating system (system call</a>) and displays the request involved, including the arguments.
So, for example, if Linux a program tries to open a file, it will show up in strace</code> as an open</code> or openat</code> call.</p>
Note: if that sounds interesting and you want to know more, check out this zine</a> and this article</a>.
And if you're not using Linux, you can try Process Monitor</a> or DTrace</a>.</em></p>
So I gave strace</code> a try (edited for brevity):</p>
$</span> strace gdalinfo /mount_nfs/path/to/file.jp2 </span># old GDAL
</span>[snip]
</span>stat</span>("</span>/mount_nfs/path/to/file.jp2</span>", {st_mode=S_IFREG|0666, st_size=115495922, ...}) = </span>0
</span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>", O_RDONLY) = </span>3
</span>fstat</span>(3, {st_mode=S_IFREG|0666, st_size=115495922, ...}) = </span>0
</span>read</span>(3, "</span>[snip]</span>"..., 8192) = </span>8192
</span>lseek</span>(3, 0, SEEK_SET) = </span>0
</span>[snip]
</span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>", O_RDONLY) = </span>-1</span> EPERM (Operation not permitted)
</span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>", O_RDONLY) = </span>-1</span> EPERM (Operation not permitted)
</span>[snip]
</span></code></pre>
Most of the output above isn't too important, but the last open</code> calls fail with EPERM</code>, which sounds like a good reason to not work.
If you have trouble reading the log, the full sequence is:</p>

gdalinfo</code> checks the file length and attributes</li>
gdalinfo</code> opens the file</li>
gdalinfo</code> checks the length and attributes of the open file</li>
gdalinfo</code> reads 8 KB from the file, which works, then rewinds to the beginning</li>
gdalinfo</code> tries to open the file again and gets a permissions-related error</li>
gdalinfo</code> tries to open the file again and gets a permissions-related error</li>
</ul>
Sure, this is a bit redundant, but it doesn't matter that much when you're reading 115 MB.
Note how the first open</code> call works, then but the next ones fail.</p>
In contrast, the newer GDAL works because it only tries to open the file once.
So what's wrong, can we only read files once?</p>
Of course not, otherwise a second gdalinfo</code> wouldn't work.
So a better theory is that we can't open the same file twice.
And now that we have a theory, we can try to confirm it using Rust</del> just kidding, let's use Python this time:</p>
# open the file once, works
</span>>>> f1 = </span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>")
</span>
</span># open it again, fails with EPERM
</span>>>> f2 = </span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>")
</span>Traceback </span>(most recent call last):
</span>  File "</span><stdin></span>", line </span>1</span>, in <module>
</span>IOError: [Errno </span>1</span>] Operation not permitted: '</span>/mount_nfs/path/to/file.jp2</span>'
</span>
</span># close the first handle
</span>>>> f1.</span>close</span>()
</span>
</span># opening it works
</span>>>> f2 = </span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>")
</span>
</span># opening it a second time fails again
</span>>>> f3 = </span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>")
</span>Traceback </span>(most recent call last):
</span>  File "</span><stdin></span>", line </span>1</span>, in <module>
</span>IOError: [Errno </span>1</span>] Operation not permitted: '</span>/mount_nfs/path/to/file.jp2</span>'
</span>
</span># and so on
</span>>>> f2.</span>close</span>()
</span>>>> f3 = </span>open</span>("</span>/mount_nfs/path/to/file.jp2</span>")
</span>>>>
</span></code></pre>
So yeah, our theory was correct.
I don't know what kind of server we're dealing with, but it doesn't let us open any file multiple times.
It even fails — maybe not surprisingly — if we're opening it from different processes.</p>
I wasn't able to find any reports of a similar problem, so one of my reasons for writing this is to make it available to search engines.</p>
The solution</h2>
I also happened to stumble upon a pretty simple workaround: add nfsvers=3</code> to the mount options (/etc/fstab</code>).
It appears that the problem only occurs when using NFSv4, and not NFSv3.</p>
A final twist</h2>
All this sounded somewhat familiar, so I searched through my inbox and found I had already reported this to the cloud provider in March 2020.
I even took a tcpdump</code> capture and found out that the server was returning NFS4ERR_PERM</code> on the failing call:</p>

This indicates that the requester is not the owner.  The operation was not allowed because the caller is neither a privileged user (root) nor the owner of the target of the operation.</p>
</blockquote>
But don't confuse this with NFS4ERR_ACCESS</code>:</p>

This indicates permission denied.  The caller does not have the correct permission to perform the requested operation.  Contrast this with NFS4ERR_PERM (Section 13.1.6.2), which restricts itself to owner or privileged user permission failures.</p>
</blockquote>
Hear this, Google?
If you've got a weird NFS server that doesn't allow you to open files a second time and yields NFS4ERR_PERM</code>, try downgrading to NFSv3.</p>
And no, the provider never answered back.</p>

By the way, if your files sometime go away or can't be reopened, please consider buying me a coffee</a>.</p>


Decoding Firefox session store data
2022-06-12T00:00:00+00:00
If you're using Firefox (and you should), you might have wanted to read its session restore files, perhaps to recover some lost tabs, re-import an old session after a refresh, or even track your tab hoarding habit.</p>
Unfortunately, last time I checked, there wasn't much information available about the session restore format.
I've previously used lz4json</code></a> to decode them, but it's a good prompt for a post, and I'd rather not keep an extra AUR package anyway.</p>
Finding the files</h2>
Under ~/.mozilla/firefox</code> on Linux, or %APPDATA%\Mozilla\Firefox</code> on Windows, you should have a profiles.ini</code>, an installs.ini</code> and one or more randomly-named subdirectories.
The default profile is marked as such in the two INI files.
Of course, the better way is to open about:support</code> and copy the path from there.</p>
In there, there should be a directory called sessionstore-backups</code>, with a couple of files:</p>
$</span> ls sessionstore-backups
</span>previous.jsonlz4
</span>recovery.baklz4
</span>recovery.jsonlz4
</span>upgrade.jsonlz4-20220518214245
</span>upgrade.jsonlz4-20220530093943
</span>upgrade.jsonlz4-20220606212503
</span></code></pre>
I don't know the specifics, but these are backup versions, more or less recent, some of them saved during browser upgrades.
Looking at the last modified dates, it appears that the most recent one is called recovery.jsonlz4</code>.</p>
The lz4json</code> format</h2>
The jsonlz4</code> extension is a good hint, but a good first step is to run the file</code> utility, which tries to guess the type of a file:</p>
$</span> file recovery.jsonlz4
</span>recovery.jsonlz4:</span> Mozilla lz4 compressed data, originally 145030214 bytes
</span></code></pre>
file</code> recognizes thousands of file formats, but it's relatively shallow.
This means that the uncompressed length is likely to be easily accessible.
Let's look at the file contents:</p>
$ hexyl recovery.jsonlz4 | head -n7
</span>┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
</span>│00000000│ 6d 6f 7a 4c 7a 34 30 00 ┊ 46 fc a4 08 f2 21 7b 22 │mozLz400┊F××•×!{"│
</span>│00000010│ 76 65 72 73 69 6f 6e 22 ┊ 3a 5b 22 73 65 73 73 69 │version"┊:["sessi│
</span>│00000020│ 6f 6e 72 65 73 74 6f 72 ┊ 65 22 2c 31 5d 2c 22 77 │onrestor┊e",1],"w│
</span>│00000030│ 69 6e 64 6f 77 73 22 3a ┊ 5b 7b 22 74 61 62 09 00 │indows":┊[{"tab_0│
</span>│00000040│ 62 65 6e 74 72 69 65 0c ┊ 00 f3 29 75 72 6c 22 3a │bentrie_┊0×)url":│
</span>│00000050│ 22 68 74 74 70 73 3a 2f ┊ 2f 6d 61 74 72 69 78 2e │"https:/┊/matrix.│
</span></code></pre>
We can actually see some JSON</a> in there.
LZ4</a> is designed to be as fast as possible, so it doesn't do anything fancy like Huffman coding</a>, yielding partially-readable text as output.</p>
Before the JSON, though, there is a line that looks a bit strange.
The file starts with the "mozLz40\0"</code> magic number</a>, followed by what appear to be six non-ASCII bytes, then the start of the JSON.
We can expect the six bytes to contain the file length, or maybe a checksum.
These are usually found either at the beginning or at the end of a file.</p>
Luckily, file</code> was nice to tell us the uncompressed data size.
145030214</code>, converted to hex</a> is 0x08A4FC46</code>, which is actually visible as 46 fc a4 08</code> in the hexyl</code></a> output above.</p>
We can also peek</a> at the implementation of file</code>.
It appears to confirm that the uncompressed size is a little-endian</a> 32-bit number, followed directly by the compressed data.
I assume that the >8</code> annotation means "starting from offset 8" or something similar to that.</p>
Writing some code</h2>
This is a Rust</a>-related blog, so of course I'll be using that.
Fortunately, we already have libraries for everything we're going to need, so it's going to be easy.
We can use lz4_flex</code></a> for LZ4 decoding, serde</code></a> and serde_json</code></a> for JSON decoding, and anyhow</code></a> to handle errors in a nicer way.
We'll also pull in memmap2</code></a> to map the file in memory (which is optional, but saves a bit of RAM) and url</code></a> for URL parsing.</p>
# Cargo.toml
</span>[package]
</span>name </span>= "</span>tabs</span>"
</span>version </span>= "</span>0.1.0</span>"
</span>edition </span>= "</span>2021</span>"
</span>
</span>[dependencies]
</span>anyhow </span>= "</span>1.0</span>"
</span>lz4_flex </span>= "</span>0.9</span>"
</span>memmap2 </span>= "</span>0.5</span>"
</span>serde </span>= { </span>version </span>= "</span>1.0</span>", </span>features </span>= ["</span>derive</span>"] }
</span>serde_json </span>= "</span>1.0</span>"
</span>url </span>= "</span>2.2</span>"
</span></code></pre>
lz4_flex</code> has a nifty helper function</a> to decode a size-prepended block, exactly what our file uses.
So if we're lucky, it should be enough to open the file, read it, then call the decompression function.
With some imports and command line handling omitted, it simply comes to:</p>
let</span> file = File::open(&path)?;
</span>let</span> mmap = </span>unsafe </span>{ MmapOptions::new().</span>map</span>(&file)? };
</span>let</span> buf = lz4_flex::decompress_size_prepended(&mmap[</span>8</span>..])?;
</span>let</span> buf = String::from_utf8(buf)?;
</span>
</span>// check if it worked
</span>println!("</span>{}</span>", &buf[..</span>64</span>]);
</span></code></pre>
$</span> cargo run</span> --release</span> -- </span>~</span>/.mozilla/firefox/ou63gnwj.default
</span># snip
</span>{</span>"</span>version</span>"</span>:[</span>"</span>sessionrestore</span>"</span>,1],</span>"</span>windows</span>"</span>:[{</span>"</span>tabs</span>"</span>:[{</span>"</span>entries</span>"</span>:[
</span></code></pre>
I won't show it here, but the way the JSON is structured, the session has a list of windows, each window has a list of tabs, and each tab has a list of history entries.
We only care for the last entry, which is the one currently displayed.</p>
This is pretty easy to parse with serde</code>.</p>
#[</span>derive</span>(Deserialize)]
</span>struct </span>Entry {
</span>    </span>url</span>: String,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>Tab {
</span>    </span>entries</span>: Vec<Entry>,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>Window {
</span>    </span>tabs</span>: Vec<Tab>,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>SessionStore {
</span>    </span>windows</span>: Vec<Window>,
</span>}
</span>
</span>// ...
</span>
</span>let</span> session = serde_json::from_slice::<SessionStore>(&buf)?;
</span>let mut</span> domains = HashMap::<_, </span>u32</span>>::new();
</span>for</span> window in session.windows {
</span>    </span>for</span> tab in window.tabs {
</span>        </span>if let </span>Some(entry) = tab.entries.</span>last</span>() {
</span>            </span>let</span> url = Url::parse(&entry.url)?;
</span>            </span>// skip about:blank, about:reader etc.
</span>            </span>if let </span>Some(host) = url.</span>host_str</span>() {
</span>                *domains.</span>entry</span>(host.</span>to_string</span>()).</span>or_default</span>() += </span>1</span>;
</span>            }
</span>        }
</span>    }
</span>}
</span></code></pre>
For this example, I'm grabbing every tab in every window, making sure it's not empty, taking the last entry, then sticking the domain of each URL in a HashMap</code>, in order to count them.</p>
Note: If you know Rust, that snippet looks a bit nicer written in a functional/iterator-based style.</em></p>
Finally, we collect</code></a> the entries into a Vec</code>, sort them, and display the most common domains:</p>
let mut</span> domains = domains.</span>into_iter</span>().collect::<Vec<_>>();
</span>domains.</span>sort_unstable_by_key</span>(|</span>p</span>| Reverse(p.</span>1</span>));
</span>for </span>(domain, count) in domains.</span>into_iter</span>().</span>take</span>(</span>10</span>) {
</span>    println!("</span>{} {}</span>", domain, count);
</span>}
</span></code></pre>
# sample output, truncated
</span>$</span> cargo run</span> --release</span> -- </span>~</span>/.mozilla/firefox/ou63gnwj.default
</span>github.com</span> 1150
</span>www.youtube.com</span> 213
</span>twitter.com</span> 206
</span>news.ycombinator.com</span> 109
</span></code></pre>
I may or may not have a tab hoarding problem.</em></p>
Full code:</p>
use </span>std::cmp::Reverse;
</span>use </span>std::collections::HashMap;
</span>use </span>std::fs::File;
</span>use </span>std::path::PathBuf;
</span>use </span>std::{env, process};
</span>
</span>use </span>memmap2::MmapOptions;
</span>use </span>serde::Deserialize;
</span>use </span>url::Url;
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>Entry {
</span>    </span>url</span>: String,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>Tab {
</span>    </span>entries</span>: Vec<Entry>,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>Window {
</span>    </span>tabs</span>: Vec<Tab>,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize)]
</span>struct </span>SessionStore {
</span>    </span>windows</span>: Vec<Window>,
</span>}
</span>
</span>fn </span>main</span>() -> anyhow::Result<()> {
</span>    </span>let mut</span> args = env::args_os().collect::<Vec<_>>();
</span>    </span>if</span> args.</span>len</span>() != </span>2 </span>{
</span>        eprintln!("</span>Usage: </span>{}</span> <profile></span>", args[</span>0</span>].</span>to_string_lossy</span>());
</span>        process::exit(</span>1</span>);
</span>    }
</span>
</span>    </span>let mut</span> path = PathBuf::from(args.</span>remove</span>(</span>1</span>));
</span>    path.</span>push</span>("</span>sessionstore-backups/recovery.jsonlz4</span>");
</span>
</span>    </span>let</span> file = File::open(&path)?;
</span>    </span>let</span> mmap = </span>unsafe </span>{ MmapOptions::new().</span>map</span>(&file)? };
</span>    </span>let</span> buf = lz4_flex::decompress_size_prepended(&mmap[</span>8</span>..])?;
</span>
</span>    </span>let</span> session = serde_json::from_slice::<SessionStore>(&buf)?;
</span>    </span>let mut</span> domains = HashMap::<_, </span>u32</span>>::new();
</span>    </span>for</span> window in session.windows {
</span>        </span>for</span> tab in window.tabs {
</span>            </span>if let </span>Some(entry) = tab.entries.</span>last</span>() {
</span>                </span>let</span> url = Url::parse(&entry.url)?;
</span>                </span>// println!("{url}"); // uncomment this to show all URLs
</span>                </span>if let </span>Some(host) = url.</span>host_str</span>() {
</span>                    *domains.</span>entry</span>(host.</span>to_string</span>()).</span>or_default</span>() += </span>1</span>;
</span>                }
</span>            }
</span>        }
</span>    }
</span>
</span>    </span>let mut</span> domains = domains.</span>into_iter</span>().collect::<Vec<_>>();
</span>    domains.</span>sort_unstable_by_key</span>(|</span>p</span>| Reverse(p.</span>1</span>));
</span>    </span>for </span>(domain, count) in domains.</span>into_iter</span>().</span>take</span>(</span>10</span>) {
</span>        println!("</span>{} {}</span>", domain, count);
</span>    }
</span>
</span>    Ok(())
</span>}
</span></code></pre>
Conclusion</h2>
We've uncompressed a Firefox session backup and printed the most common domains in the open tabs.</p>
I actually think this is quite a poor choice of format for sessions like mine.
Perhaps you've noticed that my session was 142 MB uncompressed, which is not insignificant.
Worse, it's on-disk size is 42 MB, and Firefox tends to write it every couple of seconds (but only as long as it changes, I hope).
That's pretty bad, not only for performance (since the file must be rewritten every time you scroll, navigate to another page, or type something in a form), but can also reduce the lifespan of an SSD drive.</p>
Because of this, people ended up with workarounds</a> to keep the session on a RAM drive</a>, at the cost of durability in case of a power failure or a crash.</p>
I don't know the reasoning behind this design, but I suspect the vast majority of users have less than four tabs.
Maybe the Firefox engineers wanted to avoid calling into SQLite while restoring the session.</p>

By the way, if you've ever felt anxious about your ever-growing tab list, please consider buying me a coffee</a>.</p>


Debugging a mysterious Python crash
2022-05-28T00:00:00+00:00
I recently wanted to prepare a Jupyter notebook with some example code and ran into an interesting problem: trying to display a Matplotlib chart made the IPython kernel crash.</p>
Gathering more info</h2>
Luckily, the crash was easy to reproduce outside of Jupyter:</p>
$</span> python3</span> -c </span>'</span>import matplotlib.pyplot as plt; plt.axes()</span>'
</span>Illegal</span> instruction
</span></code></pre>
This usually means one of two things:</p>

some compilers / standard libraries produce an ud2</code> instruction in order to abort the program execution, e.g. core::intrinsics::abort()</code> in Rust.</li>
the program tried to execute an instruction that is not available on the CPU it was running on.</li>
</ul>
I've encountered the former one much more often than the latter, but many popular Python libraries code call into optimized C.
To be honest, I was pretty sure this was caused by my mess of old, distro-supplied packages (from CentOS 7), in combination with some installed using pip</code>.
Of course, it's better to check than to guess:</p>
$</span> gdb</span> --args</span> python3</span> -c </span>'</span>import matplotlib.pyplot as plt; plt.axes()</span>'
</span>[snip]
</span>(</span>gdb</span>) r
</span>Starting</span> program: /usr/bin/python3</span> -c</span> import</span>\ </span>matplotlib.pyplot</span>\ </span>as</span>\ </span>plt</span>\;\ </span>plt.axes</span>\(\)
</span>[Thread</span> debugging using libthread_db enabled]
</span>Using</span> host libthread_db library "</span>/usr/lib/libthread_db.so.1</span>".
</span>[snip]
</span>
</span>Thread</span> 1 "</span>python3</span>" received signal SIGILL, Illegal instruction.
</span>0x00007ffff556545c</span> in dgemm_kernel_PILEDRIVER () </span>from</span> /usr/local/lib64/python3.6/site-packages/numpy/core/../../numpy.libs/libopenblasp-r0-8a0c371f.3.13.so
</span></code></pre>
Heads-up: for reasons I'll show below, I'm reproducing the issue on my own computer. I'm trying to doctor the outputs, but there might be some inconsistencies.</em></p>
It appears to be crashing in a dgemm</code> kernel in OpenBLAS, docs courtesy of LAPACK:</p>
DGEMM  performs one of the matrix-matrix operations
</span>
</span>C := alpha*op( A )*op( B ) + beta*C,
</span>
</span>where  op( X ) is one of
</span>
</span>op( X ) = X   or   op( X ) = X**T,
</span>
</span>alpha and beta are scalars, and A, B and C are matrices, with op( A )
</span>an m by k matrix,  op( B )  a  k by n matrix and  C an m by n matrix.
</span></code></pre>
That's basically a fancy matrix multiplication.
More importantly, notice the Piledriver reference in the function name.
Piledriver is an AMD microarchitecture from around 2012–2014, which is hopefully not what I'm running on.</p>
On the other hand, function names aren't always accurate, especially in highly-optimized code.
We can ask GDB to disassemble the current function in order to check the place where the code crashed and it will output a 4890-line monstruosity:</p>
(gdb) disassemble
</span>Dump of assembler code for function dgemm_kernel_PILEDRIVER:
</span>   </span>0x00007ffff555fe00 </span><</span>+</span>0</span>>:    </span>sub    </span>$</span>0x60</span>,</span>%</span>rsp
</span>   </span>0x00007ffff555fe04 </span><</span>+</span>4</span>>:    </span>mov    </span>%</span>rbx</span>,</span>(%</span>rsp</span>)
</span>   </span>0x00007ffff555fe08 </span><</span>+</span>8</span>>:    </span>mov    </span>%</span>rbp</span>,</span>0x8</span>(%</span>rsp</span>)
</span>   </span>[</span>snip</span>]
</span>   </span>0x00007ffff5565450 </span><</span>+</span>22096</span>>:    </span>vmovddup </span>-</span>0x20</span>(%</span>rsi</span>,</span>%</span>rbp</span>,</span>8</span>)</span>,</span>%</span>xmm1
</span>   </span>0x00007ffff5565456 </span><</span>+</span>22102</span>>:    </span>vmovups </span>-</span>0x80</span>(%</span>rdi</span>,</span>%</span>rax</span>,</span>8</span>)</span>,</span>%</span>xmm0
</span>=> </span>0x00007ffff556545c </span><</span>+</span>22108</span>>:    </span>vfmaddpd </span>%</span>xmm4</span>,</span>%</span>xmm1</span>,</span>%</span>xmm0</span>,</span>%</span>xmm4
</span>   </span>0x00007ffff5565462 </span><</span>+</span>22114</span>>:    </span>vmovddup </span>-</span>0x18</span>(%</span>rsi</span>,</span>%</span>rbp</span>,</span>8</span>)</span>,</span>%</span>xmm2
</span>   </span>0x00007ffff5565468 </span><</span>+</span>22120</span>>:    </span>vfmaddpd </span>%</span>xmm5</span>,</span>%</span>xmm2</span>,</span>%</span>xmm0</span>,</span>%</span>xmm5
</span></code></pre>
The crashing opcode was vfmaddpd</code>.
I've never encountered it before — not exactly surprising — but notice how it has four operands, which is pretty rare in x86</code> instructions.
That said, it's pretty easy to guess what it does if you can unpack its mnemonic:</p>

v</code>: vector (SIMD) instruction, meaning it operates not on scalars, but on vector registers with multiple values</li>
fma</code>: fused multiply-add, that is, a multiplication and addition in one step, for better performance, precision, or both</li>
p</code>: packed, meaning it uses all the values in the registers (as opposed to the scalar ones, which only touch a single value in a vector register)</li>
d</code>: double-precision, i.e. 64-bit floating-point numbers</li>
</ul>
Peeking at the docs</a>, the instruction name is Multiply and Add Packed Double-Precision Floating-Point</code>:</p>
VFMADDPD </span>dest</span>, </span>src1</span>, </span>src2</span>, </span>src3 # dest = (src1 </span>* </span>src2) </span>+ </span>src3
</span></code></pre>
It doesn't matter for us, but xmm</code> are 128-bit registers, so it's working on pairs double</code>s.</p>
This is an FMA4</code> (four-operand FMA) instruction, which has a bit of a weird history.
It was introduced by AMD, but Intel never implemented it.
Instead, Intel added their own FMA3</code> (three-operand) instructions, which look like vfmadd213pd xmm0, xmm1, xmm2</code> and don't have a separate destination.
If you're wondering, 213</code> specifies the operand order, the one here doing xmm0 = xmm1 * xmm0 + xmm2</code>.</p>
In any case, AMD dropped FMA4 in 2017 with the Zen microarchitecture, which probably caused some confusion because the instructions still work, but give the wrong results sometimes.</p>
By this point, we have a theory: our CPU does not support FMA4, but OpenBLAS (used by numpy</code>, used by matplotlib</code>) thinks it does, picks up that code path and happily crashes.</p>
Opteron</h2>
I don't have the /proc/cpuinfo</code> output any more (see below), but the model name was AMD Opteron 63xx class</code>, or something similar.
This is a virtual machine, so I'm assuming the hypervisor was reporting an older CPU model.</p>
Software can detect CPU features in a fine-grained way (using the cpuid</code> instruction), but OpenBLAS only checks for the CPU model here</a> and then again here</a>.
There actually was an Opteron 6300</a> series, which matches the reported model name and did support FMA4</code>.</p>
I also tried some code</a> that checks specifically for FMA4</code>.
I can't run it again, but indeed, it reported no FMA4</code> support.</p>
The fix and a twist</h2>
Knowing the problem, the fix was pretty simple.
Fortunately, OpenBLAS supports overriding the CPU-specific code paths through an environment variable, and setting OPENBLAS_CORETYPE=ZEN</code> made it work.
I still filed an issue</a> against the OpenBLAS repo,
in case anyone else runs into this.</p>
Unfortunately, one day later, I can't run my notebook because the /</code> partition is mounted read-only, the file system might be corrupted, and pretty much everything is broken. But for once, it's not my fault 😅.</p>
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/home/user']
</span></code></pre>
Yes, I could mount tmpfs</code> into /tmp</code>, but it doesn't matter.</p>
And the Opteron magically turned into an EPYC:</p>
processor       : 0
</span>vendor_id       : AuthenticAMD
</span>cpu family      : 23
</span>model           : 49
</span>model name      : AMD EPYC 7502P 32-Core Processor
</span>stepping        : 0
</span>microcode       : 0x1000065
</span>cpu MHz         : 2500.001
</span>cache size      : 512 KB
</span>physical id     : 0
</span>siblings        : 1
</span>core id         : 0
</span>cpu cores       : 1
</span>apicid          : 0
</span>initial apicid  : 0
</span>fpu             : yes
</span>fpu_exception   : yes
</span>cpuid level     : 16
</span>wp              : yes
</span>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm art rep_good nopl extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core retpoline_amd ssbd ibrs ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 virt_ssbd arat npt nrip_save umip spec_ctrl
</span>bogomips        : 5000.00
</span>TLB size        : 1024 4K pages
</span>clflush size    : 64
</span>cache_alignment : 64
</span>address sizes   : 40 bits physical, 48 bits virtual
</span>power management:
</span></code></pre>
Conclusion</h2>
Just a couple of things:</p>

feature detection is better than version checking not only when writing JavaScript, but even when targeting low-level code</li>
it's a good idea to provide escape hatches from optimized code paths</li>
the CPU is a lie</li>
</ul>

And by the way, if you enjoyed this short post, please consider buying me a coffee</a>.</p>


Blogroll
2022-05-24T00:00:00+00:00
Alex</a></p>


Building Rust code for my OpenWrt Wi-Fi router
2022-05-08T00:00:00+00:00
I recently got interested in running Rust code on my router (stay tuned for a future blog post?).
This is supposed to be easy, but I never tried it, so let's see how it goes.</p>
A test project</h2>
Hello, world</em>s are somewhat boring, so I'd like to build something more realistic — a DNS client.
We're not going to implement DNS here, but rather piggy-back on the trust-dns-resolver</code></a> crate, which looks pretty good.</p>
After skimming the trust-dns</code> and tokio</code> docs, and a cargo new delve</code> (shouts to dig</code> and drill</code>), we have this:</p>
[package]
</span>name </span>= "</span>delve</span>"
</span>version </span>= "</span>0.1.0</span>"
</span>edition </span>= "</span>2021</span>"
</span>
</span>[dependencies]
</span>anyhow </span>= "</span>1.0</span>"
</span>tokio </span>= { </span>version </span>= "</span>1.18</span>", </span>features </span>= ["</span>net</span>", "</span>rt-multi-thread</span>"] }
</span>trust-dns-resolver </span>= { </span>version </span>= "</span>0.21</span>", </span>features </span>= [
</span>    "</span>dns-over-rustls</span>",
</span>    "</span>tokio</span>",
</span>] }
</span></code></pre>
use </span>std::env;
</span>
</span>use </span>tokio::runtime;
</span>use </span>trust_dns_resolver::{
</span>    config::{ResolverConfig, ResolverOpts},
</span>    proto::{rr::RecordType, xfer::DnsRequestOptions},
</span>    TokioAsyncResolver,
</span>};
</span>
</span>async </span>fn </span>run</span>() -> Result<(), anyhow::Error> {
</span>    </span>let</span> resolver =
</span>        TokioAsyncResolver::tokio(ResolverConfig::cloudflare_tls(), ResolverOpts::default())?;
</span>    </span>let</span> query = env::args().</span>nth</span>(</span>1</span>).</span>unwrap</span>();
</span>
</span>    </span>let</span> response = resolver
</span>        .</span>lookup</span>(query, RecordType::A, DnsRequestOptions::default())
</span>        .await?;
</span>
</span>    </span>for</span> address in response.</span>iter</span>() {
</span>        println!("</span>{}</span>", address);
</span>    }
</span>    Ok(())
</span>}
</span>
</span>fn </span>main</span>() -> Result<(), anyhow::Error> {
</span>    </span>let</span> runtime = runtime::Builder::new_multi_thread().</span>enable_all</span>().</span>build</span>()?;
</span>    runtime.</span>block_on</span>(async { tokio::spawn(</span>run</span>()).await })?
</span>}
</span></code></pre>
The code is pretty simple.
It creates a tokio</code> multi-threaded Runtime</code>, sets up a trust-dns</code> TokioAsyncResolver</code>, then queries the Cloudflare DNS server for the hostname given in the command line.
RecordType::A</code> means asking for an IPv4 address record.</p>
If you're wondering about the block_on</code> / spawn</code> dance, apparently tokio</code> prefers that you don't do a lot of work in block_on</code>, but rather spawn a root future and do your stuff from there.
It doesn't really matter in this case, and trust-dns-resolver</code> helpfully provides a synchronous resolver, but it will matter in the project I have in mind.</p>
And it does appear to work fine:</p>
$</span> cargo run</span> --release</span> -- google.com
</span>   </span>[snip]
</span>    </span>Finished</span> release </span>[</span>optimized</span>]</span> target(s) </span>in</span> 9.66s
</span>     </span>Running </span>`</span>target/release/delve</span> google.com`
</span>142.250.179.174
</span></code></pre>
Inside my router</h2>
My router is an AVM FRITZ!Box 4040 running OpenWrt.
As far as embedded systems go, OpenWrt is pretty close to a Linux PC.
I've already enabled SSH and added my key, but I don't know what architecture it's running.
Fortunately, uname -a</code> works just as expected:</p>
root@OpenWrt:~#</span> uname</span> -a
</span>Linux</span> OpenWrt 4.14.171 </span>#0 SMP Thu Feb 27 21:05:12 2020 armv7l GNU/Linux
</span>root@OpenWrt:~#</span> cat /proc/cpuinfo
</span>processor</span>	: 0
</span>model</span> name	: ARMv7 Processor rev 5 (v7l)
</span>BogoMIPS</span>	: 67.03
</span>Features</span>	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
</span>CPU</span> implementer	: 0x41
</span>CPU</span> architecture: 7
</span>CPU</span> variant	: 0x0
</span>CPU</span> part	: 0xc07
</span>CPU</span> revision	: 5
</span>
</span>processor</span>	: 1
</span>model</span> name	: ARMv7 Processor rev 5 (v7l)
</span>BogoMIPS</span>	: 67.03
</span>Features</span>	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
</span>CPU</span> implementer	: 0x41
</span>CPU</span> architecture: 7
</span>CPU</span> variant	: 0x0
</span>CPU</span> part	: 0xc07
</span>CPU</span> revision	: 5
</span>
</span>processor</span>	: 2
</span>model</span> name	: ARMv7 Processor rev 5 (v7l)
</span>BogoMIPS</span>	: 67.03
</span>Features</span>	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
</span>CPU</span> implementer	: 0x41
</span>CPU</span> architecture: 7
</span>CPU</span> variant	: 0x0
</span>CPU</span> part	: 0xc07
</span>CPU</span> revision	: 5
</span>
</span>processor</span>	: 3
</span>model</span> name	: ARMv7 Processor rev 5 (v7l)
</span>BogoMIPS</span>	: 67.03
</span>Features</span>	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
</span>CPU</span> implementer	: 0x41
</span>CPU</span> architecture: 7
</span>CPU</span> variant	: 0x0
</span>CPU</span> part	: 0xc07
</span>CPU</span> revision	: 5
</span>
</span>Hardware</span>	: Generic DT based system
</span>Revision</span>	: 0000
</span>Serial</span>		: 0000000000000000
</span>root@OpenWrt:~#</span> ldd</span> --version
</span>musl</span> libc (armhf)
</span>Version</span> 1.1.24
</span>Dynamic</span> Program Loader
</span>Usage:</span> ldd </span>[</span>options</span>] [</span>--</span>]</span> pathname
</span></code></pre>
So it appears a 4-core ARMv7 CPU with hardware floating-point support, running a MUSL-based distro.</p>
Cross-compiling</h2>
Rust supports dozens of targets, but ARMv7 is pretty common, so it's hopefully well-supported.
I'm not sure how the target is called, so rustup target list</code> is handy:</p>
$</span> rustup target list | </span>rg</span> armv7
</span>armv7-linux-androideabi
</span>armv7-unknown-linux-gnueabi
</span>armv7-unknown-linux-gnueabihf
</span>armv7-unknown-linux-musleabi
</span>armv7-unknown-linux-musleabihf </span># sounds like a winner
</span>armv7a-none-eabi
</span>armv7r-none-eabi
</span>armv7r-none-eabihf
</span>$</span> rustup target add armv7-unknown-linux-musleabihf
</span>info:</span> downloading component '</span>rust-std</span>' for '</span>armv7-unknown-linux-musleabihf</span>'
</span>info:</span> installing component '</span>rust-std</span>' for '</span>armv7-unknown-linux-musleabihf</span>'
</span></code></pre>
Great, let's try it!</p>
$</span> cargo build</span> --release --target</span> armv7-unknown-linux-musleabihf
</span>   </span>[snip]
</span>error:</span> linking with `</span>cc</span>` failed: exit status: 1
</span>   </span>[snip</span> half a screenful of errors]
</span></code></pre>
Oh, we also need a linker.
This would normally be a version of the BFD linker (ld</code>).
Unlike clang</code> and rustc</code>, gcc</code> and ld</code> only support one target at a time, so I'd have to track down or compile a compatible version.
My Linux distro actually has one, available in AUR as muslcc-arm-linux-musleabihf-cross-bin</code>.</p>
But since we're compiling pure-Rust code — did you notice the fancy rustls</code> feature of trust-dns-resolver</code>? — the LLVM linker, lld</code>, will do the trick with less work.</p>
The correct incantation is then:</p>
$</span> CARGO_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_LINKER=rust-lld cargo build</span> --release --target</span> armv7-unknown-linux-musleabihf
</span>   </span>[snip]
</span>    </span>Finished</span> release </span>[</span>optimized</span>]</span> target(s) </span>in</span> 9.50s
</span></code></pre>
Is that all?
That was surpisingly easy!</p>
And by the way, you can also set the linker in .config/cargo.toml</code>, like this:</p>
[target.armv7-unknown-linux-musleabihf]
</span>linker </span>= "</span>rust-lld</span>"
</span></code></pre>
Testing</h2>
We could test using QEMU, but that seems too much of a hassle, as the real hardware is already available.</p>
$</span> scp target/armv7-unknown-linux-musleabihf/release/delve root@192.168.2.1:</span>~
</span>scp:</span> Connection closed
</span></code></pre>
The router is running OpenSSH 8.0p1, but scp</code> is deprecated, and the version on my PC uses SFTP by default.
The -O</code> flag reverts to the deprecated protocol:</p>
$</span> scp</span> -O</span> target/armv7-unknown-linux-musleabihf/release/delve root@192.168.2.1:</span>~
</span>delve</span>                                        100% 8598KB 176.1KB/s   00:48
</span></code></pre>
That's a larger binary than I'd like, but 176 KB/s seems pretty slow.
I know my network is faster that this, but it could be the file system:</p>
root@OpenWrt:~#</span> mount
</span>/dev/root</span> on /rom type squashfs (ro,relatime)
</span>proc</span> on /proc type proc (rw,nosuid,nodev,noexec,noatime)
</span>sysfs</span> on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
</span>tmpfs</span> on /tmp type tmpfs (rw,nosuid,nodev,noatime)
</span>/dev/mtdblock14</span> on /overlay type jffs2 (rw,noatime)
</span>overlayfs:/overlay</span> on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
</span>tmpfs</span> on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
</span>devpts</span> on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
</span>debugfs</span> on /sys/kernel/debug type debugfs (rw,noatime)
</span></code></pre>
Oh, okay, let's copy it to /tmp</code> instead:</p>
$</span> scp</span> -O</span> target/armv7-unknown-linux-musleabihf/release/delve root@192.168.2.1:/tmp
</span>delve</span>                                        100% 8598KB  11.6MB/s   00:00
</span></code></pre>
More importantly, does it work?</p>
root@OpenWrt:~#</span> /tmp/delve news.ycombinator.com
</span>209.216.230.240
</span></code></pre>
Honestly, I was more suprised to see it working than you are.</p>
What about the binary size?</h2>
Our executable packs in quite a bit: Tokio, an async DNS client, and a TLS implementation for DNS-over-TLS.
But at 8.5 MB, it's pretty large for a Wi-Fi router with 128 MB of flash.</p>
Let's see if we can get it to weigh less, testing with the PC version.
I know that a large part of the binary must be the symbols.
I'd normally use strip -s</code>, but cargo</code> can do this by itself.
This is also a good excuse to try the custom profiles feature.</p>
Let's start by adding a new profile to Cargo.toml</code>:</p>
[profile.minsize]
</span>inherits </span>= "</span>release</span>"
</span></code></pre>
, then build it with cargo build --profile minsize</code>.</p>
There's a lot of resources out there with tips for reducing the binary sizes (it's a common complaint), so I'll just list each thing I've tried, incrementally:</p>

baseline: 8.5 MB</li>
strip = true</code>: 3.2 MB</li>
lto = "thin"</code>: 3.1 MB</li>
lto = "fat"</code>: 2.6 MB (didn't expect this!)</li>
opt-level = "s"</code>: 2.3 MB (didn't expect this either)</li>
panic = "abort"</code>: 2.1 MB</li>
switch to the single-thread tokio</code> runtime: 2.0 MB</li>
cargo build -Z build-std=panic_abort,std --profile minsize --target x86_64-unknown-linux-gnu</code> (you'll need nightly or RUSTC_BOOTSTRAP=1</code> for this): 1.9 MB</li>
disable the system-config</code> feature of trust-dns-resolver</code>: 1.9 MB</li>
</ul>
Rebuilding it for the router, we get a 1.4 MB binary, which is still a bit large, but workable.</p>
Bonus: packaging for OpenWrt</h2>
I was curious about making a binary package for OpenWrt, which appears to use a format inspired by Debian (ipkg</code>).
Reading the docs isn't fun, but we can find where the package manager (okpg</code>) downloads stuff from (/etc/opkg/distfeeds.conf</code>), then get a random package from there.</p>
I picked attr_20170915-1_arm_cortex-a7_neon-vfpv4.ipk</code>, which appears to be a .tar.gz</code> archive, containing three files:</p>

control.tar.gz</code></li>
data.tar.gz</code></li>
debian-binary</code>, a text file saying 2.0</code></li>
</ul>
control.tar.gz</code> has one metadata file, control</code>, and some pre- and post-install scripts we can copy over or ignore:</p>
Package: attr
</span>Version: 20170915-1
</span>Depends: libc, libattr
</span>Source: feeds/packages/utils/attr
</span>License: LGPL-2.1 GPL-2.0
</span>LicenseFiles: doc/COPYING doc/COPYING.LGPL
</span>Section: utils
</span>Maintainer: Maxim Storchak <m.storchak@gmail.com>
</span>Architecture: arm_cortex-a7_neon-vfpv4
</span>Installed-Size: 11080
</span>Description:  Extended attributes support
</span> This package provides xattr manipulation utilities
</span> - attr
</span> - getfattr
</span> - setfattr
</span></code></pre>
Interestingly, my router uses vfpv4</code> packages, even though only vfpv3</code> appears in /proc/cpuinfo</code>.
Well, whatever makes it happy.
The Installed-Size</code> field is also strange, as it doesn't seem to match the size of the files.
It's optional optional, so it doesn't matter too much.</p>
We can make a similar file:</p>
Package: delve
</span>Version: 0.0.1
</span>Depends: libc
</span>License: MIT
</span>Section: utils
</span>Architecture: arm_cortex-a7_neon-vfpv4
</span>Installed-Size: 1414464
</span>Description: DNS testing tool
</span></code></pre>
Finally, data.tar.gz</code> contains the installed tree.
Putting things together, we have:</p>
.
</span>├── control
</span>├── control.tar.gz
</span>├── data.tar.gz
</span>├── debian-binary
</span>├── delve_0.0.1_arm_cortex-a7_neon-vfpv4.ipk
</span>└── usr
</span>    └── bin
</span>        └── delve
</span>
</span>2 directories, 6 files
</span></code></pre>
And on the router:</p>
root@OpenWrt:/tmp#</span> opkg install delve_0.0.1_arm_cortex-a7_neon-vfpv4.ipk
</span>Installing</span> delve (0.0.1) </span>to</span> root...
</span>Configuring</span> delve.
</span>root@OpenWrt:/tmp#</span> delve sci-hub.se
</span>186.2.163.219
</span></code></pre>
If you ever try this, make sure to actually compress the archives using gzip</code> (or tar czf</code>).
I forgot that tar</code> only guesses the format when extracting, and opkg install</code> greeted me with a fun Segmentation fault</code> message.</p>
Naming apologies</h2>
I just realized that the delve</code> name is already taken by a Go debugger.
Not that it matters, as nobody will be using this, but it's a common source of complaints.</p>
Final words</h2>
To sum up, we wrote a simple DNS client, tweaked things a little to reduce the binary size, cross-compiled it for an ARM Wi-Fi router, packaged and tested it there.
Surprisingly, it actually worked.</p>
Thanks for staying with me, and if you found this interesting please consider buying me a coffee</a>.</p>


Some eye candy
2022-03-30T00:00:00+00:00
</a></p>
False color composite of Sentinel-1 stack from the end of May 2021 around the Saloum River in Senegal.
VH (cross-polarization) backscatter, VV (co-polarization) backscatter and VV coherence, contrast-stretched and mapped to RGB.
Data courtesy of the European Space Agency.</p>
Click the image for a larger (23 MB) version.</p>


On self-modifying executables in Rust
2022-01-23T00:00:00+00:00
This is a short post prompted by a question on /r/rust</a> from May 2019, asking about a way for a Rust program to modify itself.</p>
Think of how much fun it would be to store a high score list directly inside the a executable.
When you make a copy (on floppy disk, to make things more realistic) and pass it to your friends, your high scores are persisted.</p>
Errata</h2>

the variable should probably be marked as #[repr(C)]</code>.</li>
as pointed out</a> by a reader, unsafe { ptr::read_volatile(&RUN_COUNT) }</code> is a better alternative to static mut</code>.</li>
you probably shouldn't do this except as a party trick.</li>
</ul>
Diving right in</h2>
As a proof of concept, we'll write a program that counts how many times it has been run.
This is hardly the only approach, but probably the simplest way is to store the counter in a separate section of the executable.</p>
#[</span>link_section </span>= "</span>count</span>"]
</span>#[</span>used</span>]
</span>static mut </span>RUN_COUNT</span>: </span>u32 </span>= </span>0</span>;
</span></code></pre>
#[link_section]</code> tells the linker to put the variable in a section called count</code>.
#[used]</code></a> means that the variable must be kept by the linker, even if could be optimized away.
Similarly, mut</code> is required because otherwise the compiler will propagate the variable value to the code that reads it.</p>
I'm not sure if the combination above the best way to accomplish this, or if it's guaranteed to work.
It was the only working method I found back then, and it still seems to be the case today.</p>
Since the code is so short, I'll just paste the rest of it here:</p>
use </span>memmap2::MmapOptions;
</span>use </span>object::{File, Object, ObjectSection};
</span>use </span>std::env;
</span>use </span>std::error::Error;
</span>use </span>std::fs::{</span>self</span>, OpenOptions};
</span>
</span>fn </span>get_section</span>(</span>file</span>: &File, </span>name</span>: &</span>str</span>) -> Option<(</span>u64</span>, </span>u64</span>)> {
</span>    </span>for</span> section in file.</span>sections</span>() {
</span>        </span>match</span> section.</span>name</span>() {
</span>            Ok(n) </span>if</span> n == name => {
</span>                </span>return</span> section.</span>file_range</span>();
</span>            }
</span>            _ => {}
</span>        }
</span>    }
</span>    None
</span>}
</span>
</span>fn </span>main</span>() -> Result<(), Box<dyn Error>> {
</span>    </span>let</span> run_count = </span>unsafe </span>{ </span>RUN_COUNT </span>};
</span>    println!("</span>Previous run count: </span>{}</span>", run_count);
</span>    </span>let</span> exe = env::current_exe()?;
</span>    </span>let</span> tmp = exe.</span>with_extension</span>("</span>tmp</span>");
</span>    fs::copy(&exe, &tmp)?;
</span>
</span>    </span>let</span> file = OpenOptions::new().</span>read</span>(</span>true</span>).</span>write</span>(</span>true</span>).</span>open</span>(&tmp)?;
</span>    </span>let mut</span> buf = </span>unsafe </span>{ MmapOptions::new().</span>map_mut</span>(&file) }?;
</span>    </span>let</span> file = File::parse(&*buf)?;
</span>
</span>    </span>if let </span>Some(range) = </span>get_section</span>(&file, "</span>count</span>") {
</span>        assert_eq!(range.</span>1</span>, </span>4</span>);
</span>        </span>let</span> base = range.</span>0 </span>as </span>usize</span>;
</span>        buf[base..(base + </span>4</span>)].</span>copy_from_slice</span>(&(run_count + </span>1</span>).</span>to_ne_bytes</span>());
</span>
</span>        </span>let</span> perms = fs::metadata(&exe)?.</span>permissions</span>();
</span>        fs::set_permissions(&tmp, perms)?;
</span>        fs::rename(&tmp, &exe)?;
</span>    } </span>else </span>{
</span>        fs::remove_file(&tmp)?;
</span>    }
</span>
</span>    Ok(())
</span>}
</span></code></pre>
We look up the current executable, and map it into memory using memmap2</code></a>.
Then we get a list of sections using the object</code></a> crate, search for one called count</code> — beware of libraries using the same name! — then replace the variable value in the memory-mapped view of the file.</p>
All this happens on a copy of our program.
Self-modifying programs were fine under MS-DOS, but modern operating systems won't let it fly.
Renaming or overwriting is fine, since the original file is still unchanged.</p>
You can see a nice diagram showing the ELF (the executable format on my platform) structure here</a>.
But we're not here to learn about executable formats.
We're here to clear out a tiny bit of backlog and see a working example, so does it work?</p>
$</span> cargo run</span> --release
</span>   </span>Compiling</span> self-modify v0.1.0 (/home/grayshade/Projects/self-modify)
</span>    </span>Finished</span> release </span>[</span>optimized</span>]</span> target(s) </span>in</span> 0.30s
</span>     </span>Running </span>`</span>target/release/self-modify</span>`
</span>Previous</span> run count: 0
</span>$</span> target/release/self-modify
</span>Previous</span> run count: 1
</span>$</span> target/release/self-modify
</span>Previous</span> run count: 2
</span>$</span> target/release/self-modify
</span>Previous</span> run count: 3
</span></code></pre>
It does!</p>
Note that I'm on Linux.
The object</code> crate should work with MacOS and Windows executables too, but on Windows the fs::rename</code> call will probably fail.
There you can rename the current executable to some temporary name, but I don't want to deal with that stuff again</a>.</p>
Bonus section</em></h2>
I mentioned above that declaring our variable as mut</code> is necessary because the compiler will optimize it otherwise.
How did I figure that out?</p>
First, let's simplify the code and give the variable a more noticeable value:</p>
#[</span>link_section </span>= "</span>count</span>"]
</span>#[</span>used</span>]
</span>static mut </span>RUN_COUNT</span>: </span>u32 </span>= </span>0x55aa55aa</span>;
</span>
</span>fn </span>main</span>() {
</span>    </span>let</span> run_count = </span>unsafe </span>{ </span>RUN_COUNT </span>};
</span>    println!("</span>Previous run count: </span>{}</span>", run_count);
</span>}
</span></code></pre>
$</span> cargo build</span> --release
</span>$</span> objdump</span> -d</span> target/release/self-modify
</span></code></pre>
[snip]
</span>0000000000011160 </span><_ZN11self_modify4main17hd56d204cd8f62812E>:
</span>   </span>11160</span>:       </span>48 83 </span>ec </span>48             </span>sub    </span>$</span>0x48</span>,</span>%</span>rsp
</span>   </span>11164</span>:       8b </span>05 </span>ee 7e </span>03 00       </span>mov    </span>0x37eee</span>(%</span>rip</span>)</span>,</span>%</span>eax        </span># </span>49058 </span><__TMC_END__>
</span>   1116a:       </span>89 44 24 04             </span>mov    </span>%</span>eax</span>,</span>0x4</span>(%</span>rsp</span>)
</span>   1116e:       </span>48 </span>8d </span>44 24 04          </span>lea    </span>0x4</span>(%</span>rsp</span>)</span>,</span>%</span>rax
</span>   </span>11173</span>:       </span>48 89 44 24 08          </span>mov    </span>%</span>rax</span>,</span>0x8</span>(%</span>rsp</span>)
</span>   </span>11178</span>:       </span>48 </span>8b </span>05 49 51 03 00    </span>mov    </span>0x35149</span>(%</span>rip</span>)</span>,</span>%</span>rax        </span># 462c8 <_etext</span>+</span>0x423</span>>
</span>   1117f:       </span>48 89 44 24 10          </span>mov    </span>%</span>rax</span>,</span>0x10</span>(%</span>rsp</span>)
</span>   </span>11184</span>:       </span>48 </span>8d </span>05 </span>e5 </span>57 03 00    </span>lea    </span>0x357e5</span>(%</span>rip</span>)</span>,</span>%</span>rax        </span># </span>46970 </span><_DYNAMIC</span>+</span>0x280</span>>
</span>   1118b:       </span>48 89 44 24 18          </span>mov    </span>%</span>rax</span>,</span>0x18</span>(%</span>rsp</span>)
</span>   </span>11190</span>:       </span>48 </span>c7 </span>44 24 20 02 00    </span>movq   </span>$</span>0x2</span>,</span>0x20</span>(%</span>rsp</span>)
</span>   </span>11197</span>:       </span>00 00
</span>   </span>11199</span>:       </span>48 </span>c7 </span>44 24 28 00 00    </span>movq   </span>$</span>0x0</span>,</span>0x28</span>(%</span>rsp</span>)
</span>   111a0:       </span>00 00
</span>   111a2:       </span>48 </span>8d </span>44 24 08          </span>lea    </span>0x8</span>(%</span>rsp</span>)</span>,</span>%</span>rax
</span>   111a7:       </span>48 89 44 24 38          </span>mov    </span>%</span>rax</span>,</span>0x38</span>(%</span>rsp</span>)
</span>   111ac:       </span>48 </span>c7 </span>44 24 40 01 00    </span>movq   </span>$</span>0x1</span>,</span>0x40</span>(%</span>rsp</span>)
</span>   111b3:       </span>00 00
</span>   111b5:       </span>48 </span>8d 7c </span>24 18          </span>lea    </span>0x18</span>(%</span>rsp</span>)</span>,</span>%</span>rdi
</span>   111ba:       ff </span>15 48 </span>4f </span>03 00       </span>call   </span>*</span>0x34f48</span>(%</span>rip</span>)        # </span>46108 </span><_etext</span>+</span>0x263</span>>
</span>   111c0:       </span>48 83 </span>c4 </span>48             </span>add    </span>$</span>0x48</span>,</span>%</span>rsp
</span>   111c4:       c3                      </span>ret
</span>        ...
</span>[snip]
</span></code></pre>
Ookay, assembly is hard to read.
What happens without the mut</code>?</p>
0000000000011160 </span><_ZN11self_modify4main17hd56d204cd8f62812E>:
</span>   </span>11160</span>:       </span>48 83 </span>ec </span>48             </span>sub    </span>$</span>0x48</span>,</span>%</span>rsp
</span>   </span>11164</span>:       c7 </span>44 24 04 </span>aa </span>55 </span>aa    movl   </span>$</span>0x55aa55aa</span>,</span>0x4</span>(%</span>rsp</span>)
</span>   1116b:       </span>55
</span>   1116c:       </span>48 </span>8d </span>44 24 04          </span>lea    </span>0x4</span>(%</span>rsp</span>)</span>,</span>%</span>rax
</span>   </span>11171</span>:       </span>48 89 44 24 08          </span>mov    </span>%</span>rax</span>,</span>0x8</span>(%</span>rsp</span>)
</span>   </span>11176</span>:       </span>48 </span>8b </span>05 </span>4b </span>51 03 00    </span>mov    </span>0x3514b</span>(%</span>rip</span>)</span>,</span>%</span>rax        </span># 462c8 <_etext</span>+</span>0x423</span>>
</span>   1117d:       </span>48 89 44 24 10          </span>mov    </span>%</span>rax</span>,</span>0x10</span>(%</span>rsp</span>)
</span>   </span>11182</span>:       </span>48 </span>8d </span>05 </span>e7 </span>57 03 00    </span>lea    </span>0x357e7</span>(%</span>rip</span>)</span>,</span>%</span>rax        </span># </span>46970 </span><_DYNAMIC</span>+</span>0x280</span>>
</span>   </span>11189</span>:       </span>48 89 44 24 18          </span>mov    </span>%</span>rax</span>,</span>0x18</span>(%</span>rsp</span>)
</span>   1118e:       </span>48 </span>c7 </span>44 24 20 02 00    </span>movq   </span>$</span>0x2</span>,</span>0x20</span>(%</span>rsp</span>)
</span>   </span>11195</span>:       </span>00 00
</span>   </span>11197</span>:       </span>48 </span>c7 </span>44 24 28 00 00    </span>movq   </span>$</span>0x0</span>,</span>0x28</span>(%</span>rsp</span>)
</span>   1119e:       </span>00 00
</span>   111a0:       </span>48 </span>8d </span>44 24 08          </span>lea    </span>0x8</span>(%</span>rsp</span>)</span>,</span>%</span>rax
</span>   111a5:       </span>48 89 44 24 38          </span>mov    </span>%</span>rax</span>,</span>0x38</span>(%</span>rsp</span>)
</span>   111aa:       </span>48 </span>c7 </span>44 24 40 01 00    </span>movq   </span>$</span>0x1</span>,</span>0x40</span>(%</span>rsp</span>)
</span>   111b1:       </span>00 00
</span>   111b3:       </span>48 </span>8d 7c </span>24 18          </span>lea    </span>0x18</span>(%</span>rsp</span>)</span>,</span>%</span>rdi
</span>   111b8:       ff </span>15 </span>4a 4f </span>03 00       </span>call   </span>*</span>0x34f4a</span>(%</span>rip</span>)        # </span>46108 </span><_etext</span>+</span>0x263</span>>
</span>   111be:       </span>48 83 </span>c4 </span>48             </span>add    </span>$</span>0x48</span>,</span>%</span>rsp
</span>   111c2:       c3                      </span>ret
</span>        ...
</span></code></pre>
Did you spot the movl   $0x55aa55aa,0x4(%rsp)</code> on the second line?
That's our value, inlined in the compiled code.</p>
To be honest, we're getting a lot of extra code in there.
Instead, it's better to get rid of the println!</code>, simplify the code as much as possible and paste it on godbolt.org.
Click here</a> to see how that looks.</p>


Mapping my walks with OSRM and Rust
2022-01-15T00:00:00+00:00
Last week</a>, we looked at the data from my smartwatch app and how to snap it to the OpenStreetMap roads.
If you haven't already, consider reading that post before moving on.
This time, we'll be writing some code in the Rust</a> programming language.
Keep in mind that this isn't exactly production-grade code, but rather something you would normally write in Python in one afternoon.
I'll link to the crates I'm using for the benefit of readers less familiar with the Rust ecosystem.</p>
Importing the data</h2>
If you recall from last time, our data points include a timestamp, the latitude and longitude, and an accuracy radius.
We can define our point type as follows:</p>
pub struct </span>Point {
</span>    </span>pub </span>time</span>: OffsetDateTime,
</span>    </span>pub </span>geom</span>: geo_types::Point<</span>f64</span>>,
</span>    </span>pub </span>radius</span>: </span>f32</span>,
</span>}
</span></code></pre>
The OffsetDateTime</code> and Point<T></code> types are from the time</code></a> and geo-types</code></a> crates, respectively.
We'll also need zip</code></a> and csv</code></a> in order to parse the exported file.</p>
Let's see what reading the file looks like:</p>
pub fn </span>read_archive</span>(</span>path</span>: &Path) -> Result<Vec<Point>, Error> {
</span>    </span>let</span> file = File::open(path)?;
</span>    </span>let mut</span> archive = ZipArchive::new(file)?;
</span>    </span>let mut</span> latitudes = Vec::new();
</span>    </span>let mut</span> record = StringRecord::new();
</span>    {
</span>        </span>let</span> reader = archive.</span>by_name</span>("</span>raw_location_latitude.csv</span>")?;
</span>        </span>let mut</span> reader = Reader::from_reader(reader);
</span>        </span>while</span> reader.</span>read_record</span>(&</span>mut</span> record)? {
</span>            latitudes.</span>push</span>(</span>get_array_first_value</span>(&record[</span>2</span>])?);
</span>        }
</span>    }
</span>    </span>let</span> n = latitudes.</span>len</span>();
</span>    </span>let mut</span> times = Vec::with_capacity(n);
</span>    </span>let mut</span> longitudes = Vec::with_capacity(n);
</span>    {
</span>        </span>let</span> reader = archive.</span>by_name</span>("</span>raw_location_longitude.csv</span>")?;
</span>        </span>let mut</span> reader = Reader::from_reader(reader);
</span>        </span>while</span> reader.</span>read_record</span>(&</span>mut</span> record)? {
</span>            </span>let</span> time = OffsetDateTime::parse(&record[</span>0</span>], &well_known::Rfc3339)?;
</span>            times.</span>push</span>(time);
</span>            longitudes.</span>push</span>(</span>get_array_first_value</span>(&record[</span>2</span>])?);
</span>        }
</span>    }
</span>    assert_eq!(longitudes.</span>len</span>(), n);
</span>    </span>let mut</span> radiuses = Vec::with_capacity(n);
</span>    {
</span>        </span>let</span> reader = archive.</span>by_name</span>("</span>raw_location_horizontal-radius.csv</span>")?;
</span>        </span>let mut</span> reader = Reader::from_reader(reader);
</span>        </span>while</span> reader.</span>read_record</span>(&</span>mut</span> record)? {
</span>            radiuses.</span>push</span>(</span>get_array_first_value</span>(&record[</span>2</span>])?);
</span>        }
</span>    }
</span>    assert_eq!(radiuses.</span>len</span>(), n);
</span>    </span>let</span> points = izip!(times, latitudes, longitudes, radiuses)
</span>        .</span>map</span>(|(</span>time</span>, </span>lat</span>, </span>lon</span>, </span>radius</span>)| {
</span>            </span>let</span> geom = geo_types::Point::new(lon, lat);
</span>            Point { time, geom, radius }
</span>        })
</span>        .</span>collect</span>();
</span>    Ok(points)
</span>}
</span>
</span>fn </span>get_array_first_value</span><T>(</span>s</span>: &</span>str</span>) -> Result<T, </span>T::</span>Err>
</span>where
</span>    T: FromStr,
</span>    </span>T::</span>Err: std::fmt::Debug,
</span>{
</span>    </span>let</span> p = s.</span>find</span>(|</span>c</span>| c == '</span>]</span>' || c == '</span>,</span>').</span>expect</span>("</span>expected array</span>");
</span>    s[</span>1</span>..p].</span>parse</span>()
</span>}
</span></code></pre>
The code is somewhat unwieldly because it needs to read three different CSVs from the archive.
The ZIP reader needs to seek into the archive, so we can't read all of them at once.
If you're familiar with Rust, that constraint is expressed in the type system by Reader</code> having a mutable reference to ZipArchive</code>.
Instead, we open the files one at a time and scan through each.</p>
We need to remember to read the timestamps and there is one extra complication.
The values are written as JSON-like arrays (e.g. [44.XXX,657.XXX]</code>), but only the first one makes sense.
Instead of reaching for a full-blown JSON parser, we simply look for a bracket or comma and parse the number we find there.</p>
The izip</code> macro comes from itertools</code></a> and lets us iterate over the multiple collections at once.</p>
Note that the implementation keeps all the points in memory (twice, even).
Normally I would dump the points to disk and read them back, but it would make the code harder to follow in a blog post.
In any case, this doesn't dominate our memory usage.</p>
Calling into OSRM</h2>
As mentioned last time, OSRM has an HTTP server, so we'll use reqwest</code></a> to call into it and serde</code></a> to deserialize the JSON responses.</p>
First of all, we define some structs that roughly match the OSRM response:</p>
#[</span>derive</span>(Deserialize, Debug)]
</span>pub struct </span>MatchResponse {
</span>    </span>pub </span>matchings</span>: Vec<Matching>,
</span>}
</span>
</span>#[</span>derive</span>(Deserialize, Debug)]
</span>pub struct </span>Matching {
</span>    </span>pub </span>geometry</span>: String,
</span>}
</span></code></pre>
Then our client:</p>
pub struct </span>OsrmClient {
</span>    </span>client</span>: Client,
</span>    </span>base_url</span>: String,
</span>}
</span>
</span>impl </span>OsrmClient {
</span>    </span>pub fn </span>new</span>(</span>base_url</span>: String) -> </span>Self </span>{
</span>        </span>let</span> client = Client::new();
</span>        </span>Self </span>{ client, base_url }
</span>    }
</span>
</span>    </span>pub fn </span>match_map</span>(
</span>        &</span>self</span>,
</span>        </span>profile</span>: &</span>str</span>,
</span>        </span>points</span>: &[Point],
</span>    ) -> Result<MultiLineString<</span>f64</span>>, Error> {
</span>        </span>let mut</span> q = format!("</span>{}</span>match/v1/</span>{profile}</span>/polyline6(</span>", </span>self</span>.base_url);
</span>        </span>let mut</span> timestamps = String::new();
</span>        </span>let mut</span> radiuses = String::from("</span>&radiuses=</span>");
</span>        </span>let</span> coordinates = LineString::from_iter(points.</span>iter</span>().</span>map</span>(|</span>p</span>| p.geom));
</span>        </span>for</span> point in points {
</span>            write!(timestamps, "</span>{}</span>;</span>", point.time.</span>unix_timestamp</span>())?;
</span>            write!(radiuses, "</span>{}</span>;</span>", point.radius)?;
</span>        }
</span>        timestamps.</span>pop</span>();
</span>        radiuses.</span>pop</span>();
</span>        </span>let</span> coordinates =
</span>            polyline::encode_coordinates(coordinates, </span>6</span>).</span>map_err</span>(|</span>e</span>| anyhow!(e))?;
</span>        </span>let</span> pe = percent_encoding::percent_encode(
</span>            coordinates.</span>as_bytes</span>(),
</span>            percent_encoding::</span>NON_ALPHANUMERIC</span>,
</span>        );
</span>        write!(
</span>            q,
</span>            "</span>{}</span>)?geometries=polyline6&tidy=true&steps=false&timestamps=</span>",
</span>            pe
</span>        )?;
</span>        q.</span>push_str</span>(&timestamps);
</span>        q.</span>push_str</span>(&radiuses);
</span>        </span>let</span> res = </span>self</span>.client.</span>get</span>(&q).</span>send</span>()?.json::<MatchResponse>()?;
</span>        </span>let</span> mls = MultiLineString(
</span>            res.matchings
</span>                .</span>into_iter</span>()
</span>                .</span>map</span>(|</span>m</span>| polyline::decode_polyline(&m.geometry, </span>6</span>))
</span>                .collect::<Result<Vec<_>, _>>()
</span>                .</span>map_err</span>(|</span>e</span>| anyhow!(e))?,
</span>        );
</span>        Ok(mls)
</span>    }
</span>}
</span></code></pre>
Again, this is quick and dirty code.
We use the polyline</code></a> encoding of the coordinates and the corresponding crate</a> for it.
Because polyline</code> uses an encoding that contains non-alphanumeric characters, we need to encode them using percent-encoding</code></a>.
This crate doesn't have a standard error type (it uses String</code>s), so we adapt them using anyhow</code></a>, which is also appears in the rest of the code.</p>
Saving a FlatGeobuf</h2>
This isn't really required for our purposes, but I want to keep a copy of the points in a format better suited for using later.
I'd normally use GeoPackage</a>, but I want to give FlatGeobuf</a> a try.
This is a newer format, with a simpler structure and which should be simpler to write, since a pure-Rust</a> implementation is available.
Fortunately, GDAL</a> also has a reader, so it should be compatible with every application I care about at the moment.</p>
flatgeobuf</code> has a slightly strange API, but it's not too bad and I've seen worse.
It integrates closely with geozero</code></a>, which is a visitor-based API for zero-copy processing of geospatial data.</p>
fn </span>export_fgb</span>(</span>name</span>: &</span>str</span>, </span>file</span>: &Path, </span>points</span>: &[Point]) -> anyhow::Result<()> {
</span>    </span>let mut</span> fgb = FgbWriter::create(name, GeometryType::Point, |_, _| {})?;
</span>    fgb.</span>set_crs</span>(</span>4326</span>, |</span>_fbb</span>, </span>_crs</span>| {});
</span>    fgb.</span>add_column</span>("</span>time</span>", ColumnType::DateTime, |_, </span>col</span>| {
</span>        col.nullable = </span>false</span>;
</span>    });
</span>    fgb.</span>add_column</span>("</span>radius</span>", ColumnType::Float, |_, </span>col</span>| {
</span>        col.nullable = </span>false</span>;
</span>    });
</span>    </span>for</span> point in points {
</span>        </span>let</span> time = point.time.</span>format</span>(&well_known::Rfc3339)?;
</span>        fgb.</span>add_feature_geom</span>(Geometry::Point(point.geom), |</span>feat</span>| {
</span>            feat.</span>property</span>(</span>0</span>, "</span>time</span>", &ColumnValue::DateTime(&time))
</span>                .</span>unwrap</span>();
</span>            feat.</span>property</span>(</span>1</span>, "</span>radius</span>", &ColumnValue::Float(point.radius))
</span>                .</span>unwrap</span>();
</span>        })?;
</span>    }
</span>    </span>let</span> out_file = File::create(file)?;
</span>    fgb.</span>write</span>(&</span>mut </span>BufWriter::new(out_file))?;
</span>    Ok(())
</span>}
</span></code></pre>
We create a writer, configure the CRS</a> and define the two columns.
Afterwards, we can iterate over our points, create features from them, and set the column values.</p>
Reprojecting the data</h2>
I want to display the resulting tracks on a map, but the (latitude, longitude) coordinates aren't ideal, since they introduce quite a bit of distorsion:</p>
</a></p>
Instead, we will reproject our points to the Web Mercator</a> (or, more formally, EPSG:3857) projection.
While it's not really accurate, it's good enough at this latitude for my purposes.
Web Mercator has been popularized by Google Maps, and the look of it should be familiar to anyone who's ever seen an interactive map on the Web.</p>
</a></p>
For this, I'd normally use GDAL</code> or PROJ</code></a>, but GDAL's approach to threading isn't the most fortunate.
We could probably use the proj</code></a> bindings, but even those bring quite a bit of complexity.</p>
The conversion to Web Mercator should be pretty easy.
You can see a pair of formulas in the link above, but they use different bounds, and it's not obvious how to do it correctly.
Stack Overflow produces a fair bit of confusion</a> and two</a> answers</a>, but neither of them agrees with PROJ</code>.
After a bit of hit-and miss, I realized that the three implementations are using different values for the Earth radius.
I believe the correct value to use in this case is 6_378_137 m</code>.
My final implementation gives the same result (up to the precision limit) as PROJ</code> for a couple of points I've tried, so I hope it's correct.</p>
pub fn </span>epsg_4326_to_3857</span>(</span>x</span>: </span>f64</span>, </span>y</span>: </span>f64</span>) -> (</span>f64</span>, </span>f64</span>) {
</span>    </span>const </span>WGS84_EQUATORIAL_RADIUS</span>: </span>f64 </span>= </span>6_378_137.0</span>;
</span>    </span>const </span>MAX_LATITUDE</span>: </span>f64 </span>= </span>85.06</span>;
</span>
</span>    </span>let</span> x = x.</span>to_radians</span>();
</span>    </span>let</span> y = </span>if</span> y > </span>MAX_LATITUDE </span>{
</span>        std::f64::consts::</span>PI
</span>    } </span>else if</span> y < -</span>MAX_LATITUDE </span>{
</span>        -std::f64::consts::</span>PI
</span>    } </span>else </span>{
</span>        </span>let</span> y = y.</span>to_radians</span>() / </span>2.0 </span>+ std::f64::consts::</span>FRAC_PI_4</span>;
</span>        y.</span>tan</span>().</span>ln</span>()
</span>    };
</span>    (x * </span>WGS84_EQUATORIAL_RADIUS</span>, y * </span>WGS84_EQUATORIAL_RADIUS</span>)
</span>}
</span></code></pre>
There is also a helper that reprojects an entire track:</p>
fn </span>reproject_route</span>(</span>mls</span>: &</span>mut </span>MultiLineString<</span>f64</span>>) {
</span>    mls.</span>iter_mut</span>().</span>for_each</span>(|</span>ls</span>| {
</span>        ls.</span>0.</span>iter_mut</span>().</span>for_each</span>(|</span>p</span>| {
</span>            </span>let </span>(x, y) = geo::epsg_4326_to_3857(p.x, p.y);
</span>            p.x = x;
</span>            p.y = y;
</span>        });
</span>    });
</span>}
</span></code></pre>
Rasterization</h2>
Next, I want to display the tracks returned by OSRM as images.
Converting from vector data to an image is called rasterization.
I had a couple of options here (it's less flexible, but GDAL can do it), but I decided to a recently-published crate called rasterize</code></a>.</p>
First of all, rasterize</code> represents paths as a list of segments, not points, so we need a conversion function for that:</p>
fn </span>mls_to_path</span>(</span>mls</span>: MultiLineString<</span>f64</span>>) -> rasterize::Path {
</span>    </span>let</span> subpaths = mls
</span>        .</span>into_iter</span>()
</span>        .</span>filter_map</span>(|</span>ls</span>| {
</span>            SubPath::new(
</span>                ls.</span>into_iter</span>()
</span>                    .</span>map</span>(|</span>p</span>| p.</span>x_y</span>())
</span>                    .</span>tuple_windows</span>()
</span>                    .</span>map</span>(|(</span>p1</span>, </span>p2</span>)| Segment::Line(Line::new(p1, p2)))
</span>                    .</span>collect</span>(),
</span>                </span>false</span>,
</span>            )
</span>        })
</span>        .</span>collect</span>();
</span>    rasterize::Path::new(subpaths)
</span>}
</span></code></pre>
We also need to pick a couple of things:</p>

the view bounding box in map coordinates</li>
the stroke style and color</li>
a scale factor to apply, so that our output image size doesn't depend on the EPSG:3857 coordinates</li>
</ul>
struct </span>RasterizeOptions {
</span>    </span>rasterizer</span>: ActiveEdgeRasterizer,
</span>    </span>bbox</span>: BBox,
</span>    </span>transform</span>: Transform,
</span>    </span>stroke_color</span>: Arc<LinColor>,
</span>    </span>stroke_style</span>: StrokeStyle,
</span>}
</span></code></pre>
Then we can write a bit of code to rasterize a track and save the result to a file:</p>
fn </span>rasterize_route</span>(
</span>    </span>options</span>: &RasterizeOptions,
</span>    </span>output</span>: File,
</span>    </span>mut </span>mls</span>: MultiLineString<</span>f64</span>>,
</span>) -> anyhow::Result<()> {
</span>    </span>reproject_mls</span>(&</span>mut</span> mls);
</span>    </span>let</span> scene = Scene::stroke(
</span>        Arc::new(</span>mls_to_path</span>(mls)),
</span>        Arc::clone(&options.stroke_color) as Arc<dyn Paint>,
</span>        options.stroke_style,
</span>    );
</span>    </span>let</span> layer = scene.</span>render</span>(
</span>        &options.rasterizer,
</span>        options.transform,
</span>        Some(options.bbox),
</span>        None,
</span>    );
</span>    layer.</span>write_png</span>(output)?;
</span>    Ok(())
</span>}
</span></code></pre>
The result looks like this (if it looks weird for you, it's because it has a transparent background):</p>
</a></p>
Accumulating the images</h2>
At the end, I want to overlay all the tracks in the same image.
We load the files from the previous step using png</code></a>, then accumulate them into a white layer:</p>
fn </span>accumulate_images</span>(</span>bbox</span>: BBox, </span>images</span>: &[String]) -> anyhow::Result<()> {
</span>    </span>let mut</span> layer = Layer::new(bbox, Some(LinColor::new(</span>1.0</span>, </span>1.0</span>, </span>1.0</span>, </span>1.0</span>)));
</span>    </span>let mut</span> buf = vec![</span>0</span>; layer.</span>width</span>() * layer.</span>height</span>() * </span>4</span>];
</span>    </span>for</span> img in images {
</span>        {
</span>            </span>let</span> decoder = Decoder::new(File::open(&img)?);
</span>            </span>let mut</span> reader = decoder.</span>read_info</span>()?;
</span>            </span>let</span> info = reader.</span>next_frame</span>(&</span>mut</span> buf)?;
</span>            layer
</span>                .</span>iter_mut</span>()
</span>                .</span>zip</span>((&buf[..info.</span>buffer_size</span>()]).</span>chunks</span>(</span>4</span>))
</span>                .</span>for_each</span>(|(</span>pa</span>, </span>p</span>)| {
</span>                    </span>let</span> p = &p[..</span>4</span>];
</span>                    *pa = pa.</span>blend_over</span>(&ColorU8::new(p[</span>0</span>], p[</span>1</span>], p[</span>2</span>], p[</span>3</span>]).</span>into</span>());
</span>                });
</span>        }
</span>        </span>let</span> file = File::create(img)?;
</span>        layer.</span>write_png</span>(file)?;
</span>    }
</span>    Ok(())
</span>}
</span></code></pre>
The bleding operator (blend_over</code>, from rasterize</code>) keeps the existing pixels and only overwrites the transparent ones.</p>
Putting it together</h2>
Skipping over argument parsing and other initializations, we can finally read the archive and save the FlatGeobuf file:</p>
let mut</span> points = import::read_archive(archive)?;
</span>export_fgb</span>("</span>location</span>", &archive.</span>with_extension</span>("</span>fgb</span>"), &points)?;
</span>points.</span>sort_unstable_by_key</span>(|</span>p</span>| p.time);
</span></code></pre>
We then filter out the bad points and group them by date:</p>
let</span> points = points
</span>    .</span>into_iter</span>()
</span>    .</span>filter</span>(|</span>p</span>| p.radius < </span>50.0</span>)
</span>    .</span>group_by</span>(|</span>p</span>| p.time.</span>date</span>())
</span>    .</span>into_iter</span>()
</span>    .</span>map</span>(|(</span>d</span>, </span>p</span>)| (d, p.collect::<Vec<_>>()))
</span>    .collect::<Vec<_>>();
</span></code></pre>
Run them through OSRM:</p>
let</span> osrm_client = osrm::OsrmClient::new("</span>http://127.0.0.1:5000/</span>".</span>to_string</span>());
</span>let</span> routes = points
</span>    .</span>into_par_iter</span>()
</span>    .</span>filter_map</span>(|(</span>date</span>, </span>points</span>)| {
</span>        osrm_client
</span>            .</span>match_map</span>("</span>foot</span>", &points)
</span>            .</span>ok</span>()
</span>            .</span>map</span>(|</span>m</span>| (date, m))
</span>    })
</span>    .collect::<Vec<_>>();
</span></code></pre>
I'm using rayon</code></a> here in order to do multiple requests to OSRM at once.
It would be very inconsiderate to do this against another server, but I'm running my own instance in Docker.</p>
Rasterize the routes for every date into a corresponding image:</p>
let mut</span> images = routes
</span>    .</span>into_par_iter</span>()
</span>    .</span>map</span>(|(</span>date</span>, </span>mls</span>)| {
</span>        </span>let</span> raster_name = format!("</span>{date}</span>.png</span>");
</span>        </span>let</span> file = File::create(&raster_name)?;
</span>        </span>rasterize_route</span>(&rasterize_options, file, mls)?;
</span>        Ok(raster_name)
</span>    })
</span>    .collect::<Result<Vec<_>, Error>>()?;
</span></code></pre>
Again, this uses rayon</code> for parallelization.
Since I mentioned the memory usage before, there is one subtle issue here.
Every running thread will rasterize some tracks into a layer, so the memory usage will depend on the available concurrency.
If some computation takes, say, 1 GB RAM, that's not too bad until you start 32 of them at once.
So remember to pair CPUs with many cores with an appropriate amount of RAM, even if it sits mostly unused.</p>
Then we finally run the accumulation step:</p>
images.</span>sort_unstable</span>();
</span>accumulate_images</span>(bbox, &images)?;
</span></code></pre>
And just one thing left to do:</p>
$</span> ffmpeg</span> -framerate</span> 15</span> -pattern_type</span> glob</span> -i </span>'</span>*.png</span>'</span> -c</span>:v libwebp_anim</span> -lossless</span> 1</span> -quality</span> 100 gps_tracks.webp
</span></code></pre>
You can see the result below:</p>
</a></p>
Performance</h2>
I've only done rough measurements, but the whole process (except running ffmpeg</code> at the end) takes 4.5 seconds on my system.
Replacing into_par_iter</code> with into_iter</code> brings that up to 10.5 seconds.</p>
Out of the 4.5 s, calling into OSRM takes 1.3 s (6.5 s serially).
The rest of the time is spent in rasterization (which is quite fast), and PNG compression and decompression.</p>
Closing words</h2>
If you've followed through, thank you for reading.
You can find the code on GitHub</a>.
Props also to the GeoRust</a> community, which owns all the geospatial-related crates I've used here.</p>
Map data from OpenStreetMap</a></em></p>


Announcing the sd-notify crate
2022-01-12T00:00:00+00:00
This is a quick post announcing the sd-notify</code></a> crate.
sd-notify</code> is a Rust library for interacting with systemd</code> or a compatible service manager.</p>
If you're not running Linux or you don't like systemd</code>, this crate is not for you.</p>
systemd in 3 minutes</h2>
Assuming you are running systemd</code>, at some point you might find yourself with a program you want to automatically run on start-up.
Historically, this meant doing an intricate dance which includes calling fork()</code> twice and writing your PID to a file, but fortunately, with systemd</code> this is no longer required and is actually discouraged.</p>
You can take your plain console application and make a systemd</code> unit for it:</p>
[Unit]
</span>Description</span>=Monitors room temperature
</span>
</span>[Service]
</span>User</span>=stats
</span>ExecStart</span>=/usr/local/bin/monitor-temperature
</span>Restart</span>=on-failure
</span>RestartSec</span>=</span>10
</span>
</span>[Install]
</span>WantedBy</span>=default.target
</span></code></pre>
You then drop that file under /etc/systemd/system/monitor-temperature.service</code>, run systemctl daemon-reload</code>, systemctl enable --now monitor-temperature</code> and you're done.</p>
For the effort, you get logs (journalctl -efu monitor-temperature</code>), precise child process tracking (using control groups), CPU, memory and I/O accounting, automatic restarts, and a way to manage your service across all the popular Linux distros.</p>
systemd start-up types</h2>
If your daemon has a long start-up sequence, systemd</code> can tell you whether it is ready or not.
In order to do that, you can set the Type</code> clause.</p>
The simplest, and also the default, option is called simple</code>.
It means that the daemon is marked as ready right after it starts.
Another useful option is forking</code>, which means that the service does the classic double-fork dance.
You can read more about these two (any many others) in your systemd.service</code></a> documentation page.</p>
What the sd-notify</code> crate helps with is the notify</code> start-up type.
This lets you do whatever initialization you want, then notify systemd</code> that you are ready by sending a over a Unix socket.
Services can also say that they are reloading their settings, stopping, or even set a status message.
The protocol is described in detail here</a>.</p>
Using sd-notify</h2>
The basic usage is quite simple.
You do your initialization steps, then call:</p>
let </span>_ = sd_notify::notify(</span>true</span>, &[NotifyState::Ready]);
</span></code></pre>
That's all and now your daemon only shows as "running" when it finished starting up.</p>
$</span> systemctl status monitor-temperature
</span>     </span>Active:</span> activating (start) </span>since</span> Wed 2022-01-12 20:36:02 EET; </span>9s</span> ago
</span>     </span>[snip]
</span>$</span> systemctl status monitor-temperature
</span>     </span>Active:</span> active (running) </span>since</span> Wed 2022-01-12 20:36:12 EET; </span>2s</span> ago
</span>     </span>[snip]
</span></code></pre>
You can also include a status message:</p>
let </span>_ = sd_notify::notify(
</span>    </span>true</span>,
</span>    &[
</span>        NotifyState::Ready,
</span>        NotifyState::Status("</span>Monitoring room temperature</span>"),
</span>    ],
</span>);
</span></code></pre>
$</span> systemctl status monitor-temperature
</span>     </span>Active:</span> active (running) </span>since</span> Wed 2022-01-12 20:41:46 EET; </span>782ms</span> ago
</span>     </span>Status: </span>"</span>Monitoring room temperature</span>"
</span>     </span>[snip]
</span></code></pre>
There's also one advanced feature: you can retrieve file descriptors passed by the service manager, for socket-activated daemons.
You can read more on it here</a>.</p>
Final words</h2>
sd-notify</code> is written in pure Rust, has no dependencies and is available under a permissive license (MIT or Apache-2.0).
My intention is that it stays lightweight, but I'm not necessarily opposed to adding more features.</p>
If you want more functionality today, try the libsystemd</code></a> or rust-systemd</code></a> crates.</p>


Map matching using OSRM
2022-01-08T00:00:00+00:00
Hoping to be more active during the pandemic, I bought a Withings smart watch, and at some point got the idea of playing with the GPS tracks recorded when using it.
The manufacturer allows you to download your data by logging in to their web dashboard, going to settings, and choosing "Download my data".
In 10 minutes or so, you will get a ZIP archive in your inbox.</p>
The Withings data</h2>
I like to take long walks and I'm currently living in a small suburb near a larger city, so most of my walks were around the town.
Let's look first at the format of the data.
There are 38 CSVs and a small README.txt</code>:</p>
 303790 Jan  8 12:01 activities.csv
</span>   8253 Jan  8 11:53 aggregates_calories_earned.csv
</span>   8980 Jan  8 11:53 aggregates_calories_passive.csv
</span>   8591 Jan  8 11:53 aggregates_distance.csv
</span>   7029 Jan  8 11:53 aggregates_elevation.csv
</span>    447 Jan  8 11:53 aggregates_manual_spo2.csv
</span>   7274 Jan  8 11:53 aggregates_steps.csv
</span>   2091 Jan  8 11:53 bp.csv
</span>     55 Jan  8 11:53 height.csv
</span>   1576 Jan  8 11:53 manual_spo2.csv
</span>     30 Jan  8 12:01 note.csv
</span> 821784 Jan  8 11:59 raw_bed_calories-earned.csv
</span>2414611 Jan  8 11:59 raw_bed_hr.csv
</span>     21 Jan  8 11:59 raw_bed_maximum_movement.csv
</span>     21 Jan  8 11:59 raw_bed_movement_pim.csv
</span>     21 Jan  8 11:59 raw_bed_respiratory-rate.csv
</span>  59615 Jan  8 11:59 raw_bed_sleep-state.csv
</span>     21 Jan  8 11:59 raw_bed_snoring.csv
</span>7775984 Jan  8 11:53 raw_hr_hr.csv
</span>2258790 Jan  8 11:56 raw_location_altitude.csv
</span>     21 Jan  8 11:56 raw_location_direction.csv
</span>2151043 Jan  8 11:56 raw_location_gps-speed.csv
</span>2196946 Jan  8 11:56 raw_location_horizontal-radius.csv
</span>2376827 Jan  8 11:56 raw_location_latitude.csv
</span>2377039 Jan  8 11:56 raw_location_longitude.csv
</span>1992700 Jan  8 11:56 raw_location_vertical-radius.csv
</span> 293536 Jan  8 11:58 raw_spo2_auto_spo2.csv
</span>2615639 Jan  8 11:58 raw_spo2_quality_score.csv
</span>  13277 Jan  8 11:58 raw_swim_lap-pool.csv
</span>1562522 Jan  8 11:54 raw_tracker_calories-earned.csv
</span>1483782 Jan  8 11:54 raw_tracker_distance.csv
</span>1303766 Jan  8 11:54 raw_tracker_elevation.csv
</span>   5553 Jan  8 11:54 raw_tracker_lap-pool.csv
</span> 116658 Jan  8 11:54 raw_tracker_sleep-state.csv
</span>1337423 Jan  8 11:54 raw_tracker_steps.csv
</span>   3323 Jan  8 12:01 README.txt
</span>1650810 Jan  8 11:53 signal.csv
</span>  39917 Jan  8 12:01 sleep.csv
</span>   1048 Jan  8 11:53 weight.csv
</span></code></pre>
In the basic-but-better-than-last-time docs, we see:</p>
raw_location_latitude.csv: Latitude data
</span>· Duration: (seconds)
</span>
</span>raw_location_longitude.csv: Longitude data
</span>· Duration: (seconds)
</span></code></pre>
# raw_location_latitude.csv
</span>start,duration,value
</span>2022-01-06T12:50:26+01:00,[60],[44.XXX]
</span>2022-01-06T13:28:49+01:00,[60],[44.XXX]
</span>2022-01-06T13:28:59+01:00,[60],[44.XXX]
</span>
</span># raw_location_longitude.csv
</span>start,duration,value
</span>2022-01-06T12:50:26+01:00,[60],[26.XXX]
</span>2022-01-06T13:28:49+01:00,[60],[26.XXX]
</span>2022-01-06T13:28:59+01:00,[60],[26.XXX]
</span></code></pre>
The XXX</code>s are not in the original files, I've (quite pointlessly) masked the digits after the decimal point.
I don't think we care about the durations, but the timestamps are not monotonic, which might matter later.
Fortunately, the timestamps appear to be correlated across the two files.
Otherwise it would be quite annoying to match the two values.
On the other hand, we have some weird-looking entries:</p>
# raw_location_latitude.csv
</span>[snip]
</span>2022-01-06T13:45:36+01:00,"[60,60]","[44.XXX,657.XXX]"
</span>[snip]
</span>2021-11-10T13:14:42+01:00,"[60,60]","[44.XXX,2021.XXX]"
</span>[snip]
</span>2021-11-09T15:05:49+01:00,"[60,60]","[44.XXX,-217.XXX]"
</span>
</span># raw_location_longitude.csv
</span>[snip]
</span>2022-01-06T13:45:36+01:00,"[60,60]","[26.XXX,277.XXX]"
</span>[snip]
</span>2021-11-10T13:14:42+01:00,"[60,60]","[26.XXX,1448.XXX]"
</span>[snip]
</span>2021-11-09T15:05:49+01:00,"[60,60]","[26.XXX,-534.XXX]"
</span></code></pre>
At first sight, those might appear to be two measurements merged into one entry, as the duration seems legit.
But the second coordinate values are clearly bogus.
There are no entries with three values, and those with a single one don't have any outrageous outliers.
So whatever these are, we will make sure to ignore them.</p>
I won't show any code now, but let's extract the points from there, open QGIS</a> and check whether the values (for one arbitrary day) appear to be correct:</p>
</a></p>
Yes, there are a couple of outliers but overall it's fine.</p>
Map matching</h2>
My hope was to match these points to road segments obtained from OpenStreetMap data.
It took me a while to stumble across a way to do this, but I managed to find the OSRM</a> project.
OSRM is open-source and they have a demo web service, but the strict query length limitation made trying it a was of time.</p>
Looking at the docs, we find the match</code></a> endpoint.
It takes a set of coordinates and returns a list of route legs corresponding to them.
You can see an example if you follow the previous link.</p>
I found a couple of sites that offered OSM exports, but couldn't figure out quickly how to register for an API key.
In the end, I downloaded a pbf</code> file for my whole country from https://data.osm-hr.org/</a>.
It was quite small, at 247 MB.</p>
In order to use OSRM, we need to do some pre-processing steps, and also pick a profile.
foot</code> is a good choice here because I don't care about one-way streets and passable barriers.</p>
$</span> docker run</span> --rm -t -v </span>$</span>PWD</span>:/data osrm/osrm-backend osrm-extract</span> -p</span> /opt/foot.lua /data/romania.osm.pbf
</span>$</span> docker run</span> --rm -t -v </span>$</span>PWD</span>:/data osrm/osrm-backend osrm-partition /data/romania.osrm
</span>$</span> docker run</span> --rm -t -v </span>$</span>PWD</span>:/data osrm/osrm-backend osrm-customize /data/romania.osrm
</span></code></pre>
On my computer this took 33 seconds and used about 3.1 GB RAM at its peak.
Now we can start the OSRM routing service:</p>
$</span> docker run</span> --rm -p</span> 5000:5000</span> -v </span>$</span>PWD</span>:/data osrm/osrm-backend osrm-routed</span> --algorithm</span> mld</span> --max-matching-size</span> 5000 /data/romania.osrm
</span></code></pre>
It's a bit weird, but match</code> also expects a profile.
It doesn't seem to matter (as you would expect, since we picked a profile during the pre-processing), and this was slightly confusing.
Does it work?</p>
</a></p>
No, that's not great.
I tried a couple of things: displaying all the matched legs, changing the profile (I originally used the wrong one), sorting the points by timestamp, and even including the timestamps in the request:</p>
</a></p>
This is much better.
What if we also use the location accuracy, available in the raw_location_horizontal-radius.csv</code> file from the archive?</p>
</a></p>
This slows down the matching from "apparently instant" to almost 90 seconds, and introduces some errors.
Looking at the accuracy radius values, most of them are around 10 m, but some have suspicious values like 600 and 800 m.
Let's keep the points with accuracy better than 50 m:</p>
</a></p>
With the old output in green, you can see that most of the extra segments are now gone.
But some are still missing and this confused me quite a bit until I looked at the OSM data.</p>
Hint: don't try open a pbf</code> file directly; you can use GDAL</a> to convert it to a format with spatial index support:</p>
$</span> ogr2ogr romania.osm.gpkg romania.osm.pbf
</span></code></pre>
</a></p>
It's hard to see in the image, but there's a visible gap near the top-left corner.
Could my OSM export have been too old?
Looking at that site again, the other files are from 2018.
My download probably triggered an update, since now the pbf</code> shows a timestamp from today.</p>
I searched for other ways to get the OSM data and Geofabrik</a> seems to be more popular.
Let's see how it looks:</p>
</a></p>
Unfortunately, some segments are still missing or wrong, even though this time they are present in the OSM data.
Checking one of them, it had an access => private</code> tag, which makes sense.
There are some residential buildings with barriers and I might have passed near some of them without noticing it wasn't allowed.</p>
The foot</code> profile has a list of blacklisted tags:</p>
    </span>access_tag_blacklist </span>= </span>Set </span>{
</span>      '</span>no</span>',
</span>      '</span>agricultural</span>',
</span>      '</span>forestry</span>',
</span>      '</span>private</span>',
</span>      '</span>delivery</span>',
</span>    },
</span></code></pre>
I removed them, re-ran the OSRM pre-processing, and re-did the matching:</p>
</a></p>
The missing segment is a foot path through an uncultivated area between two recently-built apartment buildings:</p>
</a></p>
But overall it's quite decent, so I guess this is it for today.
Thanks for reading if you got to the end.</p>
In the next post, I will try to show these tracks in a more interesting form.</p>
Map data from OpenStreetMap</a></em></p>


Announcing the co2mon crate
2021-12-30T00:00:00+00:00
This is a couple of years too late, but I want to write a quick post announcing my first crate, co2mon</code></a>. It's a small library for reading data from Holtek and similar CO₂ USB monitors.
These are relatively popular, because the protocol was reverse-engineered</a> and it's easy to communicate with them.</p>
Linux prerequisites</h2>
If you're on Linux, you will need permissions to access the device node.
The following udev</code> rule will give permissions to every user on the system:</p>
ACTION=="add|change", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04d9", ATTRS{idProduct}=="a052", MODE:="0666"
</span></code></pre>
See the crate documentation for details on how to install it.
You can check the vendor and product id pair using the lsusb</code> command.</p>
If you get a linker error and you might need to install a libusb-1.0-0-dev</code> or libusb</code> package.</p>
Getting started</h2>
Add the co2mon</code> dependency to your Cargo.toml</code>, then copy-paste the following code and try to run it:</p>
use </span>co2mon::{Result, Sensor};
</span>use </span>std::thread;
</span>use </span>std::time::Duration;
</span>
</span>fn </span>main</span>() -> Result<()> {
</span>    </span>let</span> sensor = Sensor::open_default()?;
</span>    </span>loop </span>{
</span>        </span>match</span> sensor.</span>read</span>() {
</span>            Ok(reading) => println!("</span>{:.4}</span> °C, </span>{}</span> ppm CO₂</span>", reading.</span>temperature</span>(), reading.</span>co2</span>()),
</span>            Err(e) => eprintln!("</span>{}</span>", e),
</span>        }
</span>        thread::sleep(Duration::from_secs(</span>5</span>));
</span>    }
</span>}
</span></code></pre>
The sensor sends various types of messages, in an arbitrary order.
Sensor::read</code> waits for a pair of temperature and CO₂ measurements and returns that.
There's also a Sensor::read_one</code> method, which returns a single message.
This can be useful for versions of the sensor that also measure the relative humidity.</p>
Device support</h2>
Tested using a TFA Dostmann AIRCO2NTROL MINI</a>, but thanks to a kind contributor</a>, the crate should also work for newer versions like the COACH</a>.</p>
What to do with it</h2>
Open the windows.
The CO₂ levels you'll find in a closed room aren't exactly dangerous, but can still cause headaches, malaise and impair thinking skills.
At one point I took mine to the office and it showed an error because it only goes up to 3000 ppm.
Reading from USB still worked, giving concentrations around 3500 ppm.</p>
Having a constant airflow in an office will also help against Covid, possibly even more than wearing masks.</p>
Demo</h2>
My Grafana isn't in the best of shapes these days, but here's a screenshot:</p>
</a></p>


My Rust 2020 wish list: playing nice
2019-11-03T00:00:00+00:00
This year I decided to jot down some thoughts for the Rust 2020 call for blog posts</a>.</p>
The major caveat here is that I'm just a language enthusiast.
I'm not using Rust professionally, and my skill in using the language is somewhat mediocre.
We are a pretty diverse bunch, and other people and teams will have different wishes and expectations.</p>
I think the general theme here will be integration or "playing well", on different levels.
A lot of these here are not language changes, and most of them already exist in some form.
Probably none of them are new ideas.</p>
I'm not going to assign priorities: priorities change, new issues are discovered, and contributors work on what they're interested in, not necessarily roadmap items.
And some of the aspects I mention are requests for the community not the project members.</p>
Debugging experience</h2>
Rust got a head start here because of its similarity to C and C++, and being able to use the debugging symbols implementation from LLVM.
With a bit of patience and skill you can make gdb</code> or lldb</code> work decently enough with Rust programs.
I think, however, that we could and should do better.
We shouldn't aim to match the gdb</code> experience for C++ programs, but rather for C# or Java-like integration.</p>
A first step here would be to improve the existing debug visualizers, and make sure they play nice with the three major debuggers we have — gdb</code>, lldb</code> and vsdbg</code>.
In the long run, I hope we will find a way to call existing Debug</code> or Display</code> implementations.
This is probably very hard, since the formatting code might have been dropped by the linker from the final executable.
Is there a way to leverage DWARF for it? I don't know.</p>
Relevant issues:</p>

https://github.com/rust-lang/rust/issues/50005</a></li>
https://github.com/rust-lang/rust/issues/36503</a></li>
https://github.com/rust-lang/rust/issues/40460</a></li>
</ul>
Linking speed</h2>
This doesn't apply to all projects, but a lot of users would be happier if cargo</code> would use the LLVM linker (lld</code>) by default.
In some cases, linking is by far the slowest step in a build, more so in an incremental one.
Using lld</code> can dramatically cut down on link times, especially for large projects with a lot of debug information.</p>
My understanding of the current status is that:</p>

on MacOS it's not worth using lld</code> because it's buggy and unmaintained</li>
on Linux with GCC 9 or later it's easy to use lld</code> by adding -C link-arg=-fuse-ld=lld</code> to RUSTFLAGS</code> or your .cargo/config</code> file</li>
on Windows it should work, but there's a large chance that cargo</code> will pick up the GCC packaged by the Rust toolchain, even if you have GCC 9 installed.
I'm not sure what's the current situation with Visual Studio.</li>
</ul>
All of this info doesn't seem to be documented, but can be found floating around GitHub and other forums.
Is there any hope of seeing lld</code> as the default, at least on Linux?</p>
Visual Studio also has incremental linking.
Can we get something like that?</p>
Relevant issue: https://github.com/rust-lang/rust/issues/39915</a>.</p>
Compile cache</h2>
A nice, relatively easy win is to have a global build-cache.
Supposedly, this is hard to implement, because it needs to handle different versions of compiler, crates, cargo</code> features, conditional compilation and targets.
But cargo</code> is already great at keeping track of those within a single project, so I don't buy this argument.</p>
There's the sccache</code> project that tries to offer this, but in my experience it's a bit unreliable about tracking the compiler version, so I often need to nuke its cache manually after upgrades.
Somewhat worse, it doesn't handle anything with build scripts or procedural macros (that is, a lot of slow to compile crates), presumably because it needs to support distributed compilation.
I think a local-only mode for sccache</code> would be a huge win.</p>
There's also the disk space issue.
With incremental compilation, build files tend to take a huge amount of space.
With a global cache, the external crates could be stored only once, which would make a lot of people with 120 GB SSDs happy.</p>
System integration</h2>
Right now, cargo</code> stores the crates.io registry under ~/.cargo/registry</code>, and maybe also under ~/.cargo/git</code>.
I'm not sure myself.
I'd very much like if cargo</code> was a better team player here.
There are clear-cut guidelines for storing configuration and cached data, and cargo</code> is not following them.
This makes it painful for people using backup programs, roaming user profiles, or who just want to free up some disk space.</p>
This was quite heavily discussed and even implemented, but eventually fizzled out:</p>

https://github.com/rust-lang/rfcs/pull/1615</a></li>
https://github.com/rust-lang/rust/issues/12725</a></li>
https://github.com/rust-lang/cargo/issues/1734</a></li>
https://github.com/rust-lang/cargo/issues/1976</a></li>
https://github.com/rust-lang/cargo/pull/5183</a></li>
</ul>
Notable here is the strong opposition from someone closely associated to the language.
I believe it's in the interest of Rust to play nice with the platform expectations, instead of being its own special snowflake.
There's nothing special to ~/.cargo/git</code> here, no deeper meaning, no potential for data loss if it removed.
It's just a volatile directory, and should not be elevated to something that the users need or care to look at.</p>
Mildly relevant: https://github.com/rust-lang/rfcs/pull/2789</a>.</p>
Formal grammar</h2>
There is a working group for this, but it still seems to come up relatively often.
I think having a formal grammar and a parser crate that lives outside of rustc</code> would make a lot of people feel more at ease with regards to the viability of another compiler implementation.
Personally, I don't think we need</em> another implementation at this point, but a grammar and parser would still be useful regardless of that.</p>
Language features</h2>
I hope there will be progress on features like impl Trait</code>, async fn</code> in traits, const generics, streams, generators and perhaps specialization.
Probably everyone wants these, so there's not much to say.</p>
Maybe the next year the chalk</code> and polonius</code> work will finally bear some fruit, and perhaps there will be some resolution to the parallel compiler saga.
Speaking of which, can salsa</code> replace or supplement the rustc</code> query system?</p>
Sanitizers</h2>
Using the LLVM sanitizers seems to be a bit of a mystery.
People talk about this, but nobody seems to be doing it.
Together with miri</code>, sanitizers would be a powerful tool for dynamic checking of Rust programs.</p>
Bringing crates into std</code></h2>
This is something that comes up from time to time.
I think we should try to identify common patterns, especially in unsafe code, and offer good implementations in libstd</code>.</p>
Relevant discussion: https://internals.rust-lang.org/t/calculating-which-3rd-party-crates-are-good-candidates-for-std-inclusion-via-left-pad-index/11129</a></p>
std</code>-aware cargo</code></h2>
This is quite specific and is already requested, but I'd like to see cargo</code> be able to build libstd</code> instead of using a bundled version.
This would be very helpful for embedded projects, and will improve code generation in some setups like projects that use abort on panic.</p>
Relevant issue: https://github.com/rust-lang/cargo/issues/4959</a></p>
Popular cargo</code> plugins</h2>
We've seen cargo-vendor</code> get integrated into cargo</code>, but there are a couple of other popular plugins that should follow the same route: cargo-edit</code>, cargo-outdated</code> and cargo-tree</code> come to mind.</p>
Relevant issue: https://github.com/rust-lang/cargo/issues/4309</a></p>
Paper cuts</h2>
I'm sure that everybody has some paper cuts they would like to see fixed.
The worst I can think of right now are probably the weird interaction between cargo</code> workspaces and features, and html_root_url</code> which so many people forget to update that it's common to add a reminder about it in the manifest.</p>
A principled stance on unsafe</code></h2>
Right now we have two vocal minorities.
One, generally made of newcomers to the language, says that unsafe code is bad (and even that, by extension, the whole language is bad and useless because libstd</code> uses unsafe</code>).
The other, usually developers experienced in other languages, carelessly uses unsafe</code> wherever the opportunity arises, not caring about the fact that their code is unsound.</p>
None of the two groups are right, and that should be obvious.
A more subtle issue is that an overreaction to former group could normalize the idea that unsafe</code> is perfectly fine if you say you know what you're doing, thus validating the beliefs of the latter.</p>
crates.io and RustSec integration</h2>
The RustSec</a> project has aggregated quite a few of vulnerabilities, some of them still unfixed.
It would be really cool if crates.io could query the RustSec database and display which crate versions are vulnerable or not.</p>
The future of RLS</h2>
I was glad to watch rust-analyzer</code> make great strides in the last year, from the nifty assists that were implemented, to macro_rules</code> support and pretty decent method resolution.
Pending some chalk</code> improvements, it might become almost as precise as RLS, but with much better responsiveness.</p>
The one thing I'm worried about is visible in the contributors list</a>.
Not to discount the work of many people there, but the whole project is basically moving forward only through the effort of a couple of people.</p>
In theory, rust-analyzer</code> is an experiment to see if we can improve on RLS.
I think we're quickly approaching a point where we need to answer that question.
Is it fit to replace RLS or not? In my opinion (and that of many nightly toolchain users, I'm sure) the answer is a resounding yes.
But I would like to see some official statement here.
If it's deemed to be the way forward, maybe RLS should be put into maintenance mode, with those resources being allocated to rust-analyzer</code> instead.</p>
I believe that when rust-analyzer</code> got started, there were some quite strong differences of opinion related to the long-term suitability of the RLS design.
Hopefully we're over them by now, and we can make a decision that goes towards the best interest of the users.</p>
The RFC process</h2>
It's probably well-understood that the RFC process has some limitations that make it scale poorly.
Perhaps a better approach here, which I'm sure was proposed before, would be to:</p>

move most of the RFC discussions to separate repositories, using issues and pull requests</li>
keep a long-running summary of these discussion with pros and cons in something like a HackMD document instead of GitHub comments</li>
try to put some contingency measures in place in case some team members are unavailable or otherwise cannot vote on an RFC</li>
</ul>
The communication platform</h2>
The Rust project seems to have settled on Discord as a replacement for IRC.
I've never used that platform but, as a lot of others have pointed out, something like Matrix might be more in line with the values of the community.</p>
Community (part I)</h2>
Speaking of community, I'm pretty disappointed in how the new website project was handled.
I'm not going to repeat the complaints that others have had about the contents, lack of translations, design, and the site being more or less broken in not so subtle ways.
A lot of these were fixed in the last year.</p>
But I think the launch of the new site was rushed out unnecessarily, and became a friction point that could have been avoided by saying "okay, a lot of people are crying out about this change we're making, maybe we should stop and re-think this for a while".
Instead, we've seen helpful pull requests being closed without thanks or reason.
Is that the impression we want to leave to new contributors? If I had that experience with a project, I would be quite reluctant to get involved any further.</em></p>
Speaking of which, some of us are probably still waiting</a> for an honest post-mortem on it.
The point here is not to make heads roll, but to acknowledge that some mistakes were made, figure out what they were and what to do to prevent them from happening again.
I'd like to see some accountability and transparency on this front, more than others.</p>
To wit, there is this blurb on the the Community team</a> page:</p>

Coordination and supporting events, content creation, running the RustBridge program, and conducting the survey.</p>
</blockquote>
I agree that those are important things to care about, but the community is more than that.
We should acknowledge the regular contributors, the crate authors, the enthusiasts that help people learn the language on the various forums, even those that submit a drive-by pull request once in a blue moon.</p>
We do have the Rust Survey once a year, but no formal feedback channel otherwise.
Sure, some people will post on Discourse or Reddit, and some project members might notice and answer them, but I think there is some room for improvement here.</p>
Roadmap</h2>
I think I would like to see a rough and very high-level list or board of things that the team is actively working on, or planning to.
Maybe the roadmap for 2020 could have a section of "we will probably want these in the future, but cannot work on them now, maybe the next year".
Such a list could be updated during the year, as priorities and resources change.</p>
Community (part II)</h2>
There is a very deep rift in the community in the shape of async run-times and related projects.
Much has been said about the two major implementations, yet to me it seems more and more that they are more similar than different.
I think it's apparent here that the reasons for it are more social than technical.
Time will probably make them converge more and more on the technical side, but the current situation is not ideal.</p>
There is much to say here about various aspects, from naming to timing, and generally the reasoning behind the projects started by the now-disbanded Async Ecosystem WG (nb. async-std</code> was not an Async Ecosystem WG project, even though the same people are involved).
But these were already discussed a lot in different places, and I don't want to add anything here.
Perhaps it is more important to ask:</p>

is the status quo</em> of two very similar, but slightly incompatible run-times in the best interest of the users?</li>
if yes, how do the two projects position themselves, which users are they catering to? It's worth mentioning here that the goals of the two projects are quite similar according to their developers — offering async versions of the functionality present in the standard library. One of the two projects actually uses this as its tagline, which tends to confuse beginners.</li>
if no, what can be done to close this rift? The answer here is clearly not technical, and it will require giving up some control and some ego, and actively trying to work with the other side. This is hard to do, harder than any technical challenge that the two projects might encounter.</li>
</ol>
I also want to say that there is a certain level of misinformation and FUD floating around, most likely unintentional.
We should work on that, because it can get toxic in the long run.
In addition, I was very disappointed to see a comment on a well-intended pull request</a> that was quite aggressive towards the competing project.
It's now deleted, but it went like:</p>

After 2 hours, this issue got 17 thumbs up. After 5 hours, it has 20. I recommend closing this issue as spam.</p>
</blockquote>
(although initially it said "obvious trolling" in Russian, or something similar)</p>
That's an awful way to speak about the competing project, which actually spawned both romio</code> and the one you're working on.
And I don't think it's in the best interest of async-std</code> if its developers take this kind of position.</p>
Update:</strong> unsurprisingly, this boiled over pretty bad.</p>
Transparency</h2>
Finally, I'm not sure how well the working group structure has panned out.
There are quite a lot of WGs, but I'm not sure how much progress they have been making.</p>
I think the plan was to have regular progress meetings and keep minutes for each WG, but my impression is that these happened for a month or two, then they stopped.</p>
Maybe the recently-announced Inside Rust</a> blog will be a simple solution to this problem.</p>
The future</h2>
As somebody else</a> has mentioned, I think our greater challenges have yet to come, brought by new production users (let's not forget that the greatest mass of software developers is not visible in the Stack Overflow surveys) and maybe some internal tensions.
I still think that Rust is a quite nice local maximum of programming languages, but we must steel ourselves and prepare for new challenges, from inexperience users, to companies tying to get more control over the project, to competition for other languages like C# that are inching close and closer towards Rust and Go.</p>
Conclusion</h2>
Even though I spoke more about things that aren't quite working, we've made a nice amount of progress this year.
I'm not going to make a list here, as I'm sure others will be much better at it.</p>
And while I've been somewhat negative on a couple of points, I still think we're doing pretty well overall.
I'm only worried that social factors can prove to be a problem in the long run, and we — both the project members and the community — should focus a little on resiliency</em> in the long run.
This means playing nice towards others, and being prepared for new users, more competition, and the potential for more conflicts.</p>


Acknowledgements
2018-08-05T00:00:00+00:00
Site created with the Zola</a> static site generator. Theme based on Hyde</a>.</p>


Rust and the Case of the Redundant Comparison
2018-08-04T00:00:00+00:00
A couple of days ago I landed my second pull request</a> in the Rust Programming Language</a> repository</a>.
This is the story of how that went.</p>
This post is inspired by other</a> posts</a> about improving the Rust compiler.</p>
TL;DR:</strong> I made a 20 lines PR to the Rust standard library.
If you're so inclined, you should try doing the same.</p>
The chase after unsafe code</h2>
There's been quite a bit of noise</a> recently about the amount of unsafe code in the actix-web</code></a> framework.
I won't discuss the merits of grabbing the pitchforks as soon as someone writes code of a buggy and unidiomatic nature 1</a></sup>.
But my first encounter with unsafe</code> in Rust was a hard to reproduce, Windows-only, crashing bug in a crate.
On that code path there was a single, seemingly innocuous, unsafe</code> line.
It took me and the crate's author maybe half an hour to find it, with the unsafe</code> arrow pointing at it the whole time 2</a></sup>.</p>
To be fair, actix-web</code> now lost a large chunk of its unsafe code, although it's still not quite my cup of tea.
So I downloaded the source of hyper</code></a>, another popular, if much lower-level, HTTP library.
Unfortunately (‽), it had relatively few unsafe</code> blocks, many of them in test code.
But one idiom caught my eye:</p>
#[</span>derive</span>(Clone)]
</span>pub struct </span>Cursor<T> {
</span>    </span>bytes</span>: T,
</span>    </span>pos</span>: </span>usize</span>,
</span>}
</span>
</span>impl </span>Cursor<Vec<</span>u8</span>>> {
</span>    </span>fn </span>reset</span>(&</span>mut </span>self</span>) {
</span>        </span>self</span>.pos = </span>0</span>;
</span>        </span>unsafe </span>{
</span>            </span>self</span>.bytes.</span>set_len</span>(</span>0</span>);
</span>        }
</span>    }
</span>}
</span></code></pre>
The code is trying to clear a Vec</code> by forcefully setting its length to 0</code>.
That's a bad idea if the elements of the vector implement Drop</code>, because they won't be destroyed.
Fortunately, hyper</code> only did that for u8</code> values, but the whole thing seemed unnecessary.</p>
I almost sent a PR to nuke them, but the crate author said that set_len(0)</code> might be faster than clear()</code>.
That couldn't be true, since clear</code> has nothing to do besides setting the length, so I tried to prove it:</p>
#[</span>no_mangle</span>]
</span>pub extern fn </span>test_set_len</span>(</span>x</span>: &</span>mut </span>Vec<</span>u8</span>>) {
</span>    </span>unsafe </span>{ x.</span>set_len</span>(</span>0</span>); }
</span>}
</span>
</span>#[</span>no_mangle</span>]
</span>pub extern fn </span>test_clear</span>(</span>x</span>: &</span>mut </span>Vec<</span>u8</span>>) {
</span>    x.</span>clear</span>();
</span>}
</span></code></pre>
But I was surprised to see this in the generated code:</p>
test_set_len:
</span>    </span>movq    </span>$</span>0</span>, </span>16</span>(%</span>rdi</span>)
</span>    retq
</span>
</span>test_clear:
</span>    cmpq    </span>$</span>0</span>, </span>16</span>(%</span>rdi</span>)
</span>    </span>je      </span>.LBB4_2
</span>    </span>movq    </span>$</span>0</span>, </span>16</span>(%</span>rdi</span>)
</span>
</span>.LBB4_2:
</span>    retq
</span></code></pre>
If you're not used to reading assembly, the set_len</code> code sets the vector's length to zero (movq</code>) and returns.
clear</code>, however, compares the current length with 0</code>.
If it's empty, the code jumps to the end of the function (je</code> stands for "jump if equal").
Otherwise, the length gets set to 0</code> and the function returns.
That's a useless comparison, the same as:</p>
if</span> len != </span>0 </span>{
</span>    len = </span>0</span>;
</span>}
</span></code></pre>
LLVM is pretty good, so it's a bit surprising to see it generate this 3</a></sup>.
In its favour, the debug version is much larger, so the optimizer is still doing a great job.</p>
Do the extra two instructions matter in practice? I didn't benchmark, but my intuition says no 4</a></sup>, and the set_len(0)</code> calls were mostly in test code.
However, the crate's author seemed unconvinced and I didn't want to be the one who slowed down hyper</code>.</p>
Down the rabbit hole</h2>
Disappointed by this turn of events, I searched for the implementation of clear</code>:</p>
#[</span>inline</span>]
</span>pub fn </span>clear</span>(&</span>mut </span>self</span>) {
</span>    </span>self</span>.</span>truncate</span>(</span>0</span>)
</span>}
</span>
</span>pub fn </span>truncate</span>(&</span>mut </span>self</span>, </span>len</span>: </span>usize</span>) {
</span>    </span>unsafe </span>{
</span>        </span>// drop any extra elements
</span>        </span>while</span> len < </span>self</span>.len {
</span>            </span>// decrement len before the drop_in_place(), so a panic on Drop
</span>            </span>// doesn't re-drop the just-failed value.
</span>            </span>self</span>.len -= </span>1</span>;
</span>            </span>let</span> len = </span>self</span>.len;
</span>            ptr::drop_in_place(</span>self</span>.</span>get_unchecked_mut</span>(len));
</span>        }
</span>    }
</span>}
</span></code></pre>
The code is a bit convoluted, because of the reason described in the comment.
drop</code> can panic 5</a></sup>, so the function must decrement the length before dropping an element.
In case of a panic, the last element of the Vec</code> will be a valid one.</p>
A quick test on the Playground</a> shows that if we set the length at the end, the compiler generates the code that we expect:</p>
#[</span>inline</span>(never)]
</span>fn </span>truncate_wrong_dont_use</span><T>(</span>x</span>: &</span>mut </span>Vec<T>, </span>len</span>: </span>usize</span>) {
</span>    </span>unsafe </span>{
</span>        </span>let mut</span> last = x.</span>len</span>();
</span>        </span>while</span> len < last {
</span>            last -= </span>1</span>;
</span>            ptr::drop_in_place(x.</span>get_unchecked_mut</span>(last));
</span>        }
</span>        x.</span>set_len</span>(len);
</span>    }
</span>}
</span></code></pre>
playground::truncate_wrong_dont_use:
</span>    </span>movq    </span>$</span>0</span>, </span>16</span>(%</span>rdi</span>)
</span>    retq
</span></code></pre>
But that won't do.
At this point I filed an issue in the compiler repository, then dropped [sic!] it, waiting for someone else to come up with a fix.
Unfortunately, nobody did, so I gave it a little more thought.</p>
If we could run some code at the end, even on panics, we might be able to set the length from there.
And we can, with a helper struct which applies the change when dropped.
It sounds harder for the compiler, but there's only one way to check.
And if we scroll around the file, we find this:</p>
// Set the length of the vec when the `SetLenOnDrop` value goes out of scope.
</span>//
</span>// The idea is: The length field in SetLenOnDrop is a local variable
</span>// that the optimizer will see does not alias with any stores through the Vec's data
</span>// pointer. This is a workaround for alias analysis issue #32155
</span>struct </span>SetLenOnDrop<</span>'a</span>> {
</span>    </span>len</span>: &</span>'a mut usize</span>,
</span>    </span>local_len</span>: </span>usize</span>,
</span>}
</span>
</span>impl</span><</span>'a</span>> SetLenOnDrop<</span>'a</span>> {
</span>    #[</span>inline</span>]
</span>    </span>fn </span>new</span>(</span>len</span>: &</span>'a mut usize</span>) -> </span>Self </span>{
</span>        SetLenOnDrop { local_len: *len, len: len }
</span>    }
</span>
</span>    #[</span>inline</span>]
</span>    </span>fn </span>increment_len</span>(&</span>mut </span>self</span>, </span>increment</span>: </span>usize</span>) {
</span>        </span>self</span>.local_len += increment;
</span>    }
</span>}
</span>
</span>impl</span><</span>'a</span>> Drop </span>for </span>SetLenOnDrop<</span>'a</span>> {
</span>    #[</span>inline</span>]
</span>    </span>fn </span>drop</span>(&</span>mut </span>self</span>) {
</span>        *</span>self</span>.len = </span>self</span>.local_len;
</span>    }
</span>}
</span></code></pre>
The comment indicates that LLVM gets confused into thinking that modifying the vector's data could change its length.
And that sounds exactly like our issue.</p>
Compiling a compiler</h2>
I cloned the compiler repository and skimmed the contributing guide</a>.
I copied the sample configuration file and changed a couple of lines:</p>
diff --git 1/config.toml.example 2/config.toml
</span>index 9907341633..477c558545 100644
</span>--- 1/config.toml.example
</span>+++ 2/config.toml
</span>@@ -104,11 +104,11 @@
</span>
</span> # Instead of downloading the src/stage0.txt version of Cargo specified, use
</span> # this Cargo binary instead to build all Rust code
</span>-#cargo = "/path/to/bin/cargo"
</span>+cargo = "/usr/bin/cargo"
</span>
</span> # Instead of downloading the src/stage0.txt version of the compiler
</span> # specified, use this rustc binary instead as the stage0 snapshot compiler.
</span>-#rustc = "/path/to/bin/rustc"
</span>+rustc = "/usr/bin/rustc"
</span>
</span> # Flag to specify whether any documentation is built. If false, rustdoc and
</span> # friends will still be compiled but they will not be used to generate any
</span>@@ -250,6 +250,7 @@
</span> # means "the number of cores on this machine", and 1+ is passed through to the
</span> # compiler.
</span> #codegen-units = 1
</span>+codegen-units = 0
</span>
</span> # Whether or not debug assertions are enabled for the compiler and standard
</span> # library. Also enables compilation of debug! and trace! logging macros.
</span></code></pre>
I didn't want to download another compiler, so I filled in the paths to mine.
The codegen-units</code> setting makes the compiler produce code from multiple threads at once.
The trade-off involved is that the generated code is slower.
Initially, I went for an unoptimized build (which the comments advise against), but ended up cancelling it.</p>
The way compilers are usually built, there are a couple of stages</em>.
Stage 0 is an existing compiler, stage 1 is the compiler we are working on — built by the stage 0 one, and stage 2 is our version built by itself.
If we make the compiler generate faster — or more buggy — code, stage 1 won't be affected, but stage 2 will.
I wasn't planning to change anything in the compiler, so stage 1 was enough for me.</p>
Apparently, there is a small complication: some stage 0 artefacts like libstd</code> will still be rebuilt, triggering a cascade of rebuilds that I didn't want.
After asking around on IRC, I found the command line I wanted:</p>
./x.py --keep-stage</span> 0</span> --stage</span> 1
</span></code></pre>
This is all described in the bootstrapping documentation</a>, which I completely missed.
I also tried incremental builds (via config.toml</code>, not the command line — not sure if it matters), which didn't work out too well for me: an incremental build after touching a single file was just a tad faster than a non-incremental one including LLVM.
But there may have been something wrong with my build files, so it's still worth trying.</p>
Banishing a read</h2>
There was no need to do an initial build, but I wanted to check that everything worked.
That took about 40 minutes on my laptop.
I then added the missing method to the SetLenOnDrop</code> struct:</p>
#[</span>inline</span>]
</span>fn </span>decrement_len</span>(&</span>mut </span>self</span>, </span>decrement</span>: </span>usize</span>) {
</span>    </span>self</span>.local_len -= decrement;
</span>}
</span></code></pre>
truncate</code> took a couple of tries, but I ended up with the following:</p>
pub fn </span>truncate</span>(&</span>mut </span>self</span>, </span>len</span>: </span>usize</span>) {
</span>    </span>let</span> current_len = </span>self</span>.len;
</span>    </span>unsafe </span>{
</span>        </span>let mut</span> ptr = </span>self</span>.</span>as_mut_ptr</span>().</span>offset</span>(</span>self</span>.len as </span>isize</span>);
</span>        </span>// Set the final length at the end, keeping in mind that
</span>        </span>// dropping an element might panic. Works around a missed
</span>        </span>// optimization, as seen in the following issue:
</span>        </span>// https://github.com/rust-lang/rust/issues/51802
</span>        </span>let mut</span> local_len = SetLenOnDrop::new(&</span>mut </span>self</span>.len);
</span>
</span>        </span>// drop any extra elements
</span>        </span>for </span>_ in len..current_len {
</span>            local_len.</span>decrement_len</span>(</span>1</span>);
</span>            ptr = ptr.</span>offset</span>(-</span>1</span>);
</span>            ptr::drop_in_place(ptr);
</span>        }
</span>    }
</span>}
</span></code></pre>
The code walks a pointer backwards, destroying each element.
Meanwhile, the helper struct is also keeping track of the remaining length.
The code does seem more complex (it's counting twice), but does it work? Let's check some assembly.</p>
#[</span>no_mangle</span>]
</span>pub extern fn </span>foo</span>(</span>x</span>: &</span>mut </span>Vec<</span>u8</span>>) {
</span>    x.</span>clear</span>();
</span>}
</span>
</span>#[</span>no_mangle</span>]
</span>pub extern fn </span>bar</span>(</span>x</span>: &</span>mut </span>Vec<</span>u8</span>>) {
</span>    x.</span>truncate</span>(</span>5</span>);
</span>}
</span>
</span>#[</span>no_mangle</span>]
</span>pub extern fn </span>baz</span>(</span>x</span>: &</span>mut </span>Vec<</span>u8</span>>, </span>n</span>: </span>usize</span>) {
</span>    x.</span>truncate</span>(n);
</span>}
</span></code></pre>
The unpatched compiler version shows what we've already seen:</p>
00000000000460a0 <foo>:
</span>   460a0:       </span>48 83 </span>7f </span>10 00          </span>cmpq   </span>$</span>0x0</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460a5:       </span>74 08                   </span>je     </span>460af <foo</span>+</span>0xf</span>>
</span>   460a7:       </span>48 </span>c7 </span>47 10 00 00 00    </span>movq   </span>$</span>0x0</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460ae:       </span>00
</span>   460af:       c3                      retq
</span>
</span>00000000000460b0 <bar>:
</span>   460b0:       </span>48 83 </span>7f </span>10 06          </span>cmpq   </span>$</span>0x6</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460b5:       </span>72 08                   </span>jb     </span>460bf <bar</span>+</span>0xf</span>>
</span>   460b7:       </span>48 </span>c7 </span>47 10 05 00 00    </span>movq   </span>$</span>0x5</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460be:       </span>00
</span>   460bf:       c3                      retq
</span>
</span>00000000000460c0 <baz>:
</span>   460c0:       </span>48 39 77 10             </span>cmp    </span>%</span>rsi</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460c4:       </span>76 04                   </span>jbe    </span>460ca <baz</span>+</span>0xa</span>>
</span>   460c6:       </span>48 89 77 10             </span>mov    </span>%</span>rsi</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   460ca:       c3                      retq
</span>   460cb:       0f 1f </span>44 00 00          </span>nopl   </span>0x0</span>(%</span>rax</span>,</span>%</span>rax</span>,</span>1</span>)
</span></code></pre>
How about the patched one?</p>
0000000000084d10 <foo>:
</span>   84d10:       </span>48 </span>c7 </span>47 10 00 00 00    </span>movq   </span>$</span>0x0</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   84d17:       </span>00
</span>   84d18:       c3                      retq
</span>   84d19:       0f 1f </span>80 00 00 00 00    </span>nopl   </span>0x0</span>(%</span>rax</span>)
</span>
</span>0000000000084d20 <bar>:
</span>   84d20:       </span>48 </span>8b </span>47 10             </span>mov    </span>0x10</span>(%</span>rdi</span>)</span>,</span>%</span>rax
</span>   84d24:       </span>48 83 </span>f8 </span>05             </span>cmp    </span>$</span>0x5</span>,</span>%</span>rax
</span>   84d28:       b9 </span>05 00 00 00          </span>mov    </span>$</span>0x5</span>,</span>%</span>ecx
</span>   84d2d:       </span>48 </span>0f </span>42 </span>c8             </span>cmovb  </span>%</span>rax</span>,</span>%</span>rcx
</span>   84d31:       </span>48 89 </span>4f </span>10             </span>mov    </span>%</span>rcx</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   84d35:       c3                      retq
</span>   84d36:       </span>66 </span>2e 0f 1f </span>84 00 00    </span>nopw   %</span>cs</span>:</span>0x0</span>(%</span>rax</span>,</span>%</span>rax</span>,</span>1</span>)
</span>   84d3d:       </span>00 00 00
</span>
</span>0000000000084d40 <baz>:
</span>   84d40:       </span>48 </span>8b </span>47 10             </span>mov    </span>0x10</span>(%</span>rdi</span>)</span>,</span>%</span>rax
</span>   84d44:       </span>48 39 </span>f0                </span>cmp    </span>%</span>rsi</span>,</span>%</span>rax
</span>   84d47:       </span>48 </span>0f </span>47 </span>c6             </span>cmova  </span>%</span>rsi</span>,</span>%</span>rax
</span>   84d4b:       </span>48 89 47 10             </span>mov    </span>%</span>rax</span>,</span>0x10</span>(%</span>rdi</span>)
</span>   84d4f:       c3                      retq
</span></code></pre>
The clear()</code> call works out nicely, it's what we wanted to see (nopl</code> is a "no-operation" instruction, used to align the functions) 6</a></sup>.
truncate(n)</code> also seems fine: it takes the minimum of the current and new length, then writes it back.
The truncate(5)</code> variant does the same thing, but loads 5</code> into the ecx</code> register in an awkward way: mov %rax, %rcx</code> might have been better.
Alas, it came out pretty well.</p>
I also tested with a Drop</code> type (String</code>).
I didn't try to make sense of the assembly output, but the new code was similar.
You can see it on GitHub</a>, if you're curious.</p>
Wrapping it up</h2>
At this point I committed the code and opened a pull request on GitHub.
I got assigned a reviewer who reviewed my code in less than two hours.
He asked me to add a codegen test to make sure the compiler doesn't revert to the worse code sequence in the future.
I'd never written one, but after looking at the existing ones, I put this up:</p>
// compile-flags: -O
</span>
</span>#![</span>crate_type </span>= "</span>lib</span>"]
</span>
</span>// CHECK-LABEL: @vec_clear
</span>#[</span>no_mangle</span>]
</span>pub fn </span>vec_clear</span>(</span>x</span>: &</span>mut </span>Vec<</span>u32</span>>) {
</span>    </span>// CHECK-NOT: load
</span>    </span>// CHECK-NOT: icmp
</span>    x.</span>clear</span>()
</span>}
</span></code></pre>
The Rust compiler produces what's called LLVM IR as output.
It's similar to, but more portable and higher-level than assembly language.
The IR gets passed to LLVM, which optimizes it and outputs a binary for the platform you're targeting.
load</code> and icmp</code> are LLVM's terms for movq</code> and cmpq</code>.
The test checks that the generated code doesn't contain the two instructions.</p>
To figure that out, I used the Playground to see my function's IR:</p>
; Function Attrs: norecurse nounwind uwtable
</span>define void @test_clear(%"alloc::vec::Vec<u8>"* noalias nocapture dereferenceable(24) %x) unnamed_addr #2 {
</span>start:
</span>  %0 = getelementptr inbounds %"alloc::vec::Vec<u8>", %"alloc::vec::Vec<u8>"* %x, i64 0, i32 3
</span>  %1 = load i64, i64* %0, align 8, !alias.scope !5
</span>  %2 = icmp eq i64 %1, 0
</span>  br i1 %2, label %"_ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hf92f022d73112116E.exit", label %bb3.preheader.i.i
</span>
</span>bb3.preheader.i.i:                                ; preds = %start
</span>  store i64 0, i64* %0, align 8, !alias.scope !5
</span>  br label %"_ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hf92f022d73112116E.exit"
</span>
</span>"_ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hf92f022d73112116E.exit": ; preds = %start, %bb3.preheader.i.i
</span>  ret void
</span>}
</span></code></pre>
That's a mouthful, but you can find the length field address computation (getelementptr</code>, affectionately called gep</code>), the load and comparison, the branch and the store.
%0</code> … %5</code> are similar to variables, but are only assigned once</a>.</p>
I then tried to run my test:</p>
./x.py</span> test src/test/codegen</span> --keep-stage</span> 0</span> --stage</span> 1
</span></code></pre>
But that didn't seem to work</a>, possibly due to me using a local compiler for stage 0 (remember those config.toml</code> settings?).
So I crossed my fingers and pushed the code, waited for the relatively quick (half an hour) Travis tests to pass, pinged my reviewer and then it was out of my hands.
The Rust team uses bors</a> to merge one pull request at a time.
This is done to avoid cases when two PRs work independently, but not together.
The downside is that the whole process tends to be rather slow.
Someone from the Rust team takes the pull requests that seem "harmless", merges them into a single one, then tries to land that.
In my case this failed a couple of times, then finally went through.</p>
Aftermath</h2>
My change missed the 1.29 deadline by a day, but should be included in 1.30.
After it was merged, I was finally able to send a pull request</a> to hyper</code>, removing four measly (but a quarter of all!) unsafe</code> blocks.</p>
My changes didn't make hyper</code> safer 7</a></sup>, nor do I expect them to have a measurable impact on the compiled code.
But in the end, every little bit helps, while you might also call it a learning experience.
And if I managed, then so could you — so consider giving it a shot.
And if compilers aren't your thing, there's plenty of projects that could use a hand.</p>
^{1</sup>
As I've been known to do myself at times.</p>
</div>
^{2</sup>
Some detractors of Rust argue that, since the standard library and many crates use unsafe code, the whole language is unsafe.
unsafe</code> is unavoidable, but it's a declaration of "Here be dragons", while you can trust the compiler to have your back for the rest of the code.
Compare that to C.</p>
</div>
^{3</sup>
Compilers have "peephole optimization" phases where they clean up sequences of instructions like these.
LLVM might be missing one for this specific case.
It might be an interesting exercise to analyze binaries generated by LLVM to check if this pattern occurs more often.</p>
</div>
^{4</sup>
And we've already seen how that goes.</p>
</div>
^{5</sup>
Rust doesn't have exceptions, except panics are totally like exceptions, but more exceptional.
You're not supposed to panic without a good reason.</p>
</div>
^{6</sup>
The extra 00</code> byte between movq</code> and retq</code> must be either a no-op, or part of movq</code>.</p>
</div>
^{7</sup>
Except maybe from the pitchfork mobs.</p>
</div>}}}}}}}

`Next steps</h2> Our result looks encouraging, but we still need to do something about the compression artefacts (remember, the image we extracted is a JPEG) and, ideally, fill in the building footprints. If you'd like to read more, stay tuned for the next part!</p>`

The solution</h2>
I also happened to stumble upon a pretty simple workaround: add `nfsvers=3</code> to the mount options (/etc/fstab</code>). It appears that the problem only occurs when using NFSv4, and not NFSv3.</p>`

`Closing words</h2>If you've followed through, thank you for reading. You can find the code on GitHub</a>. Props also to the GeoRust</a> community, which owns all the geospatial-related crates I've used here.</p> Map data from OpenStreetMap</a></em></p>`

Demo</h2>
My Grafana isn't in the best of shapes these days, but here's a screenshot:</p>
</a></p>

The communication platform</h2>
The Rust project seems to have settled on Discord as a replacement for IRC. I've never used that platform but, as a lot of others have pointed out, something like Matrix might be more in line with the values of the community.</p>

Talks

Recovering information from a PDF map

About a fun little NFS bug

Decoding Firefox session store data

Debugging a mysterious Python crash

Blogroll

Building Rust code for my OpenWrt Wi-Fi router

Some eye candy

On self-modifying executables in Rust

Mapping my walks with OSRM and Rust

Announcing the sd-notify crate

Map matching using OSRM

Announcing the co2mon crate

My Rust 2020 wish list: playing nice

Acknowledgements

Rust and the Case of the Redundant Comparison

Talks

Recovering information from a PDF map

About a fun little NFS bug

The solution</h2> I also happened to stumble upon a pretty simple workaround: add nfsvers=3</code> to the mount options (/etc/fstab</code>). It appears that the problem only occurs when using NFSv4, and not NFSv3.</p>

Decoding Firefox session store data

Debugging a mysterious Python crash

Gathering more info</h2> Luckily, the crash was easy to reproduce outside of Jupyter:</p>

Conclusion</h2> Just a couple of things:</p>

Blogroll

Building Rust code for my OpenWrt Wi-Fi router

Inside my router</h2> My router is an AVM FRITZ!Box 4040 running OpenWrt. As far as embedded systems go, OpenWrt is pretty close to a Linux PC. I've already enabled SSH and added my key, but I don't know what architecture it's running. Fortunately, uname -a</code> works just as expected:</p>

Cross-compiling</h2> Rust supports dozens of targets, but ARMv7 is pretty common, so it's hopefully well-supported. I'm not sure how the target is called, so rustup target list</code> is handy:</p>

Some eye candy

On self-modifying executables in Rust

Errata</h2> the variable should probably be marked as #[repr(C)]</code>.</li>

Diving right in</h2> As a proof of concept, we'll write a program that counts how many times it has been run. This is hardly the only approach, but probably the simplest way is to store the counter in a separate section of the executable.</p>

Mapping my walks with OSRM and Rust

Announcing the sd-notify crate

Map matching using OSRM

The Withings data</h2> I like to take long walks and I'm currently living in a small suburb near a larger city, so most of my walks were around the town. Let's look first at the format of the data. There are 38 CSVs and a small README.txt</code>:</p>

Announcing the co2mon crate

Demo</h2> My Grafana isn't in the best of shapes these days, but here's a screenshot:</p>

My Rust 2020 wish list: playing nice

The communication platform</h2> The Rust project seems to have settled on Discord as a replacement for IRC. I've never used that platform but, as a lot of others have pointed out, something like Matrix might be more in line with the values of the community.</p>

Acknowledgements

Rust and the Case of the Redundant Comparison

The solution</h2>
I also happened to stumble upon a pretty simple workaround: add `nfsvers=3</code> to the mount options (/etc/fstab</code>). It appears that the problem only occurs when using NFSv4, and not NFSv3.</p>`

Demo</h2>
My Grafana isn't in the best of shapes these days, but here's a screenshot:</p>
</a></p>

The communication platform</h2>
The Rust project seems to have settled on Discord as a replacement for IRC. I've never used that platform but, as a lot of others have pointed out, something like Matrix might be more in line with the values of the community.</p>