New Tool: fswatch

During my first week at InfluxData, I have been learning by digging into the InfluxDB storage layer. InfluxDB has various interactions with the filesystem: write-ahead log, data storage and indexing. At a certain point today I had the idea, "What if I could watch file mutations as data is fed into InfluxDB?" Some searching for command-line interfaces to inotify and I found fswatch.

Demo

Easiest to explain by example:

$ brew install fswatch
$ fswatch -t -x ~/.influxdb/

In another console (NOAA sample data):

$ influx -import -path=NOAA_data.txt -precision=s -database=NOAA_water_database

Back to the first console, here's a sample of the output. Notice the directory for database NOAA_water_database is created, the directory for retention policy autogen is created, then WAL, etc. I'm using inmem index, so there's no TSI here:

Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/10 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/_series Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/_series/00 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/_series/00/0000.initializing Created PlatformSpecific Renamed Updated IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/_series/00/0000 Renamed IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/9 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/9 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/8 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/8 Created IsDir
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/9/_00001.wal Created IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/8/_00001.wal Created IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/meta/meta.dbtmp Created Renamed Updated IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/meta/meta.db Renamed IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/8/fields.idx.tmp Created Renamed Updated IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/8/fields.idx Renamed IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/9/fields.idx.tmp Created Renamed Updated IsFile
Thu Mar 29 09:39:19 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/9/fields.idx Renamed IsFile
Thu Mar 29 09:49:20 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/9/_00002.wal Created IsFile
Thu Mar 29 09:49:20 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/8/_00002.wal Created IsFile
Thu Mar 29 09:49:20 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/9/000000001-000000001.tsm.tmp Created Renamed Updated IsFile
Thu Mar 29 09:49:20 2018 /Users/jacobmarble/.influxdb/data/NOAA_water_database/autogen/9/000000001-000000001.tsm Renamed IsFile
Thu Mar 29 09:49:20 2018 /Users/jacobmarble/.influxdb/wal/NOAA_water_database/autogen/9/_00001.wal Removed Updated IsFile

About fswatch

fswatch source code is hosted on GitHub, and claims to be very multi-platform, which is nice. In the example, I used flag -x to add the event flag to the output, for example Removed Updated IsFile and Created IsDir. I also used flag -t to add timestamp to the output. There is a recursive flag -r, but it seems to be enabled by default on macOS.

Show Comments