Using FlashAir, what's the timestamp on the data files?

[biorn]

Ah, well. it was a thought...

The ResMed engineers who didn't think to put the code in to sync the clock using the cell phone network are idiots! Maybe they'll fix that in the AirSense11?

The cellular networks don't serve time, nor do cell devices use the network to use a more traditional time protocol like NTP or PTP, though one can use an app for that. Cell phones capable of time syncing use GPS Network Time Synchronization.

I was thinking more along the lines of the machine is talking to the ResMed network and could ask what time it is. From there it would be simple to write that information somewhere in the data stream. The clock doesn’t drift very far very fast, so that would fix most problems well enough. 

Having been a bemused bystander to the great network time religious wars, I can say that the takeaway is that it’s not that hard to do something reasonably intelligent, and then developers can figure out what to do from there. 

I’m wondering if you could use the FlashAir syncing to keep reasonable track. There are a whole bunch of files being created with times embedded in the names, and everything has creation times in the metadata. You ought to be able to run a script that would keep track of which flies first appeared when and then you should be able to zero in on the clock drift after watching files pop up all night. 

But, anyway, the computer nerds had invested an extraordinary amount of thought into managing times over far-flung networks, and had it pretty well figured out long before GPS was even dreamt of. The ResMed engineers obviously put some considerable thought into how to manage the problem that the date rolls to the next day in the middle of the night which you want to be all part of “one night’s sleep”. And what they came up with is not stupid. So they solved the hard problem but didn’t think about the easy problem!

Yes my first answer was in this way, there is a lot of solution to update the time from the modem but but resmed did not do it. And the new way to get the time on the 5g network is more more efficiant thant gps. (but it's not the subject)

So on the flashair you could do a simple trick. That need some investigation but on the main lines for the signal when a new file is created the flashair will change the reference time in the file to the actual time know by the flashair. Maybe there is better solutions but it's the first I got.

[cathyf]

So on the flashair you could do a simple trick. That need some investigation but on the main lines for the signal when a new file is created the flashair will change the reference time in the file to the actual time know by the flashair. Maybe there is better solutions but it's the first I got.

I'm thinking that it's better that the files that the CPAP is writing should only be read by any other applications. If I understand how FlashPap works, it is copying the files from the flash drive over the network to the computer. That would be the logical point to correct the file metadata by setting the correct UTC timestamps on the computer's copies.

Since OSCAR already has some clock drift code in it, probably a logical place to go from there is for OSCAR to compare the time stamps on the files with the time strings in the files and the time strings embedded in the file names. For example, here's my DATALOG/20210420 files:

Code:

bash-3.2$ ls -lUT
total 6232
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:41:24 2021 20210421_004122_CSL.crc
-rwxrwxrwx  1 cathy  staff      800 Apr 21 00:41:24 2021 20210421_004122_CSL.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:41:24 2021 20210421_004122_EVE.crc
-rwxrwxrwx  1 cathy  staff     1048 Apr 21 00:41:24 2021 20210421_004122_EVE.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:41:28 2021 20210421_004128_BRP.crc
-rwxrwxrwx  1 cathy  staff     7026 Apr 21 00:41:28 2021 20210421_004128_BRP.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:41:28 2021 20210421_004129_PLD.crc
-rwxrwxrwx  1 cathy  staff     3358 Apr 21 00:41:28 2021 20210421_004129_PLD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:41:28 2021 20210421_004129_SAD.crc
-rwxrwxrwx  1 cathy  staff     1266 Apr 21 00:41:28 2021 20210421_004129_SAD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:43:52 2021 20210421_004351_BRP.crc
-rwxrwxrwx  1 cathy  staff   511194 Apr 21 00:43:52 2021 20210421_004351_BRP.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:43:52 2021 20210421_004352_PLD.crc
-rwxrwxrwx  1 cathy  staff    48886 Apr 21 00:43:52 2021 20210421_004352_PLD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 00:43:52 2021 20210421_004352_SAD.crc
-rwxrwxrwx  1 cathy  staff    21594 Apr 21 00:43:52 2021 20210421_004352_SAD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 02:10:34 2021 20210421_021034_BRP.crc
-rwxrwxrwx  1 cathy  staff   859310 Apr 21 02:10:34 2021 20210421_021034_BRP.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 02:10:34 2021 20210421_021034_PLD.crc
-rwxrwxrwx  1 cathy  staff    80322 Apr 21 02:10:34 2021 20210421_021034_PLD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 02:10:34 2021 20210421_021034_SAD.crc
-rwxrwxrwx  1 cathy  staff    35630 Apr 21 02:10:34 2021 20210421_021034_SAD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 04:35:40 2021 20210421_043540_BRP.crc
-rwxrwxrwx  1 cathy  staff  1357476 Apr 21 04:35:40 2021 20210421_043540_BRP.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 04:35:40 2021 20210421_043541_PLD.crc
-rwxrwxrwx  1 cathy  staff   125308 Apr 21 04:35:40 2021 20210421_043541_PLD.edf
-rwxrwxrwx  1 cathy  staff        8 Apr 21 04:35:42 2021 20210421_043541_SAD.crc
-rwxrwxrwx  1 cathy  staff    55716 Apr 21 04:35:42 2021 20210421_043541_SAD.edf

Of course all of the time stamps on those files are according to what the time the CPAP thinks it is. If there were a way to make copies with the correct times in the file timestamps while times according to the CPAP clock are  embedded in the filename and in the data that the CPAP wrote into the file, then OSCAR is probably 95% of the way to looking at the difference and being able to correct all of the times in all of the data, right?

(I have some experience with how this works because I have always managed my data in a particular way that is similar in spirit to what flashpap does. I created a .dmg image on my computer and copied the flash drive to it. Each time I load data, I put the card in my computer, open my .dmg image, copy the files from the card to the computer, eject the flash drive and go put it back into my cpap, and then come back to my computer and load the data from the .dmg into OSCAR (and before that Sleepyhead.) In 6-1/2 years I've only done a handful of nights with the card in the computer rather than in the CPAP machine!  Since all my experience is with reading copies of my data, I know how it works when the files on the card in the machine all have time stamps that correspond to what time the CPAP thought it was when they were created and written, but the files that OSCAR (/Sleepyhead) works with are copies and so their timestamps could be manipulated/corrected if there was some way to know what the correction is. And then since the times are in two places -- the metadata on the file and the name of the file, if the copies had the different timestamps then the difference between the CPAP clock and the real time would be a known constant for each file set.

At one point in March I went two nights with the machine clock set to 12 hours in the future. I was able to see that when I copied the files to my computer and looked at them less than 12 hours before I had been asleep, my computer was perfectly happy to tell me that the creation times on the files was in the future according to the computer's clock! I actually went in and edited all of the filenames to move them back 12 hours, and all of the recognizable date/times that I could find inside the files I also edited the times back from PM to AM. It more or less worked -- I couldn't find the date/time in the settings so OSCAR thinks that it doesn't know the settings for the first day in the run, but that's minor. Here is the directory listing, and those are the creation timestamps that the cpap machine put on the files, while the filenames were all manually corrected by me.)

Code:

bash-3.2$ cd ../20210315
bash-3.2$ ls -lUT
total 5248
-rwxrwxrwx  1 cathy  staff       8 Mar 16 12:45:16 2021 20210316_004515_CSL.crc
-rwxrwxrwx@ 1 cathy  staff     800 Mar 16 12:45:16 2021 20210316_004515_CSL.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 12:45:16 2021 20210316_004515_EVE.crc
-rwxrwxrwx@ 1 cathy  staff    2728 Mar 16 12:45:16 2021 20210316_004515_EVE.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 12:46:38 2021 20210316_004639_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  397156 Mar 16 12:46:38 2021 20210316_004639_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 12:46:40 2021 20210316_004639_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   38588 Mar 16 12:46:38 2021 20210316_004639_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 12:46:40 2021 20210316_004639_SAD.crc
-rwxrwxrwx@ 1 cathy  staff   16996 Mar 16 12:46:40 2021 20210316_004639_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 13:55:54 2021 20210316_015553_BRP.crc
-rwxrwxrwx@ 1 cathy  staff   97056 Mar 16 13:55:54 2021 20210316_015553_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 13:55:54 2021 20210316_015554_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   11488 Mar 16 13:55:54 2021 20210316_015554_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 13:55:54 2021 20210316_015554_SAD.crc
-rwxrwxrwx@ 1 cathy  staff    4896 Mar 16 13:55:54 2021 20210316_015554_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:12:54 2021 20210316_021254_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  253108 Mar 16 14:12:54 2021 20210316_021254_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:12:54 2021 20210316_021254_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   25580 Mar 16 14:12:54 2021 20210316_021254_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:12:54 2021 20210316_021254_SAD.crc
-rwxrwxrwx@ 1 cathy  staff   11188 Mar 16 14:12:54 2021 20210316_021254_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:56:06 2021 20210316_025607_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  157076 Mar 16 14:56:06 2021 20210316_025607_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:56:08 2021 20210316_025608_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   16908 Mar 16 14:56:08 2021 20210316_025608_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 14:56:08 2021 20210316_025608_SAD.crc
-rwxrwxrwx@ 1 cathy  staff    7316 Mar 16 14:56:08 2021 20210316_025608_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 15:23:56 2021 20210316_032357_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  433168 Mar 16 15:23:56 2021 20210316_032357_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 15:23:58 2021 20210316_032358_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   41840 Mar 16 15:23:58 2021 20210316_032358_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 15:23:58 2021 20210316_032358_SAD.crc
-rwxrwxrwx@ 1 cathy  staff   18448 Mar 16 15:23:58 2021 20210316_032358_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 16:36:36 2021 20210316_043636_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  745272 Mar 16 16:36:36 2021 20210316_043636_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 16:36:38 2021 20210316_043637_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   70024 Mar 16 16:36:36 2021 20210316_043637_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 16:36:38 2021 20210316_043637_SAD.crc
-rwxrwxrwx@ 1 cathy  staff   31032 Mar 16 16:36:38 2021 20210316_043637_SAD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 18:41:36 2021 20210316_064136_BRP.crc
-rwxrwxrwx@ 1 cathy  staff  145072 Mar 16 18:41:36 2021 20210316_064136_BRP.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 18:41:36 2021 20210316_064137_PLD.crc
-rwxrwxrwx@ 1 cathy  staff   15824 Mar 16 18:41:36 2021 20210316_064137_PLD.edf
-rwxrwxrwx  1 cathy  staff       8 Mar 16 18:41:36 2021 20210316_064137_SAD.crc
-rwxrwxrwx@ 1 cathy  staff    6832 Mar 16 18:41:36 2021 20210316_064137_SAD.edf

[biorn]

I hope I understood all you said, cause my english is worst.

First thing is, timestamp creation of a file should never be used. (to much possiblities of corruption)
so don't care about it.

I don't remember this part of acquisition but Oscar don't should be car about the file name (the time include in the name) because each file include the start time of the data and that in this way edf format was created. Maybe I'm wrong and oscar use it as baseline. In all case it's better to have the name corrected and the crc too. (I mean oscar check it)

this part of the file:

   


And the last thinks as I can see in your screenshot, the time isn't instant so the script should be care about hours difference and not the minutes or secondes I think.

Sorry if I missunderstoud something

[cathyf]

I hope I understood all you said, cause my english is worst.

First thing is, timestamp creation of a file should never be used. (to much possiblities of corruption)
so don't care about it.

I don't remember this part of acquisition but Oscar don't should be car about the file name (the time include in the name) because each file include the start time of the data and that in this way edf format was created. Maybe I'm wrong and oscar use it as baseline. In all case it's better to have the name corrected and the crc too. (I mean oscar check it)

this part of the file:




And the last thinks as I can see in your screenshot, the time isn't instant so the script should be care about hours difference and not the minutes or secondes I think.

Sorry if I missunderstoud something

Ok, I see what you mean but I'm going someplace different. I'm trying to figure out if there is a simple way to get ResMed data so that it can synchronize properly with other data sources like pulse-oximeters, etc. If everyone is synched to the real time (i.e. the atomic clock in Colorado) then it's trivial.

The internal clock in the cpap keeps time well enough that it's going to be off by the same amount for an entire night, at least to the resolution that we need. Every file has metadata, written by the flashcard device driver, and that metadata records the creation date/time according to the ResMed's system clock -- so what time the cpap machine thought that it was when it wrote the metadata.

I think -- from what you guys are telling me -- that the FlashAir subsystem actually can know how far off the ResMed system clock is, because the FlashAir has a network connection to your router which knows the atomic-clock-in-Colorado time and it simultaneously has read access to the flash card files and their file metadata which is being written by a device driver that is looking at the cpap system clock. So, for example, on my plain old ordinary flash card that was in my cpap on April 5th, the first file that got written in the DATALOG/20210405 directory is
    named 20210405_223835_EVE.edf
    has a creation time according to the device's clock of Apr  5 22:38:36 2021
    has data inside the file that includes the string 05.04.2122.38.37768
In other words, the time that the file was created has been recorded in three different places by two different programs that were both relying on the same clock. The ResMed code that runs the CPAP machine constructed the name of the file at 22:38:35 cpap clock time. The ResMed code issued the file create command which the disk driver obeyed at 22:38:36 according to the cpap clock. And then the ResMed code constructed the string that it wrote inside the file at 22:38:37.768.

What I think is true (if I understand correctly) is that if this had been a FlashAir card there is another tiny computer that is in the FlashAir device. That computer is talking to the router, and the router is connected to the internet, so that computer can know what time it REALLY is.

If the FlashAir computer is sitting there reading the card, waiting for files to appear, then the FlashAir will know when the file appears. If the FlashAir can be programmed so that when a new DATALOG/*/*.edf file appears the FlashAir immediately retrieves the real date & time from the router, then the FlashAir can compare the real date and time that the file appeared with what the cpap clock thinks the date and time is that the file was created. And *boom* the FlashAir card knows exactly how much the cpap clock is wrong.

If the FlashAir card can know how far the CPAP's clock has drifted, then somebody who understands the FlashPap scripts and somebody who understands OSCAR could figure out how to get FlashPap to tell OSCAR how to fix up all of the times in the ResMed data so that they are correct according to the real time.

Once the ResMed data has the correct atomic-clock-in-Colorado times, then every other data stream which also has the correct atomic-clock-in-Colorado times can be easily synchronized with the ResMed data.

If someone else has already tried to figure out how to make this work and that person discovered something that makes it impossible, I'll move on. But if you guys who know more about this than I do think it might work, then I would be game to buy myself a FlashAir card and take the deep dive into the code and see if I can figure out how to get the FlashAir to tell me how to correct all of the time data that my cpap has recorded with the wrong times.


What I was hoping when I first asked the question was that the FlashAir computer was already correcting the metadata on the copies of the files that it is responsible for copying over to the computer. But I didn't actually expect that to be happening, since that sort of metadata manipulation is not the usual practice when copying files over networks. (And since I was somewhat aware of the arguments that were made back in the early days of the Internet about times and clocks, I even understand why it's not generally a good idea.)

[biorn]

ha got it.

You just want the flashair write somewhere the drift clock information.
But in fact oscar will drift all the data.

So you always got the same difference with other data like oxymeter.

The airsense will be ok but the oxymeter will be drifted

The best way rest what I was purpose. Or just adjust the clock of the airsense

[Crimson Nape]

I don't mean to rain on your parade, but I haven't read any ideas to addressing the following.

The FlashAir card is nothing more than a storage device with wireless connectivity.   It doesn't possess the logic to know when the data was actually created.   Creation time has to be provided by the device creating the data.  If a file was created a 3AM but wasn't written to the FlashAir until 7AM, are the times inaccurate at the time of creation and/or at the time the data is being transferred?  How does the FlashAir determine when data being written to it is in real-time as opposed to being historic data?

Then we have the CPAP itself.  For a Resmed, all reporting parameters are not totally functional up to one minute after the displayed starting time. When all parameters are reporting, when was the observation made as opposed to when it is being reported?  

Another consideration are time variables provided by the oximeter's sensor location and the patient's pulse rate.  Even if you were able to synchronize the CPAP and the oximeter to the exact second,  there is a lag time from when an event occurs and the time as to when the sensor reads this change.   If a patient was wearing the sensor on a toe and is experiencing bradycardia, there is going to be a larger time delta as opposed to having tachycardia and wearing the sensor on a finger.

Both the CPAP and oximeter have a ramp, or normalization period, after pressing the start button.  On the CPAP, there is a blower pressure ramp-up after staring that is separate from the Ramp option.  When does the actual start time occur?   As soon as you press the start button, or when the blower is at full pressure?   For the oximeter, it takes several seconds for it to acquire and report a pulse and SpO₂ value.  How long is the normalization time for any given recording session?  

Viatom/Wellue data files provide a starting time and date, a time duration, and file size in their header.  Based on file size and time duration determines the sampling rate of the data.  Based on the math proving this, the reported start time is that of the first recorded sample.  Since there isn't any value, that I'm aware of, for the time between power-up and sensor normalization, this is the unknown.

Based on all the variables I listed above, I don't believe that a time synchronization is feasible or possible.  I am interested on your ideas as how to address these unknowns.  It may be something simple that I'm not seeing.

Please take into consideration that I have been known to march to a different drummer though.  
- Red

[biorn]

I don't mean to rain on your parade, but I haven't read any ideas to addressing the following.

The FlashAir card is nothing more than a storage device with wireless connectivity.   It doesn't possess the logic to know when the data was actually created.   Creation time has to be provided by the device creating the data.  If a file was created a 3AM but wasn't written to the FlashAir until 7AM, are the times inaccurate at the time of creation and/or at the time the data is being transferred?  How does the FlashAir determine when data being written to it is in real-time as opposed to being historic data?

Then we have the CPAP itself.  For a Resmed, all reporting parameters are not totally functional up to one minute after the displayed starting time. When all parameters are reporting, when was the observation made as opposed to when it is being reported?  

Another consideration are time variables provided by the oximeter's sensor location and the patient's pulse rate.  Even if you were able to synchronize the CPAP and the oximeter to the exact second,  there is a lag time from when an event occurs and the time as to when the sensor reads this change.   If a patient was wearing the sensor on a toe and is experiencing bradycardia, there is going to be a larger time delta as opposed to having tachycardia and wearing the sensor on a finger.

Both the CPAP and oximeter have a ramp, or normalization period, after pressing the start button.  On the CPAP, there is a blower pressure ramp-up after staring that is separate from the Ramp option.  When does the actual start time occur?   As soon as you press the start button, or when the blower is at full pressure?   For the oximeter, it takes several seconds for it to acquire and report a pulse and SpO₂ value.  How long is the normalization time for any given recording session?  

Viatom/Wellue data files provide a starting time and date, a time duration, and file size in their header.  Based on file size and time duration determines the sampling rate of the data.  Based on the math proving this, the reported start time is that of the first recorded sample.  Since there isn't any value, that I'm aware of, for the time between power-up and sensor normalization, this is the unknown.

Based on all the variables I listed above, I don't believe that a time synchronization is feasible or possible.  I am interested on your ideas as how to address these unknowns.  It may be something simple that I'm not seeing.

Please take into consideration that I have been known to march to a different drummer though.    
- Red

I was thinking the discussion was on timezone problem. not on micro second synch. Sorry if I missunderstood something

But flashair is a connecteded object, it's work like a computer in fact. Nucleus RTOS I presume.
I don't know when the file is created I just read the screen provided in the discussion and deduct the file is created at start of recording. But yeah you could also use the file crate timestamp drift to drift the starting time in the data of the file if it's an historique file.

maybe there is a spi solution, I don't now this card.