OSCAR: Synchronization -- general question

[PurchanceToDream]

While searching for other info related to importing data, I've notice the issue of synchronization of data from different devices which presumably may have started taking data at different times.

As far as I've notice, if data from different devices has different start/stop times, the timeline of all charts is expanded to encompass both.

So wouldn't it be the case that the only issue of synchronization is whether the clocks on the devices themselves are in sync?

I suppose that can be an issue if you want the data lined up accurately to the second.  But I'm thinking that something like SPO2 drops will just all be offset by a few seconds and fairly obvious.

Is there more to it? In the posts I've seen about it, seems like it matters a lot which data you import first (cpap, oxymeter, etc.). Why would that matter at all?

For what it is worth, I find the imports into OSCAR (and same problem existed with Sleepyhead) from  CMS50F result in random small sections of data, of up to several minutes, get repeated, sometimes more than once with the same section of data. I only noticed because it ends up "padding" extra total time (rough and added 10 minutes for every hour of actual data) and I finally noticed that the end time for my O2 data would sometimes be 1-2 hours in the future when I had just taken the thing off and immediately uploaded the data. At first I thought the time was just off... but it had the correct start time. I mention this because if this is happening to you and you do not realize it, then it'll probably cause great headaches for you trying to "synchonize" data that cannot possibly be synchronized.  I have no idea whether the bug is in the CMS50F or in OSCAR/Sleepyhead. One other tidbit about it the bug: if you purge the data and import it again, it will not repeat the same sections of data nor the same total minutes of data. It is a very strange but. I cannot think of any "typical" programming error that would do this, unless maybe mismanage buffers and pointers in some C code.

[GuyScharf]

As far as I've notice, if data from different devices has different start/stop times, the timeline of all charts is expanded to encompass both.

So wouldn't it be the case that the only issue of synchronization is whether the clocks on the devices themselves are in sync?

Yes, the display area will expand to run from the minimum start time to the maximum start time.

Not all oximeters have internal clocks, so specifically setting a start time is sometimes required.

For what it is worth, I find the imports into OSCAR (and same problem existed with Sleepyhead) from  CMS50F result in random small sections of data, of up to several minutes, get repeated, sometimes more than once with the same section of data.

Yes, that error is sometimes seen when importing data directly from a CMS50x using the cable. If you import using the SpO2 assistant and the import the data file the spO2 assistant creates, all is well.

We do plan to rework oximeter import in a future release and should resolve this issue then. In the meantime, import data from the CMS50F to your PC using SpO2 Assistant, and then import the SpO2 Assistant data file into OSCAR.

[PurchanceToDream]

For what it is worth, I find the imports into OSCAR (and same problem existed with Sleepyhead) from  CMS50F result in random small sections of data, of up to several minutes, get repeated, sometimes more than once with the same section of data.

Yes, that error is sometimes seen when importing data directly from a CMS50x using the cable. If you import using the SpO2 assistant and the import the data file the spO2 assistant creates, all is well.

We do plan to rework oximeter import in a future release and should resolve this issue then. In the meantime, import data from the CMS50F to your PC using SpO2 Assistant, and then import the SpO2 Assistant data file into OSCAR.

Ah thanks! I did not know about that workaround.

Maybe once I get my apnea under control with cpap and have some energy & concentration, I'll clone the repo and fix that bug myself.

[GuyScharf]

We're always happy to have people contribute to OSCAR development!

[2SleepBetta]

Days ago, struggling with synchronizations, I installed Smart Device Assistant (SDA) 3.1.0.1 from Contec (oximeter) Support after discovering my SpO2 Assistant software, Vsn 2.4, had been writing its (almost hidden) CSV file entirely in what looks like Chinese ideographs. One other recent thread deals with that newer software. 

Below, I drone and on with observations reader/users of oximeters and other such sleep support devices may want to check into. I see no reason to start a separate thread and may be wrong in some of my recent passing observations reflected below. Hammer me if this amounts to (unintentional) hijacking  . 

That Contec foreign language software with its Chinese (?) CSV rendering was installed ca April 2019, incidental to buying a rehabbed laptop after being warned by my tech provider that my old ASUS with mechanical HD (English CSV) was on its last legs (with Vsn 2.7) with me in the middle of finishing my taxes.

As I recall, that software came from a site in Silicon Valley that, at the time and right or wrong, I saw and accepted as Contec's source of the UART code underlying my earlier versions of SA SpO2. I could not find my original Contec CD and at that hurried time so I took that errant approach to installing the Contec SpO2 S/W in the new-to-me computer (I should have found Contect Support then as I did days ago).

Correct me if wrong here: I was (mistakenly?) disappointed to find that the SDA 3.1... Vsn had data logged by the minute, rather than by the second as I believe the earlier versions had. I haven't started the old ASUS to check that old Vsn's CSV logging rate, but will. I had hoped to edit and use some other OSCAR import/emulation capability for the Contec's CSV file's synchronization.

All this and similar matters are part of (my naively and foolishly?) pushing my/the limits to get the best attainable synchronization of 3 out of 4 widely drifting device clocks--not to mention that one cannot push three buttons and use VAuto AutoOn to start all devices at the same time (nor would doing so start all their variously lagging time loggers at the same corrected time): VAuto, Accelerometer, Oximeter, Dreem 2 (D2 has good UTC time, isn't it?). Why do I care? It's to enable the best judgment call I/we can, with the tools/graphs in my/our hands, to personally and roughly score which irregular or high amplitude wave "bursts" in our/my Flow Rate wave forms are arousals? 

Which bursts reflect certain irrelevant (?) voluntary and involuntary motions, say, cough, sneeze, throat-catch/clearing, mere comfort-seeking motions vs. arousals?  To what extent do those bursts correlate with that particular sleep stage segment's unique flow rate burst(s) at/near the time of that sleep stage's change from Deep, REM or Light sleep to Light or WAKE stage? 

Lacking a real EEG to read brain waves, we have no good tool. The Dreem 2 has poor resolution (minimum 30 seconds, if not 60 or more) although it occasionally does pick up a WAKE event. But flow rate curve bursts, and motion and position curve bursts strongly indicate totally undetected arousals, arousals that often arise from low level UARS which is reflected in ONLY a preceding string of inspiratory flow rate peak deformations (no flags, most all the time).   

Later, I may check and keep some notes to look at this too: Inadvertently I discovered yesterday that my Contec CMS 50I handles and logs start up, quite possibly, with a consistent error for the first minute, depending on when  you hit the RECORD button in its "base or startup minute". Crude example: Successive "cold start ON and RECORD button presses" to check for the Slow/Fast error (make checks against smart phone time) will show, say, 10 sec. Fast, a second "quick Off then On and Record" check minutes later will show a 40 sec. Fast--a 30 second difference, regardless of the differing starting times' offsets. Hence, fast/slow checks are not in themselves consistent; the startup time fast-slow discrepancy apparently depends upon when you hit the RECORD button during or at the start of the device's initial startup window. Seemingly, the logged startup instant can be at least 30 seconds off (maybe several minutes because some SpO2 and PR curves' starts have minutes of gap between the 0-point origin and the start of the curves).

It's past time for me to be content for now with crude SpO2 synchs, especially since its curves are the most difficult to closely correlate with other graphs and to sleep-stages. It's fortunate that it has, for my uses, OK numbers and graphs and it is the least significant rest indicator of the four devices for me with a pacemaker. It is most difficult for me to synchronize the SpO2 closely because there is no distinctive and sharp marker (like there is with motion and flow rate coincident markers). I haven't found a way to reveal few-second (even 30 sec.) time synch errors for the device. Larger pulse rate rises, coincident with larger motions correlate with large flow rate bursts fairly well but SpO2 percentage much less so, although those often echo flow rate bursts some. Other users without a rate adjusting pacemaker that has a floor set for sleep, may be able to wring more info from their devices PR and SpO2 curves--may be able to see expected irregularities in pulse and SpO2 level correlate with REM sleep segments. 

Fortunately, the Dreem 2 logging is UTC based (if one cannot have the VAuto be the standard as would be best). When one uses the drift offset in OSCAR it comes with its own problems. Minor, but noted recently, the scale number size of a flow limit moves clear outside its bar graph profile when the drift offset is reset enough. 
Also, a drift reset will propagate backward, disturbing other dated synchronizations of that profile. Not sure, but I think getting back to default basis may require a Clear All  for each adjunct machine's graph and a restart of OSCAR.

Another bugaboo with the my Contec device's graphs: its curves sometimes begin at the time of the first logged time shown on the graph at the origin; but at other times the curve trace starts minutes later after 0-time. A defect in my device? Dunno. Overall, sharp pulse rate and SpO2 rises tend to correlate fairly well with motion and flow rate bursts and dips.

Again, OSCAR: what a great and amazing  sleep evaluation and guidance tool! Always advancing and being pushed by Team and users to overcome limits no one envisioned for it.  

It is always "gimme more" but that is, or once was, a key part of our counry's driving psyche and freedom: betterments and beneficial discoveries.   

  Fred, sawinglogz, guy Scharf, pholynyk, Crimson Nape and all the team. Many thanks to Mark Watkins, too, who laid OSCAR's base in SleepyHead, and most likely had team members' help to improve it in its day.

[PurchanceToDream]

Days ago, struggling with synchronizations, I installed Smart Device Assistant (SDA) 3.1.0.1 from Contec (oximeter) Support after discovering my SpO2 Assistant software, Vsn 2.4, had been writing its (almost hidden) CSV file entirely in what looks like Chinese ideographs. One other recent thread deals with that newer software. 

Below, I drone and on with observations reader/users of oximeters and other such sleep support devices may want to check into. I see no reason to start a separate thread and may be wrong in some of my recent passing observations reflected below. Hammer me if this amounts to (unintentional) hijacking  . 

That Contec foreign language software with its Chinese (?) CSV rendering was installed ca April 2019, incidental to buying a rehabbed laptop after being warned by my tech provider that my old ASUS with mechanical HD (English CSV) was on its last legs (with Vsn 2.7) with me in the middle of finishing my taxes.

As I recall, that software came from a site in Silicon Valley that, at the time and right or wrong, I saw and accepted as Contec's source of the UART code underlying my earlier versions of SA SpO2. I could not find my original Contec CD and at that hurried time so I took that errant approach to installing the Contec SpO2 S/W in the new-to-me computer (I should have found Contect Support then as I did days ago).

Correct me if wrong here: I was (mistakenly?) disappointed to find that the SDA 3.1... Vsn had data logged by the minute, rather than by the second as I believe the earlier versions had. I haven't started the old ASUS to check that old Vsn's CSV logging rate, but will. I had hoped to edit and use some other OSCAR import/emulation capability for the Contec's CSV file's synchronization.

All this and similar matters are part of (my naively and foolishly?) pushing my/the limits to get the best attainable synchronization of 3 out of 4 widely drifting device clocks--not to mention that one cannot push three buttons and use VAuto AutoOn to start all devices at the same time (nor would doing so start all their variously lagging time loggers at the same corrected time): VAuto, Accelerometer, Oximeter, Dreem 2 (D2 has good UTC time, isn't it?). Why do I care? It's to enable the best judgment call I/we can, with the tools/graphs in my/our hands, to personally and roughly score which irregular or high amplitude wave "bursts" in our/my Flow Rate wave forms are arousals? 

Which bursts reflect certain irrelevant (?) voluntary and involuntary motions, say, cough, sneeze, throat-catch/clearing, mere comfort-seeking motions vs. arousals?  To what extent do those bursts correlate with that particular sleep stage segment's unique flow rate burst(s) at/near the time of that sleep stage's change from Deep, REM or Light sleep to Light or WAKE stage? 

Lacking a real EEG to read brain waves, we have no good tool. The Dreem 2 has poor resolution (minimum 30 seconds, if not 60 or more) although it occasionally does pick up a WAKE event. But flow rate curve bursts, and motion and position curve bursts strongly indicate totally undetected arousals, arousals that often arise from low level UARS which is reflected in ONLY a preceding string of inspiratory flow rate peak deformations (no flags, most all the time).   

Later, I may check and keep some notes to look at this too: Inadvertently I discovered yesterday that my Contec CMS 50I handles and logs start up, quite possibly, with a consistent error for the first minute, depending on when  you hit the RECORD button in its "base or startup minute". Crude example: Successive "cold start ON and RECORD button presses" to check for the Slow/Fast error (make checks against smart phone time) will show, say, 10 sec. Fast, a second "quick Off then On and Record" check minutes later will show a 40 sec. Fast--a 30 second difference, regardless of the differing starting times' offsets. Hence, fast/slow checks are not in themselves consistent; the startup time fast-slow discrepancy apparently depends upon when you hit the RECORD button during or at the start of the device's initial startup window. Seemingly, the logged startup instant can be at least 30 seconds off (maybe several minutes because some SpO2 and PR curves' starts have minutes of gap between the 0-point origin and the start of the curves).

It's past time for me to be content for now with crude SpO2 synchs, especially since its curves are the most difficult to closely correlate with other graphs and to sleep-stages. It's fortunate that it has, for my uses, OK numbers and graphs and it is the least significant rest indicator of the four devices for me with a pacemaker. It is most difficult for me to synchronize the SpO2 closely because there is no distinctive and sharp marker (like there is with motion and flow rate coincident markers). I haven't found a way to reveal few-second (even 30 sec.) time synch errors for the device. Larger pulse rate rises, coincident with larger motions correlate with large flow rate bursts fairly well but SpO2 percentage much less so, although those often echo flow rate bursts some. Other users without a rate adjusting pacemaker that has a floor set for sleep, may be able to wring more info from their devices PR and SpO2 curves--may be able to see expected irregularities in pulse and SpO2 level correlate with REM sleep segments. 

Fortunately, the Dreem 2 logging is UTC based (if one cannot have the VAuto be the standard as would be best). When one uses the drift offset in OSCAR it comes with its own problems. Minor, but noted recently, the scale number size of a flow limit moves clear outside its bar graph profile when the drift offset is reset enough. 
Also, a drift reset will propagate backward, disturbing other dated synchronizations of that profile. Not sure, but I think getting back to default basis may require a Clear All  for each adjunct machine's graph and a restart of OSCAR.

Another bugaboo with the my Contec device's graphs: its curves sometimes begin at the time of the first logged time shown on the graph at the origin; but at other times the curve trace starts minutes later after 0-time. A defect in my device? Dunno. Overall, sharp pulse rate and SpO2 rises tend to correlate fairly well with motion and flow rate bursts and dips.

Again, OSCAR: what a great and amazing  sleep evaluation and guidance tool! Always advancing and being pushed by Team and users to overcome limits no one envisioned for it.  

It is always "gimme more" but that is, or once was, a key part of our counry's driving psyche and freedom: betterments and beneficial discoveries.   

  Fred, sawinglogz, guy Scharf, pholynyk, Crimson Nape and all the team. Many thanks to Mark Watkins, too, who laid OSCAR's base in SleepyHead, and most likely had team members' help to improve it in its day.