Oscar sessions seem wrong

[DanEm]

Following sound advice, my questions will now be posted here.

Please note that my condition requires I monitor the sessions’ data closely rather than daily data.

It all started when I noticed one missing session in Oscar.

After investigation and discussion, I found also the time stamp for my ResMed files on the SD card is off by approximately 5 hours and the session missing happens to have the time stamped after noon.  Does Oscar use the time stamp of the file for something related to the sessions display?

My last action was to get the exported sessions csv files from Oscar. Each time I purged all data from Oscar then added the last ResMed data to read by Oscar. The Excel files showed that some prior data was changed to zero thus the last csv I get from Oscar does not have the real data for previous sessions.

I have put those exported csv in one Excel file included with this post. The tabs are the end dates of the last row of data.
Do I have a flawed version of Oscar?  What should I do next?

[GuyScharf]

After investigation and discussion, I found also the time stamp for my ResMed files on the SD card is off by approximately 5 hours and the session missing happens to have the time stamped after noon.  Does Oscar use the time stamp of the file for something related to the sessions display?

I will pass this question on to our ResMed developer. What timezone are you in and what timezone is your AS11 set to? (That setting would be in the clinical menu.) Or just attach the SETTINGS/settings.json file (I think that's the name) to a post so we can see what is in it.

My last action was to get the exported sessions csv files from Oscar. Each time I purged all data from Oscar then added the last ResMed data to read by Oscar. The Excel files showed that some prior data was changed to zero thus the last csv I get from Oscar does not have the real data for previous sessions.

If you purge the data from OSCAR, then the data will be gone (i.e., have a value of zero). Why are you purging data from OSCAR? That is rarely useful except to resolve strange problems, and best done with the guidance of an expert here on the forum.

If you purge the first day (again), and then import from your SD card again, all data should be present.

Re: missing session

You showed me a png file and thought the last session was not shown in the session slider at the bottom left of the Daily page. That session was there -- it was the blue bar on the right -- but it was short enough that the elapsed time could not be displayed. The sessions in the bar alternate between yellow and blue. There's no significance to the color choice; it is just that they are different so that if you have a lot of short sessions the session bar doesn't completely blur together.

[DanEm]

GuyScharf
What timezone are you in and what timezone is your AS11 set to? (That setting would be in the clinical menu.) Or just attach the SETTINGS/settings.json file (I think that's the name) to a post so we can see what is in it.

It was checked.

DanEm
01-19-2022, 03:48 PM
RE: Oscar glitched or what?
My time zone is (UTC-05:00) Eastern Time (US and Canada).

The test I did was run on 2022-01-19 14:04.

The date/time stamped in the files is  2022-01-19 19:04.
So the stamp is 5 hours after the real time.

In the CurrentSettings.json file before the reboot, the parameter setting is: 
"TimeZoneFeature":{"TimeZoneOffset":"-05:00"}
This seems OK for my location.

In the CurrentSettings.json file after the reboot, the parameter setting is the same.

In Oscar's graphics the time scale is the right one (14:04) so the content of the files is OK. 
Oscar must use the date/time stamp of the files within the routine concerning the day period (from noon to noon).
But why Oscar does not add the missed session to the next day as usual?

GuyScharf
If you purge the data from OSCAR, then the data will be gone (i.e., have a value of zero). Why are you purging data from OSCAR? That is rarely useful except to resolve strange problems, and best done with the guidance of an expert here on the forum.

If you purge the first day (again), and then import from your SD card again, all data should be present.

Steps taken when I purge data

1. Start laptop without SD card
   
2. Open Oscar and purge all data
   
3. Close Oscar
   
4. Insert SD card in laptop
   
5. Copy SD card content to dedicated folder on laptop
   
6. Remove SD card from laptop
   
7. Open Oscar and import saved data.
   

I had discussion somewhere in the forum where purging data did not seem to be problematic.  I can't remember where.
If there is better way to make use of purge please tell me.

GuyScharf
You showed me a png file and thought the last session was not shown in the session slider at the bottom left of the Daily page. That session was there -- it was the blue bar on the right -- but it was short enough that the elapsed time could not be displayed. The sessions in the bar alternate between yellow and blue. There's no significance to the color choice; it is just that they are different so that if you have a lot of short sessions the session bar doesn't completely blur together.

Thank you for this info.
But the missing session comment I made is still valid.
Not in the screenshot with the red arrows

But...

It must have been something wrong I did because now when I go back to that date in Oscar the missing session is there.
See screenshot January_10_A made January 12  and January_10_B made January 29.
(Re: time stamp discussion)

[GuyScharf]

I will leave the time discussion for pholynyk to address as he knows the innards of the ResMed importer

Regarding purging, are you purging the current selected day, or are you purging ALL machine data?

Also, why are you purging the data? What are you trying to accomplish by purging data? With your description of what you are doing, I don't see that accomplishes anything except making the import process take longer. 

Are you reformatting or changing the contents of your SD card?

Does Oscar use the time stamp of the file for something related to the sessions display?

No. If it uses the timestamp of the file for anything, it would be in the import process. After the data is imported, the file timestamp information is not retained.

[DanEm]

why are you purging the data? What are you trying to accomplish by purging data? With your description of what you are doing, I don't see that accomplishes anything except making the import process take longer.

Not knowing all what Oscar is doing and the changes in data I see in the exported csv have me very cautious no to mess the data on the SD card for one thing.
So, I am not certain of Oscar's behavior at my end and that to is of some concern and it is why I made certain to protect the SD card whether on import or export..
Perhaps I am overdoing it but better be safe than sorry.
The same reasoning applies to the purge. Maybe for this one I am not on the right track.

The last purge I did was for ALL the machine data.
I did not notice anything unusual after the purge and the import of ResMed data.

To conclude, I will stop those purges and see how things evolve.

Are you reformatting or changing the contents of your SD card?

Certainly not even though I could since I have a full backup of the card.

If it uses the timestamp of the file for anything, it would be in the import process.

Got it.

[GuyScharf]

Not knowing all what Oscar is doing and the changes in data I see in the exported csv have me very cautious no to mess the data on the SD card for one thing.

I understand the concern about safety of your data.

OSCAR never writes to the SD card.

[pholynyk]

I created a debug version to check the correspondence between the filename date/time with the file's internal start date/time. They match, as long as the time zone setting matches the OSCAR computer's time zone. This is an issue that need to be resolved, but does not affect the data here. The solution is rather involved, and I am still working on it.

In the meanwhile, I'd like you to record your first session start time after noon for a couple of days and then zip up your SD card as before and upload it to our Dropbox: https://www.dropbox.com/request/REp6AH7LCGyeOcXfOokH

Much appreciated

[DanEm]

pholynyk

I'd like you to record your first session start time after noon for a couple of days

I'm not certain I get this right. That first session is usually when I go to bed for the night. Is that what you want?

If I send the last csv file I got, won't you have all the start time? You just need to filter out the ones you don't want.

So I can give you precisely what you want, maybe you can give me more details.

---

I did some experiment on the subject and this is what I did to get more information on my case of missing data.

1. Deleted all ResMed data on backup of SD card except the 2 very first sets (Dec 22 2021 & Dec 23 2021 in DATALOG).
   
2. Opened Oscar imported the above.
   
3. Nothing changed, no data was erased.
   
4. Purged all Machine Data in Oscar.
   
5. Every day after Dec 23 2021 is shown but there is no data to graph.
   
6. Exported all session csv.
   

Results are in DanEmData.csv

1. Closed Oscar and reopened it.
   
2. Imported last ResMed data saved from SD card.
   

Last day Jan 29 2022 shows and for all other days, except Dec 22 2021 & Dec 23 2021, there is no data to graph.

1. Purged all Machine Data and re-imported same.
   

Now, everything is back.

I think what I did show there is a bug in Oscar with import and export sessions.
I know now how to go back and recuperate what data is missing in the last session. It will be a long process but it will work.
Hopefully this will be useful for you guys and you can fix it.
If I can help in any way, I will be glad to do it.

DanEm

[GuyScharf]

I did some experiment on the subject and this is what I did to get more information on my case of missing data.

1. Deleted all ResMed data on backup of SD card except the 2 very first sets (Dec 22 2021 & Dec 23 2021 in DATALOG).
   

No, don't alter the contents of the Backup folder. All the consequences you report are a result of doing that. I cannot think of a situation in which deleting files from the Backup folder would be appropriate.

Try this:

1. Create a new profile
   
2. Import your SD card into the new profile
   
3. Export CSV and see what the data looks like and let us know
   

[DanEm]

[b]GuyScharf[/b]
No, don't alter the contents of the Backup folder. All the consequences you report are a result of doing that. I cannot think of a situation in which deleting files from the Backup folder would be appropriate.

The backup is only a copy I make of the SD card.  It can be re-done anytime.
I don't see any harm playing with it?

Anyway, I did what you suggested.

Compared session csv file, old profile vs new profile.

At first glance did not see differences.
Then thought of a bit by bit comparison and used SourceGear DiffMerge.

Results are in ZIP file.

Missing data for session 1642619040 was added in new profile.

Some data for sessions 1643223300, 1643254980, 1643271360, 1643289900 was changed for 0 (zero) in new profile.

The rest is the same.

Tell me if you need more.