RE: OSCAR import of Dreem 2 CSV data
As I've been looking at this some more, updated observation, which is probably a general issue when integrating tz-aware data with tz-naive xPAP data in OSCAR:
Quote:If one moves across timezones with one's xPAP machine, the times gets out of sync.
Dreem CSV includes the timezone info in its CSV, but the output format of this script (and I assume the Zeo format, and also therefore the OSCAR import code) is not timezone-aware.
I guess the best way to do that is to convert all datetimes to UTC, and then convert to localized default timezone at the end?
I can't yet figure out how best to do that though.
Caveats: I'm just a patient, with no medical training.
01-31-2020, 12:25 PM
(This post was last modified: 01-31-2020, 12:25 PM by sawinglogz.)
RE: Zeo CSV format, hypnogram import
(10-10-2019, 08:26 AM)slowriter Wrote: They say yes, it will be stable now, and that improving data export is a big focus for them.
Who were you working with? When I've asked them about data export, they've been pretty disinterested.
I'm working on a native loader for their CSV files and I'm running into a very weird situation: there are too many hypnogram samples!
For example, I have a night of 504m34s according to the start time and stop time. That *should* be 1009 or maybe 1010 samples. But the hypnogram has 1011.
I thought maybe they were using the wall clock and emitting samples at :00 and :30, as well as the end time, but that doesn't seem to match my data either.
RE: Zeo CSV format, hypnogram import
(01-31-2020, 12:25 PM)sawinglogz Wrote: Who were you working with? When I've asked them about data export, they've been pretty disinterested.
I'm working on a native loader for their CSV files and I'm running into a very weird situation: there are too many hypnogram samples!
For example, I have a night of 504m34s according to the start time and stop time. That *should* be 1009 or maybe 1010 samples. But the hypnogram has 1011.
I thought maybe they were using the wall clock and emitting samples at :00 and :30, as well as the end time, but that doesn't seem to match my data either.
Have you looked at the python script on github I posted earlier? It's author has some comments there that may help?
I've not had any problems with the resulting data.
I'll PM you on your first question.
Caveats: I'm just a patient, with no medical training.
RE: OSCAR import of Dreem 2 CSV data
Script:
https://gist.github.com/jeremyblow/542df...1284f9aec4
I forked it to add timezone support (though I don't know how well).
Caveats: I'm just a patient, with no medical training.
RE: Zeo CSV format, hypnogram import
(11-21-2019, 01:23 PM)sawinglogz Wrote: Could you PM me a copy of the CSV that makes OSCAR crash? We don't like crash bugs.
It turned out any CSV without all of the expected headers was causing an assertion failure.
I just merged in a new native Dreem loader, and fixed the Zeo loader in the process. The Zeo loader should now no longer crash on arbitrary CSV files, and both Zeo and Dreem hypnograms are now drawn the right way up.
They also both account for discontinuities in the data, and Zeo loader now handles sleep stage 6 ("Deep (2)" according to Zeo Viewer), which was a surprise.
And, because OSCAR can now parse the Dreem files directly, it gets the time zone information right.
...now if only I could figure out why there are too many hyponogram samples in Dreem files...
01-31-2020, 05:48 PM
(This post was last modified: 02-01-2020, 07:11 AM by slowriter.)
?
Got it built, and imported a few months of data.
First, thanks much for doing this; appreciated!
Now, a couple/few details.
Good: imports super fast, and appears to be correct. Also, thanks for fixing the display (now right-sided up).
Minor issue: the numbers on the sidebar are now -1, -2, etc. Is that necessary to get it display correctly? Because it is confusing, and conflicts with the tool tip legend.
[moving separate bug report to another thread]
Caveats: I'm just a patient, with no medical training.
RE: ?
(01-31-2020, 05:48 PM)slowriter Wrote: If I have focus on the OSCAR window, I can't use my Gnome keyboard shortcuts to cycle through applications.
The only way I can resolve this is if I click on the top window bar.
Caveats: I'm just a patient, with no medical training.
RE: ?
(01-31-2020, 05:48 PM)slowriter Wrote: Good: imports super fast, and appears to be correct. Also, thanks for fixing the display (now right-sided up).
Minor issue: the numbers on the sidebar are now -1, -2, etc. Is that necessary to get it display correctly? Because it is confusing, and conflicts with the tool tip legend.
You're exactly right. That's how I got the display to be right side up. (Upside down was driving me nuts!)
I suppose we could update the tool tip legend. I figured people would overlook the difference between the tooltip and the scale, but evidently not! 
...of course, if you had Zeo data, you'd also notice that there's sometimes a -3.75 for the Deep-2 stage...
In a future version we can refine some of these details, such as inverting the chart without inverting the values, and putting text labels beside it instead of numeric...
Can you start a new thread for the non-loader issues? Someone else should take a look at that.
RE: ?
(01-31-2020, 06:06 PM)sawinglogz Wrote: (01-31-2020, 05:48 PM)slowriter Wrote: Good: imports super fast, and appears to be correct. Also, thanks for fixing the display (now right-sided up).
Minor issue: the numbers on the sidebar are now -1, -2, etc. Is that necessary to get it display correctly? Because it is confusing, and conflicts with the tool tip legend.
You're exactly right. That's how I got the display to be right side up. (Upside down was driving me nuts!)
I suppose we could update the tool tip legend. I figured people would overlook the difference between the tooltip and the scale, but evidently not!
...of course, if you had Zeo data, you'd also notice that there's sometimes a -3.75 for the Deep-2 stage...
In a future version we can refine some of these details, such as inverting the chart without inverting the values, and putting text labels beside it instead of numeric...
Can you start a new thread for the non-loader issues? Someone else should take a look at that.
New thread started.
The "minor issue" I noted is definitely not a huge deal. I just can imagine it would confuse some people.
Caveats: I'm just a patient, with no medical training.
RE: OSCAR import of Dreem 2 CSV data
Regarding DREEM sleep position data, they don't seem open to putting it in their downloadable files. I really wanted to get it though so I signed up for their research program and bought the $600 special version of their headband that permits downloading ALL the raw data. Unfortunately I didn't read all the fine print where it says that results they derive from the raw data, such as sleep position, are not available. Since the raw data is from a 3 axis accelerometer, you have to write your own program produce sleep positions. Not impossible of course but that means you have to also handle noise problems, ambiguities, etc and deal with the EDF format. I've put it on the back burner for now as there are more interesting things to do.
|
