+- Apnea Board Forum - CPAP | Sleep Apnea (https://www.apneaboard.com/forums)
+-- Forum: Public Area (https://www.apneaboard.com/forums/Forum-Public-Area)
+--- Forum: Software Support Forum (https://www.apneaboard.com/forums/Forum-Software-Support-Forum)
+--- Thread: how export raw data of the flow rate? (/Thread-how-export-raw-data-of-the-flow-rate)
how export raw data of the flow rate? - BB63 - 08-11-2020
I'm not sure if this is possible, but would appreciate some input.
I'd like to export the raw (x,y) data for the flow from Oscar, so I can import it into a spreadsheet for more different types of analysis. Specifically, I'd like to be able to quantify the length of each inhale and exhale and plot this data using the spreadsheets powerful and very flexible display capability for charts. While OSCAR is fine for displaying the flow rate data, having it in a spreadsheet would let me play and hopefully learn a little more. I DO not do programming/software, so for me a spreadsheet in highly desirable because I know my way around them well.
Can the raw (x,y) data be exported from OSCAR? Or, maybe from the .edf files on the SD card have this data?
TY.
AB
PS:In looking at the hardware in my CPAP, it's NOT clear that there is direct flow rate metering, though I haven't fully dissected my CPAP yet. Since the air pump isn't a positive displacement pump, its tachometer output can't ACCURATELY quantify flow rate-so I am a little puzzled. A pressure reading can produce flow data for a specific restriction, but the body's restriction is dynamic for multiple reasons and varies widely in the short term.
RE: how export raw data of the flow rate? - sawinglogz - 08-11-2020
The Ti/Te graphs show what the (Resmed) machine is reporting as your inhale and exhale times. (Resmed only: OSCAR tries to calculate this from the flow graph for PRS1, but its algorithm very broken.)
Are you needing something beyond that?
RE: how export raw data of the flow rate? - pholynyk - 08-11-2020
The AutoSet does not report Ti and e; but the VAuto does.
The problem I see with exporting the flow or pressure data is that it comes 25 times a second, so a csv file becomes very large very fast. It's a terribly inefficient way of transporting repetitive data.
It would be much better to use one of the tools used for looking at edf files. http://www.teuniz.net has a number of tools, including the edfbrowser which I use. Also Google 'edfinfo' - I've misplaced the link - for more information about the file format.
RE: how export raw data of the flow rate? - BB63 - 08-11-2020
Yes, I do need other info. I am only needing the x, y, data points that produce the Flow Rate data. I will calculate the TI/TE from the Flow Rate raw data if needed. I mentioned TI/TE as an example of something I'd like to do, but it was probably an error on my part and probably led you to believe my interest was in the TI/TE Oscar data.
I only need/want the raw data from the Flow Rate plot.
I know noting about the format of the data stored on the SD card for my Resmed machine. The info would be helpful, but unless it's documented somewhere that accessible to me, I'd prefer not to burden others with a request for info on the data format that my Resmed produces...it's not absolutely essential for my purposes.
I only need the raw data for the Flow Rate plot.
TY
AB
RE: how export raw data of the flow rate? - Crimson Nape - 08-11-2020
Resmed's use an EDF file format. Here is a definition from Wikipedia:
Quote:European Data Format (EDF) is a standard file format designed for exchange and storage of medical time series. Being an open and non-proprietary format, EDF(+) is commonly used to archive, exchange and analyse data from commercial devices in a format that is independent of the acquisition system.
RE: how export raw data of the flow rate? - BB63 - 08-11-2020
(08-11-2020, 02:18 PM)pholynyk Wrote: The problem I see with exporting the flow or pressure data is that it comes 25 times a second, so a csv file becomes very large very fast. It's a terribly inefficient way of transporting repetitive data.
It would be much better to use one of the tools used for looking at edf files. http://www.teuniz.net has a number of tools, including the edfbrowser which I use. Also Google 'edfinfo' - I've misplaced the link - for more information about the file format.
TY pholynyk!
Datapoints for flow every 40 mSec give me the needed resolution!!! I routinely use csv's of 700,000 lines with each line having 14 to 20 cells. It does get a bit slow with files that large, but Calc (in its native environment) does FB!
I did manage to compile from the source code and had a brief look at edf browser and converted all the .edf files for yesterday to .csv files. I found much of the data is reported every 2 seconds and some data is reported 25 times per second. I did see the raw data points in the converted .csv files though!!! I couldn't produce a graphic display of the data points, which I believe edf browser promises to do. But, I didn't even look at the docs or the help file. So, the brief peek and this message is very preliminary!
TY for the pointer, I might look into other programs for viewing .edf files, but for now, edf viewer looks GREAT!
Teunis van Beelen, the author also turns out to be an EE, with experience in designing EEG & ECG medical devices. His program even contains an algorithm for removing 50 and 60 Hz power line interference in EEG plots!!! Ultimately, I want to roll my own EEG and do an FFT on it to determine which state of sleep the user is in, and to plot that in OSCAR too. However, that is beyond the capability of the Arduino microcontroller I'm working with now.
Again, many thanks for the nudge in the right direction!
AB
RE: how export raw data of the flow rate? - BB63 - 08-11-2020
(08-11-2020, 03:09 PM)Crimson Nape Wrote: Resmed's use an EDF file format. Here is a definition from Wikipedia:
Quote:European Data Format (EDF) is a standard file format designed for exchange and storage of medical time series. Being an open and non-proprietary format, EDF(+) is commonly used to archive, exchange and analyse data from commercial devices in a format that is independent of the acquisition system.
I had seen the Wikipedia article and was familiar with the EDF's ability to contain just about any form of data. It isn't viewable in a text editor though, but I am sure I can find the specification for the EDF's formatting online.
A quick question.......
If I design a datalogging system that produces output files in proper EDF, will OSCAR import and display it without the need for a programmer or a developer to make changes to the OSCAR source code?
TY so much, and be safe.
AB
RE: how export raw data of the flow rate? - pholynyk - 08-11-2020
The current OSCAR database schema requires pre-defined channel codes, so it's not possible to add arbitrary channels on the fly.
I can see that it could be very useful to have that flexibility in a new database design, though. We will keep that in mind. Ideally, it would be nice to allow both csv and edf formats to define channels.
RE: how export raw data of the flow rate? - 2SleepBetta - 08-11-2020
BB63,
I'm with you part way but tagging along far behind, not knowing much about numbers of things you and kappa mention from your engineering, medical and nutrition backgrounds here at AB. It's good to see you two pushing the envelope where you do as I dwell nearer the sleep apnea treatment core of things. I'll toss in some things below that you are likely well aware of and beyond, but an item or two may trigger a promising thought if you care to read on or make suggestions.
More to your point, I am quite sure, at minimum, that I once got some ResMed AutoSet FR data from that largest edf file, exported it to a CSV file, charted it in Excel and compared it to my Excel or a SleepyHead graph and saw they agreed.
Do you anticipate being able to accurately synchronize acceleration, flow, HR and SpO2 data, maybe even EEG, and esophageal or nasal pressure data--all to be presented in graphic form? My guess is that at least having all data gathering logged by one clock in one multifunction device is highly desirable and necessary.
The point polynyck made about huge bioemetric data files resonates. My oldest Excel (2007) limited me to 32k rows of data for charting so I reduced sampling rates to lowest levels for my devices, 8 and 16 Hz, and adjusted dead zone bands and restart bursts to record only about 8-16k rows per night. A later old version (2010) on the other laptop allows more rows, not sure how many, but I should up the X2 rate to get rid of tiny jog artifacts I see in the baseline for slight changes due to wakeups from dead times that need to be shortened. I did download R to deal with more data, as Gulf Data Concepts mentions, and have only fiddled with looking at the ResMed .edf data files briefly, but not been motivated to go farther than charting some of the AutoSense AutoSet flow data once--so it is doable (with CSV the R output, I think I used) as you intend. Comfortable with Excel back to 2.0, even its crude macros then [1986?] and later improvements, I've aged, become too lazy and taken sleep improvement almost as far as I want to try in remaining discretionary time.
The sequences from my three sleep-data recording devices suggest diagnostic possibilities but leave hanging the tough question whether a change observed is cause or effect: this due at least to different lags at startup. Further, my inexpensive oximeter senses from a heart-and-time-distant finger tip and the sampling rate is only 1 Hz as I recall, at least that is what it is reduced to in the CSV file, probably in the SpO2 Assistant presentation, too. To deal with data from three clocks it helps to start them all at once, as we do, and then to correct out their drifts in the analysis spreadsheet (the zero-point for t of x, y , z) or in OSCAR (import window for oximeter). The VAuto Easy On helps: with a huff and puff the VAuto starts as I start the accelerometer and then CMS50I all within a fraction of a second.
Trying to wring out of Excel and Oscar analyses all we can about UARS related questions, we devise crude synchronization related helps and improvised markers: "simultaneous" startup, sharp, time-spaced movements while "on the blower" with oximeter recording, time separation of the sharpest and deepest dips and rebounds in SpO2 vs. the most visually correlatable FR drops and recovery breath rebound, breaking the mask seal briefly to register a LL, doing test sets of time spaced movements, etc. (You or kappa indicated getting time uncertainty down to 0.5 second using kicks for the accelerometer synch, as I recall, if you care to elaborate how, exactly.)
Not (at this stage anyway) aware of any PLM or RLS related issues, I have experimented with wearing and plotting data from two accelerometers, one worn against my central upper forehead and one at the small of my back near CG (neither is firmly affixed, but are sufficiently so for current assessments). Range and, er, intensity of motion and frequency are much more, of course, at forehead as expected, and lumbar vs forehead visual correlations are much as expected.
There is a local Maker Group with members dabbling in all kind of hands on things, including digital electronics. I attended some of their meetings (led by a pediatrician and guided by an EE) when I thought I'd have to make my own accelerometer, using (e.g., google Adafruit) extensive component offerings for electronic DIYers. But I stumbled onto a fellow (once or still a member of the another SA forum owned by a DME) who once had a website where he had posted a sleep position graph of GDC accelerometer data. Got one, the X16-1D (AA batteries) and it soon convinced me I had to do what I did to stop any possibility of supine sleep. My OSA and AHI plummeted, so now continual fl that grade into FL and some arousals are the analysis project. Excellent AB advisers and the Autoset Rx got me well on my way to <1.0 and their advice and the VAuto's PS have dramatically reduced detectable levels of flow limits.
Related to what may be your concern about the GCD devices: It may be I dropped and damaged the more sensitive (2 g tolerant?) X2 (rechargeable) GDC unit causing its off-spec anisotropic sensitivity. I can only do a basic check of its 12,800 beat (?) +1g constant in each of 6 orientations, testing each when pointing close to nadir. How those vary at angles? No idea, but results as far as indications of sleep position, motion events and intensity of accelerations are more than sufficiently accurate at and with my knowledge level, methods for synchronization and the excellent capabilities of OSCAR--though I hope the latter may soon retain the Somnopose data (and include a way to relabel the default "Inclination" label). I've toyed with using my crude tests in 6 directions to enable scaling all 6 directions the same so as to eliminate the artifact displacements that are there and most evident in the jog that occurs in the 0-motion axis when I switch the side I sleep on. STI and probably others have a paper I got but for doing (undone) tumble tests to address the matter. But, stepping back, small errors in my graphs of position angle or a few hundredths of acceleration error are trivial compared to synchronization limitations and, as you point out, inaccuracies arising from pump flow rate data. As far as rotation position is concerned, only a single near vertical axis (x) suffices during side sleep, mostly at 75 degrees from supine. My anterior-posterior (navel to lumbar-L3/L4) axis (z) is the most sensitive detector and it has the greatest variation range when sleeping laterally--x not being able to change as much as z does with rocking motions from a base rotation angle. The superior-inferior axis (y) loafs, merely echoing other movements very slightly as I lie near 90 degrees to the vertical the whole night abed. (I think that is what "Inclination" is for in the Somnopose device usage--a measure of anterior-inferior tilt angle.)
Finally, on remembering your caveat about short trial of non-dairy, I was fully aware how lucky I'd be if it made a difference in a week. But with episodic drainage, other throat accumulations, and continuous deformations of inhalation flow graph peaks--all those together with awareness that singers swear off dairy some time before performing suggested I do a quick trial I'd never done as an adult. But I do remember the checker board drawn on my preschooler back where a large number personal allergen suspects were tested at pin pricks about 75 years ago. Most actual allergens have hidden underground or disappeared a I aged. Last things: slinging nasal mucous and "crying" copious tears, at 16, while sweating heavily and crouched inside the hot threshing chamber of a hillside grain combine that I was unclogging (almost blindlly cutting the wrapped around blooming green sweet clover from a hills of the rolling Palouse in SE WA State); then, still, my occasions when household dusts trigger up to 20 sneezes or bright sudden sunlight triggers a couple.
Yeah, some old men and their stories go on and on!
Anyway, I'm verbose as usual and hoping to see you moving forward and keeping us posted.
2SB