Reading SleepyHead Binary Files

[darral.freund]

Hello,


I just joined the forum in the hopes of getting some help with a problem I’m having.

I have a Respironics DreamStation and CMS50F pulse oximeter, and I transfer the data daily to the SleepyHead Software, V1.0.0-beta-2, running on MacBook Pro, OS X version 10.11.6.   I love that program and it works great for me.

I’ve been using the CPAP/pulse oximeter for about 10 months.  It took a while for my doctors to determine exactly my medical problem, but I actually have severe hyper-dynamic airway collapse as opposed to sleep apnea.  Because of this, my doctors are interesting in obtaining more information about my sleeping SpO2 than is reported by the Sleepyhead software.  

I’ve done a lot of matlab programming before I retired, and so I figured I could decode the binary sleepyhead files and get the information I need — I want to create a set of histograms of the percentage of time my SpO2 is at various levels over the course of several nights using the historical data I have stored in the sleepyhead binary files.

Unfortunately, accessing this data would be easy if I had a PC as I can read both the cvs file and the .SpO2 files generated by the native software of the CMS50F that runs on Windows.  But that software doesn’t run on a Mac.  I can get around that for the future by various means, but I have 10 months of historical data that I can’t access.

I bought a copy of Synalyze It Pro, and downloaded the source for Sleepyhead figuring I could decode the binary files with a bit of effort using the source as a guide.  I don’t really have any experience using C++ , and especially not with Qt.  I can see information about the structure of the data in the source, but I can’t completely unwravel it.  I can parse the header, but when I search through the binary data files, which if I’m interpreting the fields correctly the CMS50F data isn’t compressed, I can’t find any evidence of the data values that were recorded.

I’ve been unsuccessful in trying to contact the developers about this problem.  I suspect they figure I should be able to decode the format with the source, etc., but I’ve put in considerable effort and it’s definitely beyond my abilities at this point.

So, I’m wondering if anyone had successfully parsed the sleepyhead binary data files for use in other software — in my case software that I’m trying to write — or if someone had developed a grammar for Synalyze It Pro?  Any insights or suggestions would be greatly appreciated.

I will happily share whatever I write with anyone who wants it.

Thanks greatly,

Darral

[srlevine1]

Hey Darral,

Why reinvent the wheel?

Why not use Parallels to run the Sp02Assistant and grab the csv data? You can get a free 14-day full-featured trial to play with. And, you can get a cheap copy of Windows 7 to install.

Commercial Link Removed.  Do a search for Parallels Desktop or Parallels for Mac.

-----
Moderator Action: Link Removed

To maintain our status as an educational organization, the only commercial links allowed in this forum are to CPAP-related manufacturer websites.  This is stated in the Apnea Board Rules with details given in the Commercial Links Policy section.

-----

[darral.freund]

Yes,  I'm going to do that.  Or use the vmware version.  But, I have all this archival data where my treatments and conditions have changed and I would really like to get access to it. Switching to a VM now won't help me with that as I was using Sleepyhead directly to read the data.  So, I'm still sort of out of luck as there are no .SpO2 files.

Thanks,

Darral

[adrinef]

Darral,

Which file\files are you reading? The files located under Events for your oximeter device? Can you explain how you are reading the header and what fields are contained in the header?

[TBMx]

@darrel: you can see the format in the "session.cpp" .. look in the "Session::StoreEvents()"

but I did not fully understand what it is exactly that you want to see or know.
Most likely it would be easier done directly in sleepyHead.
(c++ is not that hard to learn if you have at least some knowledge in any kind of procedural language - for what you want to do QT is not that important - all you have to do is installing the framework^^)

[darral.freund]

Hi,

Thanks greatly for your reply!


I definitely thought about doing what you suggested with making changes to the source and recompiling it.  But after reading about all the issues with the latest release of Xcode, I decided that would be more difficult, especially since I’d have to really learn C++ and then probably write code to export the data in a more friendly format.

I had previously found the part of the file where you said to look (Session::StoreEvents(QString filename)), and I can read the header just fine with the following matlab listed below.  I tried it with a test file that I also processed on a PC using the native CMS50 software, and so I have the actual numbers in a spreadsheet and was also able to read the .Sp02 file using matlab.  
[fn, pn] = uigetfile('*.001','Open data file');
fid = fopen([pn fn] ,'r');
sp02 =[];
sp02.magic_number = fread(fid,1,'uint32'); 3341948587
sp02.file_version = fread(fid,1,'uint16'); 10
sp02.file_type_data = fread(fid,1,'uint16'); 1
sp02.machine_ID = fread(fid,1,'uint32'); 280801473
sp02.session_ID = fread(fid,1,'uint32'); 1478048538
sp02.start_time = fread(fid,1,'int64'); 1478048538000
sp02.end_time_dur = fread(fid,1,'int64'); 1478051057000
sp02.compression = fread(fid,1,'uint16'); 0
sp02.machine_type = fread(fid,1,'uint16'); 2

The problem is that I don’t understand C++ well enough to really figure out what it’s doing.  I can read up to the last line of code:

header << (quint16)s_machine->GetType();// Machine Type

 I think I get all the right answers up to and including the machine time, and the start and stop times are in milliseconds after the Unix datum of 1970-01-01. They convert nicely to the correct values down to the second in the test file.  

But these following lines must be doing something that I don’t understand and anything I try to read after those lines don’t appear to be incorrect based on looking at the spreadsheet.

    QByteArray databytes;
    QDataStream out(&databytes,QIODevice::WriteOnly);
    out.setVersion(QDataStream::Qt_4_6);
    out.setByteOrder(QDataStream::LittleEndian);

    out << (qint16)eventlist.size(); // Number of event categories

sp02.num_categories = fread(fid,1,'uint16’); 14358

I’ve attached the actual “.001” file is anyone wants to look at it, but otherwise I’ll seek help locally to help with decoding the file format.  I had to add a ".txt" suffix to the file.  Hopefully, that's not a no no...

Thanks greatly for your help,

Darral

[adrinef]

Darral,

I believe the 14358 is the Datasize. 


 header >> compmethod;   // Compression Method (quint16)
        header >> machtype;     // Machine Type (quint16)
        header >> datasize;     // Size of Uncompressed Data (quint32)
        header >> crc16;        // CRC16 of Uncompressed Data (quint16)



Look in the LoadEvents in session.cpp

[darral.freund]

All,

Thanks for all your help, but after spending hours on this trying to understand the data structures, etc., I'm giving up. My programming experience is with fortran and matlab mostly, and C++ is just too different for the time I have to spend on this.   I will see if I can get a grad student at the one of the research hospitals in the Denver area to do it for a fee.

Thanks again for everyones help,

Darral

[TBMx]

I send you a Private Message^^

I rewrote the code so that it reflects the actual file structure and commented what it does.
the important part is the byte-encoding - http://de.mathworks.com/help/matlab/ref/...machinefmt (l or b)

C++ is a typed language - this means in order to use a variable you first have to declare it and specify the type before you can use it.
'//' are comments
{..} is a programming block

PHP Code:

   header.setByteOrder(QDataStream::LittleEndian); // set the byte-order to little endian <-- important!

    header << (quint32)magic;      // New Magic Number
    header << (quint16)events_version; // File Version
    header << (quint16)filetype_data;  // File type 1 == Event
    header << (quint32)s_machine->id();// Machine Type 
    header << (quint32)s_session;      // This session's ID
    header << s_first; // int64 - starttime in miliseconds since 1970-01-01 00:00
    header << s_last; // int64 - endtime

    if (p_profile->session->compressSessionData()) { // check if compression is enabled in the preferences
        compress = compress_method;
    }

    header << (quint16)compress; // if you have NOT checked in the preferences to compress session data this does not concern you - otherwise you have to decompress first: ONLY the data-part is compressed - NOT the header!

    header << (quint16)s_machine->type();// Machine Type --> I believe 2 is for oxymeters

    header << datasize; // the size of the actual data AFTER the header in bytes.
    header << chk; // checksum - should 0 - if it is not it is the checksum for the compression

 // header is written to file
 
 // the data part starts from here on:
 
    out.setByteOrder(QDataStream::LittleEndian); // little endian again!

    out << (qint16)eventlist.size(); // Number of event categories

    qint16 ev_size; // it just defines the variable so we can use it frome here on

    for (i = eventlist.begin(); i != i_end; i++) { // loop through the categories (as in oxydata, CPAP-data ...)
 
        out << i.key(); // ChannelID ... as in "pulse rate, SPO2, SPO2-drops, Pulserate-changes, ...." --> it should be uint32 --> look in "machine_common.h"
 //if the events_version is below 8 this is stored as string / text
 
        ev_size=i.value().size(); // store the number of data-lists in the current category in the variable 
        out << (qint16)ev_size; // write that to file as int16


        for (int j = 0; j < ev_size; j++) { // loop through the data-lists in that category
            EventList &e = *i.value()[j]; // declaring the variable and assigning it the current data-list
            out << e.first(); // int64 - starttime in miliseconds since 1970-01-01 00:00
            out << e.last(); // int64 - endtime
            out << (qint32)e.count(); // int32 - number of records / data-points
            out << (qint8)e.type(); // int8 - if it's 0 it's a waveform / 1 for event-format
            out << e.rate(); // float - time between data-points of the raw-data on import (should be in milliseconds or maybe seconds)
            out << e.gain(); // float - multiplier to convert the stored data back to its original value
            out << e.offset(); // float - what it says^^
            out << e.Min(); // float
            out << e.Max(); // float
            out << e.dimension(); // string - the dimension in "plain text" ... it should be 0x00 terminated - meaning this are "chars" and the string ends with 0x00
            out << e.hasSecondField(); // if it has a 2nd field (don't know for sure but I believe Oxy-data has NOT)

            if (e.hasSecondField()) {
                out << e.min2();
                out << e.max2();
            }
        }
    }

 // the "meta-information" for each channel is now written to file
 
 // now the actual data-points are written to file:
    for (i = eventlist.begin(); i != i_end; i++) { // once again loop through the categories
        ev_size=i.value().size(); // number of data-lists within the current category - see above

        for (int j = 0; j < ev_size; j++) { // loop through the data-lists 
 //here the data gets stored as 16bit values ... you have to tinker around if its int16 or uint16 - you also have to find out the byte-coding - I would first assume big-endian - but maybe it really is little endian.

            if (e.hasSecondField()) {
            //if it has a second field that data is stored as well - read above on the format - it's the very same.
            }

            // Store the time delta fields for non-waveform EventLists
            if (e.type() != EVL_Waveform) {
            // if the type is waveform this does NOT apply as all waveform data is written value by value to the file.
            // but if it is event-format it is stored in a time-delta-format (to save space)
            // this means: that if there are data-points in a row with the same value just 1 data-value is stored and the time how long that was.
            // looks like the milliseconds between them 
 
            // format is: uint32 - the byte-order you have to find out.
            }
        }
    } 


I don't know matlab ... looks like a scriptlanguage, so it shouldn't be that hard to read ... if you post the code you have, maybe we can work out the hard parts^^

[adrinef]

TBMx,

Thanks for posting the code with comments.