12-12-2020, 04:05 AM
ID's and work arounds for CSV importing and other general software problems
This thread is intended to be helpful to members as a repository for contributions of all our proven methods of dealing with general software brick walls we encounter. As individuals, many of us with limited programming skills, deal with sleep apnea related data import, export and analysis difficulties, especially when first using other common or device supporting software tools.
Suggestion: Other specific program difficulties encountered in, say, a member's use of OSCAR or, say, an oximeter, would likely get faster responses from posting that person's own specific thread and calling for help. After resolution of a problem using forum help, it can be helpful to others--as I hope I will be now with this thread and first post--if you post in this thread a brief description of your problem, how it was solved, make use of any key words (for others' AB or google searches), and provide a link to your thread for all the detail. That way your amazing solution, not always well described by my (your?) thread's chosen subject, will not get buried in a thread that will disappear over the forum's time horizon, never to be found again except by clever searching.
Here's a kick off with my general software (CSV) handling problem and its solution:
I got a Dreem headband that provided sleep stage data in a CSV file. My first few imports worked perfectly, but the CSV data needed to be cleaned up due to my poor workflow and time management as a fumbling newbie.
After a few successful imports of Dreem CSV data file, as-is, into OSCAR, I decided to edit that data and do clean up edits. After doing so using Excel, OSCAR imports would indicate data had been imported but there were never any detailed graphs being generate to show sleep stages through the night.
It turned out, thanks to Crimson Nape (primarily) and others' helps, that the Excel version of a CSV file differs from the Dreem CSV file in how detailed data lines end. Oscar was seeing Excel's invisible CRLF (carriage return ASCII 13 with LF line feed ASCII 10) at the ends of all data lines of the Excel opened (and thereby blighted) CSV file. OSCAR refused it, demanding that there only be the LF that proper Dreem CSV data rows do present.
The childishly simple solution was simply to use NotePad++, but not NotePad, to open, edit and save the dreem_export.csv file as a CSV file. Then, import that NotePad++ CSV file into OSCAR.
(Along my tortuous path, I learned that the "debug log" listing item, accessible through OSCAR's Data-Advanced-Troubleshooting menu choice sequence, presents a way to expose the debug log which, I learned, traces/summarizes the internal program path that OSCAR used in setting up my immediate OSCAR session for view on screen. Not knowing much, now or then, I could at least see there that summary files for Dreem data had been prepared, but not the details files, nor their Events file holder. All this could be sniffed out, even by this software newbie, eventually, after checking matters via OSCAR's app data folder in the Documents folder. By that path I came to see the needed Dreem profile alongside my VAuto's profile, along with a Dreem Summary folder and without the missing Events folder that should have held Dreem sleep stage data. All of this was and still is almost totally Greek to me.)
A link to my struggle thread if you want to see gory details: http://www.apneaboard.com/forums/Thread-...lp-request
Here are other links to some of the related sites I visited when groping for info, hints and solutions. Note that one site, possibly linked below, indicated there are even various hidden differences among UTF-8 code page outputs and inputs, a matter I looked at not knowing what was wrong at the time.). I suffered no known virus consequences and learned a lot from such reading, but the breakthrough came from Crimson Nape putting his finger on the CRLF problem, and then my using NotePad++ to see the hidden CRLF's at ends of rows, using NotePad++. Then I could to edit Dreem csv files in NotePad++, if needed, do saves with it to CSV and then do imports of the NotePad++ digested file into OSCAR.
https://www.w3.org/International/getting...characters
https://www.w3.org/International/questio...ark#remove
https://excelribbon.tips.net/T009508_Com...tions.html
Solved: UTF-8 CSV output without BOM? - Alteryx Community
https://community.alteryx.com/t5/Alteryx...M/td-p/652
https://www.johndcook.com/blog/2019/09/07/excel-r-bom/
https://softwareengineering.stackexchang...order-mark
Saving CSV/Excel file as UTF-8 Encoded – WooCommerce Product Import Export - WebToffee
https://csv.thephpleague.com/8.0/bom/
https://stackoverflow.com/questions/6002...omatically
https://digicloud.zendesk.com/hc/en-us/a...g-csv-file
Google search: utc-8 utc-16 bom eom excel csv
Good luck finding your problem's general software solution. I hope you find and will post a summary of your findings and a link to your solution thread here.
2SB
Suggestion: Other specific program difficulties encountered in, say, a member's use of OSCAR or, say, an oximeter, would likely get faster responses from posting that person's own specific thread and calling for help. After resolution of a problem using forum help, it can be helpful to others--as I hope I will be now with this thread and first post--if you post in this thread a brief description of your problem, how it was solved, make use of any key words (for others' AB or google searches), and provide a link to your thread for all the detail. That way your amazing solution, not always well described by my (your?) thread's chosen subject, will not get buried in a thread that will disappear over the forum's time horizon, never to be found again except by clever searching.
Here's a kick off with my general software (CSV) handling problem and its solution:
I got a Dreem headband that provided sleep stage data in a CSV file. My first few imports worked perfectly, but the CSV data needed to be cleaned up due to my poor workflow and time management as a fumbling newbie.
After a few successful imports of Dreem CSV data file, as-is, into OSCAR, I decided to edit that data and do clean up edits. After doing so using Excel, OSCAR imports would indicate data had been imported but there were never any detailed graphs being generate to show sleep stages through the night.
It turned out, thanks to Crimson Nape (primarily) and others' helps, that the Excel version of a CSV file differs from the Dreem CSV file in how detailed data lines end. Oscar was seeing Excel's invisible CRLF (carriage return ASCII 13 with LF line feed ASCII 10) at the ends of all data lines of the Excel opened (and thereby blighted) CSV file. OSCAR refused it, demanding that there only be the LF that proper Dreem CSV data rows do present.
The childishly simple solution was simply to use NotePad++, but not NotePad, to open, edit and save the dreem_export.csv file as a CSV file. Then, import that NotePad++ CSV file into OSCAR.
(Along my tortuous path, I learned that the "debug log" listing item, accessible through OSCAR's Data-Advanced-Troubleshooting menu choice sequence, presents a way to expose the debug log which, I learned, traces/summarizes the internal program path that OSCAR used in setting up my immediate OSCAR session for view on screen. Not knowing much, now or then, I could at least see there that summary files for Dreem data had been prepared, but not the details files, nor their Events file holder. All this could be sniffed out, even by this software newbie, eventually, after checking matters via OSCAR's app data folder in the Documents folder. By that path I came to see the needed Dreem profile alongside my VAuto's profile, along with a Dreem Summary folder and without the missing Events folder that should have held Dreem sleep stage data. All of this was and still is almost totally Greek to me.)
A link to my struggle thread if you want to see gory details: http://www.apneaboard.com/forums/Thread-...lp-request
Here are other links to some of the related sites I visited when groping for info, hints and solutions. Note that one site, possibly linked below, indicated there are even various hidden differences among UTF-8 code page outputs and inputs, a matter I looked at not knowing what was wrong at the time.). I suffered no known virus consequences and learned a lot from such reading, but the breakthrough came from Crimson Nape putting his finger on the CRLF problem, and then my using NotePad++ to see the hidden CRLF's at ends of rows, using NotePad++. Then I could to edit Dreem csv files in NotePad++, if needed, do saves with it to CSV and then do imports of the NotePad++ digested file into OSCAR.
https://www.w3.org/International/getting...characters
https://www.w3.org/International/questio...ark#remove
https://excelribbon.tips.net/T009508_Com...tions.html
Solved: UTF-8 CSV output without BOM? - Alteryx Community
https://community.alteryx.com/t5/Alteryx...M/td-p/652
https://www.johndcook.com/blog/2019/09/07/excel-r-bom/
https://softwareengineering.stackexchang...order-mark
Saving CSV/Excel file as UTF-8 Encoded – WooCommerce Product Import Export - WebToffee
https://csv.thephpleague.com/8.0/bom/
https://stackoverflow.com/questions/6002...omatically
https://digicloud.zendesk.com/hc/en-us/a...g-csv-file
Google search: utc-8 utc-16 bom eom excel csv
Good luck finding your problem's general software solution. I hope you find and will post a summary of your findings and a link to your solution thread here.
2SB
I have no particular qualifications or expertise with respect to the apnea/cpap/sleep related content of my posts beyond my own user experiences and what I've learned from others on this site. Each of us bears the burden of evaluating the validity and applicability of what we read here before acting on it.
Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.
Of my 3 once-needed, helpful, and adjunctive devices I have listed, only the accelerometer remains operative (but now idle). My second CMS50I died, too, of old age and the so-so Dreem 2 needs head-positioning band repair--if, indeed, Dreem even supports use of it now.
